Short-term Efficiency and Adaptability Assessment of a Modified PPO Model for Pipeline Monitoring

Kefas Yunana, Ishaq Oyebisi Oyefolahan, Sulaimon Adebayo Bashir

Abstract


Abstract: Critical infrastructure of oil pipelines needs robust monitoring techniques to promptly detect and address potential leaks, faults, or anomalies that can lead to severe economic and environmental consequences. Traditional monitoring techniques often rely on static or rule-based approaches, which may struggle with the dynamic and complex operational nature of pipeline environments. This study introduces a Modified Proximal Policy Optimization-based Pipeline network Monitoring Agent (MPPOPNMA) specifically designed to enhance the short-term efficiency and adaptability of pipeline monitoring through advanced reinforcement learning techniques. The MPPOPNMA model has been optimized with tailored reward structures and hyperparameter tuning to enable rapid and accurate decision-making in response to state changes characterized by sensor data, including pressure, flow rate, and velocity. In contrast to standard PPO and DQN models, MPPOPNMA demonstrated superior performance in terms of average reward and convergence rate, indicating its ability to both maximize immediate gains and achieve efficient learning outcomes with fewer iterations. The average reward metric highlights MPPOPNMA effectiveness in consistently selecting optimal actions, enhancing its responsiveness to dynamic shifts within the pipeline system. Meanwhile, the high convergence rate emphasizes the model’s ability to quickly adapt and stabilize, underscoring its potential for real-time application where immediate action is paramount. This study thus positions MPPOPNMA as a viable and powerful tool for critical infrastructure monitoring, suggesting that reinforcement learning can advance safety, efficiency, and responsiveness within oil pipeline operations.


Full Text:

PDF

References


Badakhshan, S., Jacob, R. A., Li, B., & Zhang, J. (2023). Reinforcement Learning for Intentional Islanding in Resilient Power Transmission Systems. 2023 IEEE Texas Power and Energy Conference, TPEC 2023, 1–6. https://doi.org/10.1109/TPEC56611.2023.10078568

Boudlal, A., Khafaji, A., & Elabbadi, J. (2024). Entropy adjustment by interpolation for exploration in Proximal Policy Optimization (PPO). Engineering Applications of Artificial Intelligence, 133, 108401. https://doi.org/10.1016/J.ENGAPPAI.2024.108401

Chatterjee, D., Roy, R., & Sengupta, A. (2023). Comparison of Reinforcement Learning controller with a classical controller for an UAV. 2023 Second International Conference on Electrical, Electronics, Information and Communication Technologies (ICEEICT), 1–5. https://doi.org/10.1109/ICEEICT56924.2023.10156999

Christian, A. B., Lin, C.-Y., Tseng, Y.-C., Van, L.-D., Hu, W.-H., & Yu, C.-H. (2022). Accuracy-Time Efficient Hyperparameter Optimization Using Actor-Critic-based Reinforcement Learning and Early Stopping in OpenAI Gym Environment. 2022 IEEE International Conference on Internet of Things and Intelligence Systems (IoTaIS), 230–234. https://doi.org/10.1109/IoTaIS56727.2022.9975984

Ding, Y., Wang, X., Cao, X., Hu, H., & Bu, Y. (2023). A reinforcement learning method for optimal control of oil well production using cropped well group samples. Heliyon, 9(7), e17919. https://doi.org/10.1016/j.heliyon.2023.e17919

Gao, X., Peng, D., Kui, G., Pan, J., Zuo, X., & Li, F. (2023). Reinforcement learning based optimization algorithm for maintenance tasks scheduling in coalbed methane gas field. Computers and Chemical Engineering, 170(December 2022), 108131. https://doi.org/10.1016/j.compchemeng.2022.108131

Gléau, T. Le, Marjou, X., Lemlouma, T., & Radier, B. (2021). A Multi-agent OpenAI Gym Environment for Telecom Providers Cooperation. 2021 24th Conference on Innovation in Clouds, Internet and Networks and Workshops (ICIN), 28–32. https://doi.org/10.1109/ICIN51074.2021.9385538

Huan, H., Liu, L., Huan, B., Chen, X., Zhan, J., & Liu, Q. (2021). A theoretical investigation of modelling the temperature measurement in oil pipelines with edge devices. Measurement: Journal of the International Measurement Confederation, 168(September 2020), 108440. https://doi.org/10.1016/j.measurement.2020.108440

Huang, Z., Gong, W., & Duan, J. (2023). TBDQN: A novel two-branch deep Q-network for crude oil and natural gas futures trading. Applied Energy, 347(June), 121321. https://doi.org/10.1016/j.apenergy.2023.121321

Jaensch, F., Klingel, L., & Verl, A. (2022). Virtual Commissioning Simulation as OpenAI Gym - A Reinforcement Learning Environment for Control Systems. 2022 5th International Conference on Artificial Intelligence for Industries (AI4I), 64–67. https://doi.org/10.1109/AI4I54798.2022.00023

Jayanetti, A., Halgamuge, S., & Buyya, R. (2022). Deep reinforcement learning for energy and time optimized scheduling of precedence-constrained tasks in edge–cloud computing environments. Future Generation Computer Systems, 137, 14–30. https://doi.org/10.1016/j.future.2022.06.012

Kahnamouei, J. T., & Moallem, M. (2024). Advancements in control systems and integration of artificial intelligence in welding robots: A review. In Ocean Engineering (Vol. 312). Elsevier Ltd. https://doi.org/10.1016/j.oceaneng.2024.119294

Kaloev, M., & Krastev, G. (2021). Experiments Focused on Exploration in Deep Reinforcement Learning. ISMSIT 2021 - 5th International Symposium on Multidisciplinary Studies and Innovative Technologies, Proceedings, 351–355. https://doi.org/10.1109/ISMSIT52890.2021.9604690

Khalid, M., Wang, L., Wang, K., Aslam, N., Pan, C., & Cao, Y. (2023). Deep reinforcement learning-based long-range autonomous valet parking for smart cities. Sustainable Cities and Society, 89(June 2022), 104311. https://doi.org/10.1016/j.scs.2022.104311

Kiefer, J., & Dorer, K. (2023). Double Deep Reinforcement Learning. 2023 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC), 17–22. https://doi.org/10.1109/ICARSC58346.2023.10129640

Lee, J., Seo, S., & Ko, H. (2023). UAV-Based Data Collection and Wireless Power Transfer System with Deep Reinforcement Learning. 2023 International Conference on Information Networking (ICOIN), 400–403. https://doi.org/10.1109/ICOIN56518.2023.10048978

Liang, J., Ma, L., Liang, S., Zhang, H., Zuo, Z., & Dai, J. (2023). Data-driven digital twin method for leak detection in natural gas pipelines. Computers and Electrical Engineering, 110(174), 108833. https://doi.org/10.1016/j.compeleceng.2023.108833

Lü, X., Qian, S., Zhai, X. R., Wang, P., & Wu, T. (2024). Adaptive energy management strategy for FCHEV based on improved proximal policy optimization in deep reinforcement learning algorithm. Energy Conversion and Management, 321, 118977. https://doi.org/10.1016/J.ENCONMAN.2024.118977

Najafi, S., & Lee, C. G. (2023). A deep reinforcement learning approach for repair-based maintenance of multi-unit systems using proportional hazards model. Reliability Engineering and System Safety, 234(January 2022), 109179. https://doi.org/10.1016/j.ress.2023.109179

Nguyen, T.-H., Park, H., & Park, L. (2023). Recent Studies on Deep Reinforcement Learning in RIS-UAV Communication Networks. 2023 International Conference on Artificial Intelligence in Information and Communication (ICAIIC), 378–381. https://doi.org/10.1109/ICAIIC57133.2023.10067052

Reddy, K. S., & Xie, E. (2017). Cross-border mergers and acquisitions by oil and gas multinational enterprises: Geography-based view of energy strategy. Renewable and Sustainable Energy Reviews, 72(November 2016), 961–980. https://doi.org/10.1016/j.rser.2017.01.016

Saha, U., Jawad, A., Shahria, S., & Rashid, A. B. M. H. U. (2024). Proximal policy optimization-based reinforcement learning approach for DC-DC boost converter control: A comparative evaluation against traditional control techniques. Heliyon, 10(18). https://doi.org/10.1016/j.heliyon.2024.e37823

Silva, J. R., Euzébio, T. A. M., & Braga, M. F. (2024). Control of conventional continuous thickeners via proximal policy optimization. Minerals Engineering, 214, 108761. https://doi.org/10.1016/J.MINENG.2024.108761

Vieira, S., Rocha, A., & Chaimowicz, L. (2023). Towards sample efficient deep reinforcement learning in collectible card games. Entertainment Computing, 47(July), 100594. https://doi.org/10.1016/j.entcom.2023.100594

Wang, L., & Wang, X. (2024). An adaptive financial trading strategy based on proximal policy optimization and financial signal representation. Engineering Applications of Artificial Intelligence, 138, 109365. https://doi.org/10.1016/J.ENGAPPAI.2024.109365

Wang, X., & Guo, H. (2021). Mobility-aware computation offloading for swarm robotics using deep reinforcement learning. 2021 IEEE 18th Annual Consumer Communications and Networking Conference, CCNC 2021, 2021–2024. https://doi.org/10.1109/CCNC49032.2021.9369456

Yang, X., Zou, Y., Zhang, H., Qu, X., & Chen, L. (2023). Improved deep reinforcement learning for car-following decision-making. Physica A: Statistical Mechanics and Its Applications, 624(4800), 128912. https://doi.org/10.1016/j.physa.2023.128912

Yang, Y., Liu, Z., Saydaliev, H. B., & Iqbal, S. (2022). Economic impact of crude oil supply disruption on social welfare losses and strategic petroleum reserves. Resources Policy, 77(December 2021), 102689. https://doi.org/10.1016/j.resourpol.2022.102689

Yunana, K., Oyefolahan, I. O., & Bashir, S. A. (2022). A Framework for Critical Infrastructure Monitoring Based on Deep Reinforcement Learning Approach. Proceedings of the 5th International Conference on Information Technology for Education and Development: Changing the Narratives Through Building a Secure Society with Disruptive Technologies, ITED 2022, 1–6. https://doi.org/10.1109/ITED56637.2022.10051520

Zeng, J., Gou, F., & Wu, J. (2023). Task offloading scheme combining deep reinforcement learning and convolutional neural networks for vehicle trajectory prediction in smart cities. Computer Communications, 208(July 2022), 29–43. https://doi.org/10.1016/j.comcom.2023.05.021

Zhang, S., Zhang, A., Chen, P., Li, H., Zeng, X., Chen, S., Dong, T., Shi, P., Lang, Y., & Zhou, Q. (2024). Application of artificial intelligence hybrid models in safety assessment of submarine pipelines: Principles and methods. Ocean Engineering, 312, 119203. https://doi.org/10.1016/J.OCEANENG.2024.119203

Zhang, T., Lam, K. Y., & Zhao, J. (2023). Deep reinforcement learning based scheduling strategy for federated learning in sensor-cloud systems. Future Generation Computer Systems, 144, 219–229. https://doi.org/10.1016/j.future.2023.03.009

Zhu, J., Zhang, W., Yu, L., & Guo, X. (2024). A novel multi-attention reinforcement learning for the scheduling of unmanned shipment vessels (USV) in automated container terminals. Omega, 129, 103152. https://doi.org/10.1016/J.OMEGA.2024.103152.


Refbacks

  • There are currently no refbacks.