Energy Management Strategy Optimized by Munchausen-PER-DDQN for Hybrid Tracked Vehicle

Xiaoran LU; Yuan ZOU; Xudong ZHANG; Wei SUN; Yihao MENG; Bin ZHANG

doi:10.12382/bgxb.2024.0498

您当前的位置：

首页 >

文章列表页 >

Energy Management Strategy Optimized by Munchausen-PER-DDQN for Hybrid Tracked Vehicle

更新时间：2025-09-05

- Energy Management Strategy Optimized by Munchausen-PER-DDQN for Hybrid Tracked Vehicle
- Acta Armamentarii Vol. 46, Issue 6, Pages: 240498(2025)
- 作者机构：
  
  1. 北京理工大学机械与车辆学院, 北京 100081
  2. 中国北方车辆研究所, 北京 100072
- 作者简介：
- 基金信息：
- DOI：10.12382/bgxb.2024.0498
  CLC：
- Received：24 June 2024，
  
  Published Online：28 June 2025，
  
  Published：10 June 2025
- 稿件说明：
移动端阅览
Xiaoran LU, Yuan ZOU, Xudong ZHANG, et al. Energy Management Strategy Optimized by Munchausen-PER-DDQN for Hybrid Tracked Vehicle[J]. Acta Armamentarii, 2025, 46(6): 240498.
DOI：

Xiaoran LU, Yuan ZOU, Xudong ZHANG, et al. Energy Management Strategy Optimized by Munchausen-PER-DDQN for Hybrid Tracked Vehicle[J]. Acta Armamentarii, 2025, 46(6): 240498. DOI： 10.12382/bgxb.2024.0498.

摘要

为优化串联式混合动力履带车辆的燃油经济性及能量管理系统的离线训练用时

提出一种采用蒙乔森(Munchausen)优化算法及优先经验采样(Prioritized Experience Replay

PER)算法的双重深度Q网络(Double-Deep Q_learning Network

DDQN)的能量管理策略。通过包含发动机发电机组、动力电池组及驱动电机的模型对整车功率需求进行解算

根据功率需求

用能量管理控制策略对发动机节气门开度进行最优控制。采用蒙乔森优化算法、PER算法共同作用于离散型DDQN

同时提高网络对高影响数据的选取训练概率及对最优解的专注训练能力

在2种算法共同作用下DDQN能量管理策略的燃油经济性可实现对连续型复杂神经网络的超越

同时具有较大的离线训练用时优势。仿真实验结果表明:与基于PER的双延迟深度确定性策略梯度算法相比

新的能量管理控制策略可使得串联式混动履带车的燃油经济性平均提高4.6%

控制策略训练用时平均优化了35.3%。

Abstract

To optimize the fuel economy of the series hybrid tracked vehicle and reduce the offline training time of neural network

an energy management strategy (EMS) based on double-deep Q_learning network (DDQN) algorithm with Munchausen gradient optimization and prioritized experience replay (Munchausen-PER-DDQN) is proposed.The required power is calculated by a vehicle model which involves the engine-generator set

the battery pack and drive motor

and then the peoposed strategy is used to optimally control the throttle opening of engine based on power demand.The Munchausen gradient optimization algorithm adds log-policy to the reward to ease the learning of sub-optimal actions

and the prioritized experience replay algorithm assigns higher selection possibility to certain experience for those who have more influence on the training of the algorithm

Tthe energy management strategy based on Munchausen-PER-DDQN algorithm shows a better performance of fuel economy and training time of neural network.The simulated result shows that

compared with TD3-PER algorithm

the Munchausen-PER-DDQN algorithm achieves 35.3% improvement in neural network training time and 4.6% improvement in the fuel economy.

关键词

Keywords

references

孙逢春 , 张承宁 . 装甲车辆混合动力电传动技术 [M ] . 北京 : 国防工业出版社 , 2008 .

SUN F C , ZHANG C N . Technologies for the hybrid electric drive system of armored vehicle [M ] . Beijing : National Defense Industry Press , 2008 . (in Chinese)

侯旭朝 , 马越 , 项昌乐 . 电驱动履带车辆转向稳定性控制研究 [J ] . 机械工程学报 , 2024 , 60 ( 8 ): 233 - 244 .

HOU X Z , MA Y , XIANG C L . Research on steering stability control of electric drive tracked vehicle [J ] . Journal of Mechanical Engineering , 2024 , 60 ( 8 ): 233 - 244 .( (in Chinese)

邹渊 , 焦飞翔 , 崔星 . 等 . 地面无人平台动力源集成技术发展综述 [J ] . 兵工学报 , 2020 , 41 ( 10 ): 2132 - 2140 .

ZOU Y , JIAO F X , CUI X , et al . A review on power source technology of unmanned ground vehicle [J ] . Acta Armamentaril , 2020 , 41 ( 10 ): 2132 - 2140 . (in Chinese)

FARAJ M , BASIR O . Range anxiety reduction in battery-powered vehicles [C ] // Proceedings of the 2016 IEEE Transportation Electrification Conference and Expo.Dearborn,MI,US:IEEE , 2016 : 1 - 6 .

赵秀春 , 郭戈 . 混合动力电动汽车能量管理策略研究综述 [J ] . 自动化学报 , 2016 , 42 ( 3 ): 321 - 334 .

ZHAO X C , GUO G . Survey on energy management strategies for hybrid electric vehicles [J ] . Acta Automatic Sinica , 2016 , 42 ( 3 ): 321 - 334 . (in Chinese)

张卫青 . 混合动力汽车的发展现状及其关键技术 [J ] . 重庆理工大学学报 , 2006 , 20 ( 5 ): 19 - 22 .

ZHANG W Q . Research actuality and key technologies of hybrid electric vehicle [J ] . Journal of Chongqing Institute of Technology , 2006 , 20 ( 5 ): 19 - 22 . (in Chinese)

LEON R , MONTALEZA C , MALDONADO L , et al . Hybrid electric vehicles:a review of existing configurations and thermodynamic cycles [J ] . Thermo , 2021 , 1 ( 2 ): 134 - 150 .

WANG Y , BISWAS A , RODRIGUEZ R , et al . Hybrid electric vehicle specific engines:state-of-the-art review [J ] . Energy Reports , 2022 , 8 : 832 - 851 .

唐小林 , 郎陈佳 , 郑林洋 , 等 . 智能网联混合动力汽车能量管理研究综述 [J ] . 重庆理工大学学报 , 2023 , 37 ( 9 ): 1 - 12 .

TANG X L , LANG C J , ZHENG L Y , et al . Energy management research of intelligent connected hybrid electric vehicle:a review [J ] . Journal of Chongqing Institute of Technology , 2023 , 37 ( 9 ): 1 - 12 . (in Chinese)

PADMARAJAN B , MCGORDON A , JENNINGS P . Blended rule based energy management for PHEV:system structure and strategy [J ] . IEEE Transactions on Vehicular Technology , 2016 , 65 ( 10 ): 8757 - 8762 .

邓富昌 , 张校锋 . 基于规则的混合型燃料电池汽车能量管理策略 [J ] . 青岛大学学报 , 2023 , 38 ( 3 ): 75 - 80 .

DENG F C , ZHANG X F . Rule based energy management system of hybrid vehicle [J ] . Journal of Qingdao University , 2023 , 38 ( 3 ): 75 - 80 . (in Chinese)

TROVAO J , PEREIRINHA P , JORGE H , et al . A multi-level energy management system for multi-source electric vehicles-an integrated rule-based meta-heuristic approach [J ] . Applied Energy , 2013 , 105 : 304 - 318 .

丁阿鑫 , 张晨阳 , 沈英 . 燃料电池汽车改进型状态机能量管理策略研究 [J ] . 机械制造与自动化 , 2021 , 50 ( 2 ): 181 - 204 .

DING A X , ZHANG C Y , SHEN Y . Study on improved state machine energy management strategy for fuel cell vehicles [J ] . Machine Building & Automation , 2021 , 50 ( 2 ): 181 - 204 . (in Chinese)

MORTEZA M , MEHDI M . Development a new power management strategy for power split hybrid electric vehicles [J ] . Transportation Research Part D:Transport and Environment , 2015 , 37 : 79 - 96 .

ZOU Y , SUN F C , HU X S , et al . Combined optimal sizing and control for a hybrid tracked vehicle [J ] . Energies , 2012 , 5 ( 12 ): 4697 - 4710 .

ZHU H J , SONG Z Y , HOU J , et al . Simultaneous identification and control using active signal injection for series hybrid electric vehicles based on dynamic programming [J ] . IEEE Transactions on Transportation Electrification , 2020 , 6 ( 1 ): 298 - 307 .

JIANG H L , XU L F , LI J Q , et al . Energy management and component sizing for a fuel cell/battery/supercapacitor hybrid powertrain based on two-dimensional optimization algorithms [J ] . Energy , 2019 , 177 : 386 - 396 .

ZHANG S , HU X S , XIE S B , et al . Adaptively coordinated optimization of battery aging and energy management in plug-in hybrid electric buses [J ] . Applied Energy , 2019 , 256 : 113891 .

LIU T , ZOU Y , LIU D X , et al . Reinforcement learning of adaptive energy management with transition probability for a hybrid electric tracked vehicle [J ] . IEEE Transactions on Industrial Electronics , 2015 , 62 ( 12 ): 7837 - 7846 .

DU G D , ZOU Y , ZHANG X , et al . Energy management for a hybrid electric vehicle based on prioritized deep reinforcement learning framework [J ] . Energy , 2022 , 241 : 122523 .

CHEN H , GUO G , TANG B B , et al . Data-driven transferred energy management strategy for hybrid electric vehicles via deep reinforcement learning [J ] . Energy Reports , 2023 , 10 : 2680 - 2692 .

SINGH V , CHEN S S , SINGHANIA M , et al . How are reinforcement learning and deep learning algorithms used for big data based decision making in financial industries-a review and research agenda [J ] . International Journal of Information Management Data Insights , 2022 , 2 ( 2 ): 100094 .

AN X F , HE H W , WU J , et al . Energy management based on reinforcement learning with double deep Q-learning for a hybrid electric tracked vehicle [J ] . Applied Energy , 2019 , 254 : 113708 .

CUI H H , RUAN J G , WU C C , et al . Advanced deep deterministic policy gradient based energy management strategy design for dual-motor four-wheel-drive electric vehicle [J ] . Mechanism and Machine Theory , 2023 , 179 : 105119 .

ZHOU J H , XUE S W , XUE Y , et al . A novel energy management strategy of hybrid electric vehicle via an improved TD3 deep reinforcement learning [J ] . Energy , 2021 , 224 : 120118 .

张彬 , 邹渊 , 张旭东 , 等 . 基于TD3-PER的混合动力履带车辆能量管理 [J ] . 汽车工程 , 2022 , 44 ( 9 ): 1400 - 1409 .

ZHANG B , ZOU Y , ZHANG X D , et al . Energy management strategy based on TD3-PER for hybrid electric tracked vehicle [J ] . Automotive Engineering , 2022 , 44 ( 9 ): 1400 - 1409 . (in Chinese)

邹渊 , 张彬 , 张旭东 , 等 . 基于归一化优势函数的强化学习混合动力履带车辆能量管理 [J ] . 兵工学报 , 2021 , 42 ( 10 ): 2159 - 2169 .

ZOU Y , ZHANG B , ZHANG X D , et al . Energy management of hybrid tracked vehicle based on reinforcement learning with normalized advantage function [J ] . Acta Armamentarii , 2021 , 42 ( 10 ): 2159 - 2169 . (in Chinese) DOI: 10.3969/j.issn.1000-1093.2021.10.011 http://doi.org/10.3969/j.issn.1000-1093.2021.10.011 The energy management strategy based on reinforcement learning encounters the problem of “dimension disaster”when dealing with high-dimensional problems because of the discretization of state and control variables. For this problem, a new energy management algorithm based on deep reinforcement learning with normalized advantage function is proposed, where two deep neural networks with normalized advantage function are used to realize the continuous control of energy and eliminate the discretization of state and control variables. Based on the modeling of powertrain of a series hybrid tracked vehicle, the framework of the proposed deep reinforcement learning algorithm was built and the parameter update process was completed for the series hybrid tracked vehicle. The simulated results show that the proposed algorithm can output more refined control quantity and less output fluctuation. Compared with the deep Q-learning algorithm, the proposed algorithm improves the fuel economy of series hybrid tracked vehicle by 3.96%. In addition, the adaptability of the proposed algorithm and the optimized effect in real-time control environment are verified by the hardware-in-the-loop simulation.

SCHAUL T , QUAN J , ANTONOGLOU I , et al . Prioritized experience replay [J ] . International Conference on Learning Representations , 2016 , 1511 : 05952 .

Views

168

下载量

CNKI被引量

Alert me when the article has been cited

提交

Tools

Publicity Resources

Path Planning Method for Large-scale UAV Swarms Based on Reinforcement Learning Conflict Resolution

An Impact Angle and Field of View Constraints Guidance Law Based on Deep Reinforcement Learning

Research on Robot Navigation Method Integrating Safe Convex Space and Deep Reinforcement Learning

A Hierarchical Multi-Agent Collaborative Decision-making Method Based on the Actor-critic Framework

Deep Reinforcement Learning-Based Active Disturbance Rejection Control for Integrated Guidance and Control

Related Author

Junzhi LI

Jianxin ZHONG

Jingliang SUN

Dawei LIU

Teng LONG

Zhenlin ZHOU

Zhilin WU

Jie SONG

Related Institution

National Key Laboratory of Land and Air Based Information Perception and Control

Beijing Institute of Technology Chongqing Innovation Center

Key Laboratory of Dynamics and Control of Flight Vehicle of Ministry of Education

School of Aerospace Engineering, Beijing Institute of Technology

Research and Development Academy of Machinery Equipment

Postal code：100089
Tel：010-68963060/68962718 Fax：010-68963025 Email：bgxb@cos.org.cn
Technical support is provided by Beijing Founder electronics co., LTD 京ICP备05059581号-4 京公网安备11010802024360号
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.
Total Visits：44524 Visits Today：504

⁰