Energy Management Strategy Optimized by Munchausen-PER-DDQN for Hybrid Tracked Vehicle

doi:10.12382/bgxb.2024.0498

Abstract

Abstract:

To optimize the fuel economy of the series hybrid tracked vehicle and reduce the offline training time of neural network,an energy management strategy (EMS) based on double-deep Q_learning network (DDQN) algorithm with Munchausen gradient optimization and prioritized experience replay (Munchausen-PER-DDQN) is proposed.The required power is calculated by a vehicle model which involves the engine-generator set,the battery pack and drive motor,and then the peoposed strategy is used to optimally control the throttle opening of engine based on power demand.The Munchausen gradient optimization algorithm adds log-policy to the reward to ease the learning of sub-optimal actions,and the prioritized experience replay algorithm assigns higher selection possibility to certain experience for those who have more influence on the training of the algorithm,Tthe energy management strategy based on Munchausen-PER-DDQN algorithm shows a better performance of fuel economy and training time of neural network.The simulated result shows that,compared with TD3-PER algorithm,the Munchausen-PER-DDQN algorithm achieves 35.3% improvement in neural network training time and 4.6% improvement in the fuel economy.

Key words: series hybrid tracked vehicle, Munchausen gradient optimization algorithm, Prioritized experience replay algorithm, deep reinforcement learning, energy management strategy

CLC Number:

TJ810.2

LU Xiaoran, ZOU Yuan, ZHANG Xudong, SUN Wei, MENG Yihao, ZHANG Bin. Energy Management Strategy Optimized by Munchausen-PER-DDQN for Hybrid Tracked Vehicle[J]. Acta Armamentarii, 2025, 46(6): 240498-.

Figures/Tables 17

References 28

[1]	孙逢春, 张承宁. 装甲车辆混合动力电传动技术[M]. 北京: 国防工业出版社, 2008.
	SUN F C, ZHANG C N. Technologies for the hybrid electric drive system of armored vehicle[M]. Beijing: National Defense Industry Press, 2008. (in Chinese)
[2]	侯旭朝, 马越, 项昌乐. 电驱动履带车辆转向稳定性控制研究[J]. 机械工程学报, 2024, 60(8):233-244.
	HOU X Z, MA Y, XIANG C L. Research on steering stability control of electric drive tracked vehicle[J]. Journal of Mechanical Engineering, 2024, 60(8):233-244. ((in Chinese)
[3]	邹渊, 焦飞翔, 崔星. 等. 地面无人平台动力源集成技术发展综述[J]. 兵工学报, 2020, 41(10):2132-2140.
	ZOU Y, JIAO F X, CUI X, et al. A review on power source technology of unmanned ground vehicle[J]. Acta Armamentaril, 2020, 41(10):2132-2140. (in Chinese)
[4]	FARAJ M, BASIR O. Range anxiety reduction in battery-powered vehicles[C]// Proceedings of the 2016 IEEE Transportation Electrification Conference and Expo.Dearborn,MI,US:IEEE, 2016:1-6.
[5]	赵秀春, 郭戈. 混合动力电动汽车能量管理策略研究综述[J]. 自动化学报, 2016, 42(3):321-334.
	ZHAO X C, GUO G. Survey on energy management strategies for hybrid electric vehicles[J]. Acta Automatic Sinica, 2016, 42(3):321-334. (in Chinese)
[6]	张卫青. 混合动力汽车的发展现状及其关键技术[J]. 重庆理工大学学报, 2006, 20(5):19-22.
	ZHANG W Q. Research actuality and key technologies of hybrid electric vehicle[J]. Journal of Chongqing Institute of Technology, 2006, 20(5):19-22. (in Chinese)
[7]	LEON R, MONTALEZA C, MALDONADO L, et al. Hybrid electric vehicles:a review of existing configurations and thermodynamic cycles[J]. Thermo, 2021, 1(2):134-150.
[8]	WANG Y, BISWAS A, RODRIGUEZ R, et al. Hybrid electric vehicle specific engines:state-of-the-art review[J]. Energy Reports, 2022, 8:832-851.
[9]	唐小林, 郎陈佳, 郑林洋, 等. 智能网联混合动力汽车能量管理研究综述[J]. 重庆理工大学学报, 2023, 37(9):1-12.
	TANG X L, LANG C J, ZHENG L Y, et al. Energy management research of intelligent connected hybrid electric vehicle:a review[J]. Journal of Chongqing Institute of Technology, 2023, 37(9):1-12. (in Chinese)
[10]	PADMARAJAN B, MCGORDON A, JENNINGS P. Blended rule based energy management for PHEV:system structure and strategy[J]. IEEE Transactions on Vehicular Technology, 2016, 65(10):8757-8762.
[11]	邓富昌, 张校锋. 基于规则的混合型燃料电池汽车能量管理策略[J]. 青岛大学学报, 2023, 38(3):75-80.
	DENG F C, ZHANG X F. Rule based energy management system of hybrid vehicle[J]. Journal of Qingdao University, 2023, 38(3):75-80. (in Chinese)
[12]	TROVAO J, PEREIRINHA P, JORGE H, et al. A multi-level energy management system for multi-source electric vehicles-an integrated rule-based meta-heuristic approach[J]. Applied Energy, 2013, 105:304-318.
[13]	丁阿鑫, 张晨阳, 沈英. 燃料电池汽车改进型状态机能量管理策略研究[J]. 机械制造与自动化, 2021, 50(2):181-204.
	DING A X, ZHANG C Y, SHEN Y. Study on improved state machine energy management strategy for fuel cell vehicles[J]. Machine Building & Automation, 2021, 50(2):181-204. (in Chinese)
[14]	MORTEZA M, MEHDI M. Development a new power management strategy for power split hybrid electric vehicles[J]. Transportation Research Part D:Transport and Environment, 2015, 37:79-96.
[15]	ZOU Y, SUN F C, HU X S, et al. Combined optimal sizing and control for a hybrid tracked vehicle[J]. Energies, 2012, 5(12):4697-4710.
[16]	ZHU H J, SONG Z Y, HOU J, et al. Simultaneous identification and control using active signal injection for series hybrid electric vehicles based on dynamic programming[J]. IEEE Transactions on Transportation Electrification, 2020, 6(1):298-307.
[17]	JIANG H L, XU L F, LI J Q, et al. Energy management and component sizing for a fuel cell/battery/supercapacitor hybrid powertrain based on two-dimensional optimization algorithms[J]. Energy, 2019, 177:386-396.
[18]	ZHANG S, HU X S, XIE S B, et al. Adaptively coordinated optimization of battery aging and energy management in plug-in hybrid electric buses[J]. Applied Energy, 2019, 256:113891.
[19]	LIU T, ZOU Y, LIU D X, et al. Reinforcement learning of adaptive energy management with transition probability for a hybrid electric tracked vehicle[J]. IEEE Transactions on Industrial Electronics, 2015, 62(12):7837-7846.
[20]	DU G D, ZOU Y, ZHANG X, et al. Energy management for a hybrid electric vehicle based on prioritized deep reinforcement learning framework[J]. Energy, 2022, 241:122523.
[21]	CHEN H, GUO G, TANG B B, et al. Data-driven transferred energy management strategy for hybrid electric vehicles via deep reinforcement learning[J]. Energy Reports, 2023, 10:2680-2692.
[22]	SINGH V, CHEN S S, SINGHANIA M, et al. How are reinforcement learning and deep learning algorithms used for big data based decision making in financial industries-a review and research agenda[J]. International Journal of Information Management Data Insights, 2022, 2(2):100094.
[23]	AN X F, HE H W, WU J, et al. Energy management based on reinforcement learning with double deep Q-learning for a hybrid electric tracked vehicle[J]. Applied Energy, 2019, 254:113708.
[24]	CUI H H, RUAN J G, WU C C, et al. Advanced deep deterministic policy gradient based energy management strategy design for dual-motor four-wheel-drive electric vehicle[J]. Mechanism and Machine Theory, 2023, 179:105119.
[25]	ZHOU J H, XUE S W, XUE Y, et al. A novel energy management strategy of hybrid electric vehicle via an improved TD3 deep reinforcement learning[J]. Energy, 2021, 224:120118.
[26]	张彬, 邹渊, 张旭东, 等. 基于TD3-PER的混合动力履带车辆能量管理[J]. 汽车工程, 2022, 44(9):1400-1409.
	ZHANG B, ZOU Y, ZHANG X D, et al. Energy management strategy based on TD3-PER for hybrid electric tracked vehicle[J]. Automotive Engineering, 2022, 44(9):1400-1409. (in Chinese)
[27]	邹渊, 张彬, 张旭东, 等. 基于归一化优势函数的强化学习混合动力履带车辆能量管理[J]. 兵工学报, 2021, 42(10):2159-2169.
	ZOU Y, ZHANG B, ZHANG X D, et al. Energy management of hybrid tracked vehicle based on reinforcement learning with normalized advantage function[J]. Acta Armamentarii, 2021, 42(10):2159-2169. (in Chinese)
[28]	SCHAUL T, QUAN J, ANTONOGLOU I, et al. Prioritized experience replay[J]. International Conference on Learning Representations, 2016, 1511:05952.

名称	参数	数值
车辆参数	整车总质量/(m·kg^-1)	1500
	重力加速度g/(kg·m^-2)	9.8
	滚动阻力系数f	0.0494
	履带接地长度L/m	1.6
	迎风面积A/m²	0.91
	传动效率η	0.92
	空气阻力系数C_d	0.9
发电机参数	发电机转动惯量J_e/(kg·m²)	0.207
	峰值功率/kW	30
	反电动势系数K_e/(V·(rad/s)^-1)	1.8024
	等效阻抗系数K_x/(N·m·A^-2)	0.0098
动力电池参数	动力电池内阻R_in/Ω	0.1
	动力电池容量/(A·h)	45.5

名称	参数	数值
车辆参数	整车总质量/(m·kg^-1)	1500
	重力加速度g/(kg·m^-2)	9.8
	滚动阻力系数f	0.0494
	履带接地长度L/m	1.6
	迎风面积A/m²	0.91
	传动效率η	0.92
	空气阻力系数C_d	0.9
发电机参数	发电机转动惯量J_e/(kg·m²)	0.207
	峰值功率/kW	30
	反电动势系数K_e/(V·(rad/s)^-1)	1.8024
	等效阻抗系数K_x/(N·m·A^-2)	0.0098
动力电池参数	动力电池内阻R_in/Ω	0.1
	动力电池容量/(A·h)	45.5

参数	数值
Replay Buffer大小	8192
每个回合的训练步数	589
Batch size	64
折扣因子γ	0.99
动作网络学习率lr	0.0001
延迟更新参数d	2
软更新参数τ	0.005
优先采样权重调节因子β	0.4
控制均匀采样和贪婪抽样的超参数α	0.6
训练回合数	100

参数	数值
Replay Buffer大小	8192
每个回合的训练步数	589
Batch size	64
折扣因子γ	0.99
动作网络学习率lr	0.0001
延迟更新参数d	2
软更新参数τ	0.005
优先采样权重调节因子β	0.4
控制均匀采样和贪婪抽样的超参数α	0.6
训练回合数	100

参数	数值
Replay Buffer大小	8192
每个回合的训练步数	589
Batch size	64
折扣因子γ	0.99
动作网络学习率lr	0.0001
延迟更新参数d	2
软更新参数τ	0.005
优先采样权重调节因子β	0.4
控制均匀采样和贪婪抽样的超参数α	0.6
训练回合数	35