基于Munchausen-PER算法优化的混合动力履带车辆能量管理策略

doi:10.12382/bgxb.2024.0498

摘要/Abstract

摘要：

为优化串联式混合动力履带车辆的燃油经济性及能量管理系统的离线训练用时,提出一种采用蒙乔森(Munchausen)优化算法及优先经验采样(Prioritized Experience Replay,PER)算法的双重深度Q网络(Double-Deep Q_learning Network,DDQN)的能量管理策略。通过包含发动机发电机组、动力电池组及驱动电机的模型对整车功率需求进行解算,根据功率需求,用能量管理控制策略对发动机节气门开度进行最优控制。采用蒙乔森优化算法、PER算法共同作用于离散型DDQN,同时提高网络对高影响数据的选取训练概率及对最优解的专注训练能力,在2种算法共同作用下DDQN能量管理策略的燃油经济性可实现对连续型复杂神经网络的超越,同时具有较大的离线训练用时优势。仿真实验结果表明:与基于PER的双延迟深度确定性策略梯度算法相比,新的能量管理控制策略可使得串联式混动履带车的燃油经济性平均提高4.6%,控制策略训练用时平均优化了35.3%。

关键词: 串联式混动履带车, Munchausen优化算法, 优先经验采样算法, 深度强化学习, 能量管理策略

Abstract:

To optimize the fuel economy of the series hybrid tracked vehicle and reduce the offline training time of neural network,an energy management strategy (EMS) based on double-deep Q_learning network (DDQN) algorithm with Munchausen gradient optimization and prioritized experience replay (Munchausen-PER-DDQN) is proposed.The required power is calculated by a vehicle model which involves the engine-generator set,the battery pack and drive motor,and then the peoposed strategy is used to optimally control the throttle opening of engine based on power demand.The Munchausen gradient optimization algorithm adds log-policy to the reward to ease the learning of sub-optimal actions,and the prioritized experience replay algorithm assigns higher selection possibility to certain experience for those who have more influence on the training of the algorithm,Tthe energy management strategy based on Munchausen-PER-DDQN algorithm shows a better performance of fuel economy and training time of neural network.The simulated result shows that,compared with TD3-PER algorithm,the Munchausen-PER-DDQN algorithm achieves 35.3% improvement in neural network training time and 4.6% improvement in the fuel economy.

Key words: series hybrid tracked vehicle, Munchausen gradient optimization algorithm, Prioritized experience replay algorithm, deep reinforcement learning, energy management strategy

中图分类号:

TJ810.2

路潇然, 邹渊, 张旭东, 孙巍, 孟逸豪, 张彬. 基于Munchausen-PER算法优化的混合动力履带车辆能量管理策略[J]. 兵工学报, 2025, 46(6): 240498-.

LU Xiaoran, ZOU Yuan, ZHANG Xudong, SUN Wei, MENG Yihao, ZHANG Bin. Energy Management Strategy Optimized by Munchausen-PER-DDQN for Hybrid Tracked Vehicle[J]. Acta Armamentarii, 2025, 46(6): 240498-.

图/表 17

图1 课题组自研串联式HETV

Fig.1 Series hybrid-electric tracked vehicle

图2 动力驱动系统拓扑结构

Fig.2 Topological structure of powertrain

表1 课题组自研串联式HETV技术参数

Table 1 Parameters of CyberTank Ⅱ

名称	参数	数值
车辆参数	整车总质量/(m·kg^-1)	1500
	重力加速度g/(kg·m^-2)	9.8
	滚动阻力系数f	0.0494
	履带接地长度L/m	1.6
	迎风面积A/m²	0.91
	传动效率η	0.92
	空气阻力系数C_d	0.9
发电机参数	发电机转动惯量J_e/(kg·m²)	0.207
	峰值功率/kW	30
	反电动势系数K_e/(V·(rad/s)^-1)	1.8024
	等效阻抗系数K_x/(N·m·A^-2)	0.0098
动力电池参数	动力电池内阻R_in/Ω	0.1
	动力电池容量/(A·h)	45.5

图3 电池SOC与车速的仿真数据与实车数据对比

Fig.3 Comparison between simulated data and real data of SOC curve and vehicle speed curve

图4 基于Munchausen-PER-DDQN算法的EMS框架

Fig.4 Framework of EMS based on Munchausen-PER-DDQN algorithm

图5 Munchause-PER-DDQN算法的计算流程

Fig.5 Calculation flow of Munchause-PER-DDQN algorithm

图6 Munchausen-PER-DDQN算法训练工况

Fig.6 Training condition of Munchausen-PER-DDQN algorithm

表2 Munchause-PER-DDQN算法超参数

Table 2 Hyper parameters of Munchause-PER-DDQN algorithm

参数	数值
Replay Buffer大小	8192
每个回合的训练步数	589
Batch size	64
折扣因子γ	0.99
动作网络学习率lr	0.0001
延迟更新参数d	2
软更新参数τ	0.005
优先采样权重调节因子β	0.4
控制均匀采样和贪婪抽样的超参数α	0.6
训练回合数	100

图7 训练中的总评价曲线

Fig.7 Reward curve of training process

表3 Munchause-PER-DDQN算法超参数

Table 3 Hyper parameters of Munchause-PER-DDQN algorithm

参数	数值
Replay Buffer大小	8192
每个回合的训练步数	589
Batch size	64
折扣因子γ	0.99
动作网络学习率lr	0.0001
延迟更新参数d	2
软更新参数τ	0.005
优先采样权重调节因子β	0.4
控制均匀采样和贪婪抽样的超参数α	0.6
训练回合数	35

图8 TD3-PER、DDQN及Munchausen-PER-DDQN算法SOC曲线

Fig.8 SOC curves of TD 3-PER,DDQN and Munchausen-PER-DDQN algorithm

表4 基于TD3-PER、DDQN与Munchausen-PER-DDQN算法的训练用时与燃油消耗量

Table 4 Calculated training duration and fuel consumption of TD 3-PER,DDQN and Munchausen-PER-DDQN algorithms

序号	算法	训练用时/s	油量消耗/g
1	DDQN	510	268
2		515	275
3		521	267
4		512	263
5		511	266
6	TD3-PER	918	259
7		918	264
8		924	255
9		899	265
10		896	279
11	Munchausen-PER-DDQN	620	250
12		603	251
13		598	254
14		521	252
15		607	254

图9 训练工况中发动机工作点位对比

Fig.9 Engine working positions under training condition

图10 实车采集两种验证工况

Fig.10 Two testing conditions based on vehicle data

图11 验证工况1中发动机工作点位对比

Fig.11 Engine working positions under testing condition 1

图12 验证工况2中发动机工作点位对比

Fig.12 Engine working positions under testing condition 2

表5 基于TD3-PER、DDQN与Munchausen-PER-DDQN算法在验证工况1、2中的燃油消耗量

Table 5 Calculated fuel consumption of TD 3-PER,DDQN and Munchausen-PER-DDQN algorithms under testing conditions 1and 2

EMS算法	燃油消耗
EMS算法	验证工况1	验证工况2
DDQN	290	291
TD3-PER	281	281
Munchausen-PER-DDQN	263	275

参考文献 28

[1]	孙逢春, 张承宁. 装甲车辆混合动力电传动技术[M]. 北京: 国防工业出版社, 2008.
	SUN F C, ZHANG C N. Technologies for the hybrid electric drive system of armored vehicle[M]. Beijing: National Defense Industry Press, 2008. (in Chinese)
[2]	侯旭朝, 马越, 项昌乐. 电驱动履带车辆转向稳定性控制研究[J]. 机械工程学报, 2024, 60(8):233-244.
	HOU X Z, MA Y, XIANG C L. Research on steering stability control of electric drive tracked vehicle[J]. Journal of Mechanical Engineering, 2024, 60(8):233-244.((in Chinese)
[3]	邹渊, 焦飞翔, 崔星. 等. 地面无人平台动力源集成技术发展综述[J]. 兵工学报, 2020, 41(10):2132-2140.
	ZOU Y, JIAO F X, CUI X, et al. A review on power source technology of unmanned ground vehicle[J]. Acta Armamentaril, 2020, 41(10):2132-2140. (in Chinese)
[4]	FARAJ M, BASIR O. Range anxiety reduction in battery-powered vehicles[C]// Proceedings of the 2016 IEEE Transportation Electrification Conference and Expo.Dearborn,MI,US:IEEE, 2016:1-6.
[5]	赵秀春, 郭戈. 混合动力电动汽车能量管理策略研究综述[J]. 自动化学报, 2016, 42(3):321-334.
	ZHAO X C, GUO G. Survey on energy management strategies for hybrid electric vehicles[J]. Acta Automatic Sinica, 2016, 42(3):321-334. (in Chinese)
[6]	张卫青. 混合动力汽车的发展现状及其关键技术[J]. 重庆理工大学学报, 2006, 20(5):19-22.
	ZHANG W Q. Research actuality and key technologies of hybrid electric vehicle[J]. Journal of Chongqing Institute of Technology, 2006, 20(5):19-22. (in Chinese)
[7]	LEON R, MONTALEZA C, MALDONADO L, et al. Hybrid electric vehicles:a review of existing configurations and thermodynamic cycles[J]. Thermo, 2021, 1(2):134-150.
[8]	WANG Y, BISWAS A, RODRIGUEZ R, et al. Hybrid electric vehicle specific engines:state-of-the-art review[J]. Energy Reports, 2022, 8:832-851.
[9]	唐小林, 郎陈佳, 郑林洋, 等. 智能网联混合动力汽车能量管理研究综述[J]. 重庆理工大学学报, 2023, 37(9):1-12.
	TANG X L, LANG C J, ZHENG L Y, et al. Energy management research of intelligent connected hybrid electric vehicle:a review[J]. Journal of Chongqing Institute of Technology, 2023, 37(9):1-12. (in Chinese)
[10]	PADMARAJAN B, MCGORDON A, JENNINGS P. Blended rule based energy management for PHEV:system structure and strategy[J]. IEEE Transactions on Vehicular Technology, 2016, 65(10):8757-8762.
[11]	邓富昌, 张校锋. 基于规则的混合型燃料电池汽车能量管理策略[J]. 青岛大学学报, 2023, 38(3):75-80.
	DENG F C, ZHANG X F. Rule based energy management system of hybrid vehicle[J]. Journal of Qingdao University, 2023, 38(3):75-80. (in Chinese)
[12]	TROVAO J, PEREIRINHA P, JORGE H, et al. A multi-level energy management system for multi-source electric vehicles-an integrated rule-based meta-heuristic approach[J]. Applied Energy, 2013, 105:304-318.
[13]	丁阿鑫, 张晨阳, 沈英. 燃料电池汽车改进型状态机能量管理策略研究[J]. 机械制造与自动化, 2021, 50(2):181-204.
	DING A X, ZHANG C Y, SHEN Y. Study on improved state machine energy management strategy for fuel cell vehicles[J]. Machine Building & Automation, 2021, 50(2):181-204. (in Chinese)
[14]	MORTEZA M, MEHDI M. Development a new power management strategy for power split hybrid electric vehicles[J]. Transportation Research Part D:Transport and Environment, 2015, 37:79-96.
[15]	ZOU Y, SUN F C, HU X S, et al. Combined optimal sizing and control for a hybrid tracked vehicle[J]. Energies, 2012, 5(12):4697-4710.
[16]	ZHU H J, SONG Z Y, HOU J, et al. Simultaneous identification and control using active signal injection for series hybrid electric vehicles based on dynamic programming[J]. IEEE Transactions on Transportation Electrification, 2020, 6(1):298-307.
[17]	JIANG H L, XU L F, LI J Q, et al. Energy management and component sizing for a fuel cell/battery/supercapacitor hybrid powertrain based on two-dimensional optimization algorithms[J]. Energy, 2019, 177:386-396.
[18]	ZHANG S, HU X S, XIE S B, et al. Adaptively coordinated optimization of battery aging and energy management in plug-in hybrid electric buses[J]. Applied Energy, 2019, 256:113891.
[19]	LIU T, ZOU Y, LIU D X, et al. Reinforcement learning of adaptive energy management with transition probability for a hybrid electric tracked vehicle[J]. IEEE Transactions on Industrial Electronics, 2015, 62(12):7837-7846.
[20]	DU G D, ZOU Y, ZHANG X, et al. Energy management for a hybrid electric vehicle based on prioritized deep reinforcement learning framework[J]. Energy, 2022, 241:122523.
[21]	CHEN H, GUO G, TANG B B, et al. Data-driven transferred energy management strategy for hybrid electric vehicles via deep reinforcement learning[J]. Energy Reports, 2023, 10:2680-2692.
[22]	SINGH V, CHEN S S, SINGHANIA M, et al. How are reinforcement learning and deep learning algorithms used for big data based decision making in financial industries-a review and research agenda[J]. International Journal of Information Management Data Insights, 2022, 2(2):100094.
[23]	AN X F, HE H W, WU J, et al. Energy management based on reinforcement learning with double deep Q-learning for a hybrid electric tracked vehicle[J]. Applied Energy, 2019, 254:113708.
[24]	CUI H H, RUAN J G, WU C C, et al. Advanced deep deterministic policy gradient based energy management strategy design for dual-motor four-wheel-drive electric vehicle[J]. Mechanism and Machine Theory, 2023, 179:105119.
[25]	ZHOU J H, XUE S W, XUE Y, et al. A novel energy management strategy of hybrid electric vehicle via an improved TD3 deep reinforcement learning[J]. Energy, 2021, 224:120118.
[26]	张彬, 邹渊, 张旭东, 等. 基于TD3-PER的混合动力履带车辆能量管理[J]. 汽车工程, 2022, 44(9):1400-1409.
	ZHANG B, ZOU Y, ZHANG X D, et al. Energy management strategy based on TD3-PER for hybrid electric tracked vehicle[J]. Automotive Engineering, 2022, 44(9):1400-1409. (in Chinese)
[27]	邹渊, 张彬, 张旭东, 等. 基于归一化优势函数的强化学习混合动力履带车辆能量管理[J]. 兵工学报, 2021, 42(10):2159-2169.
	ZOU Y, ZHANG B, ZHANG X D, et al. Energy management of hybrid tracked vehicle based on reinforcement learning with normalized advantage function[J]. Acta Armamentarii, 2021, 42(10):2159-2169. (in Chinese) doi: 10.3969/j.issn.1000-1093.2021.10.011
[28]	SCHAUL T, QUAN J, ANTONOGLOU I, et al. Prioritized experience replay[J]. International Conference on Learning Representations, 2016, 1511:05952.