基于归一化优势函数的强化学习混合动力履带车辆能量管理

doi:10.3969/j.issn.1000-1093.2021.10.011

摘要/Abstract

摘要： 基于强化学习的能量管理策略由于状态变量和控制变量的离散化，处理高维问题时存在“维数灾难”的困扰。针对此问题，提出一种基于归一化优势函数的深度强化学习能量管理算法。采用两个具有归一化优势函数的深度神经网络实现连续控制，消除离散化。在对串联式混合动力履带车辆动力总成建模的基础上，完成深度强化学习能量管理算法的框架搭建和参数的更新过程，并将其应用于串联式混合动力履带车辆。仿真结果表明，该算法能够输出更为细化的控制量以及更小的输出波动性，与深度Q学习算法相比，对于串联式混合动力履带车辆的燃油经济性提升了3.96%. 通过硬件在环仿真实验验证了强化学习能量管理算法的适应性，以及在实时控制环境下的优化效果。

关键词: 履带车辆, 能量管理策略, 归一化优势函数, 连续控制, 串联式混合动力, 硬件在环仿真

Abstract: The energy management strategy based on reinforcement learning encounters the problem of “dimension disaster”when dealing with high-dimensional problems because of the discretization of state and control variables. For this problem, a new energy management algorithm based on deep reinforcement learning with normalized advantage function is proposed, where two deep neural networks with normalized advantage function are used to realize the continuous control of energy and eliminate the discretization of state and control variables. Based on the modeling of powertrain of a series hybrid tracked vehicle, the framework of the proposed deep reinforcement learning algorithm was built and the parameter update process was completed for the series hybrid tracked vehicle. The simulated results show that the proposed algorithm can output more refined control quantity and less output fluctuation. Compared with the deep Q-learning algorithm, the proposed algorithm improves the fuel economy of series hybrid tracked vehicle by 3.96%. In addition, the adaptability of the proposed algorithm and the optimized effect in real-time control environment are verified by the hardware-in-the-loop simulation.

Key words: serieshybridtrackedvehicle, energymanagementstrategy, normalizedadvantagefunction, continuouscontrol, hardware-in-the-loopsimulation

中图分类号:

TJ810.2

邹渊，张彬，张旭东，赵志颖，康铁宇，郭玉枫，吴喆. 基于归一化优势函数的强化学习混合动力履带车辆能量管理[J]. 兵工学报, 2021, 42(10): 2159-2169.

ZOU Yuan, ZHANG Bin, ZHANG Xudong, ZHAO Zhiying, KANG Tieyu, GUO Yufeng, WU Zhe. Energy Management of Hybrid Tracked Vehicle Based on Reinforcement Learning with Normalized Advantage Function[J]. Acta Armamentarii, 2021, 42(10): 2159-2169.

参考文献

［1］邓启文, 刘书雷, 沈雪石. 无人装备发展新动向及影响研究[J]. 装备学院学报, 2016, 27(1): 76-79.
DENG Q W, LIU S L, SHEN X S. New trends and influence of unmanned equipment[J]. Journal of Equipment Academy, 2016, 27(1): 76-79. (in Chinese)
[2] 陈慧岩, 张玉. 军用地面无人机动平台技术发展综述[J]. 兵工学报, 2014, 35(10): 1696-1706.
CHEN H Y, ZHANG Y. An overview of research on military unmanned ground vehicles[J]. Acta Armamentarii, 2014, 35(10): 1696-1706. (in Chinese)
[3] 孟红, 朱森. 地面无人系统的发展及未来趋势[J]. 兵工学报, 2014, 35(增刊1): 1-7.
MENG H, ZHU S. The development and future trends of unmanned ground systems[J]. Acta Armamentarii, 2014, 35(S1): 1-7. (in Chinese)
[4] 邹渊, 焦飞翔, 崔星, 等.地面无人平台动力源集成技术发展综述[J].兵工学报, 2020, 41(10): 2131-2144.
ZOU Y, JIAO F X, CUI X, et al. A review on power source technology of unmanned ground vehicles[J]. Acta Armamentarii, 2020, 41(10): 2131-2144. (in Chinese)
[5] FARAJ M, BASIR O. Range anxiety reduction in battery-powered vehicles[C]∥Proceedings of 2016 IEEE Transportation Electrification Conference and Expo. Dearborn, MI, US:IEEE, 2016:16176997.
[6] CAUX S, GAOUA Y, LOPEZ P. A combinatorial optimisation approach to energy management strategy for a hybrid fuel cell vehicle[J]. Energy, 2017, 133(15):219-230.
[7] 张卫青. 混合动力汽车的发展现状及其关键技术 [J]. 重庆理工大学学报, 2006, 20(5): 19-22.
ZHANG W Q. Research actuality and key technologies of hybrid electric vehicle[J].Journal of Chongqing Institute of Technology, 2006, 20(5): 19-22. (in Chinese)
[8] PENG J K, HE H W, XIONG R. Rule based energy management strategy for a series-parallel plug-in hybrid electric bus optimized by dynamic programming[J]. Applied Energy, 2016, 185(Part 2): 1633-1643.
[9] LIU T, HU X S. A bi-level control for energy efficiency improvement of a hybrid tracked vehicle[J]. IEEE Transactions on Industrial Informatics, 2018, 14(4):1616-1625.
[10] VAGG C, AKEHURST S, BRACE C J, et al. Stochastic dyna-mic programming in the real-world control of hybrid electric vehicles[J]. IEEE Transactions on Control Systems Technology, 2016, 24(3):853-866.
[11] WIRASINFHA S G, EMADI A. Classification and review of control strategies for plug-in hybrid electric vehicles[C]∥Proceedings of 2009 IEEE Vehicle Power and Propulsion Conference. Dearborn, MI, US:IEEE, 2009:907-914.
[12] ZHENG C H, LI W M, LIANG Q. An energy management stra-tegy of hybrid energy storage systems for electric vehicle applications[J]. IEEE Transactions on Sustainable Energy, 2018, 9(4): 1880-1888.
[13] ENANG W, BANNISTER C. Modelling and control of hybrid electric vehicles (a comprehensive review) [J]. Renewable and Sustainable Energy Reviews, 2017, 74(7):1210-1239.
[14] CHEN Z, ZHANG X, MI C C. Slide mode and fuzzy logic based powertrain controller for the energy management and battery lifetime extension of series hybrid electric vehicles [J]. Journal of Asian Electric Vehicles, 2010, 8(2): 1425-1432.
[15] HUANG Y J, WANG H, KHAJEPOUR A, et al. Model predictive control power management strategies for HEVs: a review [J]. Journal of Power Sources, 2017, 341(2):91-106.
[16] SCIARRETTA A, BACK M, GUZZELLA L. Optimal control of parallel hybrid electric vehicles [J]. IEEE Transactions on Control Systems Technology, 2004, 12(3): 352-363.
[17] XIANG C L, DING F, WANG W D, et al. MPC-based energy management with adaptive Markov-chain prediction for a dual-mode hybrid electric vehicle [J]. SCIENCE CHINA Technological Sciences, 2017, 60(5): 737-748.
[18] QI X W, LUO Y D, WU G Y, et al. Deep reinforcement learning-based vehicle energy efficiency autonomous learning system [C]∥Proceedings of IEEE Intelligent Vehicles Symposium. Los Angeles, CA, US:IEEE, 2017:1228-1233.
[19] WU J D, HE H, PENG J, et al. Continuous reinforcement learning of energy management with deep Q network for a power split hybrid electric bus [J]. Applied Energy, 2018, 222: 799-811.
[20] 孙逢春, 张承宁. 装甲车辆混合动力电传动技术 [M]. 北京:国防工业出版社, 2008:288-301.
SUN F C, ZHANG C N. Technologies for the hybrid electric drive system of armored vehicle[M]. Beijing:National Defense Industry Press,2008:288-301. (in Chinese)

[1]	张伟，刘辉，韩立金，刘宝帅，张勋，张万年. 混合动力履带车辆机电联合制动控制[J]. 兵工学报, 2022, 43(5): 969-981.
[2]	王博洋, 关海杰, 龚建伟, 陈慧岩, 赵卉菁. 面向异构履带车辆的统一运动规划方法[J]. 兵工学报, 2022, 43(2): 241-251.
[3]	徐保荣, 王涛, 梁梓. 基于操纵动作预测的履带车辆载荷谱编制方法与流程[J]. 兵工学报, 2022, 43(2): 252-259.
[4]	陈路明，廖自力，马晓军，刘春光. 基于分层控制的混合动力车辆实时能量管理策略[J]. 兵工学报, 2021, 42(8): 1580-1591.
[5]	杜明刚，张金乐，张喜明，孙亚东，毛飞鸿. 履带装甲车辆空气阻力系数试验测定[J]. 兵工学报, 2021, 42(7): 1564-1568.
[6]	李睿，项昌乐，王超，范晶晶，刘春林. 自动驾驶履带车辆鲁棒自适应轨迹跟踪控制方法[J]. 兵工学报, 2021, 42(6): 1128-1137.
[7]	张杰，马晓军，刘春光，袁东，张运银. 双侧独立电驱动履带车辆反馈线性化解耦与预测行驶控制[J]. 兵工学报, 2021, 42(4): 697-705.
[8]	李春明, 吴维, 郭智蔷, 苑士华, 陈思. 履带车辆纵向与垂向耦合动力学模型及功率特性[J]. 兵工学报, 2021, 42(3): 449-458.
[9]	赵屹东, 陈慧岩, 胡家铭. 双侧独立电驱动履带车辆自动机械变速器线控换挡控制方法[J]. 兵工学报, 2021, 42(3): 459-467.
[10]	盖江涛，刘春生，马长军，沈宏继. 考虑履带滑转滑移的电驱动履带车辆转向控制[J]. 兵工学报, 2021, 42(10): 2092-2101.
[11]	马田，王志涛，张欣，庞大千，高晓宇. 履带车辆电传动系统减速机构热特性[J]. 兵工学报, 2021, 42(10): 2170-2179.
[12]	盖江涛，生辉，周广明，刘丽芳. 串联式混合动力履带车辆急加速工况功率平衡控制策略[J]. 兵工学报, 2021, 42(10): 2180-2188.
[13]	曾根，王伟达，盖江涛，马长军，李训明，李欢欢. 双侧电机耦合驱动履带车辆斜坡转向控制策略[J]. 兵工学报, 2021, 42(10): 2189-2195.
[14]	邹天刚，闫清东，盖江涛，侯威，王志涛，帅志斌，孙雪岩. 基于扁平电机的轻混式履带车辆综合传动方案[J]. 兵工学报, 2021, 42(10): 2233-2241.
[15]	张瑞增，龚建伟，陈慧岩，王博洋，刘海鸥. 基于无人驾驶单流传动的中型履带车辆大坡道起步控制[J]. 兵工学报, 2020, 41(8): 1590-1599.