A Human-machine Collaborative Control Algorithm for Intelligent Vehicles Based on Model Prediction and Policy Learning

doi:10.12382/bgxb.2022.0815

Abstract

Abstract:

A human-machine collaborative control algorithm based on model prediction and policy learning is proposed for the optimal decision-making and high maneuvering motion control of intelligent vehicles in complex environments. The algorithm takes advantage of the human driver’s understanding of the environment and comprehensive processing ability to assist the machine in local trajectory planning at the decision planning level, including speed adjustment and dynamic path generation, to achieve the human-machine collaboration.For the timeliness of the online optimal planning and control of vehicles with high maneuverability, on the one hand, a long sampling interval and a simplified dynamics model are used to design a local trajectory planning method based on model predictive control at the local planning level in order to achieve efficient online trajectory optimization. On the other hand, a learning-based predictive control method based on rolling time-domain reinforcement learning is used to optimize the control strategy in the control layer in order to improve the computational efficiency and adaptability of online optimal control. In the driving simulation on the mountain highway with the driver in the loop, the proposed method not only complies with the driver’s acceleration and deceleration commands and steering commands to generate a safe and smooth planning trajectory for human-machine cooperation, but also can accurately control the vehicle to travel along the desired trajectory in real time. In the human-machine cooperative control mode, the time to complete the same driving task is reduced by 8.3% on average and the steering operation load is reduced by 51.1% compared with the manual driving by six ordinary drivers.

Key words: intelligent vehicles, human-machine collaboration, high maneuvering motion, reinforcement learning, model predictive control

CLC Number:

TP13

JIANG Yan, DING Yuyan, ZHANG Xinglong, XU Xin. A Human-machine Collaborative Control Algorithm for Intelligent Vehicles Based on Model Prediction and Policy Learning[J]. Acta Armamentarii, 2023, 44(11): 3465-3477.

Figures/Tables 19

Fig.1 Framework of human-machine collaborative control system for high maneuvering motion of intelligent vehicle

Fig.2 Vehicle dynamics model considering yaw, pitch and roll

Fig.3 Influence of road grade on vehicle dynamics model

Fig.4 Schematic diagram of vehicle acceleration limit

Fig.5 Schematic diagram of local driving route planning sprinkler sampling (o1 is a static obstacle, o2 is a dynamic obstacle, o2,k represents the polygonal envelope of obstacle 2 after collision processing at time k, and s is the direction of road centreline)

Fig.6 Point mass vehicle dynamics model

Fig.7 Schematic diagram of vehicle trajectory tracking error

Fig.8 Structure diagram of intelligent vehicle high maneuvering motion controller based on learning predictive control

Fig.9 Mountain road scene and driving route

Table 1 Vehicle model parameters, and planner and controller parameters

参数	数值	参数	数值
m/kg	1555	γ	0.95
h_CG/m	0.665	n_s	15
I_x/(kg·m²)	846.6	ζ	1.1
I_y/(kg·m²)	1 816	n_d	6
I_z/(kg·m²)	1 816	N_p	30
l_f/m	1.85	T_p/s	0.2
l_r/m	1.80	q_c	0.01
K_ϕ/(N·m·rad^-1)	51 600	q_l	0.1
D_ϕ/(N·m·s·rad^-1)	5 300	q_ω	5.0
K_θ/(N·m·rad^-1)	45 000	$r a x$	0.01
D_θ/(N·m·s·rad^-1)	2 600	$r a y$	0.5
C_cf/(N·rad^-1)	76 500	$r Δ a x$	0.2
C_cr/(N·rad^-1)	76 500	$r Δ a y$	1
w/m	1.7	N_c	50
a_y,_max/(m·s^-2)	4	T_c/s	0.02
a_y,_min/(m·s^-2)	-4	i_max	5
a_x,_max/(m·s^-2)	-4	/m	0.2
a_x,_min/(m·s^-2)	-6	$q e x$	0.2
w₁	0.5	$q e y$	0.2
w₂	3.0	$q e φ$	0.1
w₃	0.5	$r F x$	10^-9
w₄	0.5	$r δ f$	0.05

Table 1 Vehicle model parameters, and planner and controller parameters

参数	数值	参数	数值
m/kg	1555	γ	0.95
h_CG/m	0.665	n_s	15
I_x/(kg·m²)	846.6	ζ	1.1
I_y/(kg·m²)	1 816	n_d	6
I_z/(kg·m²)	1 816	N_p	30
l_f/m	1.85	T_p/s	0.2
l_r/m	1.80	q_c	0.01
K_ϕ/(N·m·rad^-1)	51 600	q_l	0.1
D_ϕ/(N·m·s·rad^-1)	5 300	q_ω	5.0
K_θ/(N·m·rad^-1)	45 000	$r a x$	0.01
D_θ/(N·m·s·rad^-1)	2 600	$r a y$	0.5
C_cf/(N·rad^-1)	76 500	$r Δ a x$	0.2
C_cr/(N·rad^-1)	76 500	$r Δ a y$	1
w/m	1.7	N_c	50
a_y,_max/(m·s^-2)	4	T_c/s	0.02
a_y,_min/(m·s^-2)	-4	i_max	5
a_x,_max/(m·s^-2)	-4	/m	0.2
a_x,_min/(m·s^-2)	-6	$q e x$	0.2
w₁	0.5	$q e y$	0.2
w₂	3.0	$q e φ$	0.1
w₃	0.5	$r F x$	10^-9
w₄	0.5	$r δ f$	0.05

Table 2 Performance indicators

符号	描述
t_task	完成驾驶任务总时长
L_2c	车辆在违反安全区域边界情况下的行驶距离
$E δ h$	转向操作做功,定义为 $E δ h$ = $∫ t t a s k 0$ ( $δ h 2$ + $δ · h 2$ )dt

Table 2 Performance indicators

符号	描述
t_task	完成驾驶任务总时长
L_2c	车辆在违反安全区域边界情况下的行驶距离
$E δ h$	转向操作做功,定义为 $E δ h$ = $∫ t t a s k 0$ ( $δ h 2$ + $δ · h 2$ )dt

Fig.10 Simulated results of human-machine collaborative driving in uphill scene

Fig.11 Simulated results of manual driving in uphill scene

Fig.12 Simulated results of human-machine collaborative driving in downhill scene

Fig.13 Simulated results of manual driving in downhill scene

Fig.14 Simulated results of the scene of vehicles meeting on the winding mountain road

Fig.15 Positions and trajectories of two vehicles at different times

Table 3 Statistics of performance indicators of different control methods in mountain road scene

方法	指标	测试人员
方法	指标	1	2	3	4	5	6
手动驾驶	t_task/s	268.4	244.9	247.9	233.5	227.7	248.1
	L₂_c/m	190.2	6.0	29.0	56.8	96.9	173.7
	$E δ h$	2945	2155	2089	2766	2587	1037
对比方法^[21]	t_task/s	226.1	237.4	238.7	229.3	245.6	232.4
	L₂_c/m	2.7	8.5	3.1	5.5	0	1.8
	$E δ h$	1557	1157	1256	1564	1437	975
本文方法	t_task/s	224.6	217.9	234.8	218.9	234.5	217.4
	L₂_c/m	0	0	0	0	0	0
	$E δ h$	1717	914	1011	975	1168	852

Table 3 Statistics of performance indicators of different control methods in mountain road scene

方法	指标	测试人员
方法	指标	1	2	3	4	5	6
手动驾驶	t_task/s	268.4	244.9	247.9	233.5	227.7	248.1
	L₂_c/m	190.2	6.0	29.0	56.8	96.9	173.7
	$E δ h$	2945	2155	2089	2766	2587	1037
对比方法^[21]	t_task/s	226.1	237.4	238.7	229.3	245.6	232.4
	L₂_c/m	2.7	8.5	3.1	5.5	0	1.8
	$E δ h$	1557	1157	1256	1564	1437	975
本文方法	t_task/s	224.6	217.9	234.8	218.9	234.5	217.4
	L₂_c/m	0	0	0	0	0	0
	$E δ h$	1717	914	1011	975	1168	852

Fig.16 Performance comparison of different control methods in mountain road scene

References 21

[1]	陈慧岩, 陈舒平, 龚建伟. 智能汽车横向控制方法研究综述[J]. 兵工学报, 2017, 38(6):1203-1214. doi: 10.3969/j.issn.1000-1093.2017.06.021
	CHEN H Y, CHEN S P, GONG J W. Review on the research of lateral control for intelligent vehicles[J]. Acta Armamentarii, 2017, 38(6):1203-1214. (in Chinese)
[2]	胡云峰, 曲婷, 刘俊, 等. 智能汽车人机协同控制的研究现状与展望[J]. 自动化学报, 2019, 45(7):1261-1280.
	HU Y F, QU T, LIU J, et al. Human-machine cooperative control of intelligent vehicle: recent developments and future perspectives[J]. Acta Automatica Sinica, 2019, 45(7):1261-1280. (in Chinese)
[3]	ANSARI S, NAGHDY F, DU H P. Human-machine shared driving: challenges and future directions[J]. IEEE Transactions on Intelligent Vehicles, 2022, 7(3):499-519. doi: 10.1109/TIV.2022.3154426 URL
[4]	刘俊. 智能车辆人机协同转向控制策略研究[D]. 长春: 吉林大学, 2020:13-29.
	LIU J. Research on driver-automation cooperative steering control strategy of intelligent vehicle[D]. Changchun: Jilin University, 2020:13-29. (in Chinese)
[5]	LI M J, CAO H T, SONG X L, et al. Shared control driver assistance system based on driving intention and situation assessment[J]. IEEE Transactions on Industrial Informatics, 2018, 14(11): 4982-4994. doi: 10.1109/TII.2018.2865105 URL
[6]	XU S B, PENG H. Design, analysis, and experiments of preview path tracking control for autonomous vehicles[J]. IEEE Transactions on Intelligent Transportation Systems, 2019, 21(1): 48-58. doi: 10.1109/TITS.6979 URL
[7]	JI X W, HE X K, LÜ C, et al. Adaptive-neural-network-based robust lateral motion control for autonomous vehicle at driving limits[J]. Control Engineering Practice, 2018, 76: 41-53. doi: 10.1016/j.conengprac.2018.04.007 URL
[8]	MOUSAVINEJAD E, HAN Q L, YANG F, et al. Integrated control of ground vehicles dynamics via advanced terminal sliding mode control[J]. Vehicle System Dynamics, 2017, 55(2): 268-294. doi: 10.1080/00423114.2016.1256489 URL
[9]	NIE L Z, GUAN J Y, LU C H, et al. Longitudinal speed control of autonomous vehicle based on a self‐adaptive PID of radial basis function neural network[J]. IET Intelligent Transport Systems, 2018, 12(6): 485-494. doi: 10.1049/itr2.v12.6 URL
[10]	BEAl C E, GERDES J C. Model predictive control for vehicle stabilization at the limits of handling[J]. IEEE Transactions on Control Systems Technology, 2012, 21(4):1258-1269. doi: 10.1109/TCST.2012.2200826 URL
[11]	JI J, KHAJEPOUR A, MELEK W W, et al. Path planning and tracking for vehicle collision avoidance based on model predictive control with multiconstraints[J]. IEEE Transactions on Vehicular Technology, 2016, 66(2): 952-964. doi: 10.1109/TVT.2016.2555853 URL
[12]	LINIGER A, DOMAHIDI A, MORARI M. Optimization-based autonomous racing of 1: 43 scale RC cars[J]. Optimal Control Applications and Methods, 2015, 36(5): 628-647. doi: 10.1002/oca.v36.5 URL
[13]	LIAN C Q, XU X, CHEN H, et al. Near-optimal tracking control of mobile robots via receding-horizon dual heuristic programming[J]. IEEE Transactions on Cybernetics, 2015, 46(11):2484-2496. doi: 10.1109/TCYB.2015.2478857 URL
[14]	ZHANG X L, JIANG Y, LU Y, et al. A receding-horizon reinforcement learning approach for kinodynamic motion planning of autonomous vehicles[J]. IEEE Transactions on Intelligent Vehicles, 2022, 7(3): 556-568. doi: 10.1109/TIV.2022.3167271 URL
[15]	龚建伟, 姜岩, 徐威. 无人驾驶车辆模型预测控制[M]. 北京: 北京理工大学出版社, 2020.
	GONG J W, JIANG Y, XU W. Model predictive control for self-driving vehicles[M]. Beijing: Beijing Institute of Technology Press, 2020. (in Chinese)
[16]	BERNTORP K. Derivation of a six degrees-of-freedom ground-vehicle model for automotive applications[R]. Sweden: Department of Automatic Control, Lund University, Technical Report ISRN LUTFD2/TFRT-7627-SE, 2013.
[17]	SNIDER J M. Automatic steering methods for autonomous automobile path tracking[R]. Pittsburgh,PA,US:Robotics Institute, Technical Report CMU-RITR-09-08, 2009.
[18]	CHEN P Y. Effects of the entropy weight on TOPSIS[J]. Expert Systems with Applications, 2021, 168: 114186. doi: 10.1016/j.eswa.2020.114186 URL
[19]	LINIGER A, DOMAHIDI A, MORARI M. Optimization-based autonomous racing of 1: 43 scale RC cars[J]. Optimal Control Applications and Methods, 2015, 36(5):628-647. doi: 10.1002/oca.v36.5 URL
[20]	FRISON G, DIEHL M. HPIPM:a high-performance quadratic programming framework for model predictive control[J]. IFAC-PapersOnLine, 2020, 53(2): 6563-6569. doi: 10.1016/j.ifacol.2020.12.073 URL
[21]	YUE M, FANG C, ZHANG H Z, et al. Adaptive authority allocation-based driver-automation shared control for autonomous vehicles[J]. Accident Analysis & Prevention, 2021, 160: 106301. doi: 10.1016/j.aap.2021.106301 URL