Identification of Auditory and Visual Channel Workloads of Special Vehicle Operators Based on EEG Indicators

doi:10.12382/bgxb.2024.0815

Abstract

Abstract:

In order to effectively identify the visual and auditory channel workloads of operators during the operation of a special vehicle, a machine learning-based workload recognition model is constructed from the electroencephalogram (EEG) signals acquired in a simulated driving environment. A total of 30 participants were recruited for experiment, and the visual and auditory workload states were induced by increasing the scenario complexity and administering an auditory N-back task. The experimental results show that, the power spectral densities in δ, θ, and α bands in the frontal lobe, δ and θ bands in the temporal lobe, θ band in the occipital lobe, and all four frequency bands in the parietal lobe under the auditory workload condition are significantly higher than those under the visual workload condition. Moreover, the brain network has a stronger connectivity at θ and β bands under the auditory workload condition exhibites. Notably, the θ-band power spectral density (PSD) emerges as the most effective feature for the identification of visual and auditory workload channels, enabling the random forest algorithm to achieve a maximum classification accuracy of 95.68%. Shapley additive explanations (SHAP) analysis indicates that the frontal lobe contributes most significantly to the classification outcomes. These findings demonstrate the effectiveness of EEG-based indicators in identifying the visual and auditory channel workloads, providing a theoretical foundation for the development of adaptive interaction systems.

Key words: workload identification, electroencephalogram, machine learning, adaptive interactive systems

CLC Number:

TJ811

LIU Tiancheng, CHANG Ruosong, XIE Fang, JIANG Zebin, ZHANG Yijing, MAO Ming. Identification of Auditory and Visual Channel Workloads of Special Vehicle Operators Based on EEG Indicators[J]. Acta Armamentarii, 2025, 46(5): 240815-.

Figures/Tables 12

References 32

[1]	傅斌贺, 刘维平, 聂俊峰, 等. 考虑认知行为差异的乘员信息作业绩效研究[J]. 兵工学报, 2019, 40(3): 659-665. doi: 10.3969/j.issn.1000-1093.2019.03.026
	FU B H, LIU W P, NIE J F, et al. Research on crew’s information operation performance with the difference of cognitive behavior[J]. Acta Armamentarii, 2019, 40(3): 659-665. (in Chinese)
[2]	ZHOU Y Y, HUANG S, XU Z M, et al. Cognitive workload recognition using EEG signals and machine learning: a review[J]. IEEE Transactions on Cognitive and Developmental Systems, 2022, 14(3): 799-818.
[3]	SOLÍS-MARCOS I, KIRCHER K. Event-related potentials as indices of mental workload while using an in-vehicle information system[J]. Cognition, Technology & Work, 2019, 21(1): 55-67.
[4]	GHANI U, SIGNAL N, NIAZI I K, et al. A novel approach to validate the efficacy of single task ERP paradigms to measure cognitive workload[J]. International Journal of Psychophysiology, 2020, 158: 9-15. doi: 10.1016/j.ijpsycho.2020.09.007 pmid: 33045292
[5]	CAUSSE M, FABRE E, GIRAUDET L, et al. EEG/ERP as a measure of mental workload in a simple piloting task[J]. Procedia Manufacturing, 2015, 3: 5230-5236.
[6]	DIAZ-PIEDRA C, SEBASTIÁN M V, DI STASI L L. EEG theta power activity reflects workload among army combat drivers: an experimental study[J]. Brain Sciences, 2020, 10(4): 199.
[7]	WANG Q, SMYTHE D, CAO J, et al. Characterisation of cognitive load using machine learning classifiers of electroencephalogram data[J]. Sensors, 2023, 23(20): 8528.
[8]	PLECHAWSKA-WÓJCIK M, TOKOVAROV M, KACZOROWSKA M, et al. A three-class classification of cognitive workload based on EEG spectral data[J]. Applied Sciences, 2019, 9(24): 5340.
[9]	GORJI H T, WILSON N, VANBREE J, et al. Using machine learning methods and EEG to discriminate aircraft pilot cognitive workload during flight[J]. Scientific Reports, 2023, 13(1): 2507. doi: 10.1038/s41598-023-29647-0 pmid: 36782004
[10]	GUAN K, ZHANG Z M, CHAI X K, et al. EEG based dynamic functional connectivity analysis in mental workload tasks with different types of information[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2022, 30: 632-642. doi: 10.1109/TNSRE.2022.3156546 pmid: 35239485
[11]	DIMITRAKOPOULOS G N, KAKKOS I, DAI Z X, et al. Task-independent mental workload classification based upon common multiband EEG cortical connectivity[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2017, 25(11): 1940-1949. doi: 10.1109/TNSRE.2017.2701002 pmid: 28489539
[12]	毛明, 刘勇, 胡建军. 坦克装甲车辆综合电子信息系统的总体设计研究[J]. 兵工学报, 2017, 38(6): 1192-1202. doi: 10.3969/j.issn.1000-1093.2017.06.020
	MAO M, LIU Y, HU J J. Research on the overall design of integrated electronic information system for tanks and armored vehicles[J]. Acta Armamentarii, 2017, 38(6): 1192-1202. (in Chinese) doi: 10.3969/j.issn.1000-1093.2017.06.020
[13]	MA Z, ZHANG Y Q. Driver-automated vehicle interaction in mixed traffic: types of interaction and drivers’ driving styles[J]. Human Factors: The Journal of the Human Factors and Ergonomics Society, 2024, 66(2): 544-561.
[14]	DETJEN H, FALTAOUS S, PFLEGING B, et al. How to increase automated vehicles’ acceptance through In-vehicle interaction design: a review[J]. International Journal of Human-Computer Interaction, 2021, 37(4): 308-330.
[15]	LOEW A, KONIAKOWSKY I, FORSTER Y, et al. The impact of speech-based assistants on the driver’s cognitive distraction[J]. Accident Analysis & Prevention, 2023, 179: 106898.
[16]	MAO M, XIE F, HU J J, et al. Analysis of workload of tank crew under the conditions of informatization[J]. Defence Technology, 2014, 10(1): 17-21. doi: 10.1016/j.dt.2013.12.008
[17]	WICKENS C D. Multiple resources and mental workload[J]. Human Factors: The Journal of the Human Factors and Ergonomics Society, 2008, 50(3): 449-455.
[18]	孙晓东, 金晓萍, 解芳, 等. 多模态告警和认知负荷对装甲车辆乘员反应的影响[J]. 兵工学报, 2023, 44(4): 972-981.
	SUN X D, JIN X P, XIE F, et al. Effects of multimodal warning and cognitive load on the response of armored vehicle occupants[J]. Acta Armamentarii, 2023, 44(4): 972-981. (in Chinese) doi: 10.12382/bgxb.2022.0018
[19]	DOUDOU M, BOUABDALLAH A, BERGE-CHERFAOUI V. Driver drowsiness measurement technologies: current research, market solutions, and challenges[J]. International Journal of Intelligent Transportation Systems Research, 2020, 18(2): 297-319.
[20]	JIANG Z B, LI X Y, GE L Z, et al. Using multimodal methods and machine learning to recognize mental workload: distinguishing between underload, moderate load, and overload[J]. International Journal of Human-Computer Interaction, 2024, 41(8): 1-17.
[21]	ZARJAM P, EPPS J, CHEN F. Characterizing working memory load using EEG delta activity[C]// Proceedings of the 2011 19th European Signal Processing Conference. Barcelona, Spain:IEEE, 2011: 1554-1558.
[22]	KUTAFINA E, HEILIGERS A, POPOVIC R, et al. Tracking of mental workload with a mobile EEG sensor[J]. Sensors, 2021, 21(15): 5205.
[23]	FAVRETTO M A, RETTORE ANDREIS F, COSSUL S, et al. Motor unit behaviour at high force levels in diabetic patients with peripheral neuropathy[C]// Proceedings of the IX Latin American Congress on Biomedical Engineering and the XXVIII Brazilian Congress on Engineering Biomedical Engineering. Florianópolis, Brazil: Galoá Proceedings, 2024.
[24]	NIEDERMEYER E. Electrophysiology of the frontal lobe[J]. Clinical Electroencephalography, 2003, 34(1): 5-12.
[25]	ZHOZHIKASHVILI N, ZAKHAROV I, ISMATULLINA V, et al. Parietal alpha oscillations: cognitive load and mental toughness[J]. Brain Sciences, 2022, 12(9): 1135.
[26]	BEHRMANN M, GENG J J, SHOMSTEIN S. Parietal cortex and attention[J]. Current Opinion in Neurobiology, 2004, 14(2): 212-217. doi: 10.1016/j.conb.2004.03.012 pmid: 15082327
[27]	CHIKHI S, MATTON N, BLANCHET S. EEG power spectral measures of cognitive workload: a meta-analysis[J]. Psychophysiology, 2022, 59(6): e14009.
[28]	DING Y, CAO Y Q, DUFFY V G, et al. Measurement and identification of mental workload during simulated computer tasks with multimodal methods and machine learning[J]. Ergonomics, 2020, 63(7): 896-908. doi: 10.1080/00140139.2020.1759699 pmid: 32330080
[29]	VAN TULDER G, DE BRUIJNE M. Learning cross-modality representations from multi-modal images[J]. IEEE Transactions on Medical Imaging, 2018, 38(2): 638-648.
[30]	FONSECA A, KERICK S, KING J T, et al. Brain network changes in fatigued drivers: a longitudinal study in a real-world environment based on the effective connectivity analysis and actigraphy data[J]. Frontiers in Human Neuroscience, 2018, 12: 418. doi: 10.3389/fnhum.2018.00418 pmid: 30483080
[31]	JACKSON A F, BOLGER D J. The neurophysiological bases of EEG and EEG measurement: a review for the rest of us[J]. Psychophysiology, 2014, 51(11): 1061-1071. doi: 10.1111/psyp.12283 pmid: 25039563
[32]	JUSTESEN A B, FOGED M T, FABRICIUS M, et al. Diagnostic yield of high-density versus low-density EEG: the effect of spatial sampling, timing and duration of recording[J]. Clinical Neurophysiology, 2019, 130(11): 2060-2064. doi: S1388-2457(19)31193-9 pmid: 31541983

频段	脑区	统计分析
频段	脑区	统计量	p	效应量
	额叶	2.46	0.030^*	0.963
δ	颞叶	2.22	0.046^*	0.871
	枕叶	2.15	0.053	0.842
	顶叶	2.31	0.039^*	0.906
	额叶	5.92	0.001^***	2.320
θ	颞叶	5.22	0.001^***	2.050
	枕叶	5.29	0.001^***	2.080
	顶叶	5.64	0.001^***	2.210
	额叶	2.58	0.016^*	1.010
α	颞叶	0.25	0.805	0.098
	枕叶	1.27	0.218	0.496
	顶叶	2.10	0.046^*	0.824
	额叶	1.25	0.233	0.490
β	颞叶	-1.62	0.129	0.635
	枕叶	0.53	0.602	0.208
	顶叶	2.36	0.027^*	0.927

频段	脑区	统计分析
频段	脑区	统计量	p	效应量
	额叶	2.46	0.030^*	0.963
δ	颞叶	2.22	0.046^*	0.871
	枕叶	2.15	0.053	0.842
	顶叶	2.31	0.039^*	0.906
	额叶	5.92	0.001^***	2.320
θ	颞叶	5.22	0.001^***	2.050
	枕叶	5.29	0.001^***	2.080
	顶叶	5.64	0.001^***	2.210
	额叶	2.58	0.016^*	1.010
α	颞叶	0.25	0.805	0.098
	枕叶	1.27	0.218	0.496
	顶叶	2.10	0.046^*	0.824
	额叶	1.25	0.233	0.490
β	颞叶	-1.62	0.129	0.635
	枕叶	0.53	0.602	0.208
	顶叶	2.36	0.027^*	0.927

频段	指标	统计分析
频段	指标	统计量	p	效应量
	节点度数	2.00	0.054	0.823
δ	集聚系数	0.88	0.393	0.345
	特征路径长度	2.10	0.054	0.823
	全局效率	0.88	0.391	0.345
	节点度数	4.84	0.001^***	1.900
θ	集聚系数	6.36	0.001^***	2.490
	特征路径长度	4.84	0.001^***	1.900
	全局效率	1.28	0.223	0.501
	节点度数	1.67	0.109	0.654
α	集聚系数	1.37	0.183	0.537
	特征路径长度	1.67	0.109	0.654
	全局效率	0.11	0.914	0.043
	节点度数	2.35	0.027^*	0.923
β	集聚系数	2.70	0.012^*	1.060
	特征路径长度	2.35	0.027^*	0.923
	全局效率	1.33	0.194	0.524

频段	指标	统计分析
频段	指标	统计量	p	效应量
	节点度数	2.00	0.054	0.823
δ	集聚系数	0.88	0.393	0.345
	特征路径长度	2.10	0.054	0.823
	全局效率	0.88	0.391	0.345
	节点度数	4.84	0.001^***	1.900
θ	集聚系数	6.36	0.001^***	2.490
	特征路径长度	4.84	0.001^***	1.900
	全局效率	1.28	0.223	0.501
	节点度数	1.67	0.109	0.654
α	集聚系数	1.37	0.183	0.537
	特征路径长度	1.67	0.109	0.654
	全局效率	0.11	0.914	0.043
	节点度数	2.35	0.027^*	0.923
β	集聚系数	2.70	0.012^*	1.060
	特征路径长度	2.35	0.027^*	0.923
	全局效率	1.33	0.194	0.524

频段	特征	不同分类器的准确率/%
频段	特征	SVM	DT	RF	KNN	GBDT	XGBoost
	脑网络	44.05	41.77	42.58	61.99	41.89	51.34
δ	PSD	77.36	76.16	76.92	74.89	73.67	77.67
	双模态	78.70	41.82	72.82	66.55	69.75	74.2
	脑网络	82.89	82.15	78.18	78.66	80.11	82.74
θ	PSD	89.74	92.82	95.68	95.25	92.44	92.36
	双模态	89.21	89.21	94.04	80.79	88.23	92.63
	脑网络	59.56	68.02	47.36	55.97	67.40	64.33
α	PSD	41.52	53.02	52.65	56.80	41.83	55.7
	双模态	59.11	41.87	58.75	55.94	63.45	60.27
	脑网络	64.26	41.68	49.45	72.81	41.97	67.47
β	PSD	41.40	63.12	54.92	48.76	62.72	57.98
	双模态	64.05	41.75	64.59	71.47	41.79	65.71