Remote Brain-controlled Unmanned Aerial Vehicle System Based on Brain-machine Interface and Human-machine Closed Loop

doi:10.12382/bgxb.2023.0866

Abstract

Abstract:

With the rapid evolution of modern military warfare, the remote brain-controlled unmanned aerial vehicle (UAV) systems are playing an increasingly important role in battlefield information gathering, target surveillance, and tactical deployment. This research proposes a compressed sensing control paradigm and a human-machine closed-loop control algorithm for remote brain-controlled UAV. Based on this control paradigm and algorithm, a remote brain-controlled UAV system for military applications is constructed. Online experiments conducted in this study demonstrate that eight participants successfully completed the navigation tasks using the brain-controlled UAV system based on the compressed sensing control paradigm and human-machine closed-loop control algorithm. The average task completion rate of the proposed brain-controlled UAV system is 0.95, and its average task completion time is 100.46s, which significantly outperformes the brain-controlled UAV system based on human-machine open-loop control algorithms. In the future, the proposed brain-controlled UAV system can be used for battlefield reconnaissance in military scenarios, significantly enhancing the remote-control capabilities of military personnel and expanding their battlefield awareness.

Key words: brain-controlled unmanned aerial vehicle, brain-machine interface, human-machine closed-loop, compressed sensing

CLC Number:

TJ85

LIU Siyu, ZHANG Deyu, MING Zhiyuan, LIU Mengzhen, LIU Ziyu, CHEN Qiming, ZHANG Jian, WU Jinglong, YAN Tianyi. Remote Brain-controlled Unmanned Aerial Vehicle System Based on Brain-machine Interface and Human-machine Closed Loop[J]. Acta Armamentarii, 2024, 45(9): 3191-3203.

Figures/Tables 20

Fig.1 Schematic diagram of traditional SSVEP control paradigm

Fig.2 Compressed sensing control paradigm

Fig.3 Realization process of compressed sensing control paradigm

Fig.4 Significant increase in flash stimulation-induced energy value of corresponding frequency band (taking 8.0Hz flash stimulation as an example)

Fig.5 Schematic diagram of reverse speed vector when a UAV is close to the enemy aircraft

Fig.6 Schematic diagram of the original UAV control instruction generation method

Fig.7 Control process of PID

Fig.8 Brain-controlled UAV experiment scenarios

Fig.9 Schematic diagram of traditional SSVEP control paradigm

Fig.10 Experimental flowchart

Fig.11 The power chart of brain signal induced by the two control paradigms when the subjects pay attention to the flashes of 8.0Hz

Fig.12 The average classification accuracy under the two control paradigms

Fig.13 The average ITR under two control paradigms

Fig.14 The average classification accuracy and ITR of the subjects with analysis duration of 2.0s (* denotes significant differences found aftert-tests: P<0.05)

Fig.15 The success rate of the task of completing the three experimental tasks (*:P<0.05)

Fig.16 The test completion time of completing the three experimental tasks (*:P<0.05)

Fig.17 The relationship between the number of enemy aircraft and the task completion rate of remote UAV under the two control algorithms

Fig.18 Average value of task completion rate of remote UAV under the two control algorithms(*:P< 0.05)

Fig.19 The relationship between the number of enemy aircraft and the task completion time of remote UAV under the two control algorithms

Fig.20 Average value of task completion time of remote UAV under the two control algorithms (*:P<0.05)

References 23

[1]	陈慧岩, 张玉. 军用地面无人机动平台技术发展综述[J]. 兵工学报, 2014, 35(10):1696-1706. doi: 10.3969/j.issn.1000-1093.2014.10.026
	CHEN H Y, ZHANG Y. An overview of research on military unmanned ground vehicles[J]. Acta Armamentarii, 2014, 35(10):1696-1706. (in Chinese)
[2]	张楠, 王潇, 王军力. 无人机在陆军精确打击体系中的应用研究[J]. 兵工学报, 2010, 31(增刊2):107-111.
	ZHANG N, WANG X, WANG J L. An overview of research on military unmanned ground vehicles[J]. Acta Armamentarii, 2010, 31(S2):107-111. (in Chinese)
[3]	LIU S Y, ZHANG D Y, QIAO M, et al. Mind controlled vehicle based on lidar SLAM navigation and SSVEP technology[C]//Proceedings of the 2021 9th International Winter Conference on Brain-Computer Interface. Gangwon, Korea:IEEE, 2021:1-4.
[4]	ZHAO J, LI W, LI M F. Comparative study of SSVEP-and P300-based models for the telepresence control of humanoid robots[J]. PLoS One, 2015, 10(11): e0142168.
[5]	GUO N, WANG X J, DUANMU D H, et al. SSVEP-based brain computer interface controlled soft robotic glove for post-stroke hand function rehabilitation[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2022, 30:1737-1744.
[6]	ZHANG W C, TAN C Q, SUN F C, et al. A review of EEG-based brain-computer interface systems design[J]. Brain Science Advances, 2018, 4(2):156-167.
[7]	DEHZANGI O, FAROOQ M. Portable brain-computer interface for the intensive care unit patient communication using subject-dependent SSVEP ide.pngication[J]. BioMed Research International, 2018:1-14.
[8]	CHEN X G, WANG Y J, NAKANISHI M, et al. High-speed spelling with a noninvasive brain-computer interface[C]//Proceedings of the National Academy of Sciences of the United States of America, 2015, 112(44):E6058-E6067.DOI:10.1073/pnas.1508080112.
[9]	NAKANISHI M, WANG Y J, CHEN X G, et al. Enhancing detection of SSVEPs for a high-speed brain speller using task-related component analysis[J]. IEEE Transactions on Biomedical Engineering, 2018, 65(1):104-112.
[10]	LIU B C, CHEN X G, SHI N L, et al. Improving the performance of individually calibrated SSVEP-BCI by task-discriminant component analysis[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2021, 29:1998-2007.
[11]	CHEN X G, LIU B C, WANG Y J, et al. A spectrally-dense encoding method for designing a high-speed SSVEP-BCI with 120 stimuli[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2022, 30:2764-2772. doi: 10.1109/TNSRE.2022.3208717 pmid: 36136927
[12]	李家伟, 张洪欣, 徐瑞林. 基于脑机接口的无人机编队控制系统设计[J]. 航空科学技术, 2023, 34(2):104-110.
	LI J W, ZHANG H X, XU R L. Design of UAV formation control system based on brain-computer interface data[J]. Aeronautical Science & Technology, 2023, 34(2):104-110. (in Chinese)
[13]	刘亚群. 多模态脑控无人机地面站软件系统设计与实现[J]. 现代工程科技, 2023, 2(6):45-48.
	LIU Y Q. Design and implementation of multimodal brain-controlled aerial vehicle ground station software system modern[J]. Engineering Technology, 2023, 2(6):45-48. (in Chinese)
[14]	HIRECHE A, ZENNAIA Y, AYAD R, et al. A decoding algorithm for non-invasive SSVEP-based drone flight control[C]//Proceedings of the 2021 IEEE International Conference on Bioinformatics and Biomedicine. Houston, TX, US:IEEE, 2021:3616-3623.
[15]	CHIUZBAIAN A, JAKOBSEN J, PUTHUSSERYPADY S. Mind controlled drone:an innovative multiclass SSVEP based brain computer interface[C]//Proceedings of the 2019 7th International Winter Conference on Brain-Computer Interface. Gangwon, Korea:IEEE, 2019:1-5.
[16]	ZHANG D Y, LIU S Y, ZHANG J, et al. Brain-controlled 2D navigation robot based on a spatial gradient controller and predictive environmental coordinator[J]. IEEE Journal of Biomedical and Health Informatics, 2022, 26(12):6138-6149.
[17]	CHEN Y H, YANG C, CHEN X G, et al. A novel training-free recognition method for SSVEP-based BCIs using dynamic window strategy[J]. Journal of Neural Engineering, 2021, 18(3):036007.
[18]	YIN E W, ZHOU Z T, JIANG J, et al. A dynamically optimized SSVEP brain-computer interface(BCI) speller[J]. IEEE Transactions on Biomedical Engineering, 2014, 62(6):1447-1456.
[19]	KALUNGA E K, CHEVALLIER S, et al. Online SSVEP-based BCI using riemannian geometry[J]. Neurocomputing, 2016, 191:55-68.
[20]	LAFLEUR K, CASSADY K, DOUD A, et al. Quadcopter control in three-dimensional space using a noninvasive motor imagery-based brain-computer interface[J]. Journal of Neural Engineering, 2013, 10(4):046003.
[21]	KIM S, LEE S, KANG H, et al. P300 brain-computer interface-based drone control in virtual and augmented reality[J]. Sensors, 2021, 21(17):5765.
[22]	PASZKIEL S, SIKORA M. The use of brain-computer interface to control unmanned aerial vehicle[C]//Proceedings of the Automation 2019:Progress in Automation, Robotics and Measurement Techniques. Warsaw, Poland: Springer, 2020: 583-598.
[23]	DMYTRIYEV Y, INSERO F, CARNEVALE M, et al. Brain-computer interface and hand-guiding control in a human-robot collaborative assembly task[J]. Machines, 2022, 10(8):654.