Surface Electromyography-based Human Motion Pattern Recognition Using Convolutional Neural Networks

doi:10.12382/bgxb.2023.0125

Abstract

Abstract:

Along with the rapid development of surface electromyography (sEMG)-controlled devices, such as exoskeleton robots, the application of non-stationary and aperiodic signals in advanced high-performance motion recognition system has become a notable focus in relevant fields. To achieve the cross-domain feature fusion of sEMG, a dual convolutional chains neural network based on sEMG signals is proposed. Raw sEMG signals of seven key differentiated muscles are collected and processed by feature extraction methods, and converted into energy kernel phase diagram and discrete wavelet transform coefficient feature map, which are respectively input into the CNN branch and the MobileNetV2 branch of dual convolutional chains neural networks. The extracted high-level hidden features are processed by the fusion module for full interaction. Three datasets, including the above two feature images and conventional electromyography spectrum, are prepared. Three sets of cross experiments show that the average recognition accuracy of the proposed method for six self-tested lower limb movements is 94.19%, which is significantly better than other feature combinations and network architectures. Meanwhile, seven lower limb movements are identified with 98.32% steady-state recognition accuracy in ENABL3S open source dataset, further proving that the proposed method has excellent sEMG feature capture ability and pattern recognition accuracy.

Key words: exoskeleton robots, surface electromyography, motion pattern recognition, dual convolutional chains neural network, energy kernel, wavelet transform analysis

CLC Number:

TN911.7

LIU Yali, LU Yanchi, MA Xunju, SONG Qiuzhi. Surface Electromyography-based Human Motion Pattern Recognition Using Convolutional Neural Networks[J]. Acta Armamentarii, 2024, 45(7): 2144-2158.

Figures/Tables 20

References 30

[1]	刘亚丽, 宋遒志, 赵明升, 等. 基于力位混合控制的踝关节外骨骼机器人四段式助力技术[J]. 兵工学报, 2021, 42(12): 2722-2730. doi: 10.3969/j.issn.1000-1093.2021.12.020
	LIU Y L, SONG Q Z, ZHAO M S, et al. The four-stage assisted technology of flexible ankle exoskeleton robot based on force and position hybrid control[J]. Acta Armamentarii, 2021, 42(12): 2722-2730. (in Chinese) doi: 10.3969/j.issn.1000-1093.2021.12.020
[2]	BAKIYA A, KAMALANAND K, RAJINIKANTH V, et al. Deep neural network assisted diagnosis of time-frequency transformed electromyograms[J]. Multimedia Tools and Applications, 2020, 79(15/16):11051-11067.
[3]	MENDES J J A, FREITAS M L B, SIQUEIRA H V, et al. Feature selection and dimensionality reduction: an extensive comparison in hand gesture classification by sEMG in eight channels armband approach[J]. Biomedical Signal Processing and Control, 2020, 5(59):101920.1-101920.11.
[4]	RABIN N, KAHLON M, MALAYEV S, et al. Classification of human hand movements based on EMG signals using nonlinear dimensionality reduction and data fusion techniques[J]. Expert Systems with Applications, 2020, 7(149):113281.1-113281.13.
[5]	XIONG D Z, ZHANG D H, ZHAO X G, et al. Deep learning for EMG-based human-machine interaction:a review[J]. IEEE/CAA Journal of Automatica Sinica, 2021, 8(3):512-533.
[6]	YANG Z W, JIANG D, SUN Y, et al. Dynamic gesture recognition using surface EMG signals based on multi-stream residual network[J]. Frontiers in Bioengineering and Biotechnology, 2021, 9(10):779353-779366.
[7]	OZDEMIR M A, KISA D H, GUREN O, et al. Hand gesture classification using time-frequency images and transfer learning based on CNN[J]. Biomedical Signal Processing and Control, 2022, 8(77):103787.1-103787.16.
[8]	ZHAI X L, JELFS B, CHAN R H M, et al. Self-recalibrating surface EMG pattern recognition for neuroprosthesis control based on convolutional neural network[J]. Frontiers in Neuroscience, 2017, 7(11): 379.
[9]	TAM S, BOUKADOUM M, CAMPEAU-LECOURS A, et al. A fully embedded adaptive real-time hand gesture classifier leveraging HD-sEMG and deep learning[J]. IEEE Transactions on Biomedical Circuits and Systems, 2020, 14(2): 232-243. doi: 10.1109/TBCAS.2019.2955641 pmid: 31765319
[10]	IMONYAN K, ZISSERMAN A. Two-stream convolutional networks for action recognition in videos[C]//Proceedings of the 28th Conference on Neural Information Processing Systems. Montreal, Canada: MIT Press, 2014: 1-11.
[11]	HU Y, WONG Y K, WEI W T, et al. A novel attention-based hybrid CNN-RNN architecture for sEMG-based gesture recognition[J]. Plos One, 2018, 13(10): e206049.
[12]	COTE-ALLARD U, FALL C L, DROUIN A, et al. Deep learning for electromyographic hand gesture signal classification using transfer learning[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2019, 27(4): 760-771.
[13]	陈幸, 殷跃红, 范渊杰. 基于sEMG振子模型的骨骼肌等长收缩力与固有特性的能量核表征方法[J]. 科学通报, 2014, 59(7): 561-571.
	CHEN X, YIN Y H, FAN Y J. EMG oscillator model-based energy kernel method for characterizing muscle intrinsic property under isometric contraction[J]. Chinese Science Bulletin, 2014, 59(7): 561-571. (in Chinese)
[14]	石欣, 朱家庆, 秦鹏杰, 等. 基于改进能量核的下肢表面肌电信号特征提取方法[J]. 仪器仪表学报, 2020, 41(1): 121-128.
	SHI X, ZHU J Q, QIN P J, et al. Feature extraction method of lower limb surface EMG signal based on improved energy nucleus[J]. Chinese Journal of Scientific Instrument, 2020, 41(1):121-128. (in Chinese)
[15]	董泽萍, 仇大伟, 刘静. 表面肌电人体下肢动作识别预测进展[J]. 计算机工程与应用, 2023, 60(2):1-15.
	DONG Z P, QIU D W, LIU J. Progress in human lower extremity motion recognition and prediction based on surface electromyography[J]. Computer Engineering and Applications, 2023, 60(2): 1-15. (in Chinese)
[16]	WEI W T, DAI Q F, WONG Y K, et al. Surface-electromyography-based gesture recognition by multi-view deep learning[J]. IEEE Transactions on Biomedical Engineering, 2019, 66(10): 2964-2973.
[17]	卫文韬, 李亚军. 基于双流卷积神经网络的肌电信号手势识别方法[J]. 计算机集成制造系统, 2022, 28(1):124-131.
	WEI W T, LI Y J. Surface electromyography based gesture recognition based on dual-stream CNN[J]. Computer Integrated Manufacturing Systems, 2022, 28(1): 124-131. (in Chinese)
[18]	PINZÓN-ARENAS J O, JIMÉNEZ-MORENO R, HERRERA-BENAVIDES J E. Convolutional neural network for hand gesture recognition using 8 different EMG signals[C]//Proceedings of the 22nd Symposium on Image, Signal Processing and Artificial Vision. Bucaramanga, Colombia: IEEE, 2019: 1-5.
[19]	KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017, 60(6): 84-90.
[20]	姜海燕, 李竹韵, 陈艳. 基于改进小脑模型的sEMG下肢关节力矩预测[J]. 仪器仪表学报, 2022, 43(11): 172-180.
	JIANG H Y, LI Z Y, CHEN Y. Joint torque prediction of lower limb of sEMG signals based on improved cerebellar model[J]. Chinese Journal of Scientific Instrument, 2022, 43(11): 172-180. (in Chinese)
[21]	WANG X J, DONG D P, CHI X K, et al. sEMG-based consecutive estimation of human lower limb movement by using multi-branch neural network[J]. Biomedical Signal Processing and Control, 2021, 68(7): 102781.
[22]	CHU J U, MOON I, LEE Y J, et al. A supervised feature-projection-based real-time EMG pattern recognition for multifunction myoelectric hand control[J]. IEEE/ASME Transactions on Mechatronics, 2007, 12(3): 282-290.
[23]	HU B, ROUSE E, HARGROVE L. Benchmark datasets for bilateral lower-limb neuromechanical signals from wearable sensors during unassisted locomotion in able-bodied individuals[J]. Frontiers in Robotics and AI, 2018, 5(2): 14.
[24]	HALIM A, ABDELLATIF A, AWAD M I, et al. Prediction of human gait activities using wearable sensors[J]. Proceedings of the Institution of Mechanical Engineers Part H-Journal of Engineering in Medicine, 2021, 235(6): 676-687.
[25]	PHINYOMARK A, KHUSHABA R N, IBANEZ-MARCELO E, et al. Navigating features: a topologically informed chart of electromyographic features space[J]. Journal of the Royal Society Interface, 2017, 14(137): 20170734.
[26]	SCHULTE R V, PRINSEN E C, HERMENS H J, et al. Genetic algorithm for feature selection in lower limb pattern recognition[J]. Frontiers in Robotics and AI, 2021, 8(10): 710806.
[27]	HALIM A, IBRAHIM A A, AWAD M I, et al. Optimization of sensor number for lower limb prosthetics using genetic algorithm[C]//Proceedings of 2020 International Conference on Innovative Trends in Communication and Computer Engineering. Aswan, Egypt: IEEE, 2020: 210-215.
[28]	HU B, ROUSE E, HARGROVE L. Fusion of bilateral lower-limb neuromechanical signals improves prediction of locomotor activities[J]. Frontiers in Robotics and AI, 2018, 5(6): 78.
[29]	ZHANG K G, WANG J, DE SILVA C W, et al. Unsupervised cross-subject adaptation for predicting human locomotion intent[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2020, 28(3): 646-657. doi: 10.1109/TNSRE.2020.2966749 pmid: 31944980
[30]	LIU Z J, LIN W, GENG Y L, et al. Intent pattern recognition of lower-limb motion based on mechanical sensors[J]. IEEE-CAA Journal of Automatica Sinica, 2017, 4(4): 651-660.

特征集	分类准确率/%
特征集	NinaPro DB1	NinaPro DB2	NinaPro DB5	BioPatRec DB2
Du_FS	82.4	78.4	83.7	88.2
TSD	81.0	59.6	81.9	87.7
Atzori_FS	83.7	8.1	83.9	87.6
Phin_FS1	85.4	74.4	85.8	90.9
Phin_FS2	84.3	60.0	84.4	88.5
Doswald_FS	85.3	61.7	85.2	90.6
TDAR	85.0	75.6	84.5	88.5
DWTC	85.7	67.6	86.0	90.0
DWPTC	85.9	57.2	85.8	90.3
CWTC	84.4	61.8	85.3	89.6
Hudgins_FS	82.5	37.9	83.4	88.2

特征集	分类准确率/%
特征集	NinaPro DB1	NinaPro DB2	NinaPro DB5	BioPatRec DB2
Du_FS	82.4	78.4	83.7	88.2
TSD	81.0	59.6	81.9	87.7
Atzori_FS	83.7	8.1	83.9	87.6
Phin_FS1	85.4	74.4	85.8	90.9
Phin_FS2	84.3	60.0	84.4	88.5
Doswald_FS	85.3	61.7	85.2	90.6
TDAR	85.0	75.6	84.5	88.5
DWTC	85.7	67.6	86.0	90.0
DWPTC	85.9	57.2	85.8	90.3
CWTC	84.4	61.8	85.3	89.6
Hudgins_FS	82.5	37.9	83.4	88.2

标签	数据量	比例/%
训练集	25471	80
测试集	6368	20

标签	数据量	比例/%
训练集	25471	80
测试集	6368	20

模型	CNN输入	MobileNet输入	平均准确率/%
1	EKPD	DWFM	94.19
2	EKPD	CES	92.16
3	EKPD	EKPD	90.57
4	DWFM	EKPD	93.22
5	DWFM	CES	88.28
6	DWFM	DWFM	90.06
7	CES	EKPD	92.15
8	CES	DWFM	88.07
9	CES	CES	87.53