Analysis of Soft Intelligent Edge Computing Technologies for Unmanned Systems

doi:10.12382/bgxb.2022.1166

Abstract

Abstract:

The design and deployment of light-weight neural network models are crucial for intelligent weapon systems. This paper explores the application of intelligent edge computing in unmanned systems from the perspective of building a light-weight deep neural network model, specifically focusing on parameter pruning, knowledge distillation, and parameter quantization techniques. Recent advancements in these fields are discussed, and the performance of various intelligent edge computing technologies is evaluated using object recognition as an example. Based on the advantages and disadvantages of each light-weight design method, a new framework for edge computing is proposed. With improvements in parameter quantization accuracy and introduction of knowledge distillation, the proposed framework becomes feasible for implementation. This approach provides valuable insights for the utilization of intelligent edge computing technologies in enhancing the military intelligence of unmanned systems.

Key words: unmanned equipment, deep neural network, intelligent edge computing, parameter pruning, knowledge distillation, model quantization

CLC Number:

TP301

ZHANG Kaige, LU Zhigang, NIE Tianchang, LI Zhiwei, GUO Yuqiang. Analysis of Soft Intelligent Edge Computing Technologies for Unmanned Systems[J]. Acta Armamentarii, 2023, 44(9): 2611-2621.

Figures/Tables 11

References 64

[1]	刘谋黎. 深度卷积神经网络在计算机视觉中的应用研究综述[J]. 科学大众(科技创新), 2021(7): 55-57.
	LIU M L. Applications of deep convolutional neuralnetwork in computer vision[J]. PopularScience(Technological Innovation), 2021(7):55-57. (in Chinese)
[2]	冯志伟, 丁晓梅. 自然语言处理中的神经网络模型[J]. 当代外语研究, 2022(4):98-110.
	FENG Z W, DING X M. Neural networks in nature language processing[J]. Journal of Second Language Studies, 2022(4):98-110. (in Chinese)
[3]	孙长银, 穆朝絮. 多智能体深度强化学习的若干关键科学问题[J]. 自动化学报, 2020, 46(7):1301-1312.
	SUN C Y, MU C X. Applications of deep convolutional neural network in computer vision[J]. Acta Automatica Sinica, 2020, 46(7):1301-1312. (in Chinese)
[4]	WU Z F, SHEN C H, VAN D H A. Wider or deeper:revisiting the resnet model for visual recognition[J]. Pattern Recognition, 2019, 90:119-133. doi: 10.1016/j.patcog.2019.01.006 URL
[5]	WANG C Y, LIAO H Y M, WU Y H, et al. CSPNet: a new backbone that can enhance learning capability of CNN[C]//Proceedings of CVF Conference on Computer Vision and Pattern Recognition Workshops.Seattle, WA, US:IEEE, 2020: 1571-1580.
[6]	石磊, 王毅, 成颖, 等. 自然语言处理中的注意力机制研究综述[J]. 数据分析与知识发现, 2020, 4(5):1-14.
	SHI L, WANG Y, CHENG Y, et al. Review of attention mechanism in natural language processing[J]. Data Analysis and Knowledge Discovery, 2020, 4(5): 1-14. (in Chinese)
[7]	马瑶, 智敏, 殷雁君, 等. CNN和Transformer在细粒度图像识别中的应用综述[J]. 计算机工程与应用, 2022, 58(19): 53-63. doi: 10.3778/j.issn.1002-8331.2201-0374
	MA Y, ZHI M, YIN Y J, et al. Review of applications of CNN and transformer in fine-grained image recognition[J]. Computer Engineering and Applications, 2022, 58(19):53-63. (in Chinese) doi: 10.3778/j.issn.1002-8331.2201-0374
[8]	DENG J, DONG W, SOCHER R, et al. Imagenet: a large-scale hierarchical image database[C]//Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition.Miami,FL,US:IEEE, 2009: 248-255.
[9]	薛俊韬, 马若寒, 胡超芳. 基于MobileNet的多目标跟踪深度学习算法[J]. 控制与决策, 2021, 36(8): 1991-1996.
	XUE J T, MA R H, HU C F. Deep learning algorithm based on MobileNet for multi-target tracking[J]. Control and Decision, 2021, 36(8):1991-1996. (in Chinese)
[10]	TESFAI H, SALEH H, Al-QUTAYRI M, et al. Lightweight shufflenet based CNN for arrhythmia classification[J]. IEEE Access, 2022, 10:111842-111854. doi: 10.1109/ACCESS.2022.3215665 URL
[11]	李屹, 魏建国, 刘贯伟. 模型剪枝算法综述[J]. 计算机与现代化, 2022(9):51-59.
	LI Y, WEI J G, LIU G W. Survey of model pruning algorithms[J]. Computer and Modernization, 2022(9):51-59. (in Chinese)
[12]	黄震华, 杨顺志, 林威, 等. 知识蒸馏研究综述[J]. 计算机学报, 2022, 45(3): 624-653.
	HUANG Z H, YANG S Z, LIN W, et al. Knowledge distillation: a survey[J]. Chinese Journal of Computers, 2022, 45(3): 624-653. (in Chinese)
[13]	高晗, 田育龙, 许封元, 等. 深度学习模型压缩与加速综述[J]. 软件学报, 2021, 32(1): 68-92.
	GAO H, TIAN Y L, XU F Y, et al. Survey of deep learning model compression and acceleration[J]. Journal of Software, 2021, 32(1): 68-92. (in Chinese)
[14]	HASSIBI B, STORK D G, WOLFF G J. Optimal brain surgeon and general network pruning[C]//Proceedings of IEEE International Conference on Neural Networks. San Francisco, CA, US:IEEE, 1993: 293-299.
[15]	WANG H, QIN C, ZHANG Y L, et al. Neural pruning via growing regularization:arXiv: 2012.09243[R]. Ithaca,NY, US: Cornell University, 2020:2012.09243.
[16]	FANG G F, MA X Y, SONG M L, et al. DepGraph: towards any structural pruning:arXiv:2301.12900[R]. Ithaca,NY, US: Cornell University, 2023:2301.12900.
[17]	SHIBU A, KUMAR A, JUNG H, et al. Rewarded meta-pruning:meta learning with Rewards for channel pruning arXiv:2301.11063[R]. Ithaca,NY, US: Cornell University, 2023:2301.11063.
[18]	LIN M B, JI R R, ZHANG Y X, et al. Channel pruning via automatic structure search:arXiv: 2001.08565[R]. Ithaca,NY, US: Cornell University, 2020:2001.08565.
[19]	LIN M B, JI R R, WANG Y, et al. Hrank: Filter pruning using high-rank feature map[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Seattle, WA, US:IEEE, 2020: 1529-1538.
[20]	TIAN Y L, KRISHNAN D, ISOLA P. Contrastive representation distillation:arXiv: 1910.10699[R]. Ithaca,NY, US: Cornell University, 2019:1910.10699.
[21]	YANG C G, AN Z L, CAI L H, et al. Hierarchical self-supervised augmented knowledge distillation:arXiv: 2107.13715[R]. Ithaca,NY, US: Cornell University, 2021:2107.13715.
[22]	ZAGORUYKO S, KOMODAKIS N. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer[C]//Proceedings of the International Conference on ICLR.Paris, France:ICLR, 2017.
[23]	JIN X, PENG B Y, WU Y C, et al. Knowledge distillation via route constrained optimization[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.Seoul, Korea:IEEE, 2019:1345-1354.
[24]	CHEN D F, MEI J P, ZHANG Y, et al. Cross-layer distillation with semantic calibration: arXiv:2012.03236[R]. Ithaca, NY, US: Cornele University, 2020: 2021.03236.
[25]	YIM J, JOO D, BAE J, et al. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI, US:IEEE, 2017: 4133-4141.
[26]	LIU Y F, CAO J J, LI B, et al. Knowledge distillation via instance relationship graph[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Long Beach, CA,US:IEEE, 2019: 7096-7104.
[27]	CHEN H T, WANG Y H, XU C, et al. Learning student networks via feature embedding[J]. IEEE Transactions on Neural Networks and Learning Systems, 2020, 32(1): 25-35. doi: 10.1109/TNNLS.5962385 URL
[28]	胡干, 梁兴柱. 一种高效的基于互学习的在线蒸馏系统[J]. 电子世界, 2021(23):132-134.
	HU G, LIANG X Z. An efficient online distillation system based on mutual learning[J]. Electronics World, 2021(23):132-134. (in Chinese)
[29]	田加林, 徐行, 沈复民, 等. 基于跨模态自蒸馏的零样本草图检索[J]. 软件学报, 2022, 33(9): 3152-3164.
	TIAN J L, XU X, SHEN F M, et al. Cross-modal self-distillation for zero-shot sketch-based image retrieval[J]. Journal of Software, 2022, 33(9): 3152-3164. (in Chinese)
[30]	程祥鸣, 邓春华. 基于无标签知识蒸馏的人脸识别模型的压缩算法[J]. 计算机科学, 2022, 49(6):245-253. doi: 10.11896/jsjkx.210400023
	CHENG X M, DENG C H. Compression algorithm of face recognition model based on unlabeled knowledge distillation[J]. Computer Science, 2022, 49(6):245-253. (in Chinese)
[31]	CHMIEL B, BANNER R, SHOMRON G, et al. Robust quantization: one model to rule them all[J]. Advances in Neural Information Processing Systems, 2020, 33: 5308-5317.
[32]	LI Y H, GONG R H, TAN X, et al. Brecq:pushing the limit of post-training quantization by block reconstruction:arXiv: 2102.05426[R]. Ithaca,NY, US: Cornell University, 2021:2102.05426.
[33]	NAGEL M, AMJAD R A, VAN B M, et al. Up or down? adaptive rounding for post-training quantization: arXiv:2004.10568[R]. Ithaca, NY, US: Cornell University, 2020: 2004.10568.
[34]	JACOB B, KLIGYS S, CHEN B, et al. Quantization and training of neural networks for efficient integer-arithmetic-only inference[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Salt Lake City,UT,US:IEEE, 2018: 2704-2713.
[35]	YIN P H, LYU J C, ZHANG S, et al. Understanding straight-through estimator in training activation quantized neural nets:arXiv: 1903.05662[R]. Ithaca,NY, US: Cornell University, 2019:1903.05662.
[36]	FINKELSTEIN A, FUCHS E, TAL I, et al. QFT:Post-training quantization via fast joint finetuning of all degrees of freedom[C]//Proceedings of Computer Vision-ECCV 2022 Workshops: Part VII. Cham,Germany:Springer, 2023:115-129.
[37]	FAN A, STOCK P, GRAHAM B, et al. Training with quantization noise for extreme model compression:arXiv: 2004.07320[R]. Ithaca,NY, US: Cornell University, 2020:2004.07320.
[38]	ANWAR S, HWANG K, SUNG W. Structured pruning of deep convolutional neural networks[J]. ACM Journal on Emerging Technologies in Computing Systems, 2017, 13(3):1-18.
[39]	MA X L, LIN S, YE S K, et al. Non-structured DNN weight pruning-is it beneficial in any platform?[J]. IEEE Transactions on Neural Networks and Learning Systems, 2021, 33(9):4930-4944. doi: 10.1109/TNNLS.2021.3063265 URL
[40]	LAURENT C, BALLAS C, GEORGE T, et al. Revisiting loss modelling for unstructured pruning:arXiv: 2006.12279[R]. Ithaca,NY, US: Cornell University, 2020:2006.12279.
[41]	HE Y H, ZHANG X Y, SUN J. Channel pruning for accelerating very deep neural networks[C]//Proceedings of the IEEE International Conference on Computer Vision. Venice, Italy: IEEE, 2017:1389-1397.
[42]	LI Y, GU S, MAYER C, et al. Group sparsity:the hinge between filter pruning and decomposition for network compression[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, WA, US:IEEE, 2020: 8018-8027.
[43]	BUCILUA C, CARUANA R, NICULESCU-MIZIL A. Model compression[C]//Proceedings of the 12nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.New York, NY,US:SIGKDD, 2006: 535-541.
[44]	HINTON G, VINYALS O, DEAN J. Distilling the knowledge in a neural network[C]//Proceedings of the International Conference on NIPS.Seattle, WA,US:NIPS, 2015.
[45]	GOU J P, YU B S, MAYBANK S J, et al. Knowledge distillation:a survey[J]. International Journal of Computer Vision, 2021, 129(6):1789-1819. doi: 10.1007/s11263-021-01453-z
[46]	ROMERO A, BALLAS N, KAHOU S E, et al. Fitnets: hints for thin deep nets[C]//Proceedings of International Conference on ICLR. San Diego, CA, US:ICLR, 2015.
[47]	PASSALIS N, TEFAS A. Learning deep representations with probabilistic knowledge transfer[C]//Proceedings of the European Conference on Computer Vision. Munich, Germany:ECCV, 2018: 268-284.
[48]	GUPTA S, HOFFMAN J, MALIK J. Cross modal distillation for supervision transfer[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas,NV, US:CVPR, 2016: 2827-2836.
[49]	李姜楠, 伍星, 刘竞升, 等. 基于知识蒸馏的YOLOv3算法研究[J]. 计算机工程与应用, 2022, 58(17):174-180. doi: 10.3778/j.issn.1002-8331.2101-0089
	LI J N, WU X, LIU J S, et al. Research of YOLOv3 based on knowledge distillation[J]. Computer Engineering and Applications, 2022, 58(17):174-180. (in Chinese) doi: 10.3778/j.issn.1002-8331.2101-0089
[50]	楚玉春, 龚航, 王学芳, 等. 基于YOLOv4的目标检测知识蒸馏算法研究[J]. 计算机科学, 2022, 49(6A):337-344. doi: 10.11896/jsjkx.210600204
	CHU Y C, GONG H, WANG X F, et al. Study on knowledge distillation of target detection algorithm based on YOLOv4[J]. Computer Science, 2022, 49(6A):337-344. (in Chinese)
[51]	TAILOR S A, FERNANDEZ-MARQUES J, LANE N D. Degree-quant: quantization-aware training for graph neural networks:arXiv: 2008.05000[R]. Ithaca,NY, US: Cornell University, 2020:2008.05000.
[52]	LIU Z H, WANG Y H, HAN K, et al. Post-training quantization for vision transformer[C]//Proceedings of International Conference on Neural Information Processing Systems. Montreal,Canada:NIPS, 2021.
[53]	LI H, KADAV A, DURDANOVIC I, et al. Pruning filters for efficient convnets:arXiv:1608.08710[R]. Ithaca,NY, US: Cornell University, 2016:1608.08710.
[54]	HU H, PENG R, TAI Y W, et al. Network trimming: a data-driven neuron pruning approach towards efficient deep architectures arXiv:1607.03250[R]. Ithaca,NY, US: Cornell University, 2016:1607.0325.
[55]	MOLCHANOV P, TYREE S, KARRAS T, et al. Pruning convolutional neural networks for resource efficient transfer learning arXiv:1611.06440[R]. Ithaca,NY, US: Cornell University, 2016:1611.06440.
[56]	LUO J H, WU J, LIN W. Thinet:a filter level pruning method for deep neural network compression[C]//Proceedings of the IEEE International Conference on Computer Vision.Venice,Italy:IEEE, 2017: 5058-5066.
[57]	RACHWAN J, ZUGNER D, CHARPENTIER B, et al. Winning the lottery ahead of time:efficient early network pruning[C]//Proceedings of International Conference on Machine Learning. Honolulu, HI,US:ICML, 2022:18293-18309.
[58]	KIM J H, PARK S U, KWAK N J. Paraphrasing complex network:network compression via factor transfer: arXiv:1802.04977[R]. Ithaca, NY, US: Cornell University, 2018: 1802.04977.
[59]	HUANG Z H, WANG N Y. Like what you like: Knowledge distill via neuron selectivity transfer arXiv:1707.01219[R]. Ithaca,NY, US: Cornell University, 2017:1707.01219.
[60]	HEO B, LEE M, YUN S, et al. Knowledge transfer via distillation of activation boundaries formed by hidden neurons[C]//Proceedings of the AAAI Conference on Artificial Intelligence.Honolulu,HI,US:AAAI, 2019, 33(1): 3779-3787.
[61]	PASSALIS N, TEFAS A. Learning deep representations with probabilistic knowledge transfer[C]//Proceedings of the European Conference on Computer Vision.Munich,Germany:ECCV, 2018: 268-284.
[62]	RASTEGARI M, ORDONEZ V, REDMON J, et al. Xnor-net: Imagenet classification using binary convolutional neural networks[C]//Proceedings of the European Conference on Computer Vision. Cham,Germany:Springer, 2016: 525-542.
[63]	LIU Z H, WANG Y H, HAN K, et al. Instance-aware dynamic neural network quantization[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.New Orleans,LA,US:IEEE, 2022:12434-12443.
[64]	ZHONG Y S, LIN M B, NAN G R, et al. Intraq: learning synthetic images with intra-class heterogeneity for zero-shot network quantization[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.New Orleans,LA,US:IEEE, 2022: 12339-12348.

方法	优点	缺点	子类方法	子类方法优点	子类方法缺点
知识蒸馏	精度损失较低	加速倍数受限于学生网络结构	基于反应	使用简单	没有考虑对应中间层之间的学习,精度损失大
			基于特征	精度损失较小	难以确定哪些层之间可以建立对应关系
			基于关系	精度损失较小	需要额外的关系特征提取过程
参数量化	精度损失较低,加速倍数大	不能大幅减少参数数量,加速倍数有上限	QAT	精度损失较小	训练难度大
参数量化	精度损失较低,加速倍数大	不能大幅减少参数数量,加速倍数有上限	PTQ	易于训练	精度损失较大
模型剪枝	加速倍数较大,可以大幅减少参数数量	精度损失较大	结构化剪枝	易于实施	需要确定哪些层需要被剪枝,精度丢失较大
模型剪枝	加速倍数较大,可以大幅减少参数数量	精度损失较大	非结构化剪枝	精度丢失较小	实施困难

方法	优点	缺点	子类方法	子类方法优点	子类方法缺点
知识蒸馏	精度损失较低	加速倍数受限于学生网络结构	基于反应	使用简单	没有考虑对应中间层之间的学习,精度损失大
			基于特征	精度损失较小	难以确定哪些层之间可以建立对应关系
			基于关系	精度损失较小	需要额外的关系特征提取过程
参数量化	精度损失较低,加速倍数大	不能大幅减少参数数量,加速倍数有上限	QAT	精度损失较小	训练难度大
参数量化	精度损失较低,加速倍数大	不能大幅减少参数数量,加速倍数有上限	PTQ	易于训练	精度损失较大
模型剪枝	加速倍数较大,可以大幅减少参数数量	精度损失较大	结构化剪枝	易于实施	需要确定哪些层需要被剪枝,精度丢失较大
模型剪枝	加速倍数较大,可以大幅减少参数数量	精度损失较大	非结构化剪枝	精度丢失较小	实施困难

方法	参数数量/ 10⁶	加速倍数	Top-1 误差率增加/%	Top-5 误差率增加/%
VGG-16	138.4	1.0	0	0
L1-GAP^[53]	9.2	2.5	4.62	3.10
APoZ-GAP^[54]	9.2	2.5	3.72	2.65
TE^[55]	135.7	2.7	4.12	3.94
ThiNet^[56]	9.5	2.3	1.00	0.52
ThiNet-T^[56]	1.3	4.35	9.00	6.47
GReg^[15]	10.6	8.81	6.71	5.34
DepGraph^[16]	8.61	9.05	5.20	4.83
EarlyCroP^[57]	8.5	8.53	5.85	5.22

方法	参数数量/ 10⁶	加速倍数	Top-1 误差率增加/%	Top-5 误差率增加/%
VGG-16	138.4	1.0	0	0
L1-GAP^[53]	9.2	2.5	4.62	3.10
APoZ-GAP^[54]	9.2	2.5	3.72	2.65
TE^[55]	135.7	2.7	4.12	3.94
ThiNet^[56]	9.5	2.3	1.00	0.52
ThiNet-T^[56]	1.3	4.35	9.00	6.47
GReg^[15]	10.6	8.81	6.71	5.34
DepGraph^[16]	8.61	9.05	5.20	4.83
EarlyCroP^[57]	8.5	8.53	5.85	5.22

方法	教师网络/ 参数量/ million	教师网络精度/%	学生网络/ 参数量/ million	学生网络精度/%
FT^[58]	ResNet56/0.85	72.34	ResNet20/0.27	69.84
NST^[59]	WRN-40-2/4.4	75.61	WRN-40-1/0.6	72.24
AB^[60]	WRN-40-20/4.4	75.61	WRN-40-1/0.6	72.38
PKT^[61]	ResNet110/1.7	74.31	ResNet20/0.27	70.25
CRD^[20]	ResNet56/0.85	72.34	ResNet20/0.27	71.16
HSAKD^[21]	ResNet56/0.85	72.34	ResNet20/0.27	73.26