刘谋黎 . 深度卷积神经网络在计算机视觉中的应用研究综述 [J ] . 科学大众(科技创新) , 2021 ( 7 ): 55 - 57 .

LIU M L . Applications of deep convolutional neuralnetwork in computer vision [J ] . PopularScience(Technological Innovation ), 2021 ( 7 ): 55 - 57 . (in Chinese)

冯志伟 , 丁晓梅 . 自然语言处理中的神经网络模型 [J ] . 当代外语研究 , 2022 ( 4 ): 98 - 110 .

FENG Z W , DING X M . Neural networks in nature language processing [J ] . Journal of Second Language Studies , 2022 ( 4 ): 98 - 110 . (in Chinese)

孙长银 , 穆朝絮 . 多智能体深度强化学习的若干关键科学问题 [J ] . 自动化学报 , 2020 , 46 ( 7 ): 1301 - 1312 .

SUN C Y , MU C X . Applications of deep convolutional neural network in computer vision [J ] . Acta Automatica Sinica , 2020 , 46 ( 7 ): 1301 - 1312 . (in Chinese)

WU Z F , SHEN C H , VAN D H A . Wider or deeper:revisiting the resnet model for visual recognition [J ] . Pattern Recognition , 2019 , 90 : 119 - 133 . DOI: 10.1016/j.patcog.2019.01.006 http://doi.org/10.1016/j.patcog.2019.01.006 https://linkinghub.elsevier.com/retrieve/pii/S0031320319300135 https://linkinghub.elsevier.com/retrieve/pii/S0031320319300135

WANG C Y , LIAO H Y M , WU Y H , et al. CSPNet: a new backbone that can enhance learning capability of CNN [C ] //Proceedings of CVF Conference on Computer Vision and Pattern Recognition Workshops.Seattle, WA, US:IEEE , 2020 : 1571 - 1580 .

石磊 , 王毅 , 成颖 , 等 . 自然语言处理中的注意力机制研究综述 [J ] . 数据分析与知识发现 , 2020 , 4 ( 5 ): 1 - 14 .

SHI L , WANG Y , CHENG Y , et al . Review of attention mechanism in natural language processing [J ] . Data Analysis and Knowledge Discovery , 2020 , 4 ( 5 ): 1 - 14 . (in Chinese)

马瑶 , 智敏 , 殷雁君 , 等 . CNN和Transformer在细粒度图像识别中的应用综述 [J ] . 计算机工程与应用 , 2022 , 58 ( 19 ): 53 - 63 . DOI: 10.3778/j.issn.1002-8331.2201-0374 http://doi.org/10.3778/j.issn.1002-8331.2201-0374 细粒度图像识别旨在从类别图像中辨别子类别。由于图像间只有细微差异，这使得识别任务具有挑战性。随着深度学习技术的不断进步，基于深度学习的方法定位局部和表示特征的能力越来越强，其中以卷积神经网络（CNN）和Transformer为基础的各类算法大大提高了细粒度图像识别精度，细粒度图像领域得到了显著发展。为了整理两类方法在细粒度图像识别领域的发展历程，对该领域近年来只运用类别标签的方法进行了综述。介绍了细粒度图像识别的概念，详细阐述了主流细粒度图像数据集；介绍了基于CNN和Transformer的细粒度图像识别方法及其性能；最后，总结了细粒度图像识别未来的研究方向。

MA Y , ZHI M , YIN Y J , et al. Review of applications of CNN and transformer in fine-grained image recognition [J ] . Computer Engineering and Applications , 2022 , 58 ( 19 ): 53 - 63 . (in Chinese) DOI: 10.3778/j.issn.1002-8331.2201-0374 http://doi.org/10.3778/j.issn.1002-8331.2201-0374 Fine grained image recognition aims to distinguish subcategories from category images. This makes the recognition task challenging as there are only subtle differences between images. With the continuous progress of deep learning technology, the ability of locating local and representing features based on deep learning methods is becoming stronger and stronger. Among them, various algorithms based on convolutional neural network（CNN）and transformer greatly improve the accuracy of fine-grained image recognition, and the field of fine-grained image has been significantly developed. In order to sort out the development of the two methods in the field of fine-grained image recognition, the methods that only use category labels in this field in recent years are reviewed. Firstly, the concept of fine-grained image recognition is introduced, and the mainstream fine-grained image data set is described in detail. Secondly, the fine-grained image recognition method based on convolutional neural network and visual transformer and its performance are introduced. Finally, the future research direction of fine-grained image recognition is summarized.

DENG J , DONG W , SOCHER R , et al. Imagenet: a large-scale hierarchical image database [C ] //Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition.Miami,FL,US:IEEE , 2009 : 248 - 255 .

薛俊韬 , 马若寒 , 胡超芳 . 基于MobileNet的多目标跟踪深度学习算法 [J ] . 控制与决策 , 2021 , 36 ( 8 ): 1991 - 1996 .

XUE J T , MA R H , HU C F . Deep learning algorithm based on MobileNet for multi-target tracking [J ] . Control and Decision , 2021 , 36 ( 8 ): 1991 - 1996 . (in Chinese)

TESFAI H , SALEH H , Al-QUTAYRI M , et al . Lightweight shufflenet based CNN for arrhythmia classification [J ] . IEEE Access , 2022 , 10 : 111842 - 111854 . DOI: 10.1109/ACCESS.2022.3215665 http://doi.org/10.1109/ACCESS.2022.3215665 https://ieeexplore.ieee.org/document/9923758/ https://ieeexplore.ieee.org/document/9923758/

李屹 , 魏建国 , 刘贯伟 . 模型剪枝算法综述 [J ] . 计算机与现代化 , 2022 ( 9 ): 51 - 59 .

LI Y , WEI J G , LIU G W . Survey of model pruning algorithms [J ] . Computer and Modernization , 2022 ( 9 ): 51 - 59 . (in Chinese)

黄震华 , 杨顺志 , 林威 , 等 . 知识蒸馏研究综述 [J ] . 计算机学报 , 2022 , 45 ( 3 ): 624 - 653 .

HUANG Z H , YANG S Z , LIN W , et al. Knowledge distillation: a survey [J ] . Chinese Journal of Computers , 2022 , 45 ( 3 ): 624 - 653 . (in Chinese)

高晗 , 田育龙 , 许封元 , 等 . 深度学习模型压缩与加速综述 [J ] . 软件学报 , 2021 , 32 ( 1 ): 68 - 92 .

GAO H , TIAN Y L , XU F Y , et al. Survey of deep learning model compression and acceleration [J ] . Journal of Software , 2021 , 32 ( 1 ): 68 - 92 . (in Chinese)

HASSIBI B , STORK D G , WOLFF G J . Optimal brain surgeon and general network pruning [C ] //Proceedings of IEEE International Conference on Neural Networks. San Francisco, CA, US:IEEE , 1993 : 293 - 299 .

WANG H , QIN C , ZHANG Y L , et al. Neural pruning via growing regularization:arXiv : 2012 .09243[R ] . Ithaca,NY , US : Cornell University , 2020 :2012.09243.

FANG G F , MA X Y , SONG M L , et al. DepGraph: towards any structural pruning:arXiv:2301.12900 [R ] . Ithaca,NY , US : Cornell University , 2023 :2301.12900.

SHIBU A , KUMAR A , JUNG H , et al. Rewarded meta-pruning:meta learning with Rewards for channel pruning arXiv:2301.11063 [R ] . Ithaca,NY , US : Cornell University , 2023 :2301.11063.

LIN M B , JI R R , ZHANG Y X , et al. Channel pruning via automatic structure search:arXiv : 2001 .08565[R ] . Ithaca,NY , US : Cornell University , 2020 :2001.08565.

LIN M B , JI R R , WANG Y , et al. Hrank: Filter pruning using high-rank feature map [C ] //Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Seattle, WA, US:IEEE , 2020 : 1529 - 1538 .

TIAN Y L , KRISHNAN D , ISOLA P . Contrastive representation distillation:arXiv : 1910 .10699[R ] . Ithaca,NY , US : Cornell University , 2019 :1910.10699.

YANG C G , AN Z L , CAI L H , et al. Hierarchical self-supervised augmented knowledge distillation:arXiv : 2107 .13715[R ] . Ithaca,NY , US : Cornell University , 2021 :2107.13715.

ZAGORUYKO S , KOMODAKIS N . Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer [C ] //Proceedings of the International Conference on ICLR.Paris, France:ICLR , 2017 .

JIN X , PENG B Y , WU Y C , et al. Knowledge distillation via route constrained optimization [C ] //Proceedings of the IEEE/CVF International Conference on Computer Vision.Seoul, Korea:IEEE , 2019 : 1345 - 1354 .

CHEN D F , MEI J P , ZHANG Y , et al. Cross-layer distillation with semantic calibration: arXiv:2012.03236 [R ] . Ithaca, NY , US : Cornele University , 2020 : 2021.03236.

YIM J , JOO D , BAE J , et al . A gift from knowledge distillation: Fast optimization, network minimization and transfer learning [C ] //Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI, US:IEEE , 2017 : 4133 - 4141 .

LIU Y F , CAO J J , LI B , et al. Knowledge distillation via instance relationship graph [C ] //Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Long Beach, CA,US:IEEE , 2019 : 7096 - 7104 .

CHEN H T , WANG Y H , XU C , et al . Learning student networks via feature embedding [J ] . IEEE Transactions on Neural Networks and Learning Systems , 2020 , 32 ( 1 ): 25 - 35 . DOI: 10.1109/TNNLS.5962385 http://doi.org/10.1109/TNNLS.5962385 https://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=5962385 https://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=5962385

胡干 , 梁兴柱 . 一种高效的基于互学习的在线蒸馏系统 [J ] . 电子世界 , 2021 ( 23 ): 132 - 134 .

HU G , LIANG X Z . An efficient online distillation system based on mutual learning [J ] . Electronics World , 2021 ( 23 ): 132 - 134 . (in Chinese)

田加林 , 徐行 , 沈复民 , 等 . 基于跨模态自蒸馏的零样本草图检索 [J ] . 软件学报 , 2022 , 33 ( 9 ): 3152 - 3164 .

TIAN J L , XU X , SHEN F M , et al. Cross-modal self-distillation for zero-shot sketch-based image retrieval [J ] . Journal of Software , 2022 , 33 ( 9 ): 3152 - 3164 . (in Chinese)

程祥鸣 , 邓春华 . 基于无标签知识蒸馏的人脸识别模型的压缩算法 [J ] . 计算机科学 , 2022 , 49 ( 6 ): 245 - 253 . DOI: 10.11896/jsjkx.210400023 http://doi.org/10.11896/jsjkx.210400023 将人脸识别技术移植到移动设备上时,往往需要经过模型压缩等加速算法的处理。知识蒸馏是一种实际应用较广且易于训练的模型压缩方法,现有的知识蒸馏算法需要大量带标签的人脸数据,可能会涉及身份隐私泄露等安全问题。同时,大规模采集有标签人脸数据的成本较大,而海量可采集或生成的无标签人脸数据却无法利用。为解决上述问题,通过分析知识蒸馏在人脸识别任务中的特性,提出了一种无标签知识蒸馏的间接监督训练方法。该方法可以利用海量无标签的人脸数据,避免了隐私泄露等安全隐患问题。然而,无标签人脸数据集的数据分布无法预知,存在数据分布不均衡的问题,阻碍了间接监督算法的性能提升。文中进一步提出了一种人脸内容置换的数据增强方法,通过置换人脸部分内容来平衡人脸数据分布,同时增强了人脸数据的多样性。实验结果表明,人脸识别模型被大幅度压缩时,所提算法的性能达到了先进水平,并在LFW数据集上超越了大型网络。

CHENG X M , DENG C H . Compression algorithm of face recognition model based on unlabeled knowledge distillation [J ] . Computer Science , 2022 , 49 ( 6 ): 245 - 253 . (in Chinese)

CHMIEL B , BANNER R , SHOMRON G , et al . Robust quantization: one model to rule them all [J ] . Advances in Neural Information Processing Systems , 2020 , 33 : 5308 - 5317 .

LI Y H , GONG R H , TAN X , et al. Brecq:pushing the limit of post-training quantization by block reconstruction:arXiv : 2102 .05426[R ] . Ithaca,NY , US : Cornell University , 2021 :2102.05426.

NAGEL M , AMJAD R A , VAN B M , et al. Up or down? adaptive rounding for post-training quantization: arXiv:2004.10568 [R ] . Ithaca, NY , US : Cornell University , 2020 : 2004.10568.

JACOB B , KLIGYS S , CHEN B , et al . Quantization and training of neural networks for efficient integer-arithmetic-only inference [C ] //Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Salt Lake City,UT,US:IEEE , 2018 : 2704 - 2713 .

YIN P H , LYU J C , ZHANG S , et al. Understanding straight-through estimator in training activation quantized neural nets:arXiv : 1903 .05662[R ] . Ithaca,NY , US : Cornell University , 2019 :1903.05662.

FINKELSTEIN A , FUCHS E , TAL I , et al. QFT:Post-training quantization via fast joint finetuning of all degrees of freedom [C ] //Proceedings of Computer Vision-ECCV 2022 Workshops : Part VII. Cham,Germany:Springer, 2023 : 115 - 129 .

FAN A , STOCK P , GRAHAM B , et al. Training with quantization noise for extreme model compression:arXiv : 2004 .07320[R ] . Ithaca,NY , US : Cornell University , 2020 :2004.07320.

ANWAR S , HWANG K , SUNG W . Structured pruning of deep convolutional neural networks [J ] . ACM Journal on Emerging Technologies in Computing Systems , 2017 , 13 ( 3 ): 1 - 18 .

MA X L , LIN S , YE S K , et al . Non-structured DNN weight pruning-is it beneficial in any platform ?[J ] . IEEE Transactions on Neural Networks and Learning Systems , 2021 , 33 ( 9 ): 4930 - 4944 . DOI: 10.1109/TNNLS.2021.3063265 http://doi.org/10.1109/TNNLS.2021.3063265 https://ieeexplore.ieee.org/document/9381660/ https://ieeexplore.ieee.org/document/9381660/

LAURENT C , BALLAS C , GEORGE T , et al. Revisiting loss modelling for unstructured pruning:arXiv : 2006 .12279[R ] . Ithaca,NY , US : Cornell University , 2020 :2006.12279.

HE Y H , ZHANG X Y , SUN J . Channel pruning for accelerating very deep neural networks [C ] //Proceedings of the IEEE International Conference on Computer Vision. Venice , Italy : IEEE , 2017 : 1389 - 1397 .

LI Y , GU S , MAYER C , et al . Group sparsity:the hinge between filter pruning and decomposition for network compression [C ] //Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, WA, US:IEEE , 2020 : 8018 - 8027 .

BUCILUA C , CARUANA R , NICULESCU-MIZIL A . Model compression [C ] //Proceedings of the 12nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.New York, NY,US:SIGKDD , 2006 : 535 - 541 .

HINTON G , VINYALS O , DEAN J . Distilling the knowledge in a neural network [C ] //Proceedings of the International Conference on NIPS.Seattle, WA,US:NIPS , 2015 .

GOU J P , YU B S , MAYBANK S J , et al. Knowledge distillation:a survey [J ] . International Journal of Computer Vision , 2021 , 129 ( 6 ): 1789 - 1819 . DOI: 10.1007/s11263-021-01453-z http://doi.org/10.1007/s11263-021-01453-z

ROMERO A , BALLAS N , KAHOU S E , et al . Fitnets: hints for thin deep nets [C ] //Proceedings of International Conference on ICLR. San Diego, CA, US:ICLR , 2015 .

PASSALIS N , TEFAS A . Learning deep representations with probabilistic knowledge transfer [C ] //Proceedings of the European Conference on Computer Vision. Munich, Germany:ECCV , 2018 : 268 - 284 .

GUPTA S , HOFFMAN J , MALIK J . Cross modal distillation for supervision transfer [C ] //Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas,NV, US:CVPR , 2016 : 2827 - 2836 .

李姜楠 , 伍星 , 刘竞升 , 等 . 基于知识蒸馏的YOLOv3算法研究 [J ] . 计算机工程与应用 , 2022 , 58 ( 17 ): 174 - 180 . DOI: 10.3778/j.issn.1002-8331.2101-0089 http://doi.org/10.3778/j.issn.1002-8331.2101-0089 知识蒸馏作为一种模型压缩方法，将大网络（教师网络）学到的知识传递给小网络（学生网络），使小网络获得接近大网络的精度。知识蒸馏在图像分类任务上获得不错的效果，但在目标检测上的研究较少，且有待提高。当前目标检测中主要基于特征提取层进行知识蒸馏，该类方法存在两个问题，第一，没有对教师网络传递知识的重要程度进行度量，第二，仅对特征提取层进行蒸馏，教师网络的知识未充分传递给学生网络。针对第一个问题，通过引入信息图作为蒸馏的监督信号，强化了学生网络对教师网络重点知识的学习；针对第二个问题，对特征提取层和特征融合层的输出同时进行蒸馏，使学生网络更充分地学习教师网络传递的知识。实验结果表明，以YOLOv3为检测模型，在不改变学生网络结构的基础上，平均类别精度（mAP）提升9.3个百分点。

LI J N , WU X , LIU J S , et al. Research of YOLOv3 based on knowledge distillation [J ] . Computer Engineering and Applications , 2022 , 58 ( 17 ): 174 - 180 . (in Chinese) DOI: 10.3778/j.issn.1002-8331.2101-0089 http://doi.org/10.3778/j.issn.1002-8331.2101-0089 As a model compression method, knowledge distillation transfers the knowledge from a large network（teacher network） to a small network（student network）, making the accuracy of the small network closer to that of the large network. Knowledge distillation achieves good effect in image classification, but there is less research on object detection, and it needs to be improved. The current distillation methods of object detection are mainly based on the distillation of the feature extraction layer. However, there are two problems. Firstly, the importance of knowledge transmitted by the teacher network is not measured. Secondly, only the output of the feature extraction layer is distilled. Teacher network cannot fully transfer knowledge to student network. For the first problem, information map is introduced as the supervision signal of distillation, which strengthens the learning of key knowledge of the teacher network by the student network. For the second problem, the output of the feature extraction layer and the feature fusion layer are distilled at the same time. Student model can learn more about the knowledge delivered by teacher network. Experimental results show that mAP index value can improve 9.3 percentage points without changing network structure of student network based on YOLOv3.

楚玉春 , 龚航 , 王学芳 , 等 . 基于YOLOv4的目标检测知识蒸馏算法研究 [J ] . 计算机科学 , 2022 , 49 ( 6A ): 337 - 344 . DOI: 10.11896/jsjkx.210600204 http://doi.org/10.11896/jsjkx.210600204 知识蒸馏作为一种基于教师-学生网络思想的训练方法,它通过复杂的教师网络来引导网络结构相对简单的学生网络进行训练,使得学生网络获得与教师网络相媲美的精度。知识蒸馏在自然语言处理和图像分类领域均有广泛的研究,而在目标检测领域的研究则相对较少且实验效果有待提升。目标检测的蒸馏算法主要是在特征提取层进行,而单一的特征提取层的蒸馏方式易导致学生不能充分学习教师网络知识,使得模型的精度较差。针对上述问题,通过在特征提取和目标分类与边框预测上都利用了教师网络的“知识”来指导学生网络进行训练,并提出了一种基于多尺度注意力机制的蒸馏算法,使得教师网络的知识更好地流向学生网络。实验分析表明,在YOLOv4基础上提出的蒸馏算法可有效地提高学生网络的检测精度。

CHU Y C , GONG H , WANG X F , et al. Study on knowledge distillation of target detection algorithm based on YOLOv4 [J ] . Computer Science , 2022 , 49 ( 6A ): 337 - 344 . (in Chinese)

TAILOR S A , FERNANDEZ-MARQUES J , LANE N D . Degree-quant: quantization-aware training for graph neural networks:arXiv : 2008 .05000[R ] . Ithaca,NY , US : Cornell University , 2020 :2008.05000.

LIU Z H , WANG Y H , HAN K , et al. Post-training quantization for vision transformer [C ] //Proceedings of International Conference on Neural Information Processing Systems. Montreal,Canada:NIPS , 2021 .

LI H , KADAV A , DURDANOVIC I , et al. Pruning filters for efficient convnets:arXiv:1608.08710 [R ] . Ithaca,NY , US : Cornell University , 2016 :1608.08710.

HU H , PENG R , TAI Y W , et al. Network trimming: a data-driven neuron pruning approach towards efficient deep architectures arXiv:1607.03250 [R ] . Ithaca,NY , US : Cornell University , 2016 :1607.0325.

MOLCHANOV P , TYREE S , KARRAS T , et al . Pruning convolutional neural networks for resource efficient transfer learning arXiv:1611.06440 [R ] . Ithaca,NY , US : Cornell University , 2016 :1611.06440.

LUO J H , WU J , LIN W . Thinet:a filter level pruning method for deep neural network compression [C ] //Proceedings of the IEEE International Conference on Computer Vision.Venice,Italy:IEEE , 2017 : 5058 - 5066 .

RACHWAN J , ZUGNER D , CHARPENTIER B , et al. Winning the lottery ahead of time:efficient early network pruning [C ] //Proceedings of International Conference on Machine Learning. Honolulu, HI,US:ICML , 2022 : 18293 - 18309 .

KIM J H , PARK S U , KWAK N J . Paraphrasing complex network:network compression via factor transfer: arXiv:1802.04977 [R ] . Ithaca, NY , US : Cornell University , 2018 : 1802.04977.

HUANG Z H , WANG N Y . Like what you like: Knowledge distill via neuron selectivity transfer arXiv:1707.01219 [R ] . Ithaca,NY , US : Cornell University , 2017 :1707.01219.

HEO B , LEE M , YUN S , et al . Knowledge transfer via distillation of activation boundaries formed by hidden neurons [C ] //Proceedings of the AAAI Conference on Artificial Intelligence.Honolulu,HI,US:AAAI , 2019 , 33 ( 1 ): 3779 - 3787 .

PASSALIS N , TEFAS A . Learning deep representations with probabilistic knowledge transfer [C ] //Proceedings of the European Conference on Computer Vision.Munich,Germany:ECCV , 2018 : 268 - 284 .

RASTEGARI M , ORDONEZ V , REDMON J , et al . Xnor-net: Imagenet classification using binary convolutional neural networks [C ] //Proceedings of the European Conference on Computer Vision. Cham,Germany:Springer , 2016 : 525 - 542 .

LIU Z H , WANG Y H , HAN K , et al . Instance-aware dynamic neural network quantization [C ] //Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.New Orleans,LA,US:IEEE , 2022 : 12434 - 12443 .

ZHONG Y S , LIN M B , NAN G R , et al . Intraq: learning synthetic images with intra-class heterogeneity for zero-shot network quantization [C ] //Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.New Orleans,LA,US:IEEE , 2022 : 12339 - 12348 .