Research on Wide-area Real-time Imaging Algorithm and System for Military Vehicles

doi:10.12382/bgxb.2024.0903

Abstract

Abstract:

The wide-area imaging becomes difficult due to the small overlapping area of multi-camera images,large parallax,and complex and changeable feature information in practical application scenarios when a multi-camera detection system is used to provide environmental perception for military vehicles such as tanks and armored vehicles.A wide-area real-time imaging algorithm based on ChArUco board calibration is proposed,and a wide-area visual enhancement system for military vehicles is designed for drivers to perform real-time imaging of the external environment in a closed compartment with limited vision.This algorithm takes advantage of the characteristic that the ChArUco board can still be accurately calibrated under occlusion conditions to achieve the accurate calibration of a small-overlapping multi-camera system.Projection transformation is carried out through calibration parameters to avoid the interference of image feature information and effectively deal with complex feature scenarios.At the same time,the parallax in the overlapping area is eliminated by the fusion of optical flow relationships,and a wide-area real-time imaging is achieved by using the projection look-up table and parallel acceleration optimization method.This algorithm is deployed on a mobile computing platform to form a complete wide-area visual enhancement system.The experimental results show that this system and algorithm can meet the wide-area real-time imaging requirements of various scenarios and improve the visual perception ability of military vehicles in complex environments.

Key words: military vehicle, image stitching, multi-camera calibration, ChArUco board, optimization acceleration, system design

CLC Number:

TP391.4

WU Jing, LIN Jianhua, HUANG Jiong, YANG Zheng, HUANG Feng. Research on Wide-area Real-time Imaging Algorithm and System for Military Vehicles[J]. Acta Armamentarii, 2025, 46(9): 240903-.

Figures/Tables 17

Fig.1 Algorithm flow chart

Fig.2 ChArUco board detection example

Fig.3 Camera extrinsic calibration model

Fig.4 ChArUco corner matching illustration

Fig.5 Estimated optical flow relationship(the left is the overlapping area of two adjacent images,and the right is the optical flow relationship of overlapping area of two adjacent images)

Table 1 Comparison of various algorithms for low overlap rate scenarios

重叠率/%	相机摆放夹角/(°)	基于特征点检测和匹配的拼接算法是否有效	本文算法是否有效
30	49	是	是
20	56	是	是
15	59	否	是
10	63	否	是

Fig.6 Stitching of scenes with overlap rate of 10%

Fig.7 Validation of the proposed algorithm for feature sparse scenes

Fig.8 Verification of algorithms in scene with uneven feature distribution

Fig.9 Validation of the algorithm for noise images in low-light scenes

Fig.10 Verification of algorithm in complex dynamic scenarios

Fig.11 Composition of wide-area visual enhancement system

Fig.12 Optimized and accelerated process of the algorithm

Table 2 Comparison of time consumptions before and after algorithm optimization and acceleration

算法步骤	优化前耗时/ms	优化后耗时/ms	加速比
数据传入设备端		3.12
图像投影	17.12	3.23	5.30
光流计算	105.57	14.73	7.17
图像融合	20.41	3.34	6.11
数据传回主机端		2.51
总体耗时	143.10	26.93	5.31

Fig.13 Algorithm and system road experiment 1

Fig.14 Algorithm and system road experiment 2

Table 3 Objective data evaluation of various stitching algorithms

图像区域	AutoStitch		APAP		本文算法
图像区域	SSIM↑	PSNR↑	SSIM↑	PSNR↑	SSIM↑	PSNR↑
①	0.6555	17.9066	0.6124	16.4453	0.6841	19.1727
②	0.7027	19.4580	0.6288	17.9052	0.7704	21.0269
③	0.7224	18.7417	0.8017	23.3309	0.8052	23.3833
④	0.7379	20.4495	0.7294	20.0315	0.7929	23.1795

References 21

[1]	PEI Z H, ZHAO W Y, HU L, et al. Factors affecting the situational awareness of armored vehicle occupants[J]. Sensors, 2024, 24(11):3688.
[2]	曹昭睿, 郝永平, 刘万成, 等. 紧凑折反式仿生复眼及图像快速拼接识别算法[J]. 兵工学报, 2022, 43(8):1845-1857.
	CAO Z R, HAO Y P, LIU W C, et al. Compact catadioptric bionic compound eye and fast image mosaic recognition algorithm[J]. Acta Armamentarii, 2022, 43(8):1845-1857. (in Chinese) doi: 10.12382/bgxb.2021.0393
[3]	CHEN J, YE X, GAO S L, et al. Planar wide-angle-imaging camera enabled by metalens array[J]. Optica, 2022, 9(4):431-437.
[4]	PAN R F, ZHANG Y, XU L, et al. Stitching videos from unstructured camera arrays with rectangular boundaries[J]. IEEE Access, 2024, 12:21777-21786.
[5]	QIAO K, WANG S K, JIAO G C, et al. Research on noise testing and reduction of low illumination imaging module[C]//Proceedings of the Fifth Symposium on Novel Optoelectronic Detection Technology and Application.Xi’an, China:SPIE, 2019, 11023:528-532.
[6]	FU M Y, LIANG H, ZHU C H, et al. Image stitching techniques applied to plane or 3-D models:a review[J]. IEEE Sensors Journal, 2023, 23(8):8060-8079.
[7]	裴红星, 刘金达, 葛佳隆, 等. 图像拼接技术综述[J]. 郑州大学学报(理学版), 2019, 51(4):1-10,29.
	PEI H X, LIU J D, GE J L, et al. Review of image stitching techniques[J]. Journal of Zhengzhou University(Natural Science Edition), 2019, 51(4):1-10,29. (in Chinese)
[8]	任海蕾. 基于多相机标定的全景合成方法研究[D]. 西安: 西安电子科技大学, 2019.
	REN H L. Research on panorama synthesis method based on multi-camera calibration[D]. Xi’an: Xidian University, 2019. (in Chinese)
[9]	郑文玲, 钱宏文, 杨文豪, 等. 基于多相机标定的全景实时拼接算法研究[J]. 电子设计工程, 2021, 29(17):165-169.
	ZHENG W L, QIAN H W, YANG W H, et al. Research on real-time panoramic mosaic algorithm based on multi-camera calibration[J]. Electronic Design Engineering, 2021, 29(17):165-169. (in Chinese)
[10]	CHEN H F, ZHUANG J L, LIU B Y, et al. Camera calibration method based on circular array calibration board[J]. Systems Science & Control Engineering, 2023, 11(1):2233562.
[11]	黄文文, 彭小红, 李丽圆, 等. 相机标定方法及进展研究综述[J]. 激光与光电子学进展, 2023, 60(16):9-19.
	HUANG W W, PENG X H, LI L Y, et al. Review of camera calibration methods and their progress[J]. Laser & Optoelectronics Progress, 2023, 60(16):9-19. (in Chinese)
[12]	段志瑜, 罗哉, 江文松, 等. 基于改进ArUco标签的位姿估计方法[J]. 计量学报, 2024, 45(3):364-371.
	DUAN Z Y, LUO Z, JANG W S, et al. Pose estimation method based on improved ArUco tags[J]. Acta Metrologica Sinica, 2024, 45(3):364-371. (in Chinese)
[13]	LIN W W, LIANG P D, LUO G T, et al. Research of online hand-eye calibration method based on ChArUco board[J]. Sensors, 2022, 22(10):3805.
[14]	ZHANG Z Y. Flexible camera calibration by viewing a plane from unknown orientations[C]//Proceedings of the Seventh IEEE International Conference on Computer Vision.Kerkyra, Greece:IEEE, 1999:666-673.
[15]	FISCHLER M A, BOLLES R C. Random sample consensus:a paradigm for model fitting with applications to image analysis and automated cartography[J]. Communications of the ACM, 1981, 24(6):381-395.
[16]	de JESUS RUBIO J. Stability analysis of the modified levenberg-marquardt algorithm for the artificial neural network training[J]. IEEE Transactions on Neural Networks and Learning Systems, 2020, 32(8):3510-3524.
[17]	夏丹, 周睿. 视差图像配准技术研究综述[J]. 计算机工程与应用, 2021, 57(2):18-27. doi: 10.3778/j.issn.1002-8331.2007-0040
	XIA D, ZHOU R. Survey of parallax image registration technology[J]. Computer Engineering and Applications, 2021, 57(2):18-27. (in Chinese) doi: 10.3778/j.issn.1002-8331.2007-0040
[18]	FARNEBACK G. Two-frame motion estimation based on polynomial expansion[C]//Proceedings of the Image Analysis:13th Scandinavian Conference.Berlin,Heidelberg, Germany:Springer, 2003:363-370.
[19]	BROWN M, LOWE D G. Automatic panoramic image stitching using invariant features[J]. International Journal of Computer Vision, 2007, 74(1):59-73.
[20]	ZARAGOZA J, CHIN T J, BROWN M S, et al. As-projective-as-possible image stitching with moving DLT[C]//Proceedings of 2013 IEEE Conference on Computer Vision and Pattern Recognition.Portland,OR, US:IEEE, 2013:2339-2346.
[21]	刘杰, 游品鸿, 田明, 等. 基于局部投影的视差图像拼接平滑优化[J]. 电子学报, 2022, 50(6):1451-1456. doi: 10.12263/DZXB.20200664
	LIU J, YOU P H, TIAN M, et al. Smooth optimization of parallax images mosaic based on local projection[J]. Acta Electronica Sinica, 2022, 50(6):1451-1456. (in Chinese) doi: 10.12263/DZXB.20200664