基于通道校正卷积的真彩色微光图像增强

doi:10.12382/bgxb.2022.0204

摘要/Abstract

摘要：

针对现有真彩色夜视相机所成图像亮度低、对比度低、噪声和色彩失真等问题,提出基于通道校正卷积的神经网络算法。通道校正卷积的上分支引入通道注意力块分析RGB通道之间的特征,用来代替U-Net网络中的传统卷积,实现颜色恢复并保留更多图像信息;在传统损失函数中增加Sobel损失函数和色彩损失函数,抑制噪声的同时并保护图像细节,减小色差、增强对比度。采集真实场景下的图像数据集,提升对实际数据的处理效果。实验结果表明:该算法能同时处理低照度图像的亮度、对比度、噪声和色差问题,增强效果优于目前主流算法;与传统卷积的U-Net网络相比,降低了模型复杂度,提高了运行速度,计算量减少了13.71%,参数减少了13.65%,PSNR值提升了29.20%,SSIM值提升了7.23%,色差减少了10.46%,兼顾了成像质量与成像速度。

关键词: 微光图像, 图像增强, 颜色恢复, 噪声抑制, 卷积神经网络

Abstract:

Aiming at the problems of low brightness, low contrast, noise and color distortion of images produced by existing true color night vision cameras, a neural network algorithm based on channel-calibrated convolution is proposed. The upper branch of the channel-calibrated convolution introduces a channel attention block to analyze the features between the RGB channels. This replaces the traditional convolution in the U-Net network, enabling color recovery and the retention of more image information. The Sobel loss function and color loss function are added to the traditional loss function to suppress noise, preserve image details, reduce chromatic aberration, and enhance contrast. An image dataset under real conditions is collected, which improves the processing effect of the actual data. The experimental results show that the algorithm in this paper can simultaneously deal with the brightness, contrast, noise and chromatic aberration of low-light images, and the enhancement effect is better than the existing mainstream algorithms. Compared with the traditional convolutional U-Net network, the novel method reduces the model complexity and improves operating speed, with a 13.71% reduction in computation, a 13.65% reduction in parameters, a 29.20% increase in PSNR, a 7.23% increase in SSIM, and a 10.46% decrease in chromatic aberration. The algorithm in this paper strikes a balance between imaging quality and speed.

Key words: low-light-level image, image enhancement, color restoration, noise suppression, convolutional neural network

何锦成, 韩永成, 张闻文, 何伟基, 陈钱. 基于通道校正卷积的真彩色微光图像增强[J]. 兵工学报, 2023, 44(6): 1643-1654.

HE Jincheng, HAN Yongcheng, ZHANG Wenwen, HE Weiji, CHEN Qian. True Color Low-Light Image Enhancement Based on Channel-Calibrated Convolution[J]. Acta Armamentarii, 2023, 44(6): 1643-1654.

图/表 21

图1 基于CCConv网络结构及图像处理流程

Fig.1 CCConv-based network structure and image processing flow

图2 通道注意块工作流程

Fig.2 Channel attention block workflow

图3 通道校正卷积原理框图

Fig.3 Block diagram of Channel-Calibrated Convolution

表1 网络各卷积层特征通道数

Table 1 Number of feature channels in each convolutional layer of the network

名称	L1	L2	L3	L4	L5	L6	L7	L8	L9	L10
通道数	32	64	128	256	512	256	128	64	32	3

表2 消融实验结果

Table 2 Ablation experiment results

表3 不同参数及结构在验证集上的定量对比

Table 3 Quantitative comparison of different parameters and structures on the validation set

算法	FLOPs/10⁶	参数量/10⁶	PSNR	SSIM
U-Net算法	15.5135	7.7597	20.1727	0.7673
本文算法	13.3869	6.7002	26.0631	0.8228

表4 不同损失函数的效果对比

Table 4 Comparison of different loss functions

表5 不同损失函数在验证集上的定量对比

Table 5 Quantitative comparison of different loss functions on the validation set

损失函数	l₁	l₁和l_Sobel	l₁、l_Sobel和l_yuv
PSNR	24.2915	25.3657	26.0631

表6 不同算法在测试集上的定量对比

Table 6 Quantitative comparison of different algorithms on the test set

算法	PSNR	SSIM	NIQE
Dong算法	13.0878	0.4033	10.5240
LIME算法	17.7791	0.4068	10.4590
Retinex Net算法	17.1443	0.4191	10.5295
Zero-DCE算法	10.5861	0.4143	9.1808
Enlighten GAN算法	13.3488	0.4747	5.4888
KinD算法	20.1770	0.7765	4.6527
ICE-Net算法	18.9233	0.5968	3.7195
本文算法	24.0554	0.7413	3.3008

表7 不同算法在不同场景下的效果对比

Table 7 Comparison of different algorithms in different scenarios

图4 彩色低照度成像实验系统示意图

Fig.4 Schematic diagram of the colored low-light imaging experiment system

表8 安森美KAE-02150 EMCCD参数

Table 8 ON Semiconductor KAE-02150 EMCCD parameters

参数	数值
分辨率	1984×1124
像素尺寸/μm	5.5 ×5.5
最大帧率/(帧·s^-1)	30
动态范围/dB	68
读噪声	9 e^-rms
有效图像尺寸/mm	10.56×5.94 × 12.1

表9 不同算法在不同场景下的效果对比

Table 9 Comparison of different algorithms in different scenarios

表10 不同算法在测试集上的定量对比

Table 10 Quantitative comparison of different algorithms on the test set

算法	PSNR	SSIM	NIQE
Dong算法	9.1868	0.2632	10.6330
LIME算法	13.9119	0.2901	11.3668
Retinex Net算法	15.0024	0.3725	4.7731
Zero-DCE算法	8.2234	0.2385	10.0913
Enlighten GAN算法	10.4559	0.2687	6.5092
KinD算法	16.7876	0.5432	7.0217
ICE-Net算法	14.7353	0.3917	4.9706
本文算法	24.6161	0.7316	4.5761

表11 不同算法的运行时间

Table 11 Running time of different algorithmss

算法	分辨率
算法	320×256	640×512	1280×1024
Dong算法	0.03	0.09	0.73
LIME算法	0.14	0.21	0.85
Retinex Net算法	0.01	0.03	0.10
Zero-DCE算法	0.001	0.001	0.001
Enlighten GAN算法	0.01	0.03	0.11
KinD算法	0.01	0.05	0.16
ICE-Net算法	0.01	0.02	0.09
本文算法	0.01	0.03	0.09

表12 不同算法在不同照度下的效果对比

Table 12 Comparison of different algorithms under different illumination conditions

表13 不同算法在不同照度上的定量对比

Table 13 Quantitative comparison of different algorithms for different illuminances

算法	照度/lx
算法	5×10^-1	5×10^-2	5×10^-3
Dong算法	42.2111	49.3682	51.4946
Retinex Net算法	49.0934	51.3830	52.8046
U-Net算法	30.1576	35.4009	37.4378
Zero-DCE算法	45.2154	52.2739	54.8249
Enlighten GAN算法	41.2146	45.1531	46.4799
KinD算法	31.0168	34.3155	35.3583
ICE-Net算法	63.5733	66.0383	67.4186
本文算法	24.5726	33.4530	34.6820

表14 不同算法在不同场景下的效果对比图

Table 14 Comparison of different algorithms in different scenarios

表15 不同算法在不同场景中的NIQE对比

Table 15 NIQE comparison of different algorithms in different scenarios

算法	场景1	场景2	场景3	平均值
Dong算法	4.4046	6.9270	5.7350	5.6889
LIME算法	4.1401	6.9633	5.7872	5.6302
Retinex Net算法	4.3698	4.5812	5.0547	4.6686
Zero-DCE算法	4.1582	6.5981	5.5248	5.4270
Enlighten GAN算法	3.7187	5.7758	4.7373	4.7439
KinD算法	4.3454	5.7165	5.0255	5.0291
ICE-Net算法	3.8572	5.5220	4.8662	4.7485
本文算法	4.0432	4.4108	4.0038	4.1526

表16 不同算法在不同光照及点光源场景下的效果对比

Table 16 Comparison of the effects of different algorithms in different lighting and point light source conditions

表17 不同算法在不同场景中的NIQE对比

Table 17 NIQE comparison of different algorithms in different scenarios

算法	全局照明	局部照明	点光源	平均值
Dong算法	7.4571	7.7808	6.8216	7.3532
LIME算法	8.6032	9.5750	7.3832	8.5205
Retinex Net算法	5.2697	5.3154	4.7094	5.0982
Zero-DCE算法	6.9607	7.4526	6.2209	6.8781
Enlighten GAN算法	4.9247	5.2493	4.7647	4.9796
KinD算法	5.2197	6.0173	5.4594	5.5655
ICE-Net算法	4.8540	5.0558	4.2043	4.7047
本文算法	4.3494	4.9814	4.3887	4.5732

参考文献 27

[1]	蒋云峰, 武东生, 黄富瑜. 彩色夜视技术方法综述[J]. 激光技术, 2020, 44(1):108-112.
	JIANG Y F, WU D S, HUANG F Y. Summary of color night vision technology[J]. Laser Technology, 2020, 44(1):108-112. (in Chinese) doi: 10.1016/j.optlastec.2011.06.001 URL
[2]	陈一超, 胡文刚, 武东生, 等. 三波段微光彩色夜视方法研究[J]. 应用光学, 2015, 36(3):430-434. doi: 10.5768/JAO201536.0304001
	CHEN Y C, HU W G, WU D S, et al. Triple-band low light level color night vision technology[J]. Journal of Applied Optics, 2015, 36(3):430-434. (in Chinese) doi: 10.5768/JAO201536.0304001 URL
[3]	WAXMAN A M, AGUILAR M, FAY D A, et al. Solid-state color night vision: fusion of low-light visible and thermal infrared imagery[J]. Lincoln Laboratory Journal, 1998, 11(1):41-60.
[4]	马爽, 方建安, 孙韶媛, 等. 基于伪彩色融合图像聚类的夜视图像上色算法[J]. 光学学报, 2009, 29(6):1502-1507.
	MA S, FANG J A, SUN S Y, et al. Colorizing algorithm of night-vision image based on clustering of false color fused image[J]. Acta Optica Sinica, 2009, 29(6): 1502-1507. (in Chinese) doi: 10.3788/AOS URL
[5]	赵源萌, 王岭雪, 金伟其, 等. 基于色彩传递的单波段热图像彩色化处理算法[J]. 光学学报, 2009, 29(3):654-658.
	ZHAO Y M, WANG L X, JIN W Q, et al. Single-band infrared image colorization algorithm based on color transfer[J]. Acta Optica Sinica, 2009, 29(3):654-658. (in Chinese) doi: 10.3788/AOS URL
[6]	KOYAMA S, INABA Y, KASANO M, et al. A day and night vision MOS imager with robust photonic-crystal-based RGB-and-IR[J]. IEEE Transactions on Electron Devices, 2008, 55(3):754-759. doi: 10.1109/TED.2007.914828 URL
[7]	曾桂林, 苏秉华, 牛丽红, 等. 基于神经网络的微光图像增强与复原[J]. 兵工学报, 2006, 27(4): 652-654.
	ZENG G L, SU B H, NIU L H, et al. Neural network-based low-light-level image enhancement and reconstruction[J]. Acta Armamentarii, 2006, 27(4): 652-654. (in Chinese)
[8]	GUO X J, LI Y, LING H B. LIME: low-light image enhancement via illumination map estimation[J]. IEEE Transactions on Image Processing, 2016, 26(2):982-993. doi: 10.1109/TIP.2016.2639450 URL
[9]	冯维, 吴贵铭, 赵大兴, 等. 多图像融合Retinex用于弱光图像增强[J]. 光学精密工程, 2020, 28(3):736-744.
	FENG W, WU G M, ZHAO D X, et al. Multi images fusion retinex for low light image enhancement[J]. Optics and Precision Engineering, 2020, 28(3):736-744. (in Chinese) doi: 10.3788/OPE. URL
[10]	黄慧, 董林鹭, 刘小芳, 等. 改进Retinex的低光照图像增强[J]. 光学精密工程, 2020, 28(8):1835-1849.
	HUANG H, DONG L L, LIU X F, et al. Improved retinex low light image enhancement method[J]. Optics and Precision Engineering, 2020, 28(8):1835-1849. (in Chinese)
[11]	DONG X A, WANG G, PANG Y, et al. Fast efficient algorithm for enhancement of low lighting video[C]//Proceedings of 2011 IEEE International Conference on Multimedia and Expo. New York,NY,US: IEEE, 2011:1-6.
[12]	WANG W J, WEI C, YANG W H, et al. GLADNet: Low-light enhancement network with global awareness[C]//Proceedings of the 2018 13rd IEEE International Conference on Automatic Face & Gesture Recognition. New York,NY,US: IEEE, 2018:751-755.
[13]	WEI C, WANG W J, YANG W H, et al. Deep retinex decomposition for low-light enhancement: arXiv:1808.04560[R/OL]. Ithaca, NY, US: Cornell University (2018-08-01)[2021-04-01]. https://arxiv.org/abs/1808.04560.
[14]	SONG X J, HUANG J J, CAO J Z, et al. Multi-scale joint network based on Retinex theory for low-light enhancement[J]. Signal, Image and Video Processing, 2021, 15(6):1257-1264. doi: 10.1007/s11760-021-01856-y
[15]	ZHANG Y H, ZHANG J W, GUO X J. Kindling the darkness: a practical low-light image enhancer[C]//Proceedings of the 27th ACM International Conference on Multimedia. New York, NY,US:ACM, 2019:1632-1640.
[16]	黄辉先, 陈凡浩. 基于注意力机制和Retinex的低照度图像增强方法[J]. 激光与光电子学进展, 2020, 57(20):201004.
	HUANG H X, CHEN F H. Low-illumination image enhancement method based on attention mechanism and retinex[J]. Laser & Optoelectronics Progress, 2020, 57(20):201004. (in Chinese)
[17]	JIANG Y F, GONG X Y, LIU D, et al. Enlightengan: deep light enhancement without paired supervision[J]. IEEE Transactions on Image Processing, 2021, 30:2340-2349. doi: 10.1109/TIP.2021.3051462 URL
[18]	GUO C L, LI C Y, GUO J C, et al. Zero-reference deep curve estimation for low-light image enhancement[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York,NY, US: IEEE, 2020:1777-1786.
[19]	ZHANG Y, DI X G, ZHANG B, et al. Self-supervised image enhancement network: training with low light images only: arXiv:2002.11300[EB/OL]. Ithaca, NY, US:Cornell University(2020-02-26)[2021-04-01]. https://arxiv.org/abs/2002.11300.
[20]	唐超影, 浦世亮, 叶鹏钊, 等. 基于卷积神经网络的低照度可见光与近红外图像融合[J]. 光学学报, 2020, 40(16):1610001.
	TANG C Y, PU S L, YE P Z, et al. Fusion of low-illuminance visible and near-infrared images based on convolutional neural networks[J]. Acta Optica Sinica, 2020, 40(16):1610001. (in Chinese) doi: 10.3788/AOS URL
[21]	CHEN C, CHEN Q F, XU J, et al. Learning to see in the dark[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.New York,NY, US: IEEE, 2018:3291-3300.
[22]	RONNEBERGER O, FISCHER P, BROX T. U-net: Convolutional networks for biomedical image segmentation[C]//Proceedings of the 2015 18th International Conference on Medical Image Computing and Computer-assisted Intervention. Switzerland: Springer, 2015:234-241.
[23]	李俊峰. 基于RGB色彩空间自然场景统计的无参考图像质量评价[J]. 自动化学报, 2015, 41(9):1601-1615.
	LI J F. Non-reference image quality evaluation based on natural scene statistics in RGB color space[J]. Acta Automatica Sinica, 2015, 41(9):1601-1615. (in Chinese)
[24]	WOO S, PARK J, LEE J Y, et al. CBAM: Convolutional block attention module[C]//Proceedings of the 2018 15th European Conference on Computer Vision. Switzerland: Springer, 2018:3-19.
[25]	LIU J J, HOU Q, CHENG M M, et al. Improving convolutional networks with self-calibrated convolutions[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York,NY, US: IEEE, 2020:10096-10105.
[26]	MITTAL A, SOUNDARARAJAN R, BOVIK A C. Making a "completely blind" image quality analyzer[J]. IEEE Signal Processing Letters, 2012, 20(3):209-212. doi: 10.1109/LSP.2012.2227726 URL
[27]	HILL B, ROGER T, VORHAGEN F W. Comparative analysis of the quantization of color spaces on the basis of the CIELAB color-difference formula[J]. ACM Transactions on Graphics, 1997, 16(2):109-154. doi: 10.1145/248210.248212 URL