Thick Cloud Removal for Aerial Images Based on Deep Convolutional Generative Adversarial Networks

doi:10.3969/j.issn.1000-1093.2019.07.013

Abstract

Abstract: A thick cloud removal method for aerial images based on deep convolution generative adversa- rial network (DCGAN) is proposed. In the proposed method, a region covered by cloud in image is regarded as a missing part in the image inpainting, and the adversarial learning of convolutional neural network is used to compensate the missing information. A DCGAN model including generator-discriminator is designed. The generator is an encoder-decoder structure with a joint loss function, including reconstruction loss, adversarial loss and total variation loss. The discriminator is used to measure the authenticity of the generated image with a loss function of adversarial loss. The prediction performance of network is improved by iterating the joint optimization of generator and discriminator. Poisson image editing is introduced to reduce the influences of color difference and boundary artifact. The experimental results on both simulated and real cloud images show that the proposed method is better than the classic method in terms of peak signal to noise ratio, structural similarity index measure, natural image no-reference quality evaluation algorithm and its improved algorithm, and has a small computational complexity. Key

Key words: aerialimage, thickcloudremoval, deepconvolutionalgenerativeadversarialnetwork, Poissonimageediting

CLC Number:

TP391.413

LI Congli, ZHANG Siyu, WEI Zhe, XUE Song. Thick Cloud Removal for Aerial Images Based on Deep Convolutional Generative Adversarial Networks[J]. Acta Armamentarii, 2019, 40(7): 1434-1442.

References

［1］ ZHANG C R, LI W D, TRAVIS D J. Restoration of clouded
pixels in multispectral remotely sensed imagery with cokriging[J]. International Journal of Remote Sensing, 2009, 30(9): 2173-2195.
[2］ WANG Z Y, JIN J Q, LIANG J W, et al. A new cloud removal algorithm for multi-spectral images[J]. Proceedings of SPIE, 2005, 6043: 230-240.
[3］ FENG C, MA J W, DAI Q, et al. An improved method for cloud removal in ASTER data change detection[C]∥Proceedings of 2004 IEEE International Geoscience and Remote Sensing Symposium. Anchorage, AK, US: IEEE, 2004: 3387-3389.
[4］ LI M, LIEW S C, KWOH L K. Automated production of cloud-free and cloud shadow-free image mosaics from cloudy satellite imagery[C]∥Proceedings of the XXth ISPRS Congress. Istanbul, Turkey: International Society for Photogrammetry and Remote Sensing, 2004: 12-13.
[5］ HELMER E H, RUEFENACHT B. Cloud-free satellite image mosaics with regression trees and histogram matching[J]. Photogrammetric Engineering & Remote Sensing, 2005, 71(9): 1079-1089.
[6］ LIN C H, TSAI P H, LAI K H, et al. Cloud removal from multitemporal satellite images using information cloning[J]. IEEE transactions on Geoscience and Remote Sensing, 2013, 51(1): 232-241.
[7］ ZHANG C Y, LI Z W, CHENG Q, et al. Cloud removal by fusing multi-source and multi-temporal images[C]∥Proceedings of IEEE International Geoscience and Remote Sensing Symposium. Fort Worth, TX, US: IEEE, 2017: 2577-2580.
[8］ CHEN F, ZHAO Z M, PENG L, et al. Clouds and cloud shadows removal from high-resolution aerial images[C]∥Proceedings of IEEE International Geoscience and Remote Sensing Symposium. Seoul, South Korea: IEEE, 2005:4256-4259.
[9］ MAALOUF A, CARR P, AUGEREAU B, et al. A bandelet-based inpainting technique for clouds removal from remotely sensed images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2009, 47(7): 2363-2371.

[10］ LORENZI L, MELGANI F, MERCIER G. Inpainting strategies for reconstruction of missing data in VHR images[J]. IEEE Geoscience and Remote Sensing Letters, 2011, 8(5): 914-918.
[11］ CRIMINISI A, PREZ P, TOYAMA K. Region filling and object removal by exemplar-based image inpainting[J]. IEEE Transactions on Image Processing, 2004, 13(9): 1200-1212.
[12］ HAYS J, EFROS A A. Scene completion using millions of photographs[J]. Communications of the ACM, 2008, 51(10): 87-94.
[13］ SZEGEDY C, ZAREMBA W, SUTSKEVER I, et al. Intriguing properties of neural networks[EB/OL]. [2017-06-07]. https:∥arxiv.org/pdf/1312.6199.pdf.
[14］ GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C]∥Proceedings of Advances in Neural Information Processing Systems. Montreal, Canada: Curran Associates, Inc., 2014: 2672-2680.
[15］林懿伦, 戴星原, 李力, 等. 人工智能研究的新前线: 生成式对抗网络[J]. 自动化学报, 2018, 44(5): 775-792.
LIN Y L, DAI X Y, LI L, et al. The new frontier of AI research: generative adversarial networks[J]. Acta Automatica Sinica, 2018, 44(5): 775-792. (in Chinese)
[16］赵树阳, 李建武. 基于生成对抗网络的低秩图像生成方法[J]. 自动化学报, 2018, 44(5): 829-839.
ZHAO S Y, LI J W. Generative adversarial network for generating low-rank images[J]. Acta Automatica Sinica, 2018, 44(5): 829-839. (in Chinese)
[17］孔德江, 汤斯亮, 吴飞. 时空嵌入式生成对抗网络的地点预测方法[J]. 模式识别与人工智能, 2018, 31(1): 49-60.
KONG D J, TANG S L, WU F. Location prediction via generative adversarial network with spatial temporal embedding[J]. Pattern Recognition and Artificial Intelligence, 2018, 31(1): 49-60. (in Chinese)
[18］ GUPTA A, ZOU J. Feedback GAN (FBGAN) for DNA: a novel feedback-loop architecture for optimizing protein functions[EB/OL]. [2018-04-05]. https:∥arxiv.org/pdf/1804.01694.pdf.
[19] RADFORD A, METZ L, CHINTALA S. Unsupervised representation learning with deep convolutional generative adversarial networks[EB/OL]. [2016-01-07]. https:∥arxiv.org/pdf/1511.06434.pdf.
[20］ PATHAK D, KRAHENBUHL P, DONAHUE J, et al. Context encoders: feature learning by inpainting[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, US: IEEE, 2016: 2536-2544.
[21］ STRONG D, CHAN T. Edge-preserving and scale-dependent properties of total variation regularization[J]. Inverse Problems, 2003, 19(6): 165-187.
[22］ PREZ P, GANGNET M, BLAKE A. Poisson image editing[J]. ACM Transactions on Graphics, 2003, 22(3): 313-318.
[23］ RASMUS A, BERGLUND M, HONKALA M, et al. Semi-supervised learning with ladder networks[C]∥Proceedings of the 29th Annual Conference on Neural Information Processing Systems. Montreal, Canada: Curran Associates, Inc., 2015: 3546-3554.
[24］ KRIZHEVSKY A, SUTSKEVER I, HINTON G E. Imagenet classification with deep convolutional neural networks[C]∥Proceedings of the 25th International Conference on Neural Information Processing Systems. Lake Tahoe, NV, US: Curran Associates, Inc., 2012: 1097-1105.
[25］ YANG C, LU X, LIN Z, et al. High-resolution image inpainting using multi-scale neural patch synthesis[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI, US: IEEE, 2017, 1(2): 6721-6729.
[26］ DARABI S, SHECHTMAN E, BARNES C, et al. Image melding: combining inconsistent images using patch-based synthesis[J]. ACM Transactions on Graphics, 2012, 31(4): 82.
[27］ HUANG J B, KANG S B, AHUJA N, et al. Image completion using planar structure guidance[J]. ACM Transactions on graphics (TOG), 2014, 33(4): 129.
[28］ YEH R, CHEN C, LIM T Y, et al. Semantic image inpainting with perceptual and contextual losses[EB/OL]. [2016-07-26]. https:∥arxiv.org/pdf/1607.07539.pdf.
[29］ KINGMA D, BA J. Adam: a method for stochastic optimization[EB/OL]. [2017-01-30]. https:∥arxiv.org/pdf/1412.6980.pdf.
[30］ WANG Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600-612.
[31］ MITTAL A, MURALIDHAR G S, GHOSH J, et al. Blind image quality assessment without human training using latent quality factors[J]. IEEE Signal Processing Letters, 2012, 19(2):75-78.
[32］ ZHANG L, ZHANG L, BOVIK A C. A feature-enriched completely blind image quality evaluator[J]. IEEE Transactions on Image Processing, 2015, 24(8):2579-2591.

第40卷第7期
2019 年7月兵工学报ACTA
ARMAMENTARIIVol.40No.7Jul.2019