强振平,何丽波,陈旭,徐丹(西南林业大学大数据与智能工程学院, 昆明 650224;云南大学信息学院, 昆明 650091)
目的 图像修复是计算机视觉领域研究的一项重要内容，其目的是根据图像中已知内容来自动地恢复丢失的内容，在图像编辑、影视特技制作、虚拟现实及数字文化遗产保护等领域都具有广泛的应用价值。而近年来，随着深度学习在学术界和工业界的广泛研究，其在图像语义提取、特征表示、图像生成等方面的应用优势日益突出，使得基于深度学习的图像修复方法的研究成为了国内外一个研究热点，得到了越来越多的关注。为了使更多研究者对基于深度学习的图像修复理论及其发展进行探索，本文对该领域研究现状进行综述。方法 首先对基于深度学习图像修复方法提出的理论依据进行分析；然后对其中涉及的关键技术进行研究；总结了近年来基于深度学习的主要图像修复方法，并依据修复网络的结构对现有方法进行了分类，即分为基于卷积自编码网络结构的图像修复方法、基于生成式对抗网络结构的图像修复方法和基于循环神经网络结构的图像修复方法。结果 在基于深度学习的图像修复方法中，深度学习网络的设计和训练过程中的损失函数的选择是其重要的内容，各类方法各有优缺点和其适用范围，如何提高修复结果语义的合理性、结构及细节的正确性，一直是研究者们努力的方向，基于此目的，本文通过实验分析总结了各类方法的主要特点、存在的问题、对训练样本的要求、主要应用领域及参考代码。结论 基于深度学习图像修复领域的研究已经取得了一些显著进展，但目前深度学习在图像修复中的应用仍处于起步阶段，主要研究的内容也仅仅是利用待修复图像本身的图像内容信息，因此基于深度学习的图像修复仍是一个极具挑战的课题。如何设计具有普适性的修复网络，提高修复结果的准确性，还需要更加深入的研究。
Survey on deep learning image inpainting methods
Qiang Zhenping,He Libo,Chen Xu,Xu Dan(College of Big Data and Intelligent Engineering, Southwest Forestry University, Kunming 650224, China;School of Information Science and Engineering, Yunnan University, Kunming 650091, China)
Objective Inpainting is the process of reconstructing lost or deteriorated parts of images and videos. This reconstruction process is an important research area in the field of computer vision, and its purpose is to automatically repair lost content according to the known content of the images and videos. Inpainting has extensive application value in the fields of image editing, film and television special effect production, virtual reality, and digital cultural heritage protection. Deep learning has been widely studied in the academic and industrial fields in recent years. Its advantages in image semantic extraction, feature representation, and image generation have become increasingly prominent, leading to the increasing attention to and popularity of research on image inpainting based on deep learning. This study reviews the current research status of image inpainting based on deep learning to enable researchers to explore its theory and development. Method This paper first discusses the issue of image inpainting and summarizes the advantages and disadvantages of the commonly used methods by comparing their application results in image restoration in large areas. The theoretical basis of image inpainting based on deep learning is then analyzed, and the key technologies of image inpainting based on deep learning, which include the generation network based on auto encoder, the general training methods of deep network, and the training methods based on convolutional auto encoder network, are studied. This paper also summarizes some image inpainting methods based on deep learning that have been proposed in recent years and classifies these methods into three categories according to the architecture of their repairing network:image inpainting methods based on deep convolutional auto encoder architecture, image inpainting methods based on generative adversarial network (GAN) architecture, and image inpainting based on recurrent neural network (RNN) architecture. The basic structure of the generation network based on autoencoder is described, many improved networks and their loss functions are analyzed, and the experimental results based on different loss functions are provided. For the image inpainting methods based on GAN, the basic structure and process architecture of the GAN are described, and the experimental results based on some classical methods are presented. For the image inpainting methods based on RNN, the RNN model is analyzed, especially the methods based on the PixelRNN model, and the experimental results based on MNIST and CIFAR-10 datasets are provided. Result The design of deep learning networks and the selection of training loss functions are important in image inpainting based on deep learning methods. Each method has its merits, demerits, and application ranges. However, the main direction of the research is how to improve the rationality of semantics, the correctness of structure, and the detail of the repaired image. On the basis of this purpose, this paper summarizes and analyzes the characteristics, existing issues, requirements for training samples, application fields, and reference codes of these methods through experiments. Conclusion Although remarkable progress has been made in the field of image inpainting based on deep learning, the application of deep learning in image restoration remains in its infancy research focuses on using own image content information to restore images and still has demerits and own adaptation range. Consequently, image inpainting based on deep learning remains a challenging subject. How to improve the adaptability of the repairing network and the correctness of repairing results still requires further studies. This paper indicates the developing prospects from the following aspects:1) Further research may focus on how to design an adaptive network based on both semantic and texture networks. 2) The quality of inpainting images must be improved by studying the loss function of the repair network; the study of distance measurement based on different application purposes is especially critical. 3) Further research may focus on image inpainting methods of specific types, such as improving the generalization capability of the methods on small datasets through designing targeted training network structure and performing processes, such as fine tuning. 4) As processing power, such as GPUs, increases, the inpainting methods that train high-resolution images directly are also worth studying. 5) For some complex scenes, utilizing human-computer interaction strategies to repair images is still worth studying to promote their practical application and enrich digital image restoration technologies.