Image stitching by combining optimal seam and multi-resolution fusion
Objective: Image stitching technique can synthesize a panoramic image from multiple successive images, and can be applied in many military and civil applications. Ghost problem exists in the overlapping areas between two images being stitched when moving objects exist, or registration error occurs during image stitching. The issues of stitching line, and color inconsistency, etc, occur when camera exposure time and illumination changes during imaging. These factors may affect the panoramic image if the images are simply fused, so the improved image fusion technique, which is one of key technologies in image stitching, can be used to solve the above problems. Since Optimal seam algorithm is one of effective methods, an improved optimal seam algorithm based on differential image weighting is proposed to solve ghost problem where seams pass through moving object or inaccurate registration areas in classical optimal seam algorithm. Partition fusion algorithm based on multi-resolution fusion and weighted average fusion is presented to solve the stitching line problem caused by the change of exposure time and illumination. Method: Firstly, the images being stitched are mapped to cylindrical surface after registration, where Harris corner is used to find correspondences between images. Secondly, the overlapping areas between the image being stitched and the fused image are calculated, and then partitioned into three areas, including an optimal seam search area and two transition regions. The optimal seam search area is set to occupies three fifth, and both transition regions occupy one fifth separately in the paper. Thirdly, Differential image weighted optimal seam algorithm is proposed to search optimal seam in seam search regions. Not only considering the difference of color and structure, the metric of computing optimal seam for each pixel is also weighted by image difference. The weighting coefficient is set to infinity if this difference is above a threshold, so the moving object region if image difference is big can be bypassed when searching seam line. After the metric image is computed, dynamic programming algorithm is used to search the optimal seam in this metric image. Finally, the mask image is generated based on the obtained optimal seam. Because there are a lot of invalid areas due to the mapping to cylindrical surface, the method of extending image is adopted to fill these areas using the nearby pixels before combining two images. Multi-resolution image fusion algorithm is performed in the whole overlapping region after image extension, and then weighted average fusion algorithm is adopted to eliminate stitching line in transition regions. Result: In this paper, several image sequences captured by mobile phone at the crossing are used to test the proposed algorithm, and the proposed algorithm is compared with the algorithm of dynamic image mosaic via SIFT and dynamic programming and the stitching algorithm implemented in OpenCV. There are a large number of moving objects(e.g. cars, pedestrians, etc.) in the images, and experimental results demonstrate that, the probability of optimal seam passing through moving objects and inaccurate registration areas is obviously reduced due to the improved optimal seam search algorithm, and it is as low as possible to pass through moving objects when the optimal seam is difficult to circumvent them. The stitching line is effectively eliminated by combining the algorithms of multi-resolution fusion and weighted average fusion, and the quality of the panoramic image using our algorithm is better than that using multi-resolution fusion algorithm only, especially in the scenario the difference of illumination exists. The stitching result is the same or better than using OpenCV, although there are some distortions in panoramic image due to misregistration. The comparison of computational cost between the proposed algorithm and the dynamic image mosaic algorithm is also presented. Conclusion: The proposed optimal seam algorithm can avoid the problem, where seam line passes through moving object region or registration error exists. The multi-resolution image fusion algorithm is applied to non-strict overlapping image fusion through image extension after region partition, and high-quality panoramic image is synthesized. However, our algorithm has the following shortcomings: the code must be optimized to meet real-time requirements, and global registration optimization can be used to reduce distortion problem in the panoramic image during the process of image stitching.