1Institute of Computer Science and Technology, Peking University, Beijing, China2Cooperative Medianet Innovation Center, Shanghai, China
ABSTRACTIn this paper, we propose a novel hierarchical image comple-tion approach using regularity statistics, considering structurefeatures. Guided by dominant structures, the target image isused to generate reference images in a self-reproductive wayby image data enhancement. The structure-guided image dataenhancement allows us to expand the search space for sam-ples. A Markov Random Field model is used to guide theenhanced image data combination to globally reconstruct thetarget image. For lower computational complexity and moreaccurate structure estimation, a hierarchical process is imple-mented. Experiments demonstrate the effectiveness of ourmethod comparing to several state-of-the-art image comple-tion techniques.
Index Terms— Image completion, structure detection,perspective transformation, image inpainting
1. INTRODUCTIONImage completion or image inpainting aims to fill the missingparts of an image and make the reconstructed image looknatural. This important topic in image processing gains atten-tions with the popularity of digital life. And it is widely usedin image editing applications such as watermark removal,panorama generation and cultural heritage restoration.
In the literature, image completion methods can be classi-fied into two main categories. The first category is diffusion-based which propagates structures into the missing region.Bertalmio et al. [1] take use of the geometric and photomet-ric information and propagate Laplacian descriptors along theisophote direction. The main defect of diffusion-based meth-ods is the blurring artifacts when missing regions is large .
The second category concerns examplar-based methods.The main idea is to sample the pixels/patches in the knownparts of the image and copy them to the missing region. Ac-cording to inpainting strategies, exemplar-based methods canbe categorized as greedy methods [2, 3] and global methods[4, 5, 6, 7, 8]. Greedy methods each time fill one pixel/patchby searching the best matches as samples and iteratively com-plete the missing region while global methods fill all missingpixels simultaneously by optimizing energy functions.
∗Corresponding authorThis work was supported by National High-tech Technology R&D Program(863 Program) of China under Grant 2014AA015205, National Natural Sci-ence Foundation of China under contract No. 61472011 and Beijing NaturalScience Foundation under contract No.4142021.
Among state-of-the-art global methods, Markov RandomField (MRF) model is widely used to build the energy func-tion, for it is effective to realize global image consistencyby defining the relationship of local adjacent pixels. In [5],candidate samples are searched throughout the whole image,leading to considerable processing time. To constrain thesearch space, Ruzic et al. [9] divide the image into sev-eral regions based on their context and search candidate sam-ples in similar regions. Meanwhile, He and Sun [8] limit thesearch space to only 60 candidates using the statistics of patchoffsets, obtaining gains in both speed and quality. In all ofthe works above, the basic operation is pixel/patch transla-tions. However, broken structures cannot be restored by sim-ply shifting the known pixels/patches into the unknown re-gions when image scenes contain complex transformations.Unfortunately, transformation operations other than transla-tion are seldom available in MRF-based methods.
In fact, patch transformations such as rotation, scaling [7]and perspective transform [10] have been taken into accountin coherence-based methods. The main problem for this kindof method is that more constraints need to be enforced whenincreasing the degrees of freedom or the result would fall intolocal optimum and suffer structure distortions.
In this paper, we use the MRF model to better realizeglobal image consistency. At the same time, the search spaceis enriched by uniform structure-guided image data enhance-ment, without giving too much degrees of freedom, thusleading to fewer structure distortions. Specifically, dominantstructure are extracted based on patch regularity and used asguidance to generate the enhanced images for reference. Tocombine the information of multiple reference images, wepropose a hierarchical MRF-based image completion methodusing regularity statistics. Finally, we validate our method bycomparing with state-of-the-art image completion algorithmson both man-made scenes and natural scenes.
The rest of this paper is organized as follows: Section 2describes the proposed image completion approach. Experi-mental results are shown in Section 3 and concluding remarksare given in Section 4.
Fig. 1. Flow chart of the proposed hierarchical structure-guided image completion via regularity statistics.
as Ω, and its contour is indicated by δΩ. The source regions,on the other hand, are denoted as Φ = I − Ω. Our goal is tofill Ω seamlessly using the information of Φ. Fig. 1 shows themain procedure of the proposed method.
2.1. Structure-Guided Image Data Enhancement2.1.1. Dominant Structure Line Detection
Since human eyes are sensitive to structure consistency, bydetecting and preserving linear structures, the completionquality can be well improved. Along structure lines, imagepatches demonstrate high regularity and this can be a signif-icant cue on the determination of the lines. Inspired by [8],we detect the regularity using patch offsets. The frequencyof the matched patches’ relative spacial offsets is calculated.The most frequent ones form a set of dominant offsets andare allocated to unknown pixels for completion. The offsetsextraction will be discussed in detail in Section 2.2. To take astep further, we analyse the dominant offsets which are likelydistributed along dominant structure lines in the offset space,as shown in Fig. 2(c). We use a RANSAC-based votingapproach [11] to detect the best fitting line as a dominantstructure line. We repeat the RANSAC process over the out-liers to search multiple dominant structure lines (the red linein Fig. 2(c)) until the number of inliers is less than a giventhreshold.
(a) Input image I (b) Matched features (c) Structure lines
(d) Information quality (e) H(I)
H−1 H−2
H H2
(f) Enhanced results
Fig. 2. Structure-guided image data enhancement.
2.1.2. Perspective Shift TransformationThe ubiquitous foreshortening effects make the results ofMRF-based image completion methods degrade severely, forthey only perform translation operation. We put forward theconcept of Perspective Shift in addition to traditional transla-tion. Objects are shifted in a way that satisfies foreshorteningeffects. To accomplish this task, we estimate a homographymatrix that performs an image registration transformation.
We begin with Speeded Up Robust Features (SURF, [12])points detection and compute SURF descriptors for each fea-ture point k. Then, these feature points are matched (as shownin Fig. 2(b)) under two spacial constraints. Guided by thedominant structure line l, ki and kj are matched if their vector−−→kikj satisfies the distance constraint λmin < |
−−→kikj | < λmax
and the angle constraint dπ(−−→kikj , l) < λθ, where dπ(·, ·) is
the included angle of two lines. λmin ensures no feature pointis matched with itself and λmax is considered based on theidea of local similarity.
Then we perform a RANSAC-based voting algorithmover the matched feature points to find the best fitting trans-formation matrix H. We repeat the RANSAC process overthe outliers to obtain multiple perspective shift transforma-tions. To find the optimal one, we define two measurements:
Information quantity. We use the percentage of the per-spectively shifted known information in the missing regionsto measure the information quantity:
Rquantity(H) = |H(Φ) ∩ Ω|/|Ω|, (1)
where H(·) is the perspective shift operation and |Ω| is thepixel number in the source region Ω.
Information quality. Textures after an ideal perspectiveshift operation should match those in the original image. Weconcentrate on the outer boundary of Ω with a width of λmax
pixels (denoted as ∆Ω) and define the information quality as:
Rquality(H) =
√|∆Ω|/
∑p∈∆Ω
(I(p)− (H(I))(p))2. (2)
Next, the optimal H is obtained by solving:
1712
maxH
Rquality(H) s.t. Rquantity(H) > λquantity. (3)
To demonstrate how dominant structure lines guide the trans-formation estimation, we enumerate lines of different anglesto impose the angle constraint and obtain corresponding H.The information quality for each H is calculated and Fig. 2(d)shows how angle restriction affects the information quality,which validates that dominant structure lines are reliable toguide the algorithm to find H with relatively high Rquality(H).
Moreover, if a certain image registration transformationexists, Hi could be possibly valid perspective shift transfor-mation matrices as well (as shown in Fig. 2(f)). Intuitively,H2 represents a double perspective shift operation and H−1
represents an inverse perspective shift operation. We enrichthe reference images with Hi(I), |i| ∈ 1, 2, 3, 4.
2.2. Global Completion via Regularity StatisticsIn this section, we introduce our MRF-based image comple-tion method. The MRF-based algorithms [5, 8] treat imagecompletion as a labelling problem. In our work, we adoptthe same idea of patch offset statistics as [8], which benefitsfrom better texture and structure preservation. The target im-age I0 and its enhanced results form a set of reference imagesdenoted as SI = I0, ..., IW . We denote patch offsets ass = (u, v, w), where (u, v) is the coordinates of the patchoffsets and w ∈ 0, 1, ...,W indicates the w-th referenceimage. We match similar patches and calculate their offsetsby:
s = argmins
∥Ψ(p+ s)−Ψ(p)∥22, (4)
where p = (x, y, w) is the position of a patch and Ψ(p) is thepatch centered at (x, y) in Iw. We argue that offsets near thedominant structure lines contribute more to the completionprocess, and optimize the offset histogram within I0 by giv-ing higher weights to those offsets. Given the statistics of allpatch offsets from I0 to Ii, we pick out the most Ki frequentones and finally acquire a total number of K =
∑Wi=0 Ki
dominant offsets denoted as Ss = si(i = 1, ...,K). Thenimage completion is realized by seeking the optimal labellingL(p) = i ∈ 1, ...,K and copying the pixel value at p+ sito the pixel at p.
2.2.1. Image Completion Model Based on MRF PriorIn this section, we describe our MRF energy function. Com-pared to the definition of [8], image gradient are taken intoaccount to better preserve structure. Moreover, we reinforcethe boundary treatment by using patch difference rather thanpixel difference to accomplish better boundary consistency.
Given K dominant offsets, we define the energy functionto evaluate the labelling:
E(L) =∑
(p,q)∈N4
E(L(p), L(q)) + α∑p∈Ω
Ed(L(q)), (5)
where L(x) = i is the labelling that assigns the i-th dominantoffset to the pixel at p, N4 is the neighborhood system, α = 2is the weight to combine two energy terms:
Smoothness term: E(L(p), L(q)) penalizes the discon-tinuity within nearby pixels. It is defined as (for simplicity,we denote i = L(p), j = L(q)):
where G is the Gaussian weighting matrix and ⊗ is the point-wise product operator. Only the known pixels (Ψ ∩ Φ) arecalculated.
Once the MRF graph is given, the energy optimization canbe achieved using multi-label graph-cuts algorithm.
2.2.2. Hierarchical ImplementationWe propose a hierarchical implementation for low computa-tional complexity. Furthermore, the algorithm would be lesssensitive to noises and local singularities, thus dominant off-sets would be more reliable to demonstrate image regularity.To be specific, the target image is downsampled and com-pleted using the proposed method to obtain a labelling maps(p) = si(if L(p) = i) which describes the offset assign-ments. Then we multiply dominant offsets by two and up-sample the labelling map using the nearest interpolation. Tocorrect small misalignments, we propose an outside-in offsetrefinement algorithm based on pixel priority. The pixel pri-ority is calculated according to [2]. The labelling map of thepixel with the highest priority is refined:
s(p) = arg mins∈N5(s(p))
d(Ψ(p+ s),Ψ(p)), (9)
where N5(s) = N4(s) ∪ s. At each iteration of refining,the labelling map changes by at most one pixel but the finaladjustment can be large thanks to the hierarchical process.Fig. 3 demonstrates that misalignments are corrected.
In the end, Poisson fusion [13] is used to hide seams.
3. EXPERIMENTAL RESULTS AND ANALYSISThe proposed method is implemented on Visual Studio 2013platform. In the experiment, we set λmin = 10, λmax =200, λπ = π/8, λquantity = 0.2. The number of dominantoffsets are K0 = 60,Ki = 10(i ∈ 1, ...,W). Our imagecompletion approach is tested on varied images of man-madescenes1 and natural/semi-natural scenes2. The results are
Fig. 3. The misalignments are corrected by offset map refine-ment. The first row: (a) original image, (b) completion resultbefore refining and (c) after refining. The second row: patcheswith misalignments in (b). The third row: corresponding re-fined patches in (c).
compared to state-of-the-art image completion methods. Thewhole pictures and more experimental results can be foundon our website3.
We compare our approach with Photoshop Content AwareFill [4, 6], Offset-Based method [8] and Planar StructureGuidance method [10]. Fig. 4 shows the results. In theman-made scenes, the structures crack in Photoshop’s andHe’s results, for both methods search patches in only trans-
lation transformation space. Compared to Photoshop andHe’s method, our approach enhances the target image data toallow for a broader perspective transformation search spaceand suffers fewer artifacts. Meanwhile, Huang’s method al-lows for a search space of more degrees of freedom and mainstructures are preserved. As shown in Fig. 5, compared toHuang’s results, our approach suffers fewer distortions thanksto the perspective shift. In the semi-natural scenes, Huang’sresults suffer blurring artifacts. Our results owns better visualquality, which demonstrates the superiority of the proposedmethod.
(a) Huang[10] (b) Our result (c) Huang[10] (d) Our result
Fig. 5. Comparisons with Huang’s work for the local images.Our approach suffers less structure line distortions
4. CONCLUSIONGiven a target image with missing regions, the dominantstructure lines of it is detected and used to guide the imagedata enhancement to obtain several transformed versions ofthe target image in a self-reproductive way. These enhancedimages are combined to reconstruct the target image using theproposed regularity-statistics-based approach. The hierarchi-cal implementation accelerates the algorithm and works formore robust structure feature detection. We validate the effec-tiveness of our method by comparisons with state-of-the-artimage completion methods.
1714
5. REFERENCES
[1] M. Bertalio, G. Sapiro, V. Caselles, and C. Ballester,“Image inpainting,” in Computer Graphics Proceed-ings. ACM SIGGRAPH, 2000, pp. 417 – 424.
[2] A. Criminisi, P. Perez, and K. Toyama, “Region fillingand object removal by exemplar-based image inpaint-ing,” IEEE Transactions on Image Processing, vol. 13,pp. 1200 – 1212, September 2004.
[3] O. Le Meur, M. Ebdelli, and C. Guillemot, “Hierarchi-cal super-resolution-based inpainting,” IEEE Transac-tions on Image Processing, vol. 22, pp. 3779 – 3790,October 2013.
[4] Y. Wexler, E. Shechtman, and M. Irani, “Space-timecompletion of video.,” IEEE Transactions on PatternAnalysis and Machine Intelligence, vol. 29, no. 3, pp.463–476, March 2007.
[5] N. Komodakis and G. Tziritas, “Image completion us-ing efficient belief propagation via priority schedulingand dynamic pruning,” IEEE Transactions on ImageProcessing, vol. 16, no. 11, pp. 2649 – 2661, Novem-ber 2007.
[6] C. Barnes, E. Shechtman, A. Finkelstein, and D. B.Goldman, “Patchmatch: A randomized correspondencealgorithm for structural image editing,” ACM Transac-tions on Graphics, vol. 28, no. 3, pp. 341–352, August2009.
[7] S. Darabi, E. Shechtman, C. Barnes, D. B. Goldman,and P. Sen, “Image melding: Combining inconsistentimages using patch-based synthesis,” ACM Transactionson Graphics, vol. 31, no. 4, pp. 82:1–82:10, July 2012.
[8] K. He and J. Sun, “Image completion approaches usingthe statistics of similar patches,” IEEE Transactions onPattern Analysis and Machine Intelligence, vol. 36, pp.2423 – 2435, December 2014.
[9] T. Ruzic and A. Pizurica, “Context-aware patch-basedimage inpainting using markov random field modeling,”IEEE Transactions on Image Processing, vol. 24, no. 1,pp. 444 – 456, November 2014.
[10] J. B. Huang, S. B. Kang, N. Ahuja, and J. Kopf, “Im-age completion using planar structure guidance,” ACMTransactions on Graphics, vol. 33, no. 4, pp. 129:1–129:10, July 2014.
[11] M. A. Fischler and R. C. Bolles, “Random sample con-sensus:a paradigm for model fitting with appfication toimage analysis and automated cartography,” Communi-cations of the ACM, vol. 24, no. 6, pp. 381–395, 1981.
[12] H. Bay, T. Tuytelaars, and L. V. Gool, “Surf: Speededup robust features,” Proc. European Conf. ComputerVision, vol. 110, no. 3, pp. 404–417, 2006.
[13] P. Perez, M. Gangnet, and A. Blake, “Poisson imageediting,” ACM Transactions on Graphics, vol. 22, pp.313 – 318, July 2003.
