Manuscript submitted to doi:10.3934/xx.xx.xx.xxAIMS’ JournalsVolume X, Number 0X, XX 200X pp. X–XX

LOCAL BLOCK OPERATORS AND TV REGULARIZATION

BASED IMAGE INPAINTING

Wei Wan, Haiyang Huang and Jun Liu∗

Laboratory of Mathematics and Complex Systems (Ministry of Education of China),School of Mathematical Sciences, Beijing Normal University, Beijing 100875, P.R. China

Abstract. In this paper, we propose a novel image blocks based inpaint-ing model using group sparsity and TV regularization. The block matching

method is employed to collect similar image blocks which can be formed as

sparse image groups. By reducing the redundant information in these groups,we can well restore textures missing in the inpainting areas. We built a varia-

tional framework based on a local SVD operator for block matching and group

sparsity. In addition, TV regularization is naturally integrated in the modelto reduce artificial effects which are caused by image blocks stacking in the

block matching method. Besides, enforcing the sparsity of the representation,

the SVD operators in our method are iteratively updated and play the roleof dictionary learning. Thus it can greatly improve the quality of the restora-

tion. Moreover, we mathematically show the existence of a minimizer for the

proposed inpainting model. Convergence results of the proposed algorithm arealso given in the paper. Numerical experiments demonstrate that the proposed

model outperforms many benchmark methods such as BM3D based image in-painting.

1. Introduction. Image inpainting is an important and active topic in image pro-cessing and computer vision. The purpose of image inpainting is to fill in the in-formation for damaged or occluded regions of an image and to restore the integrityof the image by using the observed information. There are many applications inprotection of cultural relics, restoring aged or damaged photographs and films, textremoval and scratch removal, and so on.

Most inpainting techniques which have been proposed in recent years can beclassified into three groups: partial differential equation (PDE) based methods,block based methods and sparsity based methods.

PDE based methods try to fill in the missing region by propagating informationfrom the known region into the missing region. In [1], the first inpainting model(known as the BSCB inpainting model) based on the PDE method was presented byBertalmio et al., who aim to smoothly propagate information from the surroundingareas in the isophotes (i.e.,level lines of equal gray values) direction. This encour-aged Chan and Shen [2] to develop a total variation (TV) inpainting model forcartoon images, which is closely connected to the classical TV denoising model ofRudin, Osher and Fatemi [3]. In [4], Chan and Shen proposed the Curvature-Driven

2010 Mathematics Subject Classification. Primary: 94A08; Secondary: 68U10.Key words and phrases. Image inpainting, Sparsity based methods, Block based methods, TV

Regularization, SVD.∗ Corresponding author: jliu@bnu.edu.cn.

1

2 WEI WAN, HAIYANG HUANG AND JUN LIU

Diffusions (CDD) inpainting model, aiming at using the curvature of the isopho-tos to realize the Connectivity Principle. In [5], Chan, Kang and Shen presenteda variational inpainting model based on Euler’s elastica energy, which combinesCDD and BSCB. Other works include the Mumford-Shah-Euler inpainting model[6], inpainting methods based on the Cahn-Hilliard equation [7][8], and inpaintingwith the TV-stokes equation [9]. In general, variational and PDE based methodsshow good performance in geometric structure images (cartoon or structure) withsmall image gaps, but fail in the presence of texture, since the diffusion manner willoversmooth the missing region and cause a blurring effect.

Block based methods basically fill in the missing regions by copying blocks fromthe known regions. Block based methods are non-local in the sense that the wholeimage may be scanned in search of a matching block. They provide impressiveresults in texture inpainting. In [10], Criminisi et al. presented an exemplar basedinpainting algorithm in which filling order is critical to propagate both textureand structure information. Aujol et al. [11] illustrated the ability of the exemplarbased methods to reconstruct geometric features in a global variational framework.In [12], a geometrically guided exemplar based inpainting method for the jointrestoration of texture and geometry was presented by Cao et al.. In [13], Demanetet al. pointed out that the problem of block based inpainting can be regarded asfinding a correspondence map F from the inpainting region to the available region.Then, each pixel value x in the inpainting region can be represented by the mapF and the available region. Thus the map F should be chosen such that the blockcentered at y shall be similar to the one centered at F (x) as much as possible. In[14], Kawai et al. proposed an image inpainting model based on the block similarityconsidering brightness changes of sample textures and introducing spatial localityas an additional constraint. Meanwhile, Wexler et al. [15] presented a method forspace-time video completion and synthesis using various space-time blocks. Thismethod is capable of constructing large space-time regions of missing informationcontaining complex dynamic scenes. In [16], Arias et al. proposed a variationalframework for non-local image inpainting. In order to avoid local minima, theyuse several different patch sizes and incorporate the multi-scale approach in thealgorithm. However, such image blocks based methods may contain some redundantinformation and lead to undesirable artificial texture.

In recent years, sparsity based methods have become popular for image inpaint-ing. The main idea of sparsity based methods is to find a sparse representation of animage using appropriate basis functions, such as discrete cosine transform (DCT),wavelets, SVD. Then the missing pixels can be estimated by adaptively shrinkingthe sparse coefficients. Elad et al. [17] developed an image inpainting methodbased on morphological component analysis (MCA), which can fill in the inpaint-ing regions with both cartoon and texture layers effectively. In [18], an adaptiveinpainting algorithm using sparse reconstructions was proposed by Guleryuz et al..In [19], Fadili et al. introduced an expectation maximization (EM) algorithm forimage inpainting based on sparse representations. Xu et al. [20] proposed two typesof patch sparsity of natural image and applied them to the block based inpaintingmethod. The approach in [21] uses low-rank matrix completion for a non-local imageinpainting model. In [22], Li et al. proposed a universal variational framework forsparsity based image inpainting. Theoretically they proved the convergence of theproposed algorithm by using the properties of non-expensive operators. However,fixed basis functions are adopted in most of the mentioned sparsity methods. This

LOCAL BLOCK OPERATORS AND TV REGULARIZATION BASED IMAGE INPAINTING 3

has some limitations for sparse representations for all kinds of image blocks. Liu etal. [23] developed an image denoising method based on the group sparsity and basisupdating. Together with TV regularization, the quality of the restorations can begreatly improved. Here, we extend our previous work [23] to image inpainting withmatrix completion.

Inspired by matrix completion, we propose a novel inpainting model which com-bines sparsity based methods with block based methods. By using the SVD of patchmatrices, we can get some local basis vectors called local SVD operators. Updatingthese local SVD operators can effectively promote the sparsity of the representation.Using TV we can reduce the artificial ring effect caused by image block stacking.We will prove that the proposed model is well-defined. To solve the proposed modelefficiently, the splitting method is applied. We can get the convergence results forthe splitting algorithm using a variational inequality method.

The rest of the paper is organized as follows. In Section 2, the related inpaintingwork is presented. In Section 3, we outline our proposed inpainting model. Theexistence of a minimizer for the proposed model is proved in Section 4. Section 5contains the algorithm and some details about the implementation. In Section 6,we show some convergence results of our proposed algorithm. Experimental resultsare summarized in Section 7. Finally, we conclude this work in Section 8.

2. Related Work. Throughout this paper, images are denoted as matrices f ∈R√N×√N or vectors in RN such as f ∈ RN . Let us write vec : R

√N×√N → RN

as an operator for which f = vec(f). We rearrange the matrix f ∈ R√N×√N as

a vector f by stacking the columns of the f . We denote the latent clear imagesby f or f , while g or g are the observed images. Let Ω = 1, 2, · · · , N ⊂ R2

denotes the image region, O ⊂ Ω is the hole or inpainting region, and Oc := Ω \Ostands for the available region. The size of the image patch is

√n ×√n. We take

Pij ∈ Rn×N as some extra binary matrices and Pijg ∈ Rn stands for a vectorizedith most similar image block to the image patch at j ∈ Ω.

2.1. Matrix Completion. Matrix completion is a good inpainting tool for low-rank images and textures. In [24], Emmanuel et al. demonstrate that a low-rankmatrix can be perfectly recovered from an incomplete set of sampled entries. Foran incomplete low-rank matrix g, one can recover the desired matrix f by solvingthe following constrained optimization problem:

minf

rank(f)

s.t. fij = gij , (i, j) ∈ Oc,

where 1 ≤ i ≤√N, 1 ≤ j ≤

√N , and the constraint condition requires that f

should be closed to g in the known region Oc.Since this optimization problem is non-convex and NP-hard, it is not easy to be

optimized by the known algorithms. Thus, the nuclear norm is usually consideredas an alternative to approximate the rank of matrix. The related optimizationproblem is given by

minf‖f‖∗

s.t. fij = gij , (i, j) ∈ Oc,

where ‖ · ‖∗ is the nuclear norm defined as the sum of the singular values of f . Thisconvex problem can be efficiently solved by singular value thresholding [25].

4 WEI WAN, HAIYANG HUANG AND JUN LIU

2.2. Variational Framework for Image Group and Block based Method.In [23], Liu et al. developed an image denoising method based on group sparsityand basis updating. By putting all the similar patches together, the restorationproblem for a local image group centered at j can be written as

minXj

1

2‖Mj −Xj‖2F + µj‖Xj‖∗,

where Mj = [P1jg,P2jg, ...,PIjjg], Xj = [P1jf,P2jf, ...,PIjjf ], and Ij is thenumber of similar patches which is obtained by a block matching method. Applyingthe SVD on Mj , i.e. Mj = UjΣMj (Vj)′, one can get the following equivalentproblem

minΣXj

1

2‖ΣMj − ΣXj‖2F + µj‖ΣXj‖1,

where ΣMj ∈ Rn×Ij is a diagonal matrix, Uj ∈ Rn×n and Vj ∈ RIj×Ij are or-thogonal unitary matrices, and the coefficient matrix ΣXj may not be a diagonalmatrix on the basis Uj and Vj . Using the definition of ΣMj and the property ofthe Kronecker product, then a local SVD operator can be denoted as

Tj =

Ij∑i=1

vji ⊗ ((Uj)′Pij),

where vji is the i-th column of (Vj)′.

Liu et al. proposed the following denoising variational model

minf

1

2

J∑j=1

‖Tj(f − g)‖2 +

J∑j=1

µj‖Tjf‖p + µTV (f)

,

where p can be chosen as 0 or 1. Moreover, in order to enforce the sparsity repre-sentation, the basis functions Uj and Vj can be set to iteratively update during theimplementation. It has been shown that the quality of restorations can be greatlyimproved.

3. The Proposed Method. Inspired by matrix completion and the variationalframework for group and block based methods, we can construct a new inpaintingvariational model. The inpainting problem is different from the early mentionedGaussian noise removal since it is more ill-posed. In fact, the inpainting problemis related to the salt or pepper noise removal. To apply the Gaussian noise re-moval methods, we shall split the inpainting problem into a denoising and synthesissubproblems. In the first iteration of denoising subproblem, we utilize cubic in-terpolation to fill in the intensity at missing regions to get an initial guess for theinpainting result and regard this initial result as a noisy image, then we can useblock matching method to get a better denoised image. In block matching, wecan use l2 distance to measure the similarity between the reference patch and allthe patches in a search window and arrange the patches whose distance from thereference are smaller than a threshold value. In our current implementation codes,we set this threshold as 10σ2n, where σ2 is an estimated noise variance in our de-noising subproblem and n is the square of image patch size. Then a given numberof patches are selected as the most similar patches for the reference patch. Aftergetting the denoising results, the synthesis of the known and missed pixels can be

LOCAL BLOCK OPERATORS AND TV REGULARIZATION BASED IMAGE INPAINTING 5

well guided by the second subproblem. These two steps can be updated iterativelyand thus we can get some good inpainting results.

In the next, we shall give the proposed model. Let g ∈ RN be a vector version ofthe observed image and Pij ∈ Rn×N be a patch extractor. Entries of Pij are 0 and1 and there is one and only one 1 in each row. Suppose there are Ij blocks that areclose to the jth reference patch, we form a patch matrix as

Mj = [P1jg,P2jg, ...,PIjjg].

Then, the inpainting problem for each image block centered at j can be written as

minXj

1

2‖Wj (Mj −Xj)‖2F + µj‖Xj‖∗, (1)

where Wj is the matrix with element entries 1 for the indices of Pijg in Oc and0 in O. In order to get a well-defined inpainting model, we replace 0 with a verysmall positive number ε in O to get a better theoretical property. In the numericalimplementation, ε can be set to a very small value such as 10−20. Here Xj =[P1jf,P2jf, ...,PIjjf ], µj is a regularization parameter, and the symbol ” ” standsfor the multiplication of the corresponding elements of two matrices.

This convex problem does not have a closed-form solution due to existence ofWj . To solve this optimization problem efficiently, we can add a new auxiliary

variable Xj

to substitute the transformed coefficients Xj , and derive the followinginpainting model by penalty techniques

minXj ,X

j

1

2‖Wj (Mj −X

j)‖2F +

σ

2‖Xj −Xj‖2F + µj‖Xj‖∗, (2)

where σ is a penalty parameter and Xj

= [P1j f ,P2j f , ...,PIjj f ]. The SVD of Xj

is given by

Xj

= UjΣX

j (Vj)′,

where ΣX

j ∈ Rn×Ij is a diagonal matrix and Uj ∈ Rn×n,Vj ∈ RIj×Ij are orthog-

onal unitary matrices. If we choose Uj and Vj as basis, then we get

Xj = UjΣXj (Vj)′

for any Xj ∈ Rn×Ij .Note that the coefficient matrix ΣXj may not be a diagonal matrix. Then inspired

by problem (2), we can rewrite the model as

minΣXj ,ΣXj

1

2‖Wj (Mj −X

j)‖2F +

σj2‖Σ

Xj − ΣXj‖2F + µj‖ΣXj‖1. (3)

Next, we will give a global variational formulation for image inpainting based onthe above local image blocks restoring problem (3).

For an image f ∈ RN , let us review Xj

= [P1j f ,P2j f , ...,PIjj f ] and ΣX

j =

(Uj)′(Xj)Vj . Let (Vj)′ = [vj1, v

j2, · · · , v

jIj

], then we have

ΣX

j = (Uj)′[P1j f ,P2j f , ...,PIjj f ][vj1, vj2, · · · , v

jIj

]′

=

Ij∑i=1

(Uj)′Pij f(vji )′,

6 WEI WAN, HAIYANG HUANG AND JUN LIU

and the vectorized form as follows

vec(ΣX

j ) =

Ij∑i=1

(vji ⊗ ((Uj)′Pij))f .

according to vec(AXB) = (BT ⊗A)vec(X). Here vec is a operator defined at thebeginning of section 2.

Now, we define the local SVD operator as

Tj =

Ij∑i=1

vji ⊗ ((Uj)′Pij). (4)

Hence, we have

vec(ΣX

j ) = Tj f , vec(ΣXj ) = Tjf.

In addition, we can define Tjw as follows

vec(Wj Mj) = diag(vec(Wj))vec(Mj)

= (wj1,w

j2, · · · ,w

jIj

)

P1jgP2jg

...PIjjg

=

Ij∑i=1

wjiP

ijg

:= Tjwg.

Based on the above analysis, we propose the following image inpainting model:

minf,f

1

2

J∑j=1

‖Tjw(g − f)‖2 +

J∑j=1

µj‖Tjf‖p + µTV (f)

s.t f = f, (5)

where µ, µj are positive regularization parameters, p can be set as 0 or 1, and TVis the discrete isotropic TV operator with the following discrete expression

TV (f) =∥∥∥√|(I⊗D1)f |2 + |((D2)′ ⊗ I)f |2

∥∥∥1

= ‖∇f‖2,1 ,

where ‖A‖2,1 :=N∑i=1

√a2i1 + a2

i2,∀A = (aij)N×2 ∈ RN×2, D1,D2 are two 1D differ-

ence matrices with respect to x and y directions.Here Tj

w is an image patch based operator for the fidelity and Tj is a patchbased transform operator which can make Tjf to be sparse. Let us consider theproperties of Tj

w and Tj . The main properties are summarized in the followingpropositions.

Proposition 1. For Tj, we have (Tj)′Tj =∑Iji=1(Pij)′Pij and ker(

∑Jj=1(Tj)′Tj)

= 0, i.e.,∑Jj=1(Tj)′Tj is invertible.

Proposition 2. For Tjw, we have (Tj

w)′Tjw =

∑Iji=1(Pij)′(wj

i )′wj

iPij and∑J

j=1(Tjw)′Tj

w is not invertible when the inpaiting mask w is binary (0 for inpaint-

ing area O and 1 otherwise).

LOCAL BLOCK OPERATORS AND TV REGULARIZATION BASED IMAGE INPAINTING 7

Proposition 3. If the inpainting mask w takes values ε > 0 in O (inpainting area),

then we have ker(∑Jj=1(Tj

w)′Tjw) = 0.

These three propositions can be easily obtained by directly calculating the matrixtensor products. We omit them here.

4. Existence of a Minimizer. In this section we will prove the existence of aminimizer for the proposed model. Let us consider the energy functional for p = 1.In this case, the model is convex. Otherwise, for p < 1, it becomes non-convex andthis problem would be more difficult. Let us set

E(f) =1

2

J∑j=1

‖Tjw(g − f)‖2 +

J∑j=1

µj‖Tjf‖p + µTV (f).

It is easy to see that problem (5) is equivalent to the following problem

minf

E(f).

We can show the existence of a minimizer for the above minimization problemin the BV space by some standard discussions since it is a generalized ROF model.

Theorem 1. Suppose T jw,Tj are bounded linear operators, then there exists a

unique minimizer for the variational problem

minf∈BV (Ω)

E(f) (?)

if ker(J∑j=1

(Tjw)′Tj

w) = 0.

The proof of this theorem is a standard discussion which can be found in manyreferences such as [26]. For completeness of this paper, we list some keynotes inAppendix 9.1.

5. Algorithms. To solve model (5), we add some auxiliary variables α = [α1, α2, · · ·, αJ ] ∈ RnIj×J and αj ∈ RnIj , and we get a constrained minimization problem

minα,f,f

1

2

J∑j=1

‖Tjw(g − f)‖2 +

J∑j=1

µj‖αj‖p + µTV (f)

s.t. f = f, αj = Tjf. (6)

We use the standard augmented Lagrangian method to produce the following scheme:

(αk, fk, fk) = arg minα,f ,f

12

J∑j=1

‖Tjw(g − f)‖2 +

J∑j=1

µj‖αj‖p + µTV (f)

+η12

J∑j=1

‖αj −Tjf − λk−1j ‖2 + η2

2

J∑j=1

‖Tj f −Tjf − γk−1j ‖2

,

λkj = λk−1j + δ1(Tjfk − αkj ),

γkj = γk−1j + δ2(Tjfk −Tj fk).

(7)

8 WEI WAN, HAIYANG HUANG AND JUN LIU

By applying an alternating minimization algorithm, (7) becomes

fk = arg mind

12

J∑j=1

‖Tjw(g − f)‖2 + η2

2

J∑j=1

‖Tjfk−1 −Tj f − γk−1j ‖2

,

αk = arg minα

J∑j=1

µj‖αj‖p + η12

J∑j=1

‖αj −Tjfk−1 − λk−1j ‖2

,

fk = arg minf

µTV (f) + η1

2

J∑j=1

‖αkj −Tjf − λk−1j ‖2

+η22

J∑j=1

‖Tjf −Tj fk − γk−1j ‖2

,

λkj = λk−1j + δ1(Tjfk − αkj ),

γkj = γk−1j + δ2(Tjfk −Tj fk).

(8)

These subproblems can be solved as follows.1) f−subproblem: This has a closed-form solution

fk =

J∑j=1

(Tjw)′Tj

wg + η2

J∑j=1

(Tj)′Tjfk−1 − η2

J∑j=1

(Tj)′γk−1j

J∑j=1

(Tjw)′Tj

w + η2

J∑j=1

(Tj)′Tj

. (9)

2) α−subproblem: In the case of p = 1, it can be solved by soft thresholding.The optimal result is given by

αkj = S(Tjfk−1 + λk−1j ,

µjη1

), (10)

where S is a shrinkage operator and S(f, µ) = f|f | max|f | − µ, 0.

In the case of p = 0, it can be solved by hard thresholding. Similarly,

αkj = H(Tjfk−1 + λk−1j ,

√2µjη1

), (11)

where H(f, µ) =

0, |f | ≤ µf, |f | > µ

.

3) f− subproblem: This can be solved easily by the split Bregman iterations.We can rewrite it as a ROF model with a local SVD operator,

fk = arg minf

µTV (f) +η1 + η2

2

J∑j=1

‖Tjf − yj‖2 , (12)

where yj = η1η1+η2

(αkj − λk−1j ) + η2

η1+η2(Tj fk + γk−1

j ).

We summarize the proposed algorithm in algorithm 1:

Algorithm 1 (local SVD operator based image inpainting).Input: g and inpainting mask.Output: inpainting result f, f .Setting initial value f0 = g, γ0

j = λ0j = 0. Let k = 1, do

step 1. Block matching and basis updating: Find Ij for each block centered at jaccording to fk−1 with block matching method, and one can get a new Tj and Tj

w.step 2. Image inpainting: restoring fk by (9).step 3. Sparsity Regularization: computing αk according to (10) or (11).

LOCAL BLOCK OPERATORS AND TV REGULARIZATION BASED IMAGE INPAINTING 9

step 4. TV Regularization: updating fk by solving (12).step 5. Update Lagrangian multiplier: renewing λkj and γkj by formulation

λkj = λk−1j + δ1(Tjfk − αkj ),

γkj = γk−1j + δ2(Tjfk −Tj fk).

step 6. Stopping Criterion: if ‖fk−fk−1‖2‖fk−1‖2 < tolerance or the maximum iteration

number of this algorithm is reached, then the algorithm is terminated; else, k = k+1and go to step 1.

6. Convergence Analysis. In this section, we will prove the convergence of ouralgorithm for p = 1 if the local block operators Tj ,Tj

w are fixed. As for p = 0,the model would become non-convex and our numerical experiments show it is stillconvergent during practical computation.

Now set

ε(α, f , f) =1

2

J∑j=1

‖Tjw(g − f)‖2 +

J∑j=1

µj‖αj‖1 + µTV (f).

We have the following convergence results.

Theorem 2. Assume (α∗, f∗, f∗) is the minimizer of problem (6). Suppose thatδ1η1 = δ2η2, 0 < δ1 < 2, 0 < δ2 < 2, then the sequence (αk, fk, fk) generated by theiteration scheme (7) satisfies

limk→∞

ε(αk, fk, fk) = ε(α∗, f∗, f∗).

For the details of proof, please find it in appendix 9.2.

Theorem 3. Under the assumption of Theorem 2. If ker(J∑j=1

(Tjw)′Tj

w) = 0,

then the sequence (αk, fk, fk) generated by the iteration scheme (7) is convergentand lim

k→∞fk = f∗, lim

k→∞fk = f∗.

The details of proof also can be found in appendix 9.3.

7. Experiment result. In this section, we numerically demonstrate the supe-rior performance of our proposed inpainting model on natural inpainting problems.For comparison, we shall compare it with five representative and related inpaint-ing methods: cubic interpolation (matlab function, short for Cubic), a classicalTV inpainting model [2], a coherence transport method [27] (short for CTM, thesource codes are available at https://github.com/maerztom/inpaintBCT), the Beta-Bernoulli process factor analysis method [28] (short for BPFA), and a sparsity basedmethod IDI-BM3D in [22]. All the experiments are run under Windows 7 and MAT-LAB R2012b with Intel Core i5-5200U CPU@2.80GHz and 8GB memory.

7.1. Parameter selection. In general, we set penalty parameters η1 = 9, η2 =1×10−3, and time steps δ1 = 1×10−6, δ2 = 9×10−3, which satisfy the convergencecondition. The TV regularization parameter µ is set as 0.1. Besides, in order tospeed up the algorithm, firstly, we adopt the same method as IDI-BM3D to initializethe proposed model using the cubic interpolation results. In addition, we set the

10 WEI WAN, HAIYANG HUANG AND JUN LIU

patch size to 4, 6, 8 or 10 according to different inpainting areas. Without special

statements, the stopping criterion of our algorithm is ‖fk−fk−1‖2‖fk−1‖2 < 1× 10−6.

One may note that we choose greatly different values for penalty parametersη1, η2. Here both η1 and η2 are penalty parameters, but they play much differentroles in our inpainting problem. η1 is a penalty parameter to force Tjf to equal toαj , and this term together with sparsity constraint can make f smooth (denoising).

η2 is a penalty parameter which controls a fidelity term between f and f . In the f -subproblem (denoising), sparsity plays a dominant role and thus we take a relativelylarge value for η1. On the other hand, in the f subproblem (synthesis), fk can beregarded as the weighted average of g and fk−1, and η2 controls the weight betweenthem. We found numerically that when η2 is small, then the weight of fk−1 will besmall and the results will be good and stable.

7.2. Uniformly distributed missing pixels. In this section, we mainly consideran inpainting model with 80% of the pixels missing in a random manner. Thisproblem is equivalent to removing the Salt and Pepper noise with 80% noise level.Only 20% of pixels are available. We compare our proposed-`1 and -`0 methodscorresponding to p = 1 and p = 0 with Cubic, CTM, BPFA and IDI-BM3D. Thefirst row of Fig. 1 shows the test images: Monarch, Lena and Barbara. Thecontaminated images are shown in the second row. The missing pixels are indicatedin black. We observe that Cubic, CTM and BPFA keep image edges well, but theresults contain obvious artifacts near edges. The results of IDI-BM3D comparewell, but our proposed-`0 method gives better PSNR (see Table 1), especially forimages containing more texture information (see Fig. 2 for the local details ofBarbara). Moreover, it can be seen that the proposed-`0 method outperforms theproposed-`1 method, which confirms that a non-convex `0 model could generatemuch better restoration results than convex models. Our proposed-`0 method showsbetter performance than all the test methods in both visual result and quality indexPSNR.

Table 1. PSNR values of the different methods on filling randomlymissing pixels.

Image CTM Cubic BPFA IDI-BM3D Proposed-`1 Proposed-`0Monarch 23.01 24.18 24.49 26.63 25.15 27.25

Lena 27.21 27.40 28.32 29.63 28.50 29.98Barbara 25.65 26.24 27.08 28.62 27.59 29.69

7.3. Text Removal and Scratch Removal. In this section, we demonstrate theperformance of our proposed method on text and scratch removal. We mainly com-pare the proposed method with some related method such as Cubic interpolationmethod, TV[2] based PDE method, dictionaries learning based BPFA[28] (a param-eter free Bayesian algorithm, that learns dictionaries for inpainting from corrupteddata) and block matching and sparsity based IDI-BM3D[22] method. The test im-ages include Barbara, Hill and Baboon. The inpainting region is represented bywhite colour as shown in the second row of Fig. 3. The results of Cubic, BPFAand TV restore smooth areas well, however, they can not keep details on textureregions. We observe that IDI-BM3D and our proposed-`0 inpainting model havebetter visual quality, but our proposed-`0 method gives more texture information,

LOCAL BLOCK OPERATORS AND TV REGULARIZATION BASED IMAGE INPAINTING 11

such as the kerchief of Barbara and the windows of Hill (see more details in Fig.4), and gets higher PSNRs (see Table 2).

Table 2. PSNR values of different inpainting methods on text andscratch removal.

Image Cubic TV BPFA IDI-BM3D Proposed-`1 Proposed-`0Barbara 33.25 34.58 37.28 40.16 38.26 40.98

Hill 33.30 33.44 33.84 35.38 34.54 35.61Baboon 35.87 35.86 35.39 37.77 36.80 38.03

7.4. Numerical Evidence of Convergence. Relative errors ||fk−I||2||I||2 (I is the

groundtruth image) are shown in Fig. 5 and Fig. 6.In Fig.5, we show the relative error for the proposed-`1 model. The first col-

umn, i.e. Fig5 (a),(c),(e). illustrates the relative error curves of Monarch, Lenaand Barbara in section 7.2, respectively. In the second column, the relative errorcurves of Barbara, Hill and Baboon in section 7.3 are displayed in Fig.5 (b),(d),(f),respectively. Similar results are demonstrated in Fig.6 for the proposed-`0 method.

8. Conclusion. In this paper we propose a novel inpainting model based on blocksparsity and TV regularization. The local SVD operator is effective in promoting asparse representation. To solve the proposed model efficiently, we present a splittinginpainting algorithm. We mathematically prove the existence of a minimizer for theproposed inpainting model. Moreover, convergence analysis is given by a variationalinequality method. The numerical experiments demonstrate that our proposedinpainting model performs well compared to some state-of-the-art inpainting results.

However, our method is not very fast due to the block matching and a numberof SVD decompositions. Generally speaking, for an image with size 256× 256, ourproposed model takes about 16 seconds to perform one iteration with our unopti-mized matlab codes and CPU implementation. A possible research is to employ atraining method to produce a good basis to greatly accelerate the algorithm.

9. Appendix.

9.1. Proof of Theorem 1.

Proof. Let fn be a minimizing sequence for (?), i.e. a sequence such that

E(fn)→ inff∈BV

E(f).

Since E(f) is coercive, we get that fn is bounded in BV (Ω). Thus, there exists a

subsequence fn (which we still relabel by n) and f in BV (Ω) such that fnBV−w∗ f ,

fnL1(Ω)→ f .

12 WEI WAN, HAIYANG HUANG AND JUN LIU

Finally, by using the lower semicontinuity of the L2-norm and the lower semi-continuity of total variation, one can get

1

2

J∑j=1

‖Tjw(g − f)‖2 ≤ lim inf

n→∞

1

2

J∑j=1

‖Tjw(g − fn)‖2,

µTV (f) ≤ lim infn→∞

µTV (fn),

J∑j=1

µj‖Tjf‖1 = limn→∞

J∑j=1

µj‖Tjfn‖1,

that is,

E(f) ≤ lim infn→∞

E(fn).

Therefore, f is minimizer of E(f).The uniqueness can be easily obtained by the fact that the functional is strictly

convex if ker(J∑j=1

(Tjw)′Tj

w) = 0.

9.2. Proof of Theorem 2.

Proof. This proof is motivated by [29, 30]. Let

L(α, f , f ;β, ξ) =1

2

J∑j=1

‖Tjw(g − f)‖2 +

J∑j=1

µj‖αj‖1 + µTV (f)

+

J∑j=1

< βj , αj −Tjf > +η1

2

J∑j=1

‖αj −Tjf‖2

+

J∑j=1

< ξj ,Tj f −Tjf > +

η2

2

J∑j=1

‖Tj f −Tjf‖2.

Then L has at least one saddle point denoted by (α∗, f∗, f∗;β∗, ξ∗) which satisfies

L(α∗, f∗, f∗;β, ξ) ≤ L(α∗, f∗, f∗;β∗, ξ∗) ≤ L(α, f , f ;β∗, ξ∗),∀α, f , f,β, ξ. (13)

Let βk = −η1λk, ξk = −η2γ

k, i.e. βkj = −η1λkj , ξ

kj = −η2γ

kj , then the iteration

scheme (7) becomes(αk, fk, fk) = arg min

α,f ,f

L(α, f , f ;βk−1, ξk−1),

βkj = βk−1j + δ1η1(αkj −Tjfk),

ξkj = ξk−1j + δ2η2(Tj fk −Tjfk).

(14)

Denote αjk = αkj −α∗j , dk = dk − d∗, fk = fk − f∗, βkj = βkj − β∗j , ξkj = ξkj − ξ∗j . The

first inequality of (13) implies α∗j = Tjf∗,Tj f∗ = Tjf∗, and we get

β∗j = β∗j + δ1η1(α∗j −Tjf∗),

ξ∗j = ξ∗j + δ2η2(Tj f∗ −Tjf∗).

LOCAL BLOCK OPERATORS AND TV REGULARIZATION BASED IMAGE INPAINTING 13

Following the second and third equations in (14), we get

βkj = βk−1j + δ1η1(αkj −Tj fk),

ξkj = ξk−1j + δ2η2(Tj ˆfk −Tj fk).

Taking norms and squaring on both sides of these two equations and summationwith respect to j from 1 to J , we have

J∑j=1

‖βkj ‖2 −J∑j=1

‖βk−1j ‖2

=δ21η

21

J∑j=1

‖αkj −Tj fk‖2 + 2δ1η1

J∑j=1

< βk−1j , αkj −Tj fk >, (15)

and

J∑j=1

‖ξkj ‖2 −J∑j=1

‖ξk−1j ‖2

=δ22η

22

J∑j=1

‖Tj ˆfk −Tj fk‖2 + 2δ2η2

J∑j=1

< ξk−1j ,Tj ˆfk −Tj fk > . (16)

By the second inequality in (13), (α∗, f∗, f∗) is a minimizer of L(·, ·, ·;β∗, ξ∗).According to the well known variational inequality, we have

J∑j=1

µj‖αj‖1 −J∑j=1

µj‖α∗j‖1 + η1

J∑j=1

< α∗j −Tjf∗, αj − α∗j > +

J∑j=1

< β∗j , αj − α∗j >

≥ 0, (17)

µTV (f)− µTV (f∗) + η1

J∑j=1

< Tjf∗ − α∗j ,Tj(f − f∗) > −J∑j=1

< β∗j ,Tj(f − f∗) >

+η2

J∑j=1

< Tjf∗ −Tj f∗,Tj(f − f∗) > −J∑j=1

< ξ∗j ,Tj(f − f∗) >≥ 0, (18)

J∑j=1

< Tjw(f∗ − g),Tj

w(f − f∗) > +

J∑j=1

< ξ∗j ,Tj(f − f∗) >

+η2

J∑j=1

< Tj f∗ −Tjf∗,Tj(f − f∗) >≥ 0. (19)

14 WEI WAN, HAIYANG HUANG AND JUN LIU

Similarly, the first equation in (14) is characterized by the variational inequality

J∑j=1

µj‖αj‖1 −J∑j=1

µj‖αkj ‖1 + η1

J∑j=1

< αkj −Tjfk, αj − αkj >

+

J∑j=1

< βk−1j , αj − αkj >≥ 0, (20)

µTV (f)− µTV (fk) + η1

J∑j=1

< Tjfk − αkj ,Tj(f − fk) >

−J∑j=1

< βk−1j ,Tj(f − fk) > +η2

J∑j=1

< Tjfk −Tj fk,Tj(f − fk) >

−J∑j=1

< ξk−1j ,Tj(f − fk) >≥ 0, (21)

J∑j=1

< Tjw(fk − g),Tj

w(f − fk) > +

J∑j=1

< ξk−1j ,Tj(f − fk) >

+η2

J∑j=1

< Tj fk −Tjfk,Tj(f − fk) >≥ 0. (22)

Taking αj = αkj in (17), f = fk in (18), f = fk in (19), αj = α∗j in (20), f = f∗ in

(21), f = f∗ in (22) and summing, one can get

J∑j=1

< βk−1j , αkj −Tj fk > +

J∑j=1

< ξk−1j ,Tj ˆfk −Tj fk >

≤ −η1

J∑j=1

‖αkj −Tj fk‖2 − η2

J∑j=1

‖Tj ˆfk −Tj fk‖2(23)

By (15), (16) and (23), we have

J∑j=1

‖βkj ‖2 +J∑j=1

‖ξkj ‖2 −

(J∑j=1

‖βk−1j ‖2 +

J∑j=1

‖ξk−1j ‖2

)≤ δ1(δ1 − 2)η2

1

J∑j=1

‖αkj −Tj fk‖2 + δ2(δ2 − 2)η22

J∑j=1

‖Tj ˆfk −Tj fk‖2

≤ 0

(24)

since δ1η1 = δ2η2, 0 < δ1 < 2, 0 < δ2 < 2. The above inequality implies thatJ∑j=1

‖βkj ‖2 +J∑j=1

‖ξkj ‖2

is monotone non-increasing. Thus it has a limit since it

has a lower bound 0. Taking limits for (24) implies that limk→∞

‖αkj −Tjfk‖2 = 0,

limk→∞

(αkj −Tjfk) = 0,(25)

LOCAL BLOCK OPERATORS AND TV REGULARIZATION BASED IMAGE INPAINTING 15

and limk→∞

‖Tj fk −Tjfk‖2 = 0,

limk→∞

(Tj fk −Tjfk) = 0.(26)

Next, we will show that limk→∞

ε(α, f , f) = ε(α∗, f∗, f∗). From the second in-

equality of (13), one can obtain

ε(α∗, f∗, f∗) ≤ε(αk, fk, fk) +

J∑j=1

< β∗j , αkj −Tjfk > +

η1

2

J∑j=1

‖αkj −Tjfk‖2

+

J∑j=1

< ξ∗j ,Tj fk −Tjfk > +

η2

2

J∑j=1

‖Tj fk −Tjfk‖2,

and thus

ε(α∗, f∗, f∗) ≤ lim infk→∞

ε(αk, fk, fk)

by taking lim inf on both sides of the above inequality.On the other hand, setting αj = α∗j in (20), f = f∗ in (21), f = f∗ in (22) and

summing and taking lim sup, we can get

ε(α∗, f∗, f∗) ≥ lim supk→∞

ε(αk, fk, fk)

Therefore, we have

limk→∞

ε(αk, fk, fk) = ε(α∗, f∗, f∗). (27)

9.3. Proof of Theorem 3.

Proof. According to the second inequality of (13) with the first order convex opti-mization conditions, we have

−β∗j ∈ µj∂‖α∗j‖1, (28)

J∑j=1

(Tj)′β∗j +

J∑j=1

(Tj)′ξ∗j ∈ ∂µTV (f∗), (29)

J∑j=1

(Tjw)′Tj

w(f∗ − g) +

J∑j=1

(Tj)′ξ∗j = 0. (30)

16 WEI WAN, HAIYANG HUANG AND JUN LIU

Thus,

ε(αk, fk, fk) +J∑j=1

< β∗j , αkj −Tjfk > +

J∑j=1

< ξ∗j ,Tj fk −Tjfk >

≥ 12

J∑j=1

‖Tjw(fk − g)‖2 +

J∑j=1

µj‖α∗j‖1 + µTV (f∗) +J∑j=1

< −β∗j , αkj − α∗j >

+J∑j=1

< β∗j + ξ∗j ,Tj(fk − f∗) > +

J∑j=1

< β∗j , αkj −Tjfk >

+J∑j=1

< ξ∗j ,Tj fk −Tjfk >

= ε(α∗, f∗, f∗) + 12

J∑j=1

‖Tjw(fk − g)‖2 − 1

2

J∑j=1

‖Tjw(f∗ − g)‖2

+J∑j=1

< ξ∗j ,Tj fk −Tjf∗ >

= ε(α∗, f∗, f∗) +J∑j=1

<Tj

w fk+Tj

w f∗

2 −Tjwg,T

jw(fk − f∗) >

+J∑j=1

< Tjw(g − f∗),Tj

w(fk − f∗) >

= ε(α∗, f∗, f∗) + 12‖T

jw(fk − f∗)‖2.

(31)Taking limits on both sides of the above inequality, together with (25), (26) and(27), we have

limk→∞

Tjw(fk − f∗) = 0.

By using the continuity of Tjw and ker(

J∑j=1

(Tjw)′Tj

w) = 0, we have

limk→∞

fk = f∗.

Then, by the continuity of Tj and Tj f∗ = Tjf∗, we have

limk→∞

Tjfk = Tjf∗,

according to the second equation in (26). Finally, from the continuity of Tj and

ker(J∑j=1

(Tj)′T) = 0, we have

limk→∞

fk = f∗.

Acknowledgement. Haiyang Huang and Jun Liu were supported by The NationalKey Research and Development Program of China (2017YFA0604903). Jun Liu wasalso supported by the National Natural Science Foundation of China (Grant No.11871035).

REFERENCES

[1] M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester,Image inpainting, Siggraph, 4 (2005),417–424.

[2] T. F. Chan and J. Shen, Mathematical models for local nontexture inpaintings, SIAM Journalon Applied Mathematics, 62 (2001), 1019–1043.

LOCAL BLOCK OPERATORS AND TV REGULARIZATION BASED IMAGE INPAINTING 17

[3] L. I. Rudin, S. Osher and E. Fatemi, Nonlinear total variation based noise removal algorithms,Physica D Nonlinear Phenomena, 60 (1992), 259–268.

[4] T. F. Chan and J. Shen, Nontexture inpainting by curvature-driven diffusions, Journal of

Visual Communication & Image Representation, 12 (2001), 436–449.[5] T. F. Chan, S. H. Kang and J. Shen, Euler’s elastica and curvature based inpainting, SIAM

Journal on Applied Mathematics, 63 (2002), 564–592.[6] S. Esedoglu and J. Shen, Digital inpainting based on the mumford-shah-euler image model,

European Journal of Applied Mathematics, 13 (2002), 353–370.

[7] A. L. Bertozzi, S. Esedoglu and A. Gillette, Inpainting of binary images using the cahn-hilliardequation, IEEE Transactions on Image Processing, 16 (2007), 285–291.

[8] M. Burger, L. He and C. B. Nlieb, Cahn-hilliard inpainting and a generalization for grayvalue

images, SIAM Journal on Imaging Sciences, 2 (2009), 1129–1167.[9] X. C. Tai, S. Osher and R. Holm, Image inpainting using a tv-stokes equation, Springer,

Berlin, Heidelberg, 2007.

[10] A. Criminisi, P. Perez and K. Toyama, Region filling and object removal by exemplar-basedimage inpainting, IEEE Transactions on Image Processing, 13 (2004), 1200–1212.

[11] J. F. Aujol, S. Ladjal and S. Masnou, Exemplar-based inpainting from a variational point of

view, SIAM Journal on Mathematical Analysis, 42 (2010), 1246–1285.[12] F. Cao, Y. Gousseau, S. Masnou and P. Prez, Geometrically guided exemplar-based inpaint-

ing, SIAM Journal on Imaging Sciences, 4 (2009), 1143–1179.[13] L. Demanet, B. Song and T. Chan, Image inpainting by correspondence maps: a deterministic

approach, Variational Level Set Methods, Prod. Of Workshop in Int”l Conf. Image Proc.,

(2003), 1100.[14] N. Kawai, T. Sato and N. Yokoya, Image inpainting cosidiering brightness change and spatial

locality of textures and its evaluation, Pacific Rim Symposium on Advances in Image and

Video Technology, Springer, Berlin, Heidelberg, 5414 (2009), 271–282.[15] Y. Wexler, E. Shechtman and M. Irani, Space-time video completion, Proceedings of the 2004

IEEE Computer Society Conference on Computer Vision and Pattern Recognition, (2004).

[16] P. Arias, G. Facciolo, V. Caselles and G. Sapiro, A variational framework for exemplar-basedimage inpainting, International Journal of Computer Vision, 93 (2011), 319–347.

[17] M. Elad, J. L. Starck, P. Querre and D. L. Donoho, Simultaneous cartoon and texture im-

age inpainting using morphological component analysis (mca), Applied and ComputationalHarmonic Analysis, 19 (2005), 340–358.

[18] O. G. Guleryuz, Nonlinear approximation based image recovery using adaptive sparse recon-structions and iterated denoising-part II: adaptive algorithms, IEEE Transactions on Image

Processing, 15 (2006), 555–571.

[19] M. J. Fadili, J. L. Starck and F. Murtagh, Inpainting and zooming using sparse representa-tions, The Computer Journal, 52 (2009), 64–791.

[20] Z. Xu and J. Sun, Image inpainting by patch propagation using patch sparsity, IEEE Trans-actions on Image Processing, 19 (2010), 1153–1165.

[21] W. Li, L. Zhao, Z. Lin, D. Xu and D. Lu, Non-local image inpainting using low-rank matrix

completion, Computer Graphics Forum, 34 (2015), 111–122.

[22] F. Li and T. Zeng, A universal variational framework for sparsity-based image inpainting,IEEE Transactions on Image Processing, 23 (2014), 4242–4254.

[23] J. Liu and S. Osher, Block matching local svd operator based sparsity and tv regularizationfor image denoising, Journal of Scientific Computing.

[24] E. J. Candes and B. Recht, Exact matrix completion via convex optimization, Foundations

of Computational Mathematics, 9 (2009), 717–772.

[25] J. -F. Cai, E. J. Candes and Z. Shen, A singular value thresholding algorithm for matrixcompletion, SIAM Journal on Optimizaition, 20 (2010), 1956–1982.

[26] G. Aubert and P. Kornprobst, Mathematical problems in image processing: partial differentialequations and the calculus of variations, Springer, 2006.

[27] F. Bornemann and T. Marz, Fast image inpainting based on coherence transport, Journal of

Mathematical Imaging and Vision, 28 (2007), 259–278.[28] M. Zhou, H. Chen, J. Paisley, L. Ren, L. Li, Z. Xing, D. Dunson, G. Sapiro and L. Carin,

Nonparametric Bayesian Dictionary Learning for Analysis of Noisy and Incomplete Images,

IEEE Transactions on Image Processing, 21 (2012), 130–144.[29] R. Glowinski, Augmented lagrangian and operator splitting methods in nonlinear mechanics,

SIAM, Philadelphia, 1989.

18 WEI WAN, HAIYANG HUANG AND JUN LIU

[30] C. Wu and X. -C. Tai, Augmented lagrangian method, dual Methods, and split BregmanIteration for rof, vectorial tv, and high order models, SIAM Journal on Imaging Sciences, 3

(2012), 300–339.

E-mail address: weiwan@mail.bnu.edu.cn

E-mail address: hhywsg@bnu.edu.cn

E-mail address: jliu@bnu.edu.cn

LOCAL BLOCK OPERATORS AND TV REGULARIZATION BASED IMAGE INPAINTING 19

Clean:

Mask:

CTM[27]:

Cubic:

BPFA[28]:

IDI-BM3D[22]:

Proposed-`1:

Proposed-`0:

Figure 1. Filling in the missing pixels by different inpainting methods.

20 WEI WAN, HAIYANG HUANG AND JUN LIU

(a) Clean (b) Mask (c) CTM[27] (d) Cubic

(e) BPFA[28] (f) IDI-BM3D[22] (g) Proposed-`1 (h) Proposed-`0

Figure 2. Comparison of details between different inpainting methods.

LOCAL BLOCK OPERATORS AND TV REGULARIZATION BASED IMAGE INPAINTING 21

Original:

Mask:

Cubic:

TV[2]:

BPFA[28]:

IDI-BM3D[22]:

Proposed-`1:

Proposed-`0:

Figure 3. Scratch and text removal by different inpainting methods.

22 WEI WAN, HAIYANG HUANG AND JUN LIU

(a) Clean (b) Mask (c) Cubic (d) TV[2]

(e) BPFA[28] (f) IDI-BM3D[22] (g) Proposed-`1 (h) Proposed-`0

(i) Clean (j) Mask (k) Cubic (l) TV[2]

(m) BPFA[28] (n) IDI-BM3D[22] (o) Proposed-`1 (p) Proposed-`0

Figure 4. Comparison of details between different inpainting methods.

LOCAL BLOCK OPERATORS AND TV REGULARIZATION BASED IMAGE INPAINTING 23

0 5 10 15 20 25 30

iteration number

0.112

0.114

0.116

0.118

0.12

0.122

0.124

0.126

rela

tive

erro

r

(a)

0 5 10 15 20 25 30 35 40 45 50

iteration number

0.024

0.026

0.028

0.03

0.032

0.034

0.036

0.038

0.04

0.042

rela

tive

erro

r

(b)

0 5 10 15 20 25 30

iteration number

0.072

0.073

0.074

0.075

0.076

0.077

0.078

0.079

0.08

0.081

rela

tive

erro

r

(c)

0 5 10 15 20 25

iteration number

0.039

0.04

0.041

0.042

0.043

0.044

0.045

rela

tive

erro

r

(d)

0 5 10 15 20 25 30

iteration number

0.082

0.084

0.086

0.088

0.09

0.092

0.094

0.096

rela

tive

erro

r

(e)

0 2 4 6 8 10 12 14 16 18 20

iteration number

0.028

0.0285

0.029

0.0295

0.03

0.0305

0.031

rela

tive

erro

r

(f)

Figure 5. The relative error curves as functions of the iterationnumber on our experiments for the proposed-`1 method.

24 WEI WAN, HAIYANG HUANG AND JUN LIU

0 10 20 30 40 50 60 70

iteration number

0.085

0.09

0.095

0.1

0.105

0.11

0.115

0.12

0.125

rela

tive

erro

r

(a)

0 5 10 15 20 25 30 35 40

iteration number

0.015

0.02

0.025

0.03

0.035

0.04

rela

tive

erro

r

(b)

0 10 20 30 40 50 60 70

iteration number

0.06

0.062

0.064

0.066

0.068

0.07

0.072

0.074

0.076

0.078

rela

tive

erro

r

(c)

0 5 10 15 20 25 30 35

iteration number

0.034

0.035

0.036

0.037

0.038

0.039

0.04

0.041

0.042

rela

tive

erro

r

(d)

0 10 20 30 40 50 60 70

iteration number

0.06

0.065

0.07

0.075

0.08

0.085

0.09

0.095

rela

tive

erro

r

(e)

0 5 10 15 20 25 30

iteration number

0.024

0.0245

0.025

0.0255

0.026

0.0265

0.027

0.0275

0.028

0.0285

0.029

rela

tive

erro

r

(f)

Figure 6. The relative error curves as functions of the iterationnumber on our experiments for the proposed-`0 method.