Face Hallucination via Similarity Constraints

Hongliang Li, Senior Member, IEEE,Linfeng Xu, Member, IEEE, and Guanghui LiuFace Hallucination via Similarity ConstraintsOutlineIntroductionProposed MethodFramework of the Proposed MethodSimilarity Constraints ComputationLR-LR Similarity ConstraintLR-HR Similarity ConstraintHR Smoothness ConstraintSpatial SimilarityExperimentsConclusion

IntroductionIn many cases the face images captured by live cameras are often of low resolutions due to the environment or equipment limitations.

In order to generate a high resolution face image effectively, a lot of methods have been presented in the last decade.IntroductionIn this letter, a new face hallucination approach based on similarity constraints is proposed to hallucinate a high resolution face image from an input low-resolution face image.

The proposed method formulates the face hallucination as a local linear filtering progress based on training LR-HR face image pairs.Proposed MethodA. Framework of the Proposed MethodLet ZL and ZH denote the low resolution and high resolution training face images, respectively, where ZL is downsampled from ZH by an integer factor.

Assume IL be an input low-resolution face image, while IH represents its high-resolution face image to be hallucinated.Framework of the Proposed MethodFig.1. Framework of our face hallucination approach.

Framework of the Proposed MethodThree stages are involved in this work. We first search a LR-HR face database for all patches that are stored beforehand.The similarities between the input patch and each pair of LR-HR face patches are measured under different constraint conditions.Finally, we hallucinate a high-resolution image by inferring the lost details within the input low-resolution image.Framework of the Proposed MethodAssume each image has been divided into N overlapping patches with identical spacing.

Let denote the set of pairs of training LR-HR patches, i and j are patch indices.For an input LR face patch IL(i), our goal is to utilize the training patch pairs to recover the missing high frequency details in the hallucinated patch IH(i).

Framework of the Proposed Method and are the mean values of the input LR patch IL(i) and the HR patch IH(j), respectively.

The second term (ZH(j) - )is to perform the normalization by subtracting the mean from the HR patch.

is defined as a filter kernel that depends on IL, ZL, and ZH.

Framework of the Proposed MethodCij is to ensure that the sum of is equal to one.Here represents the neighborhood of patch i.It is noticed that there are four terms defined in the kernel W, which perform the similarity constraints, i.e., LR-LR similarity , LR-HRsimilarity , smoothness constraint and spatial similarity .

Proposed MethodB. Similarity Constraint Computation1) LR-LR Similarity ConstraintGiven a LR training face image, we have stored its corresponding HR training image beforehand.

It means that all the missing high-frequency details in the LR image can be accurately estimated from its HR one.Similarity Constraints ComputationThe control parameter 1 adjusts the range of intensity similarity, which means that smaller allows large changes between the two LR patches.

A straightforward computation of S is their Euclidean distance, which may result in poor performance in the case of the significant lighting variation or noise corruption.

Similarity Constraints ComputationThe distance can be expressed as

where the operation denotes the l-norm distance.

Similarity Constraints Computation2) LR-HR Similarity ConstraintThe LR-HR constraint is designed to measure the similarity between an input photo patch IL(i) and a HR patch ZH(j).

Since HR patches usually contain a great of high frequency contents that are missed for the LR patches, it is difficult to compare their similarity directly based on their difference.

Similarity Constraints ComputationWe design a new descriptor called local appearance similarity (LAS) descriptor to measure the similarity between LR and HR patches.

This descriptor is generated based on patch pairs similarity within a local region, which is illustrated in Fig. 2.Similarity Constraints ComputationFig. 2. Illustration of computation.

Similarity Constraints ComputationGiven a LR patch IL(i) and a HR training patch ZH(j), i.e., the patches marked with solid yellow line, the LR-HR constraint is defined to measure the similarity between them.

The final LAS descriptor for a patch is the concatenation of the matrix elements in terms of the raster scan order.

Similarity Constraints ComputationLet and denote the 1 x d dimensional LAS descriptors for patches IL(i) and ZH(j).

Similarity Constraints ComputationThe parameter 2 and s adjust the descriptors similarity, and denote the neighborhoods of patches IL(i) and IH(j), respectively.

In our work, we set unless otherwise specified. The final LAS descriptor will be a 25-dimensional vector.

Similarity Constraints Computation3) HR Smoothness ConstraintWe tend to design a constraint to answer if those similar patches have good compatibilities with the neighboring ones.

We call as a smoothness term, which aims to impose the smoothness constraint between neighboring hallucinated patches.

Similarity Constraints ComputationThe HR smoothness constraint can be formulated as

where t and l denote the top and left overlapping regions for pairs of patches ZH(j) IH(it) and ZH(j) IH(il), respectively.Here, 3 is used to control the range of smoothness variation.

Similarity Constraints Computation4) Spatial SimilarityIt is reasonable to assign small constraints for those patches that are far from the hallucinating patch IH(i).

We define a new constraint to compute the similarity between ZH(j) and IL(i) based on the spatial distance.

Similarity Constraints ComputationThe parameter 4 adjusts the spatial similarity.D(i,j) is a spatial window function defined by the set of the neighborhood of ti (i.e., ).

ExperimentsGiven an input LR face image, we divide it into a number of overlapping patches with the size of 4 4. The overlapping pixel is set to 3, which corresponds to 12 pixels in the HR face image.

We employ laplacian cost function, i.e., l = 1, to compute the similarity constraints.ExperimentsWe first perform the evaluation on a large number of face images taken from FERET face database. About 1200 images of 873 persons were selected as training images and 300 images of 227 persons for testing.

We compare our method with the state-of-the-art methods, which include the general bicubic interpolation, Liu et al. [3], Wang et al. [4], Ma et al. [7], and Zhang et al. [11].ExperimentsFig. 3. (a) Some examples of face hallucination results. (b) Locally enlarged results for the last two face images.

ExperimentsIn addition, we also evaluate our proposed method on some face images taken from the CMU+MIT face database.

Fig. 4. Experimental results on some LR face images.

ExperimentsWe also performthe objective evaluation on our method. Two quantitative parameters are used to measure the similarity between the original HR face image and the hallucinated one, namely peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM).

The default parameters in SSIM are set to Kssim=[0.05 0.05](constant term), window=8(local window size), and Lssim=100(dynamic range of the pixel values), which were recommended by the authors.ExperimentsHowever, as discussed in [11] and [12], we also found a similar phenomenon that PSNR and SSIM are not always consistent with the human perceptual quality.

ConclusionInspired by our guided synthesis framework, this method provides an effective way to infer the missing high frequency details within the input LR face image based on the similarity constraints.Given the training set, four constraint functions are designed to learn the lost information from the most similar training examples.Experimental evaluation demonstrates the good performance of the proposed method on the face hallucination task.Thank you for your listening