Real-Time Exemplar-Based Face Sketch Synthesis
Pipeline illustration
Note: containing animations
Yibing Song1 Linchao Bao1 Qingxiong Yang1 Ming-Hsuan Yang2
1City University of Hong Kong2University of California at Merced
Our assumption: a database containing photo-sketch pairs
1. photo database 2. sketch database
Aligned
Coarse Sketch GenerationStep 1: KNN search
p
Test photo patch Test photo
Training photo dataset
𝑻 𝒑𝑻 𝒑
𝑻 𝒑
Matched photo patch
Relative position
Similarly
Matched photo patch
Relative position
∆𝒑𝑲[ ]∆𝒑 =
Test photo patch
Matched photo patch
Matched photo patch
Matched photo patch
𝒙𝒑𝟏 ∙ +𝒙𝒑
𝟐 ∙ +𝒙𝒑𝑲 ∙ ¿
2. Compute linear mapping function defined by
Coarse Sketch GenerationStep 2: Linear Estimation from Photos
Matched sketch pixel
p
Matched sketch pixel
Test photo
𝑺𝑷 +∆𝒑𝟏
❑
𝑺𝑷 +∆𝒑𝟐
❑
Matched sketch pixel𝑺𝑷 +∆𝒑𝑲
❑
𝒙𝒑𝟏 ∙ +𝒙𝒑
𝟐 ∙ +𝒙𝒑𝑲 ∙ ¿
Estimation on pixel p
Repeat for every pixel
Coarse sketch
Coarse Sketch GenerationStep 3: Apply Linear Mapping to Sketches
𝑬𝒑
Because: coarse sketch image is not natural. is not a good similarity measurement between p and r.
Denoising: State-of-the-art Image Denoising Algorithms
Coarse sketch
Nonlocal Means (NLM)
p
r
𝑆𝑝𝑁𝐿𝑀=¿ 𝐸𝑟
𝑤(𝑝 ,𝑟 )+⋯
For all pixels in the neighbor of p:
Little improvement
After NLM
q
𝐸𝑞𝑤(𝑝 ,𝑞)+¿
[NLM] A. Buades, B. Coll and J.-M. Morel, A non-local algorithm for image denoising, CVPR 2005.
Motivation – BM3D
BM3D groups correlated patches in the noisy image to create multiple estimations.
Our idea for sketch denoising: group highly similar sketch estimations.
How BM3D works
[BM3D] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3D transform-domain collaborative filtering,” IEEE Trans. Image Process., vol. 16, no. 8, pp. 2080-2095, August 2007.
𝑤(𝑝 ,𝑞) ∙
Proposed Spatial Sketch Denoising Algorithm (SSD)
Test photo
q
𝑺𝒒+∆𝒒𝟏
❑
p
Matched sketch
𝑺𝒑+∆𝒒𝟏
❑
Similarly ,
𝑺𝒑+∆𝒒𝟐
❑
𝑺𝒒+∆𝒒𝟐
❑
,
𝑺𝒒+∆𝒒𝑲
❑
𝑺𝒑+∆𝒒𝑲
❑𝒙𝒒𝟏 ∙ +𝒙𝒒
𝟐 ∙ +𝒙𝒒𝑲 ∙ ¿
𝑬𝒑𝒒
p
Estimations from pixels in local region
r 𝑬𝒑𝒓
Averaging estimations to generate output sketch value.
Nonlocal Means (NLM):
𝑆𝑝𝑁𝐿𝑀=¿ 𝐸𝑞 +⋯𝐸𝑟𝑤(𝑝 ,𝑟 )∙+¿
Proposed SSD:
𝑆𝑝𝑆𝑆𝐷=¿ +⋯+¿1 ∙𝑬𝒑
𝒒 1 ∙𝑬𝒑𝒓
p
Proposed SSD is robust to
Input 5x5 local region
11x11 local region
17x17 local region
23x23 local region
Note: When is sufficient large (i.e., >100), the proposed SSD can effectivelysuppress noise while preserving facial details like the tiny eye reflections (see close-ups).
Robustness to the region size - the only parameter involved