Post on 14-Apr-2018
transcript
7/30/2019 19 Texture
1/50
Texture Synthesis
15-463: Computational PhotographyAlexei Efros, CMU, Fall 2005
Darren Green (www.darrensworld.com)
7/30/2019 19 Texture
2/50
Texture
Texture depicts spatially repeating patterns
Many natural phenomena are textures
radishes rocks yogurt
7/30/2019 19 Texture
3/50
Texture Synthesis
Goal of Texture Synthesis: create new samples ofa given texture
Many applications: virtual environments, hole-filling, texturing surfaces
7/30/2019 19 Texture
4/50
The Challenge
Need to model the whole
spectrum: from repeated tostochastic texture
repeated
stochastic
Both?
7/30/2019 19 Texture
5/50
Efros & Leung Algorithm
Assuming Markov property, compute P(p|N(p))
Building explicit probability tables infeasible
pp
Synthesizing a pixel
non-parametricsampling
Input image
Instead, we search the input image for all similarneighborhoods thats our pdf forp
To sample from this pdf, just pick one match atrandom
7/30/2019 19 Texture
6/50
Some Details
Growing is in onion skin order
Within each layer, pixels with most neighbors are
synthesized first If no close match can be found, the pixel is not
synthesized until the end Using Gaussian-weightedSSD is very important
to make sure the new pixel agrees with its closest
neighbors
Approximates reduction to a smaller neighborhood
window if data is too sparse
7/30/2019 19 Texture
7/50
Neighborhood Window
input
7/30/2019 19 Texture
8/50
Varying Window Size
Increasing window size
7/30/2019 19 Texture
9/50
Synthesis Results
french canvas rafia weave
7/30/2019 19 Texture
10/50
More Resultswhite bread brick wall
7/30/2019 19 Texture
11/50
Homage to Shannon
7/30/2019 19 Texture
12/50
Hole Filling
7/30/2019 19 Texture
13/50
Extrapolation
7/30/2019 19 Texture
14/50
Summary
The Efros & Leung algorithm
Very simple
Surprisingly good results
Synthesis is easier than analysis!
but very slow
7/30/2019 19 Texture
15/50
pp
Observation:
Image Quilting [Efros & Freeman]
neighbor pixels are highly correlated
Input image
non-parametricsampling
BB
Idea:Idea: unit of synthesis = blockunit of synthesis = block Exactly the same but now we want P(B|N(B))
Much faster: synthesize all pixels in a block at once
Not the same as multi-scale!
Synthesizing a block
7/30/2019 19 Texture
16/50
Input texture
B1 B2
Random placement
of blocks
block
B1 B2
Neighboring blocks
constrained by overlap
B1 B2
Minimal error
boundary cut
7/30/2019 19 Texture
17/50
min. error boundary
Minimal error boundary
overlapping blocks vertical boundary
__ ==22
overlap error
7/30/2019 19 Texture
18/50
Our Philosophy
The Corrupt Professors Algorithm:
Plagiarize as much of the source image as you can Then try to cover up the evidence
Rationale: Texture blocks are by definition correct samples of
texture so problem only connecting them together
7/30/2019 19 Texture
19/50
7/30/2019 19 Texture
20/50
7/30/2019 19 Texture
21/50
7/30/2019 19 Texture
22/50
7/30/2019 19 Texture
23/50
7/30/2019 19 Texture
24/50
7/30/2019 19 Texture
25/50
Failures(ChernobylHarvest)
7/30/2019 19 Texture
26/50
input image
Portilla & Simoncelli
Wei & Levoy Our algorithm
Xu, Guo & Shum
7/30/2019 19 Texture
27/50
Portilla & Simoncelli
Wei & Levoy Our algorithm
Xu, Guo & Shum
input image
7/30/2019 19 Texture
28/50
Portilla & Simoncelli
Wei & Levoy Our algorithm
input image
Xu, Guo & Shum
P liti l T t S th i !
7/30/2019 19 Texture
29/50
Political Texture Synthesis!
MS Di it l I P (DEMO)
7/30/2019 19 Texture
30/50
MS Digital Image Pro (DEMO)
7/30/2019 19 Texture
31/50
Fill Order
7/30/2019 19 Texture
32/50
Fill Order
In what order should we fill the pixels? choose pixels that have more neighbors filled
choose pixels that are continuations of lines/curves/edges
Criminisi, Perez, and Toyama. Object Removal by Exemplar-based Inpainting,Proc. CVPR, 2003.
Exemplar based Inpainting demo
http://www.research.microsoft.com/~antcrim/papers/Criminisi_cvpr2003.pdfhttp://www.research.microsoft.com/~antcrim/papers/Criminisi_cvpr2003.pdf7/30/2019 19 Texture
33/50
Exemplar-based Inpainting demo
http://research.microsoft.com/vision/cambridge/i3l/patchworks.htm
Application: Texture Transfer
http://research.microsoft.com/vision/cambridge/i3l/patchworks.htmhttp://research.microsoft.com/vision/cambridge/i3l/patchworks.htm7/30/2019 19 Texture
34/50
++ ==
Application: Texture Transfer
Try to explain one object with bits andpieces of another object:
Texture Transfer
7/30/2019 19 Texture
35/50
Texture Transfer
Constraint
Texture sample
Texture Transfer
7/30/2019 19 Texture
36/50
Take the texture from oneimage and paint it onto
another object
Texture Transfer
Same as texture synthesis, except an additional constraint:1. Consistency of texture
2. Similarity to the image being explained
7/30/2019 19 Texture
37/50
==++
7/30/2019 19 Texture
38/50
7/30/2019 19 Texture
39/50
Image Analogies
Aaron Hertzmann1,2
Chuck Jacobs2
Nuria Oliver
2
Brian Curless3
David Salesin2,3
1New York University
2Microsoft Research
3University of Washington
Image Analogies
7/30/2019 19 Texture
40/50
Image Analogies
A A
B B
7/30/2019 19 Texture
41/50
Blur Filter
7/30/2019 19 Texture
42/50
Blur Filter
Edge Filter
7/30/2019 19 Texture
43/50
Edge Filter
Artistic Filters
7/30/2019 19 Texture
44/50
A A
B B
Artistic Filters
Colorization
7/30/2019 19 Texture
45/50
Colorization
Texture-by-numbers
7/30/2019 19 Texture
46/50
Texture by numbers
A A
B B
Super-resolution
7/30/2019 19 Texture
47/50
p
A A
Super-resolution (result!)
7/30/2019 19 Texture
48/50
p ( )
B B
Video Matching [Sand & Teller, 2004]
7/30/2019 19 Texture
49/50
g [ , ]
7/30/2019 19 Texture
50/50
MotionMagnificationMotionMagnification
Ce Liu Antonio Torralba William T. Freeman
Frdo Durand Edward H. Adelson
Computer Science and Artificial Intelligence Laboratory
Massachusetts Institute of Technology