7/30/2019 19 Texture

1/50

Texture Synthesis

15-463: Computational PhotographyAlexei Efros, CMU, Fall 2005

Darren Green (www.darrensworld.com)

7/30/2019 19 Texture

2/50

Texture

Texture depicts spatially repeating patterns

Many natural phenomena are textures

radishes rocks yogurt

7/30/2019 19 Texture

3/50

Texture Synthesis

Goal of Texture Synthesis: create new samples ofa given texture

Many applications: virtual environments, hole-filling, texturing surfaces

7/30/2019 19 Texture

4/50

The Challenge

Need to model the whole

spectrum: from repeated tostochastic texture

repeated

stochastic

Both?

7/30/2019 19 Texture

5/50

Efros & Leung Algorithm

Assuming Markov property, compute P(p|N(p))

Building explicit probability tables infeasible

pp

Synthesizing a pixel

non-parametricsampling

Input image

Instead, we search the input image for all similarneighborhoods thats our pdf forp

To sample from this pdf, just pick one match atrandom

7/30/2019 19 Texture

6/50

Some Details

Growing is in onion skin order

Within each layer, pixels with most neighbors are

synthesized first If no close match can be found, the pixel is not

synthesized until the end Using Gaussian-weightedSSD is very important

to make sure the new pixel agrees with its closest

neighbors

Approximates reduction to a smaller neighborhood

window if data is too sparse

7/30/2019 19 Texture

7/50

Neighborhood Window

input

7/30/2019 19 Texture

8/50

Varying Window Size

Increasing window size

7/30/2019 19 Texture

9/50

Synthesis Results

french canvas rafia weave

7/30/2019 19 Texture

10/50

More Resultswhite bread brick wall

7/30/2019 19 Texture

11/50

Homage to Shannon

7/30/2019 19 Texture

12/50

Hole Filling

7/30/2019 19 Texture

13/50

Extrapolation

7/30/2019 19 Texture

14/50

Summary

The Efros & Leung algorithm

Very simple

Surprisingly good results

Synthesis is easier than analysis!

but very slow

7/30/2019 19 Texture

15/50

pp

Observation:

Image Quilting [Efros & Freeman]

neighbor pixels are highly correlated

Input image

non-parametricsampling

BB

Idea:Idea: unit of synthesis = blockunit of synthesis = block Exactly the same but now we want P(B|N(B))

Much faster: synthesize all pixels in a block at once

Not the same as multi-scale!

Synthesizing a block

7/30/2019 19 Texture

16/50

Input texture

B1 B2

Random placement

of blocks

block

B1 B2

Neighboring blocks

constrained by overlap

B1 B2

Minimal error

boundary cut

7/30/2019 19 Texture

17/50

min. error boundary

Minimal error boundary

overlapping blocks vertical boundary

__ ==22

overlap error

7/30/2019 19 Texture

18/50

Our Philosophy

The Corrupt Professors Algorithm:

Plagiarize as much of the source image as you can Then try to cover up the evidence

Rationale: Texture blocks are by definition correct samples of

texture so problem only connecting them together

7/30/2019 19 Texture

19/50

7/30/2019 19 Texture

20/50

7/30/2019 19 Texture

21/50

7/30/2019 19 Texture

22/50

7/30/2019 19 Texture

23/50

7/30/2019 19 Texture

24/50

7/30/2019 19 Texture

25/50

Failures(ChernobylHarvest)

7/30/2019 19 Texture

26/50

input image

Portilla & Simoncelli

Wei & Levoy Our algorithm

Xu, Guo & Shum

7/30/2019 19 Texture

27/50

Portilla & Simoncelli

Wei & Levoy Our algorithm

Xu, Guo & Shum

input image

7/30/2019 19 Texture

28/50

Portilla & Simoncelli

Wei & Levoy Our algorithm

input image

Xu, Guo & Shum

P liti l T t S th i !

7/30/2019 19 Texture

29/50

Political Texture Synthesis!

MS Di it l I P (DEMO)

7/30/2019 19 Texture

30/50

MS Digital Image Pro (DEMO)

7/30/2019 19 Texture

31/50

Fill Order

7/30/2019 19 Texture

32/50

Fill Order

In what order should we fill the pixels? choose pixels that have more neighbors filled

choose pixels that are continuations of lines/curves/edges

Criminisi, Perez, and Toyama. Object Removal by Exemplar-based Inpainting,Proc. CVPR, 2003.

Exemplar based Inpainting demo

http://www.research.microsoft.com/~antcrim/papers/Criminisi_cvpr2003.pdfhttp://www.research.microsoft.com/~antcrim/papers/Criminisi_cvpr2003.pdf

7/30/2019 19 Texture

33/50

Exemplar-based Inpainting demo

http://research.microsoft.com/vision/cambridge/i3l/patchworks.htm

Application: Texture Transfer

http://research.microsoft.com/vision/cambridge/i3l/patchworks.htmhttp://research.microsoft.com/vision/cambridge/i3l/patchworks.htm

7/30/2019 19 Texture

34/50

++ ==

Application: Texture Transfer

Try to explain one object with bits andpieces of another object:

Texture Transfer

7/30/2019 19 Texture

35/50

Texture Transfer

Constraint

Texture sample

Texture Transfer

7/30/2019 19 Texture

36/50

Take the texture from oneimage and paint it onto

another object

Texture Transfer

Same as texture synthesis, except an additional constraint:1. Consistency of texture

2. Similarity to the image being explained

7/30/2019 19 Texture

37/50

==++

7/30/2019 19 Texture

38/50

7/30/2019 19 Texture

39/50

Image Analogies

Aaron Hertzmann1,2

Chuck Jacobs2

Nuria Oliver

2

Brian Curless3

David Salesin2,3

1New York University

2Microsoft Research

3University of Washington

Image Analogies

7/30/2019 19 Texture

40/50

Image Analogies

A A

B B

7/30/2019 19 Texture

41/50

Blur Filter

7/30/2019 19 Texture

42/50

Blur Filter

Edge Filter

7/30/2019 19 Texture

43/50

Edge Filter

Artistic Filters

7/30/2019 19 Texture

44/50

A A

B B

Artistic Filters

Colorization

7/30/2019 19 Texture

45/50

Colorization

Texture-by-numbers

7/30/2019 19 Texture

46/50

Texture by numbers

A A

B B

Super-resolution

7/30/2019 19 Texture

47/50

p

A A

Super-resolution (result!)

7/30/2019 19 Texture

48/50

p ( )

B B

Video Matching [Sand & Teller, 2004]

7/30/2019 19 Texture

49/50

g [ , ]

7/30/2019 19 Texture

50/50

MotionMagnificationMotionMagnification

Ce Liu Antonio Torralba William T. Freeman

Frdo Durand Edward H. Adelson

Computer Science and Artificial Intelligence Laboratory

Massachusetts Institute of Technology