Image Warping
http://www.jeffrey-martin.com
[Some slides from K. Padalkar, S. Avidan, A. Shamir, A. Efros, S. Seitz, and Y. Wang]
Image Manipula-on and Computa-onal PhotographyCS294-‐69 Fall 2011
Robert Carroll
What is a warp?
What is a warp?
Original
What is a warp?
Original Warped
Warps are spatial transformations:Points map to points, but colors don’t change.
Applications
Panorama Stitching Resizing/Retargeting
Wide-Angle Distortion Shape Manipulation
Video Stabilization, Stereoscopic Depth Editing, Object Removal, ...
[Igarashi et al. 09]
Applications
Panorama Stitching Resizing/Retargeting
Wide-Angle Distortion Shape Manipulation
Video Stabilization, Stereoscopic Depth Editing, Object Removal, ...
[Igarashi et al. 09]
Image Transformations
f
x
f
x
f
x
Tf
x
image warping: change domain of image
g(x) = f(T(x))
image filtering: change range of imageg(x) = T(f(x))
T
Image Transformations
image filtering: change range of imageg(x) = T(f(x))
T
T
f
f g
g
image warping: change domain of image
g(x) = f(T(x))
Types of Warps
rotation aspect perspective cylindrical
Mesh:
Parametric:
Discrete:
-‐ Globally defined by small # of parameters
-‐ Smooth (rubber sheet) transformaAon
-‐ Each output pixel can be an arbitrary input pixel
[Barnes et al. 09]
[Wang et al. 09]
The Image Resizing/RetargeAng Problem
The Image Resizing/RetargeAng Problem
Image Retarge-ng Objec-ves
1. Change size
2. Preserve the important content and structures3. Limit ar*facts created
Tradi-onal Methods
•Letterbox wastes pixels•Squeeze/Hybrid introduce distortions•Cropping removes important parts
original squeeze crop hybridletterbox
Many Exis-ng Resizing Methods
Shai Avidan and Ariel Shamir, Proc. SIGGRAPH, 2007
Seam Carving for Content-Aware Image Resizing
Seam Carving Method
Seam Carving Method
• “Seam Carving for Content-‐Aware Image Resizing,” S. Avidan and A. Shamir, Proc. SIGGRAPH, 2007
Seam Carving Method
• “Seam Carving for Content-‐Aware Image Resizing,” S. Avidan and A. Shamir, Proc. SIGGRAPH, 2007
• In Photoshop called “content aware scaling”
Seam Carving Method
• “Seam Carving for Content-‐Aware Image Resizing,” S. Avidan and A. Shamir, Proc. SIGGRAPH, 2007
• In Photoshop called “content aware scaling”• Main idea: Remove the least noHceable pixels
– How? Define an “energy func-on” that measures how perceptually no-ceable each pixel is
Seam Carving Method
• “Seam Carving for Content-‐Aware Image Resizing,” S. Avidan and A. Shamir, Proc. SIGGRAPH, 2007
• In Photoshop called “content aware scaling”• Main idea: Remove the least noHceable pixels
– How? Define an “energy func-on” that measures how perceptually no-ceable each pixel is
• Remove the pixels with “low energy” and avoid removing pixels with “high energy”– How? Define a criterion for picking which pixels to remove
Possible Energy Func-ons
• Edgeness– Gradient magnitude
• Entropy• HOG (Histogram of Gradient)• Saliency• …Caveat: No single energy function performs well across all images
Possible Energy Func-ons
• Edgeness– Gradient magnitude
• Entropy• HOG (Histogram of Gradient)• Saliency• …Caveat: No single energy function performs well across all images
Pixel Removal Criteria• Op#mal: remove the k pixels with lowest energy• Output image no longer rectangular
Input imageEnergy image
Output image
Pixel Removal Criteria• Remove k pixels with lowest energy in each row• No visual coherence between adjacent rows
Pixel Removal Criteria• Column: Remove whole column with lowest energy• Frequently introduces arHfacts
Seam Defini-on
Seam Defini-on• VerAcal Seam
is an 8-‐connected path of pixels in an n x m image from top to boQom, containing one, and only one, pixel in each row of the image:
Seam Defini-on• VerAcal Seam
is an 8-‐connected path of pixels in an n x m image from top to boQom, containing one, and only one, pixel in each row of the image:
Seam Defini-on• VerAcal Seam
is an 8-‐connected path of pixels in an n x m image from top to boQom, containing one, and only one, pixel in each row of the image:
Seam Energy
• Energy of a Seam
• Minimum Energy Seam
Pixel Removal Criteria• Seam: Remove the verHcal curve of lowest energy
Pixel Removal Effec-veness
How to Efficiently Compute Best Seam?
• Use Dynamic Programming to find lowest energy seam in linear Hme
1. Forward Pass (top row to boWom row for finding verHcal seam)– Define M(i,j) = cumula-ve energy at (i,j)– M(1,j) = e(1,j)– M(i,j) = e(i,j) + min(M(i-‐1,j-‐1), M(i-‐1,j), M(i-‐1, j+1))
– Find minimum value in last row: minj M(n,j)
2. Backward Pass (boWom row to top row)– Trace back path from pixel in boZom row with min value to top row
Seams
Seams over energy image Seams over input image
Shrink Image in 1 Dimension
Shrink Image in 1 Dimension
• Change the image from size n × m to n × mʹ′– assume mʹ′ < m
Shrink Image in 1 Dimension
• Change the image from size n × m to n × mʹ′– assume mʹ′ < m
• Remove m-‐mʹ′ = c seams successively
Shrink Image in 1 Dimension
• Change the image from size n × m to n × mʹ′– assume mʹ′ < m
• Remove m-‐mʹ′ = c seams successively
Shrink Image in 1 Dimension
• Change the image from size n × m to n × mʹ′– assume mʹ′ < m
• Remove m-‐mʹ′ = c seams successively
Seam Carving
Shrink Image in 1 Dimension
• Change the image from size n × m to n × mʹ′– assume mʹ′ < m
• Remove m-‐mʹ′ = c seams successively
Scaling
Shrink Image in Both Dimensions:Op-mal Seam Ordering
Shrink Image in Both Dimensions:Op-mal Seam Ordering
• Change the image from size n × m to nʹ′ × mʹ′– assume mʹ′ < m and nʹ′ < n
Shrink Image in Both Dimensions:Op-mal Seam Ordering
• Change the image from size n × m to nʹ′ × mʹ′– assume mʹ′ < m and nʹ′ < n
• What is the best order for seam carving?– Remove verHcal seams first? – Horizontal seams first? – Alternate between the two?
Op-mal Seam Ordering
Op-mal Seam Ordering
• Solve opHmizaHon problem:
Op-mal Seam Ordering
• Solve opHmizaHon problem:
Op-mal Seam Ordering
• Solve opHmizaHon problem:
where k = r+c, r = (m−mʹ′), c = (n−nʹ′) and αi is a parameter that determines if at step i we remove a horizontal or verAcal seam: α ∈ {0,1}
Op-mal Seam Ordering
Op-mal Seam Ordering
• Transport map– Matrix of size n × m– Each element T(r,c) holds the minimal cost needed to obtain an image of size n−r × m−c
Op-mal Seam Ordering
• Transport map– Matrix of size n × m– Each element T(r,c) holds the minimal cost needed to obtain an image of size n−r × m−c
Enlarging Images
Enlarging Images� Method 1: Compute the opAmal verAcal (horizontal) seam s in image and duplicate the pixels in s by averaging them with their le` and right neighbors (top and boQom in the horizontal case)
� O`en will choose the same seam at each iteraAon, producing noAceable stretching arAfact
Enlarging Images� Method 1: Compute the opAmal verAcal (horizontal) seam s in image and duplicate the pixels in s by averaging them with their le` and right neighbors (top and boQom in the horizontal case)
� O`en will choose the same seam at each iteraAon, producing noAceable stretching arAfact
Enlarging Images� Method 1: Compute the opAmal verAcal (horizontal) seam s in image and duplicate the pixels in s by averaging them with their le` and right neighbors (top and boQom in the horizontal case)
� O`en will choose the same seam at each iteraAon, producing noAceable stretching arAfact
Enlarging Images
� Method 2: To enlarge width by k, compute top k verAcal seams (for removal) and duplicate each of them
Content Amplifica-on
Content Amplifica-on� Scale the image; this will scale everything, “content” as well as “non-‐content”
� Shrink the scaled-‐image using seam carving, which will (hopefully) carve out the non-‐content part
Content Amplifica-on� Scale the image; this will scale everything, “content” as well as “non-‐content”
� Shrink the scaled-‐image using seam carving, which will (hopefully) carve out the non-‐content part
Content Amplifica-on� Scale the image; this will scale everything, “content” as well as “non-‐content”
� Shrink the scaled-‐image using seam carving, which will (hopefully) carve out the non-‐content part
Object Removal
Object Removal� User marks the target object to be removed� Force seams to pass through marked pixels� To obtain the original image size, use seam inserAon
Object Removal� User marks the target object to be removed� Force seams to pass through marked pixels� To obtain the original image size, use seam inserAon
Object Removal� User marks the target object to be removed� Force seams to pass through marked pixels� To obtain the original image size, use seam inserAon
Object Removal
� One shoe removed (and image enlarged to original size)
input result
Object Removal� Object marking to prevent unwanted results: mark regions where seams must not pass
Object Removal� Object marking to prevent unwanted results: mark regions where seams must not pass
Object Removal� Object marking to prevent unwanted results: mark regions where seams must not pass
Object Removal� Object marking to prevent unwanted results: mark regions where seams must not pass
Mul--‐Size Images
H VInput
Mul--‐Size Images• Methods menAoned so far are not real-‐Ame
H VInput
Mul--‐Size Images• Methods menAoned so far are not real-‐Ame• We calculate best seam, remove it, calculate the next
seam based on new image, etc.
H VInput
Mul--‐Size Images• Methods menAoned so far are not real-‐Ame• We calculate best seam, remove it, calculate the next
seam based on new image, etc.• Compute Index map, V, of size n × m that encodes, for
each pixel, the index of the seam that removed it, i.e., V(i, j) = t means pixel (i, j) was removed by the tth seam removal iteraAon
H VInput
Failures
� Too much content� No space for seam to avoid content
Optimized Scale-and-Stretch for Image Resizing
1Yu-Shuen Wang, 2Chiew-Lan Tai, 3Olga Sorkine, 1Tong-Yee Lee
1National Cheng KungUniversity, Taiwan
3New York University2Hong Kong University of Science & Technology
Scale-and-stretch warp
• Allow important regions to uniformly scale• Find optimal local scaling factors by global optimization• Result: preserve the shape of important regions, distort non-important ones
importance map
Importance map
saliency map[Itti et al. 98]
x =
importance map
gradients only importance map
The warping mechanism
• Grid mesh, preserve the shape of the important quads
• Optimize the location of mesh vertices, interpolate image
quads with high importance uniform scaling
quads with low importance allow non-uniform scaling
The warping mechanism
• Grid mesh, preserve the shape of the important quads
• Optimize the location of mesh vertices, interpolate image
Deformation Energy• Per quad
sf – uniform scaling
f
vi
vj
vʹ′i
vʹ′j
Deformation Energy• Total deformation energy
• Quadratic energy in vʹ′, can minimize by solving a linear system of equations
importance weights
Constraints
•Corner vertices•Horizontal/vertical sliding
Deformation energy
• Problem: grid lines bend a lot; warp not so smooth
Deformation energy
• Problem: grid lines bend a lot; warp not so smooth
Grid line bending energy
• Attempt to keep original edge orientations but allow length scaling
note thenonlinearfactor
f
vi
vj
vʹ′i
vʹ′j
Energy minimization
• Total energy is nonlinear in vʹ′
• Iterative minimization with tricks• Keep sf and lij as additional variables• Do alternating minimization steps Fix sf and lij and optimize vʹ′ Compute new sf and lij• Sparse direct solver for the linear system and reuse the matrix factorization to gain speed.
Grid line bending energy
• With the bending term added
Results
Results
original SC indirect SC
Scale-and-Stretch
Mesh can move in 2D, with seam carving pixels only move in 1D
original SC indirect SC
Scale-and-Stretch
Results
original SC indirect SC
Scale-and-Stretch
Results
Optimizing Content-Preserving Projections for Wide-Angle Images
Robert CarrollManeesh Agrawala
University of California, Berkeley
Aseem AgarwalaAdobe Systems, Inc.
Rectilinear Lens
© Keith Cooper
Fisheye
© Flickr user kirainet
Cylindrical Panorama
© Flickr user Seb Przd
Viewing Sphere
Viewing Sphere
Viewing Sphere
Viewing Sphere
Courtesy of Flickr user brokendrum70
Cartography
Linear Perspective Projection
Linear Perspective Projection
Stereographic Projection
Stereographic Projection
Cylindrical Projection
Mercator Projection
Optimized Projection
Optimized Projection
GoalGiven a wide-angle image, produce a projection that preserves
straight lines in the scene and the shapes of objects
Our Approach
Mesh the viewing sphere
De!ne mapping constraints
Optimize weighted energy function
Viewing Sphere Mesh
Viewing Sphere Mesh
Viewing Sphere Mesh
Viewing Sphere Mesh
Three Properties
1. Conformality
2. Straight Lines
3. Smoothness of mapping
h
k
Conformality
k
h
Conformality
Lines
Lines
Lines
Should be zero
Lines
Should be zero
Lines
Should be zero
Smoothness
Smoothness
Smoothness
Smoothness
What’s left?
• Cannot satisfy all constraints exactly
– De!ne weighted least-squares energy terms
• Iterative non-linear optimization
Etotal= w1Econformality + w2Elines + w3Esmoothness
Lines
Should be zero
Lines
Should be zero
Lines
Should be zero
Lines
fixed n
Lines
fixed n
�(1� �)
Lines
Should be zero
Lines
fixed �
Lines
fixed �
Lines
fixed n
Lines
fixed n
Lines
fixed �
Lines
fixed �
Implementation Details
• Converges in 8 iterations• Up to 160,000 vertices• 15s - 1m, depending on !eld of view
Results
Input/Output
Input/Output
Perspective Mercator
Our ResultStereographic
Perspective Mercator
Our ResultStereographic
Perspective Mercator
Our ResultStereographic
Image CourtesyFlickr user
Aldo
Perspective Mercator
Our Result
Stereographic
ImageCourtesyJeff Chien
PerspectiveMercator
Our ResultStereographic
Input (© Flickr user Mike Schinkel)
Our ResultZorin
Our Result
Zelnik-Manor et al. MultiPlane Zelnik-Manor et al. MultiView
Mercator
Our Result
Zelnik-Manor et al. MultiPlane Zelnik-Manor et al. MultiView
Mercator
Our Result
Zelnik-Manor et al. MultiPlane Zelnik-Manor et al. MultiView
Mercator
Limitations
• Must be cropped
Limitations
• Some stretching near poles
Limitations
• Some stretching near poles
Geodesic Grid
• Lines that stretch across the full field of view
Limitations
© Flickr user Editor B
Thanks!
