1
Image Stitching and Composition
Lots of slides from from Bill Freeman, Alyosha Efros, Steve Seitz, Rick Szeliski, A. Agrawal , R. Raskar, Hong Chen, and Shmuel Peleg.
© prof. dmartin
• Stitching:– Capturing different portions of a scene with an
overlap region viewed in both images.– Combine the two images together.– The seam between the images should be
invisible.
• Composition:– Cut a portion of a scene from one image and
paste it in another image.
Image Stitching and CompositionImage Stitching and Composition
Overlap Regions
Image StitchingImage Stitching Image StitchingImage Stitching
• Naïve stitching might seem unnatural due to parallax, lens distortion, scene motion, different exposure, different illuminations
2
Image CompositionImage Composition
Target ImageSource Images
Topics to be CoveredTopics to be Covered
• Alpha Blending• Multi-band blending• Optimal-cut composing• Blending in gradient domain
Image StitchingImage Stitching Alpha Blending (feathering)Alpha Blending (feathering)
01
01
+
=Encoding transparency
I(x,y) = (αR, αG, αB, α)
Iblend = Ileft + Iright
3
Effect of Window SizeEffect of Window Size
0
1 left
right0
1
Effect of Window SizeEffect of Window Size
0
1
0
1
Good Window SizeGood Window Size
0
1
“Optimal” Window: smooth but not ghosted
Left Image Right Image
Left + Right Narrow Transition Wide Transition Multibands blend
Alpha blending is problematic in the Alpha blending is problematic in the presence of high and low frequencies:presence of high and low frequencies:
4
What is the Optimal Window?What is the Optimal Window?
• To avoid seams– window >= size of largest prominent feature
• To avoid ghosting– window <= 2*size of smallest prominent feature
• Solution: Multi-band blending (Burt & Adelson 83):
– Decompose the image into multi-band freq.– Blend each band appropriately
Multiband PyramidMultiband Pyramid
expand
-
expand
expand
-
- =
=
=
Gaussian Pyramid
LaplacianPyramid
Pyramid BlendingPyramid Blending
0
1
0
1
0
1
Left pyramid Right pyramidblend
Multiband BlendingMultiband Blending
5
Example: blend apple and orangeExample: blend apple and orange laplacianlevel
4
laplacianlevel
2
laplacianlevel
0
left pyramid right pyramid blended pyramid
Multiband BlendingMultiband Blending Multiband Blending: exampleMultiband Blending: example
6
© prof. dmartin
Multiband Blending: exampleMultiband Blending: example Optimal CutOptimal Cut• Imperfect registration or presence of moving
objects generate ghost objects• Solution: Optimal Cut
Optimal Cut Optimal Cut (Davis, 1998)(Davis, 1998)• Segment the mosaic
– Single source image per segment– Avoid artifacts along boundries– Dijkstra’s algorithm
Input texture
B1 B2
Random placement of blocks
block
B1 B2
Neighboring blocksconstrained by overlap
B1 B2
Minimal errorboundary cut
EfrosEfros & Freeman, 2001& Freeman, 2001
7
min. error boundary
Minimal error boundaryMinimal error boundaryoverlapping blocks vertical boundary
__ ==22
overlap error
Optimal Cut Optimal Cut -- AlgorithmAlgorithm• e(k,j) – penalty for cutting the kth row at the jth column • Cut trajectory is monotonic and continues• Can be solved using Dynamic Programming:
e(k,j)= I1(k,j)-I2(k,j-1) j
k
I1I2
E(k,j)=e(k,j)+MINE(k-1,j-1), E(k-1,j), E(k-1,j+1)
Graph CutGraph Cut
• What if we want similar “cut-where-things-agree” idea, but for non monotonic cut or closed regions?
• Dynamic programming can’t handle loops• Solution: GraphCut
Graph Cut Graph Cut (simple example (simple example Boykov&JollyBoykov&Jolly, ICCV, ICCV’’01)01)
n-links
s
t a cuthard constraint
hard constraint
Minimum cost cut can be computed in polynomial time(max-flow/min-cut algorithms)
8
Graph Cut Composition Graph Cut Composition ((KwatraKwatra et al, 2003)et al, 2003) Graph Cut: Graph Cut: ss--tt GraphGraph
• G=(V,E) • a source node s, and a sink node t• Directed edge (i,j)∈E from node i to node j• Each edge has a non-negative capacity C(i,j)• C(i,j)=0 for non-exist edges.
s t2 3
1
6 3i j
C(i,j)
Flow in Flow in ss--tt GraphGraph• A Flow is a real value f(i,j) that assign a real
value to edge (i,j) under the constraints:– Capacity constraint : f(i,j) ≤ C(i,j)– Flow balance constraint:
for each i∉s,t flow_in = flow_outTotal flow:
s t2/2 1/3
1/1
1/62/3i j
f(i,j)/C(i,j)
( ) ( ), ,i j
f s i f j t f= − =∑ ∑
ss--tt CutCut
• A cut is a partition of V into two exclusive subsets S and T, s.t. s∈S, and t ∈T
s t2 3
1
6 3 s t2 3
1
6 3
s-t cutnot s-t cut
9
Capacity of Capacity of ss--tt CutCut
• A capacity of a s-t cut is the accumulated capacities of edges from S towards T:
s t2 3
1
6 3
( ),
([ , ]) ,i S j T
C S T c i j∈ ∈
= ∑
([ , ]) 2 3C S T = +
Max Flow = Min cutMax Flow = Min cut
• Maximum flow is the flow that has maximum value among all possible flow assignments
• Minimum cut is the s-t cut whose capacity is minimum among all possible s-t cut
• maximum flow = minimum cut
• Polynomial time solution
Back to Blending: Common ProblemsBack to Blending: Common Problems
Global Intensity Difference
Horizontal Misalignment
Vertical Misalignment
Previous ApproachesPrevious Approaches• Optimal Seam
– Search for a curve in the overlap region on which the differences between the images is minimal
– Poorly handles global intensity differences
10
Previous ApproachesPrevious Approaches
• Alpha Blending– Smooth the transition by weighting alpha mask
as a function of the distance from the seam– Poorly handles misalignments
Previous ApproachesPrevious Approaches
• Pyramid Blending– Combine different frequency bands with different
alpha masks– Poorly handles misalignments
Blending in Gradient DomainBlending in Gradient Domain• In Pyramid Blending, we decomposed our
image into 2nd derivatives (Laplacian) and a low-res image
• Let us now look at 1st derivatives (gradients):– No need for low-res image – captures everything (up to a constant)
• Algorithm: – Differentiate– Blend– Reintegrate
Gradient Domain Blending (1D)Gradient Domain Blending (1D)
Twosignals
Regularblending
Blendingderivatives
bright
dark
11
Comparisons: Comparisons: Levin et al, 2004Levin et al, 2004 BasicsBasics
• Images as scalar fieldsR2 -> R
Vector FieldVector Field
• A vector function G: R2 →R2
• Each point (x,y) is associated with a vector (u,v)
G(x,y)=[ u(x,y) , v(x,y) ]
Gradient FieldGradient Field• Partial derivatives of scalar field • Direction
– Maximum rate of change of scalar field
• Magnitude– Rate of change
,yI
xII
∂∂
∂∂
=∇
),( yxI
12
Continuous Continuous v.sv.s. Discrete. Discrete
Image I(x,y) Ix Iy
Continues case → Derivative
Discrete case → Finite differences
[ ]
[ ] IyI
IxI
T ∗−→∂∂
∗−→∂∂
110
110
Editing in Gradient DomainEditing in Gradient Domain
• Given vector field G=(u(x,y),v(x,y)) (pasted gradient) in a bounded region Ω. Find the values of I in Ω that optimize:
Ω∂∗
Ω∂Ω=−∇∫∫ IIwithGI
I
2min
G=(u,v)
I*I
I
I*
Ω
Intuition Intuition -- What if G is null?What if G is null?
• 1D:
• 2D:
x1 x2
Ω∂∗
Ω∂Ω=∇∫∫ IIwithI
I
2min
What if What if GG is not null?is not null?
• 1D case
Seamlessly paste onto
- Add a linear function so that the boundary condition is respected- Gradient error is equally distributed all over Ω in order to respect the boundary condition
13
2D case2D case
From Perez et al. 2003
2D case2D case
From Perez et al. 2003
2D case2D case How do we reconstruct from gradient field?How do we reconstruct from gradient field?
• I minimizes the integral:
( ) dydxGIE∫∫ ∇ ,
( )22
2, ⎟⎟⎠
⎞⎜⎜⎝
⎛−
∂∂
+⎟⎠⎞
⎜⎝⎛ −
∂∂
=−∇=∇ vyIu
xIGIGIE
14
Calculus of VariationsCalculus of Variations: Euler: Euler--Lagrange EquationLagrange Equation
• I must satisfy
• Substituting E we get:
0=∂∂
−∂∂
−∂∂
yx IE
dyd
IE
dxd
IE
022 2
2
2
2
=⎟⎟⎠
⎞⎜⎜⎝
⎛∂∂
−∂∂
+⎟⎟⎠
⎞⎜⎜⎝
⎛∂∂
−∂∂
yv
yI
xu
xI
Calculus of VariationsCalculus of Variations: Euler: Euler--Lagrange EquationLagrange Equation
2
2
2
22
yI
xII
∂∂
+∂∂
=∇
GI div2 =∇
yv
xuGdiv
∂∂
+∂∂
=
(laplacian)
Poisson EquationPoisson Equation
yv
xuGdivI
∂∂
+∂∂
==∇2
• Second order PDE• Iterative solution (Conjugate Gradient)
Alternative Derivation (discrete notation)Alternative Derivation (discrete notation)
⎟⎟⎠
⎞⎜⎜⎝
⎛=⎟⎟
⎠
⎞⎜⎜⎝
⎛vu
Iy
x
DD
⎥⎥⎥⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢⎢⎢⎢
⎣
⎡
−−
−−
−−
−
=
1111
1111
1111
11
xD[ ]*110* −=∂∂x
IxDIx
=∗∂∂
• Let Dx - Toeplitz matrix
15
( ) ( ) ⎟⎟⎠
⎞⎜⎜⎝
⎛=⎟⎟
⎠
⎞⎜⎜⎝
⎛vu
I Ty
Tx
y
xTy
Tx DD
DD
DD
• Normal equation:
( ) vuI Ty
Txy
Tyx
Tx DDDDDD +=+
[ ] [ ]*011*110, −=−⇒ flipDNote Tx
[ ]*121 −=xTx DD
GdivI =∇ 2 Numerical SolutionNumerical Solution
• Discretize Laplacian
=+≡∇ yTyx
Tx DDDD2 [ ] ∗
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡−=∗
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡−+−
010141010
12
1121
⎥⎥⎥⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢⎢⎢⎢
⎣
⎡
−−
−−
−−
−
14111411
1141111411
114111141
1141
Sparse Toeplitz Matrix
Comments:– A is sparse.– A is symmetric and can be inverted.– If Ω is rectangular, A is a Teoplitz matrix.– Size of A is ∼NxN.– Impractical to form or store A.– Impractical to invert A
( ) vuI Ty
Txy
Tyx
Tx DDDDDD +=+
bI =A
Iterative Solution: Conjugate GradientIterative Solution: Conjugate Gradient
• Solves a linear system Ax=b (in our case x=I)• A is square, symmetric, positive semi-definite.• Advantages:
– Fast!– No need to store A but calculating Ax – In our case Ax can be calculated using a single
convolution.– Can deal with constraints.
16
Conjugate gradientConjugate gradient
• For each step i: – Take the residual d(i)=b-Ax(i) (= -gradient)– Make it A-orthogonal to the previous ones – Find minimum along this direction
• Needs at most N iterations.• Matlab command:
x=cgs(A,b)A can be a function handle afun
such that afun(x) returns A*x
Solving Poisson equation with boundary conditionsSolving Poisson equation with boundary conditions
Ω
*ΩS T
( ) ( )ΩΩ
+=+ SDDDDDDDD yTyx
Txy
Tyx
Tx I s.t. ∗∗ ΩΩ
= TI
Ω∇S
• Define a circumscribing square Π=Ω∪Ω*– Let Ω⊂ Π denotes the edited image area.– Let Ω*= Π-Ω denotes the surrounding area.
yyxxk ∂∗∂+∂∗∂=
( ) ∗ΩΩ∗= IIkAI U
Ω
*ΩS T( ) *ΩΩ∗= TSkb U
x=cgs(<computeAI>,b)
• The above requirements can be express as a linear set of equations:
Ω∇S
[ ] [ ]bAI
TSDDDD
IIDDDD yyxxyyxx =⇒⎟
⎟⎠
⎞⎜⎜⎝
⎛ +=⎟
⎟⎠
⎞⎜⎜⎝
⎛ +⇒
∗∗ Ω
Ω
Ω
Ω
Image stitchingImage stitching
17
Gradient Domain CompositionGradient Domain Composition Cut & Paste Cut & Paste Paste in Gradient Domain Paste in Gradient Domain
Another exampleAnother example
18
Transparent CloningTransparent CloningTransparent Cloning
I SΩ Ω∇ = ∇2
S TI Ω ΩΩ
∇ + ∇∇ = ( )max ,I S TΩ Ω Ω∇ = ∇ ∇
Transparent CloningTransparent Cloning
Shadow Transferring Shadow Transferring
19
Other Gradient Domain ApplicationsOther Gradient Domain Applications
Changing local illuminationChanging local illuminationDefect concealmentDefect concealment
High Dynamic Range CompressionHigh Dynamic Range Compression
Small exposure: Dark inside
High Dynamic Range CompressionHigh Dynamic Range Compression
Large exposure: Outside Saturated
20
Software Tone Mapping
Short Exposure
Long Exposure
High Dynamic Range CompressionHigh Dynamic Range CompressionShadow RemovalShadow Removal
Shadow RemovalShadow Removal
Null gradient on boundary and integrateshadow removal
Shadow RemovalShadow Removal
21
CompositingCompositing
• Compositing images– Have a clever blending function
Alpha blendingblend different frequencies differentlyGradient based blending
– Choose the right pixels from each imageDynamic programming – optimal seamsGraph-cuts
• Put it all together:– Interactive Digital Photomontage
Interactive Digital PhotomontageInteractive Digital Photomontage
Aseem Agarwala, Mira Dontcheva, Maneesh Agrawala, Steven Drucker, Alex Colburn, Brian Curless, David Salesin, Michael Cohen, “Interactive Digital Photomontage”, SIGGRAPH 2004
• Combining multiple photos• Find seams using graph cuts• Combine gradients and integrate
22
23
photomontageset of originals
Source images Brush strokes Computed labeling
Composite
Brush strokes Computed labeling
SummarySummaryStitching & Compositing
• Alpha Blending• Multi-band blending• Optimal-cut composing• Blending in gradient domain • Applications in Gradient Domain• Photomontage
24
T H E E N DT H E E N D