1
Improved Seam Carving for Videofor Video Resizing
Michael Rubinstein MERL & The Interdisciplinary Center
Ariel ShamirThe Interdisciplinary Center
Shai AvidanAdobe Systems Inc.
Media Size Change
• Water Ski
(c) ariel shamir
2
Previous Work (Size Change)
• Cropping & Scaling:
– [Wang et al. 2004]
– [Fan et al. 2003]
– [Liu and Gleicher 2006]
– [Deselaers et a. 2008]
• Segment & Recombine
– [Tao et al. 2007]
?
(c) ariel shamir
• Non linear scaling (Warping)
– [Wolf et al. 2007]
• Time manipulations:
– [Pritch et al. 2008] – Object based video
– [Chen and Sen 2008] – Graph cut
Overview
1. Extending Seam Carving Operator
to Video
2. Defining a New Energy Measure
(c) ariel shamir
?[Wolf et al. 2007]
3
A Seam
(c) ariel shamir
Seam Carving
(c) ariel shamir
4
Finding the Seam?
(c) ariel shamir
The Optimal Seam
(c) ariel shamir
)(minarg* sEs
s =||||)( III yxE∂∂+
∂∂=
5
Pixel Attribute Dynamic Programming
( ))1,1(),,1(),1,1(min),(),( +−−−−+= jijijijiEji MMMM
5 8 12 3
9 2 3 9
( ))1,1(),,1(),1,1(min),(),( ++ jijijijiEji MMMM
(c) ariel shamir
7 3 4 2
4 5 7 8
Video?
• Naive… frame by frame independently y p y
Time
(c) ariel shamir
6
Jittery Results
(c) ariel shamir
Naïve (2): Global Projection
• Reduction of the video problem to image p gseam carving by using projection of maximum energy through time:
(c) ariel shamir
7
Naïve (2): Global Projection
• Reduction of the video problem to image p gseam carving by using projection of maximum energy through time:
(c) ariel shamir
Results
(c) ariel shamir
Rescale Retarget
8
Results
Rescale
(c) ariel shamir
Retarget
Problems?
• More complex scenes:– Object movement
– Camera movement
(c) ariel shamir
Global Seams
9
More Complex Scenes
• More complex scenes:– Object movement
– Camera movement
(c) ariel shamir
Global Seams
More Complex Scenes
• Seams should adapt and change through time!
(c) ariel shamir
10
Global Solution
(c) ariel shamir
Video Cube
One 3D Seam
(c) ariel shamir
11
What is the challenge?
• Dynamic Programming no longer works in 3D!• Video Cubes:
– Schodl et al. Siggraph 2000, Video Textures
– Kwatra et al. Siggraph 2003, Graph cut textures
– Rav-Acha et al CVPR 2005
(c) ariel shamir
Rav Acha et al. CVPR 2005, Video Mosaics
– Wang et al. Siggraph 2005, Interactive video cutout
– Chen and Sen EG 2008 (short papers), Video Carving
What is the challenge?
• Dynamic Programming no longer works in 3D!• Use Graph Cut:
– Schodl et al. Siggraph 2000, Video Textures
– Kwatra et al. Siggraph 2003, Graph cut textures
– Rav-Acha et al CVPR 2005
(c) ariel shamir
Rav Acha et al. CVPR 2005, Video Mosaics
– Wang et al. Siggraph 2005, Interactive video cutout
– Chen and Sen EG 2008 (short papers), Video Carving
12
What is the challenge?
• How to Define a Seam from a Cut?
(c) ariel shamir
Kwatra et al. Siggraph 2003, Graph cut textures
Constraints
1. Seams should be monotonic!i.e. one pixel in each row
(c) ariel shamir
13
Constraints
1. Seams should be monotonic!2. Seams should be connected!
(c) ariel shamir
ConnectedPiecewise
Piecewise vs. Connected
(c) ariel shamir
14
Standard Construction
pi,j pi,j+1∂y
∂x
∂y
pi,j pi,j+1
∂y ∂y
∂x
S T
(c) ariel shamir
pi+1,j pi+1,j+1
∂xpi+1,j pi+1,j+1∂x
New Construction
• This construction guarantees monotonic and connected seams
• This construction creates seams that are equivalent to the
pij pi,j+1∞
∞
∞
E1
(c) ariel shamir
equivalent to the dynamic programming approach
(Proofs in the paper)
pi+1,j pi+1,j+1∞
∞E1
15
3D Graph Construction
Frame t+2 Time
(c) ariel shamir
Frame t
Frame t+1
Video Cube
3D Graph Cut
(c) ariel shamir
Video Cube
16
Not Enough!
(c) ariel shamir
Also on Images!
(c) ariel shamir
17
Also on Images!
(c) ariel shamir
Changes in Image
(c) ariel shamir
18
Changes in Image
(c) ariel shamir
Changes in Image
(c) ariel shamir
19
Energy Inserted (not only removed)
(c) ariel shamir
Energy Reduced
Energy Increased
Changes in Energy
(c) ariel shamir
20
Tracking Inserted Energy
pi,j-1 pi,j pi,j+1
pi-1,j-1 pi-1,j
pi,j
pi-1,j+1pi-1,j-1
(c) ariel shamir
• Three possibilities When Removing Pixel Pi,j
Pixel Pi,j : Left Seam
pi,j-1 pi,j pi,j+1
pi-1,j-1 pi-1,j
pi,j
pi-1,j+1pi-1,j-1
(c) ariel shamir
21
Pixel Pi,j : Right Seam
pi,j+1pi,jpi,j-1
pi-1,j+1pi-1,j
pi,j
pi-1,j-1 pi-1,j+1
(c) ariel shamir
Pixel Pi,j : Vertical Seam
pi,j-1 pi,j pi,j+1
pi-1,jpi-1,j-1
pi,j
pi-1,j+1pi-1,j
(c) ariel shamir
22
Energy Function
E
(c) ariel shamir
New Forward Looking Energy Function
E
(c) ariel shamir
23
Backward (SIG 07)
(c) ariel shamir
Forward (SIG 08)
(c) ariel shamir
24
Backward
(c) ariel shamir
Forward
(c) ariel shamir
25
Backward
(c) ariel shamir
Forward
(c) ariel shamir
26
Backward Expand
(c) ariel shamir
Forward Expand
(c) ariel shamir
27
Graph Cut Formulation
pij pi,j+1∞
∞
∞-LU -LU+LU +LU
+LR
(c) ariel shamir
pi+1,j pi+1,j+1∞
∞+LR
Results
(c) ariel shamir
28
Results
(c) ariel shamir
Rescale Retarget
Results
Rescale
(c) ariel shamir
Retarget
29
Timing
• Typical video sequences have a resolution of yp q400 x 300 and 400 frames.
• 2 Seconds per seam using multi-resolutions of three levels (details in the paper)
• Total preprocessing time to enable between
(c) ariel shamir
50% to 150% change in aspect ratio takes around 10 minutes.
• Resizing can be performed in real time (multi size videos)
Multi-size Video DEMO?
(c) ariel shamir
30
Multi-size Video
(c) ariel shamir
Limitations
• Nature Scene
(c) ariel shamir
31
…Object Removal
(c) ariel shamir
Summary
1. Graph Cut2. Forward Energy
(c) ariel shamir
?
32
Contributions
• Extend the seam carving operator from images to video while maintaining its simplicity.
• Formulate using graph cut
• Allowing vertical and horizontal content aware size change and define multi-size video
(c) ariel shamir
• A novel forward energy for better content preservation with
Future
• Quality improvements (Energy, Smoothing)y p ( gy, g)
• Faster (recent GPU-based graph-cut optimizations)
• Use of multiple operatorsA t ti
(c) ariel shamir
• Automatic
• Other applications (domains?)…
33
(c) ariel shamir
Thank you!
XT Plane
(c) ariel shamir
Time
34
Monotonic: One Cut in Each Row
pi,j pi,j+1∞
∂y ∂y
∂x
S T
(c) ariel shamir
pi+1,j pi+1,j+1∞∂x
Defining a Seam from a Cut
1. Seams should be monotonic2. Seams should be connected!
(c) ariel shamir
35
Defining a Seam from a Cut
1. Seams should be monotonic2. Seams should be connected!
(c) ariel shamir