Dynamosaicing Dynamosaicing Mosaicing of Dynamic Scenes 2007. 11. 16 (Fri) Young Ki Baik Computer...

DynamosaicingDynamosaicing

Mosaicing of Dynamic Scenes

2007. 11. 16 (Fri)

Young Ki Baik

Computer Vision LabSeoul National University

Dynamosaicing:Mosaicing of Dynamic scenes

ReferencesReferencesDynamosaicing: Mosaicing of Dynamic Scenes

• Alex Rav-Acha, Yael Pritch, Dani Lischinski and Shumuel Peleg (PAMI 2007.10)

www.vision.huji.ac.il/dynmos

http://www.vision.huji.ac.il/dynmos


What is Mosaicing in Vision?What is Mosaicing in Vision?Are you getting the whole picture?

Compact camera FOV = 50 x 35°= 50 x 35°



Compact camera FOV = 50 x 35°= 50 x 35°Human FOV = = 200 x 135°200 x 135°



Compact camera FOV = 50 x 35°= 50 x 35°Human FOV = 200 x 135°= 200 x 135° Panoramic Mosaic Panoramic Mosaic = = 360 x 180°360 x 180°


What is Mosaicing in Vision?What is Mosaicing in Vision?How does mosaicing work?

??


What is Mosaicing in Vision?What is Mosaicing in Vision?How does mosaicing work?

New wide FOV cameraNew wide FOV camera New image planeNew image plane




DynamosaicingDynamosaicingPurpose:

The wide scene movie from the normal video data Synthesis of new movie


DynamosaicingDynamosaicingContribution:

Found out new interesting application.

(50%)

Proposed new constancy for image registration.

(20%)

Introduced various video editing trick with simple idea.

(20%)

Applied Min-cut algorithm to video data in order to overcome some artifacts.

(10%)


DynamosaicingDynamosaicingHow does Dynamosaicing work?

Video alignment An initial task of Dynomosaicing

Evolving time front For making wide screen movie and editing videos

Time front optimization : Graph-cut (Min-cut) For seamless dynamosaicing


Video Alignment Video Alignment using Video Extrapolationusing Video ExtrapolationSpace time volume

tt

xx

tt

xx

kk

Stationary CameraStationary Camera Panning CameraPanning Camera

x : Image x axisx : Image x axist : Time axist : Time axis

kk : Frame index : Frame index


Video Alignment Video Alignment using Video Extrapolationusing Video ExtrapolationPurpose of Video Alignment

Changing video data from panning camera like the video taken by stationary camera

It is similar to conventional 2D static mosaicing. (Image registration)

tt

xx

Aligned videoAligned video

tt

xx

kk

Unaligned videoUnaligned video


Video Alignment Video Alignment using Video Extrapolationusing Video ExtrapolationWhat is the differences?

Dynamosaicing can treats Dynamic scenes.

Conventional mosaicing Only treat static objects.

Dynamosaicing Permit the presence of moving objects.

Brightness constancyBrightness constancyBrightness constancyBrightness constancy

Dynamic constancyDynamic constancyDynamic constancyDynamic constancy




- Only consider intensity values of images




- Only consider intensity values of images


Video Alignment Video Alignment using Video Extrapolationusing Video ExtrapolationDynamic Constancy

Assumption :

If If two different space time blocks are similar …If If two different space time blocks are similar …

Then their continuations are also Then their continuations are also similar!!!!similar!!!!Then their continuations are also Then their continuations are also similar!!!!similar!!!!



Aligned space time volume and space time block

ttxxk-1k-1

Assumption :Assumption :

we have already computed the aligned data up to k-1.we have already computed the aligned data up to k-1.

xx

yy

tt tt

k-1k-1



Aligned space time volume and space time block

ttxxk-1k-1

Space time block is same as Space time block is same as a 3D window.a 3D window.

xx

yy

tt tt

k-1k-1



Estimating k-th frame (Extrapolation)

Using previous Using previous aligned time volumealigned time volume

IIestest (x,y) (x,y)

IIestest : Estimated image for k-th frame : Estimated image for k-th frame

If you want to predict If you want to predict the value of x, y on Ithe value of x, y on Iestest……If you want to predict If you want to predict the value of x, y on Ithe value of x, y on Iestest……




Space time block is assigned to target (x,y) position Space time block is assigned to target (x,y) position

of k-1 frame .of k-1 frame .

k-1k-1xx

yy

tt

xx

tt

k-1k-1

7x7x7 7x7x7

space space

time blocktime block

7x7x7 7x7x7

space space

time blocktime block




Matching all space time block of space time volumeMatching all space time block of space time volume

between k-2 frame and initial framebetween k-2 frame and initial frame

k-1k-1xx

yy

tt

xx

tt

k-1k-1




Matching cost : SSD (sum of square differences)Matching cost : SSD (sum of square differences)

2

,,

,,,,, tyx

dstsrcdstsrc tyxtyxd WWWW

W : space-time blockW : space-time blockW : space-time blockW : space-time block




If you found out best matched W…If you found out best matched W…

~ Intensity value of next frame is assigned to the value ~ Intensity value of next frame is assigned to the value of wanted position on k-th frame. of wanted position on k-th frame.

xx

tt

k-1k-1 kk



Registration k-th frame

Matching between real k-th frame and estimated k Matching between real k-th frame and estimated k frame…frame…

IIestest (x,y) (x,y)IIcapcap (x,y) (x,y)



Updating space time volume

xx

tt

kk

IIcapcap (x,y) (x,y) of k-th frameof k-th frame


Video Alignment Video Alignment using Video Extrapolationusing Video ExtrapolationMasking Unpredictable region

There exist some unpredictable region… Raising hand, changing direction…

1

,,

,,

2

2

2

d

d

Wyx

W

EstCap

yxIyxI

yxIyxI

W2d : 2D window

Ix, Iy : color differences with neighborhood

W2d : 2D window

Ix, Iy : color differences with neighborhood


Video Alignment Video Alignment using Video Extrapolationusing Video ExtrapolationOverall algorithm

•Obtain The motion of the first k frames by using Obtain The motion of the first k frames by using conventional optical flow.conventional optical flow.•Obtain The motion of the first k frames by using Obtain The motion of the first k frames by using conventional optical flow.conventional optical flow.

•Align all frame in the space time volume.Align all frame in the space time volume.•Align all frame in the space time volume.Align all frame in the space time volume.

•Compute the motion parameters between captured and Compute the motion parameters between captured and estimated frames.estimated frames.•Compute the motion parameters between captured and Compute the motion parameters between captured and estimated frames.estimated frames.

•Estimate the next new frame by extrapolation from the Estimate the next new frame by extrapolation from the previous frames.previous frames.•Estimate the next new frame by extrapolation from the Estimate the next new frame by extrapolation from the previous frames.previous frames.


Evolving time frontEvolving time frontWhat is evolving time front?

For making wide screen movie and editing videos

tt

xx

tt

xx

kk

Stationary CameraStationary Camera Panning CameraPanning Camera


Evolving time frontEvolving time frontStationary camera case

New video are created by editing time front

tt

xx

Stationary CameraStationary Camera

tt

xx

Stationary CameraStationary Camera








Evolving time frontEvolving time frontPanning camera case

New wide FOV video are created by editing time front as follows…

tt

xx

Panning CameraPanning Camera

tt

xx





tt

xx





tt

xx






Time front optimizationTime front optimizationWhat is the problem of time front ?

There exist some artifacts from seams at the middle of moving objects.

Linear stitchingLinear stitching Using Min-cutUsing Min-cut


Time front optimizationTime front optimizationWhat is the problem of time front ?

There exist some artifacts from seams at the middle of moving objects.

tt

xx

tt

xx

Linear stitchingLinear stitching Using Min-cutUsing Min-cut


Time front optimizationTime front optimizationSingle Time Front as 3D Min-cut

Assumtion

SSll (x,y) = V(x,y, M (x,y) = V(x,y, Mkk(x,y) )(x,y) )SSll (x,y) = V(x,y, M (x,y) = V(x,y, Mkk(x,y) )(x,y) )

tt

xx

tt

xxSource space time volumeSource space time volumeTarget space time volumeTarget space time volume

kkllSSll MMkk

VV



Cost function

E(M) = EE(M) = Eshapeshape(M) + (M) + ααEEstitch-3dstitch-3d(M)(M)

αα : balances between the two E : balances between the two E

((α α =0.3 when gray values were between 0~255)=0.3 when gray values were between 0~255)



Cost function


tt

xxSource space time volumeSource space time volume

kkM’M’

kk

VV

EEshapeshape(M)(M)

MMkk

EEshapeshape(M(Mkk)=)=ΣΣ D( D(M’M’k k , , MMkk))

M’M’kk :: Predefined(or user defined) Predefined(or user defined)

~~~~ time front function time front function

MMkk: : Desired time front functionDesired time front function



Cost function


tt

xx

kk

VV

MMkk

EEstitch-3dstitch-3d(M) (M)

EEstitch-3dstitch-3d(M)=(M)=ΣΣ(x,y)(x,y) Σ Σ(x’,y’)(x’,y’)ЄЄ N(x’,y’) N(x’,y’)

ΣΣ(M(x,y)~(M(x,y)~kk~M(x’,y’))~M(x’,y’)) 0.5*ssd(0.5*ssd(VV (x,y,k)(x,y,k),, V(x,y,k+1) V(x,y,k+1)))

+0.5*ssd(+0.5*ssd(VV (x’,y’,k)(x’,y’,k),, V(x’,y’,k+1)V(x’,y’,k+1)))



Cost function

Preserve user defined time front function

Preserve the color for each pair of spatial neighboring output pixels (x,y) and (x’,y’).


EEshapeshape(M)(M)

EEstitch-3dstitch-3d(M) (M)



We can just apply cost function to Min-cut. We can make seamless video with 3D Min-cut . In order to accomplish more reliable results, We need to consider relation between past and

future.


Time front optimizationTime front optimizationEovolving Time Front as 4D Min-cut

Cost function

Temporal consistency of M in both past and future.


EEtemporaltemporal(M) (M)

EEstitch-4dstitch-4d(M)=(M)= Σ Σll EEstitch-3dstitch-3d(M)(M)

++ΣΣ(x,y,t)(x,y,t) E Etemporaltemporal(x,y,t)(x,y,t)



Cost function

Preserving Temporal Consistency


EEtemporaltemporal(x,y,t)= (x,y,t)= ΣΣ((MMt(x,y)~t(x,y)~kk~~MMt+1(x,y))t+1(x,y))

0.5*ssd(0.5*ssd(VV (x,y,k-1)(x,y,k-1),, V(x,y,k) V(x,y,k)))

+0.5*ssd(+0.5*ssd(VV (x’,y’,k)(x’,y’,k),, V(x’,y’,k+1)V(x’,y’,k+1)))



We can just apply cost function to Min-cut. We can make seamless video with 4D Min-cut .

StitchStitch StitchStitch

ShapeShape ShapeShape TemporalTemporal

3D Min-cut3D Min-cut 4D Min-cut4D Min-cut


Final resultsFinal results


Final resultsFinal results


ConclusionConclusionVideo alignment

An initial task of DynomosaicingEvolving time front

For making wide screen movie and editing videosTime front optimization : Graph-cut (Min-cut)

For seamless dynamosaicing

DiscussionDiscussionContribution

• Found out new interesting application.• Proposed new constancy for image registration. • Introduced various video editing with simple idea.• Applied Min-cut algorithm to overcome some artifact.



Cost function

M(x,y) : Desired time front functionM(x,y) : Desired time front function

M’(x,y) : Predefined(or user defined) time front M’(x,y) : Predefined(or user defined) time front functionfunction

|| || : l|| || : l11 norm norm


yx

shape yxMyxMME,

,,



Cost function


2

21),(

),(

1,,,,2

1

1,,,,2

1,,,

kyxVkyxV

kyxVkyxVyxyxEyxM

yxMkspartial

yx yxNyxdstitch yxyxEME

, ),(),(3 ,,,

M(x,y) <= M(x’, y’)M(x,y) <= M(x’, y’)M(x,y) <= M(x’, y’)M(x,y) <= M(x’, y’)



Cost function

Temporal consistency of M in both past and future.


tyxtemporal

lldstitchdstitch tyxEMEME

,, frameoutput 34 ,,

EEtemporaltemporal(M) (M)



Cost function

Preserving Temporal Consistency


2

21,

1,

2,,1,,2

1

1,,,,2

1,,

1

kyxVkyxV

kyxVkyxVtyxEyxM

yxMktemporal

t

t

MMtt(x, y) <= M(x, y) <= Mt+1t+1(x, y)(x, y)MMtt(x, y) <= M(x, y) <= Mt+1t+1(x, y)(x, y)

Date post:	16-Jan-2016
Category:	Documents
Upload:	martina-craig
View:	214 times
Download:	0 times

Dynamosaicing Dynamosaicing Mosaicing of Dynamic Scenes 2007. 11. 16 (Fri) Young Ki Baik Computer...

Documents