Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video...

transcript

Video SegmentationTal Kramer, Shai Bagon

Advanced Topics in Computer Vision Spring 2006

Video Segmentation

April 30th, 2006

Introduction

• Recognition and Segmentation• Min Cut Max Flow• Single Image Methods

– GrabCut– Lazy Snapping– …

Lazy Snapping

• Interactive User Interface

Lazy Snapping

• Energy minimization

Lazy Snapping

• Energy minimization

Lazy Snapping

• Boundary overriding

Lazy Snapping

• Boundary overriding

Motivation

• Obvious Next Step• Video Cut & Paste• Video Manipulation and Editing

Introduction

• Frame by Frame– Time Consuming and Tedious

• Error With Simple Methods– Fast motions– Deforming silhouettes – Changing topologies

Introduction

• Two Papers– Video Object Cut

and Paste– Video Cutout

Video Object Cut and Paste

Yin Li, Jian Sun, Heung-Yeung Shum

Overview

Pre-segmentation

• Pre-Segmentation to All Frames

Key Frames

• Picking Key Frames.

Key Frames

• User Fore/Background Segmentation

3D Graph Cut Segmentation

• 3D Graph – G=(V,A)

• Labeling– Foreground = 1 – Background = 0

• Volume Between Successive Key Frames

3D Graph Construction

• 2 Kinds of Arcs:– AI – Intra

Frames (BLUE)

– AT – Inter Frame (RED)

3D Graph Construction

• Minimizing Equation:

• E1 – Global Color Models

• E2 – Penalizing Spatially

• E3 – Penalizing Temporally

Likelihood Energy

• GMMs Decide Label• In Key Frames:

GMM• Gaussian Mixture

• Distance is Measured By:

Prior Energies

• E2, E3 Are the Same

• Distance of Adjacent Regions.

• β = (2 E (||cr – cs||2 ))-1

Prior Energies

• λ1 = 24

• λ2 = 12

3D Graph Segmentation

Errors

• Global Colors• Similarity to

Background• Thin Areas

Error Overriding

• Video tubes• Manual corrections

Video Tubes

• Local Color Models• Put Two Windows• Tracking Algorithm• Key Frames to

Fixing graph cut segmentation

• Minimizing:

Overriding Brush

• Fixing Boundary Manually

Manual error overriding

• Soften hard segmentation

Matting

Coherent Matting

• Boundary is not 0/1• Prevent Bolting Pixels• Smooth Paste

Coherent Matting

Example

Video CutOut

J. WANG, P. BHAT, A. COLBURN, M. AGRAWALA, M. COHEN. Interactive Video Cutout. ACM Trans. on Graphics

(Proc. of SIGGAPH2005), 2005

Video Cutout introduction

What’s new?• Different user interface• 3D graph formation• Refinement mechanism

System overview

3D Graph construction

• Hierarchical graph nodes:1. Frame by frame mean shift

segmentation2. Aggregating segments across

frames

• Pixel 26-neighborhood induce links• Lower level links induce higher level

3D Graph construction

• Stroking foreground and background over the 3D spatio-temporal volume

• Not segmenting any frame

User Interface

• Graph construction– User input propagates upward – Min cut uses yellow nodes

3D Min cut/Max flow

• Weights / Energy function– The energy function:

– Data term: color similarity to F/B model– Link term: cut likelihood

, , , , , ,i i i i j i ji nghbrs i j

E D x c L x x c c

3D Min cut/Max flow

• Terms in energy function

Graph Cut Energy function

L LinkDF ForegroundDB Background

DB,L Pix. history DB,G Color DF,G Color LL Local temporal LG Gradient

Data weight

• User input generates color model (GMM)

• Infinite weight preserves marked pixels

• Data weight = abiding to F/B color model

Data weight

White – high probability ForegroundBlack – Low probability Foreground

Data weight

White – high probability BackgroundBlack – Low probability Background

• Strong gradients segment border

• Link cost encourage cut at edges

Link weight Graph Cut Energy function

Link weight

White – low cut probabilityBlack – high cut probability

• Pixel span: (xo, yo, t)t>0

Data weight Graph Cut Energy function

• Local background model• Assuming camera is stabilized, video is

registered• Extracting “clean plate” • Weight per pixel span

• d(zi) = minimum color distance {“clean plate”, B marked pixel}.

• “Clean plate” cannot be always trusted

• Weight:

NumFramese

Data weight

White – high probability BackgroundBlack – Low probability Background

• Link span: links between two adjacent pixel spans

• Strong edges exists within segment

• Small change over time• Local temporal link cost penalize

strong temporal gradient

Link weight

White – low cut probabilityBlack – high cut probability

• Energy function

3D Min cut/Max flow

E D x c L x x c c

Iterative process

• The user refines the cut• Adds F/B strokes• Graph is re-computed

Nth iterationN+1th iteration

Post Processing

Post processing

• Binary cut obtained• Edges need refinement

• A pixel-level min cut around edges• Color model obtained form

boundary• Uniform edge cost = small

Refinement

Matting

• Soften hard segmentation• Evaluate α Channel

Matting

• Refinement fixed boundary locally

• Global 3D mesh• α Channel along

mesh normals

Results

Performance

• Pros– Online 3D min cut – Spatio temporal smooth cut

• Cons– Does not handle shadows– Ignore motion blur (LPF to avoid

temporal aliasing)– Cannot separate translucent objects

Summary

ComparisonVideo object cut

and pasteVideo Cutout

Features

•Graph nodes

2D segmentation

Frame base interface

3D segmentation in 2 stages

spatial-temporal manipulation

Performance

•Preprocessing•Artist time•Post processing•Total

4-5 min25 min

?30 min

25 min10 sec per Min cut

30 min60 min

Questions?

• The total energy:

• Foreground and background terms:

• Background terms:

• Link terms:

Energy function

E D x c L x x c c

;B Fi i

B F B F

D DD x B D x F

D D D D

2 , 2 ,1B i B L B GD x B D D

3 31i j G LL x x L L 3 0.3

NumFramese

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video...

Documents