Date post: | 24-Dec-2015 |
Category: |
Documents |
Upload: | nelson-cannon |
View: | 216 times |
Download: | 2 times |
Webcam-synopsis:Webcam-synopsis:Peeking Around the WorldPeeking Around the World
Young Ki Baik (CV Lab.)Young Ki Baik (CV Lab.)2008. 4. 4 (Fri)2008. 4. 4 (Fri)
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
ReferencesReferencesWebcam Synopsis : Peeking Around the World
• Yael Pritch, Alex Rav-Acha, Avital Gutman Shmuel Peleg• (ICCV 2007)
Non-Chronological Video Synopsis and Indexing• Yael Pritch, Alex Rav-Acha, Shmuel Peleg (PAMI 2008)
Background Cut• J. Sun, W. Zhang, X. Tang, and H. Shum (ECCV 2006)
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
What is Video-Synopsis?What is Video-Synopsis?
Video synopsis is compact video data reducing spatio-temporal redundancy in input video.
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
Demo (Final result)Demo (Final result)
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
How can we make Video Synopsis?How can we make Video Synopsis?Basic concept
• Detect activity tubes… in input video I.
• Put activity tubes… to the S… considering relation between - tubes and background. - tube and other tubes.
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
SynopsisSynopsisProperties of video synopsis S
• The video synopsis S should be substantially shorter than the original video I.
• Maximum “activity” from the original video should appear in the synopsis video.
• The dynamics of the objects should be preserved in the synopsis video. (for example, regular fast-forward may fail to preserve the dynamics of fast objects.)
• Visible seams and fragmented objects should be avoided.
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
OBJECT-BASED OBJECT-BASED SYNOPSISSYNOPSIS
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
Object-based synopsis constructionObject-based synopsis constructionOnline Phase
• This phase is done in real time during video capture.• Preparing step for response phase.
Response Phase • started with user query.• This phase may take a few minutes, depending on
the activity in the time period of interest.
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
Overall flowOverall flow
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
ONLINE PHASEONLINE PHASE
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
Video AcquisitionVideo AcquisitionVideo Volume Data
y
t
x
I(x, y, t) : 3D Space Time Volume
(x, y) : Spatial coordinates of
the image
t : Frame index
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
Background Video ConstructionBackground Video ConstructionTemporal Median
y
t
x
Sorting by intensity
Selecting median value
4 minutes
Temporal Window
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
Background Video ConstructionBackground Video Construction
y
t
x
y
t
xI B
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
Moving Objects(tubes) ExtractionMoving Objects(tubes) ExtractionUsing Background Cut (ECCV 2006)
• Using min-cut to get a smooth segmentation of foreground objects.
• Only considering 2D image (not video).
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
Moving Objects(tubes) ExtractionMoving Objects(tubes) ExtractionExtracting moving objects
• We have information for…
I
I(x, y, t) : 3D Space Time Volume
B(x, y, t) : 3D Space Time Volume for background
B
Current ~
: Current image = I (x, y, current time)
: Current background image = B (x, y, current time)
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
Moving Objects(tubes) ExtractionMoving Objects(tubes) ExtractionSimplify Object detection problem to…
• Binary labeling f : Foreground = 1, Background = 0 • Obtaining the desirable labeling f … → with Gibbs energy function.
1 2,
( ) ,r r sr I r s Neighbors
E f E f E f f
: target position of image
: neighboring of target position (4 Neighbors)
: user defined weight
r
s
Color term Contrast term
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
Moving Objects(tubes) ExtractionMoving Objects(tubes) ExtractionExtracting moving objects
• Unary term (or color term)
1 2,
( ) ,r r sr I r s Neighbors
E f E f E f f
d I r B r
11
1
01
d kE
k d otherwise
2
1 1 2 10 -
0
d k
E d k k d k
otherwise
1 2, : user defined thresholdsk k
Foreground energy Background energy
Color differences between image and background
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
Moving Objects(tubes) ExtractionMoving Objects(tubes) ExtractionUsing Background Cut (ECCV 2006)
• Using min-cut to get a smooth segmentation of foreground objects.
• Only considering 2D image (not video).
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
Moving Objects(tubes) ExtractionMoving Objects(tubes) ExtractionExtracting moving objects
• Binary term (or contrast term)
1 2,
( ) ,r r sr I r s Neighbors
E f E f E f f
: user defined weight
2
2 ( , )r s r sE f f f f Exp I r I s
In case of same labeling, energy is decreased.In case of different labeling, energy defends on differences between neighboring intensities.
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
Moving Objects(tubes) ExtractionMoving Objects(tubes) ExtractionUsing min-cut algorithm…
- Moving objects are extracted.
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
Moving Objects(tubes) ExtractionMoving Objects(tubes) ExtractionExtracting moving objects
• Constructing a mask of all foreground pixels in space-time volume.
• Applying a 3D morphological dilation on mask.• Finally we can obtain activity tubes (or objects).
x
t
Example of tubes
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
Moving Objects(tubes) ExtractionMoving Objects(tubes) ExtractionObject queue
• Obtained activity tubes are saved in queue.
End of online phase…
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
RESPONSE PHASERESPONSE PHASE
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
Response phaseResponse phaseUser query
• “I would like to watch in one minute a synopsis of the video from this camera captured during the last hour.”
• “I would like to watch in five minutes a synopsis of the last week.”…
Input Input videovideo SynopsisSynopsis
When user query occurred, response phase are started.
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
Response phaseResponse phaseUser query
• “I would like to watch in one minute a synopsis of the video from this camera captured during the last hour.”
• “I would like to watch in five minutes a synopsis of the last week.”…
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
Creating time lapse background videoCreating time lapse background videoTime lapse background video (Bout)
• The background of the synopsis video
• It should represent the background changes over time.• Day-night transitions, etc.
• It should represent the background of the activity tubes.
BackgroundVideo
Time lapseBGvideo
Sampling uniformly
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
Creating time lapse background videoCreating time lapse background videoTime lapse background video
• A temporal activity histogram Ha
• A uniform temporal histogram Ht
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
Creating time lapse background videoCreating time lapse background videoTime lapse background video
• A temporal activity histogram Ha
• A uniform temporal histogram Ht
• Interpolating the two histograms
Hi = Ha + (1- λ)Ht
BackgroundVideo (b)
Time lapseBGvideo
Sampling with Hi
Bout
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
Selecting tubes and stitchingSelecting tubes and stitchingDefinition
Input Input videovideo SynopsisSynopsis
ˆ
M
b
b
b̂b M
: temporal mapping (or time shift)
: tube with time segment
: shifted tube with time segment
,s eb b bt t t
ˆ ˆ ˆ,s eb b bt t t
ˆM b b
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
Select tubes and stitchingSelect tubes and stitchingEnergy for temporal mapping M
ˆM b b
, ,
Q
: Queue
: user selected weights
,
ˆ ˆ
ˆ ˆ ˆ ˆ , ,
a sb Q
t cb b Q
E M E b E b
E b b E b b
b
b: target tube
: other tube
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
Select tubes and stitchingSelect tubes and stitchingGraph
SynopsisSynopsis
t
Longest tube tube tube
tube
tube
Node number = number of tubeNode number = number of tube
Label = number of frame tLabel = number of frame t
label
label
label
label
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
Select tubes and stitchingSelect tubes and stitchingEnergy for temporal mapping M
• Unary term
• Activity Cost
ˆ ˆ
a sb QE b E b
SynopsisSynopsis
ˆx,y,t b
0 tube region inside the Sˆ = , , , , tube region outside the SaE b I x y t B x y t
Only pixel that were not entered into the synopsis are added to the activity cost.
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
Select tubes and stitchingSelect tubes and stitchingEnergy for temporal mapping M
• Unary term
• Consistency with background
ˆ ˆ
a sb QE b E b
SynopsisSynopsis
b̂
b̂
ˆˆx,y b ,t
ˆ = , , , ,out
s outbt t
E b I x y t B x y t
: border of the mapped activity tube
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
Select tubes and stitchingSelect tubes and stitchingEnergy for temporal mapping M
• Binary term
• Collision Cost SynopsisSynopsis
b b
ˆ ˆˆ ˆx,y,t t t
ˆ = , , , , , , , ,c b bE b I x y t B x y t I x y t B x y t
,ˆ ˆ ˆ ˆ, ,t cb b Q
E b b E b b
This expression give a low penalty to pixel whose color is similar to the background…
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
Select tubes and stitchingSelect tubes and stitchingEnergy for temporal mapping M
• Binary term
• Temporal Consistency CostSynopsisSynopsis
ˆtE b
,ˆ ˆ ˆ ˆ, ,t cb b Q
E b b E b b
Preserving the chronological order of events
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
Select tubes and stitchingSelect tubes and stitchingE(M) is minimized by Min-cut algorithm.
Stitching the synopsis video• To make more reliable result… → The α–Poisson Image Blending
End of response phase…
Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World
ConclusionConclusionThe method to creating a short video that is a synopsis
of and video stream has been presented.
DiscussionDiscussionContribution
• Found out new interesting application.Can we find more contribution?
• Finding higher dimensional problem and Solving it. • 3D volume animation, etc.