Date post: | 03-Jul-2015 |
Category: |
Technology |
Upload: | xavier-giro |
View: | 304 times |
Download: | 1 times |
AUTOMATIC KEYFRAME SELECTION
BASED ON
MUTUAL REINFORCEMENT ALGORITHM
C. Ventura, X. Giró-i-Nieto, V. Vilaplana et al
1. Introducing the problem
Automatic selection of the representative
keyframe
2
1.1. What is the application?3
1.2. Current implementation4
ARBITRARY SAMPLING
MANUAL SELECTION
BY
PROFESSIONAL
2. Designing the system
2 scenarios:
Intra-clip mode
Inter-clip mode
Database
Textual search
to retrieve
related videos
5
2.1. General scheme6
Reranking
Frame
extraction
visual
features
Textual
search
Similarity
graph
Mutual
Reinforcement
Inter-clip
mode
21
3
12
3
2.2. Intra-clip mode
Mutual Reinforcement Algorithm (Joshi04)
Gets the frame with maximum coverage
(Joshi04) D. Joshi et al. The story picturing engine: finding elite
images to illustrate a story using mutual reinforcement. In MIR ‘04
7
2.3. Inter-clip mode
Reranking (based on Liu11)
(Liu11) C. Liu et al. Query sensitive dynamic web
video thumbnail generation. In ICIP ‘11
8
INTRA INTER
INPUT FRAMESREPRESENTATIVE KEYFRAMES
FROM TEXTUAL SEARCHER
FRAME HAAR WAVELET MORPHOLOGY
2.4. Post-processing block
9 Text filtering
Goal: To avoid representative keyframes with
textual captions
3. Experiments
Qualitative evaluation
Quantitative evaluation
MOS Test
Experimental dataset
10
3.1. Qualitative evaluation
Intra-clip mode
11
3.1. Qualitative evaluation
Inter-clip mode
Ranking scores
after mutual
reinforcement
(INTRA-CLIP
MODE)
Representative
keyframes of the
retrieved videos
12
3.1. Qualitative evaluation
Inter-clip mode
Final ranking scores after reranking:
13
3.2. Quantitative evaluation
MOS (Mean Opinion Score) test
Performed by TVC professionals
Scores
14
EXCELLENTGOODBAD
NON
ACCEPTABLEACCEPTABLE
NEWS DOMAIN
3.2. Quantitative evaluation
Database
15
POLITICS
INTERNATIONAL
ECONOMY
MORNING SHOW DOMAIN
INTERVIEW
DISCUSSION
3.2. Quantitative evaluation
MOS test
4 different approaches
Intra-clip
Inter-clip
Random
Current
16
3.2. Quantitative evaluation17
NEWS
MORNING
SHOW
3.2. Quantitative evaluation18
NEWSMORNING
SHOW
3. Experiments
Database and results are available on:
imatge.upc.edu
19
4. Conclusions
Keyframe selection based on mutual reinforcement algorithm To get the frame with maximum coverage within the
video in the intra-clip approach
Inter-clip approach Textual similarity to retrieve related videos
Linear fusion to get the new ranking scores
MOS test The semi-manual system (from TVC) can be
replaced by the automatic approach.
Inter-clip approach outperforms intra-clip in controlled environments.
20
21
2.3. Inter-clip mode
Textual search
2 modalities:
Textual searcher binary
TF-IDF descriptors
22