Date post: | 21-Dec-2015 |
Category: |
Documents |
View: | 218 times |
Download: | 2 times |
Alan F. SmeatonDublin City University
&Paul Over
NIST
The TREC-2002 Video Track: Overview
2
1. Introduction and Context
– Last year’s talk…– gave an intro to video coding & compression;
– highlighted predominant access mechanism as manual tagging via metadata
– noted emerging automatic approaches are based on shot boundary detection, feature extraction and keyframe identification, followed by feature searching with keyframe browsing
– noted there is no test collection of video
– provided an overview of what 12 groups did on 11 hours of video in shot boundary detection and searching tasks
– Last year was TV101, this year is TV201
3
New this year (1)
– More participants and data: – 17 participating teams (up from 12), – 73 hours (up from 11)
– Shot boundary determination (SBD)– new measures– 3-week test window
– New semantic feature extraction task– features defined jointly by the participants– task is to identify shots with those features
– Several groups donated extracted features– identified features from test videos early– shared their output (in MPEG-7 defined by IBM) in time for
others to use as part of their search systems
4
New this year (2)
– 25 topics for the search task, – developed by NIST – 4 weeks between release and submission– text, video, image and/or audio
– Average precision added as measure – new emphasis on ranking
– A common set of shot definitions– donated by CLIPS-IMAG, formatted by DCU– common units of retrieval for feature and search tasks– allowed pooling for assessment
5
New this year (3)
– Searching was:– Interactive: full human access and iterations, or – Manual: a human with no knowledge of the test data gets
one shot to formulate the topic as a search query– No fully automatic topic-to-query translation
– Elapsed search time was added as a measure of effort for interactive search, groups gathered data on searcher characteristics
Shots Features
SBD Feature Extr. Searching
6
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China) X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
The 17 groups and the tasks they completed
7
Video Data
– Difficult to get video data for use in TREC because ©
– Used mainly Internet Archive– advertising, educational, industrial, amateur films 1930-
1970 – produced by corporations, non-profit organisations,
trade groups, etc.– Noisy, strange color, but real archive data– 73.3 hours partitioned as follows:
4.85
5.07
23.26
40.12Search test
Feature development(training and validation)
Feature test
Shot boundary test
8
2. Shot Boundary Detection task
– Not a new problem, but a challenge because of gradual transitions and false positives caused by photo flashes, rapid camera or object movement
– 4 hours, 51 minutes of documentary and educational material
– Manually created ground truth of 2,090 transitions (thanks Jonathan) with 70% hard cuts, 25% dissolves, rest are fades to black and back, etc.
– Up to 10 submissions per group, measured using precision and recall, with a bit of flexibility for matching gradual transitions
9
2001: Recall and precision for cuts
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Recall
Pre
cis
ion
CLI PSFudanI BMI CMKMMSRADCUJ HUAPLMB_ FrequencyMediaMillUMDLAMP
10
2002: Recall and precision for cuts
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Recall
Pre
cis
ion
CLI PS (10)Fudan (10)I BM (5)I CMKM (6)MSRA (10)TZI -Ubremen (1)NUS (2)RMI T (10)The data must have
gotten a little harder?
11
2001: Gradual Transitions
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Recall
Pre
cis
ion
CLI PSFudanI BMI CMKMMSRADCUJ HUAPLMB_ FrequencyMediaMillUMDLAMP
12
2002: Gradual Transitions
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Recall
Pre
cis
ion
CLI PS (10)Fudan (10)I BM (5)I CMKM (6)MSRA (10)TZI -Ubremen (1)NUS (2)RMI T (10)
Still room for improvement.Precision/recall knobs working for some systems
13
2001: Frame-recall & -precision for GTs
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Recall
Pre
cis
ion
CLI PSFudanI BMI CMKMMSRADCUJ HUAPLMB_ FrequencyMediaMillUMDLAMP
14
2002: Frame-recall & -precision for GTs
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Recall
Pre
cis
ion
CLI PSFudanI BMI CMKMMSRATZI -UbremenNUSRMI T
So, who did what ?
The approaches….
16
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China) X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
CLIPS-IMAG (Fr):
Refined 2001 SBD system which is based on frame comparisons, filters photo “flashes”, several runs with parameters varied
Shot Boundary Detection:
17
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China) X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
Fudan University (China):
An update on their 2001 SBD based on frame-frame comparisons, also filters photo flashes and has fade-in/-out detection
Shot Boundary Detection:
18
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China) X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
Shot Boundary Detection:
IBM Research (US):
Presentation to follow
19
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China) X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
Shot Boundary Detection:
Imperial College London (UK):
Frame comparison based on colour histograms, extended for gradual transitions, and also addressing photo flashes
20
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China) X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
Shot Boundary Detection:
Microsoft Research Asia (China):
Based on 2001 SBD but refined to address gradual transitions … also based on frame differences
21
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China) X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
Shot Boundary Detection:
National Univ. of Singapore:
Used wavelet coefficients to detect potential transitions and filtered for flashbulbs, object and camera motion, with an adaptive threshold for different video types
22
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China) X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
Shot Boundary Detection:
RMIT University (Aus.):
Used each frame as a query to a window of frames, and based on rank positions of other frames in this window, did SBD
23
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China) X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
Shot Boundary Detection:
University of Bremen (D):
Another based on comparing frames using histograms, with adaptive thresholding
24
3. Feature Extraction
– FE is – interesting itself but when it serves to help video
navigation and search then its importance increases
– Objective was to – begin work on benchmarking FE – allow exchange of feature detection output among
participants
– Task is as follows: – given small standard dataset (5.02 hours, 1,848 shots)
with common shot bounds, – locate up to 1,000 shots for each of 10 binary features– Feature frequency varied from “rare” to “everywhere”
25
The Features
1. Outdoors2. Indoors3. Face - 1+ human face with nose, mouth, 2 eyes4. People - 2+ humans, each at least partially visible5. Cityscape - city/urban/suburban setting6. Landscape - natural inland setting with no human
development such as ploughing or crops7. Text Overlay - large enough to be read8. Speech - human voice uttering words9. Instrumental Sound - 1+ musical instruments10. Monologue - 1 person, partially visible, speaking for a
long time without interruption
26
05
1015202530
True shots contributed uniquely by each run
– Small values imply lots of overlap between runs– Likely due to relative size of result set (1,000 shots) and total
test set (1,848 shots)
27
AvgP by feature (runs at median or above)
0
0.2
0.4
0.6
0.8
Feature
Avera
ge p
recis
ion
CMU_ r1
A_ CMU_ r2
CLIPS-LIT_ GEOD
CLIPS-LIT-LIMSU
DCUFE2002
Eurecom1
Fudan_ FE_ Sys1
Fudan_ FE_ Sys2
IBM-1
IBM-2
MediaMill1
MediaMill2
MSRA
UnivO_ MT1
UnivO_ MT2
Avg Prec Cap
Outd
oors
Indo
ors
Face
Peop
le
City
scap
e Land
sca
pe Text
ov
erla
ySp
eec
h
Inst
rum
enta
l
so
und
Mon
olo
gRandom baseline
28
Groups and Features
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China) X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
Outd
oors
In
doors
F
ace
People
Cit
ysca
pe
L
andsc
ape
T
ext
ove
rlay
Speech
M
usi
c
M
onolo
gue
29
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China) X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
Groups and Features
Carnegie Mellon University (US):
Hand-label feature training data, extract low-level image or audio features and combine in a SVM training process
Outd
oors
In
doors
F
ace
People
Cit
ysca
pe
La
ndsc
ape
T
ext
Ove
rlay
Speech
M
usi
c
M
onolo
gue
30
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China) X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
Groups and Features
CLIPS-IMAG (Fr):
4 features, face/people using CMU’s publicly available tool, speech/monologue using LIMSI’s ASR transcript
Outd
oors
In
doors
F
ace
People
Cit
ysca
pe
La
ndsc
ape
T
ext
Ove
rlay
Speech
M
usi
c
M
onolo
gue
31
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China) X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
Groups and Features
Dublin City University (Irl):
Speech/music detection based on energy peaks in encoded bitstream, face detection based on skin masks
Outd
oors
In
doors
F
ace
People
Cit
ysca
pe
La
ndsc
ape
T
ext
Ove
rlay
Speech
M
usi
c
M
onolo
gue
32
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China) X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
Groups and Features
Fudan University (China):
Colour histograms and edge direction histograms, trained on sample frames for I/O/C/L; face/people based on skin colour, motion and shape, text block detection and audio detection based on audio features
Outd
oors
In
doors
F
ace
People
Cit
ysca
pe
La
ndsc
ape
T
ext
Ove
rlay
Speech
M
usi
c
M
onolo
gue
33
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China) X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
Groups and Features
IBM Research (US):
Presentation to followO
utd
oors
In
doors
F
ace
People
Cit
ysca
pe
La
ndsc
ape
T
ext
Ove
rlay
Speech
M
usi
c
M
onolo
gue
34
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China) X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
Groups and Features
Institut Eurecom (Fr):
Visual feature detection based on classifying 16x16 pixel macroblocks, directly from MPEG encoding
Outd
oors
In
doors
F
ace
People
Cit
ysca
pe
La
ndsc
ape
T
ext
Ove
rlay
Speech
M
usi
c
M
onolo
gue
35
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China) X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
Groups and Features
Mediamill/TNO (NL):
Feature detection on all features using an active learning technique
Outd
oors
In
doors
F
ace
People
Cit
ysca
pe
La
ndsc
ape
T
ext
Ove
rlay
Speech
M
usi
c
M
onolo
gue
36
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China) X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
Groups and Features
Microsoft Research Asia (China):
Operated on multiple keyframes per shot, visual feature identification used colour moments and edge direction histograms; audio feature detection based on SVM with inputs from low-level audio analysis
Outd
oors
In
doors
F
ace
People
Cit
ysca
pe
La
ndsc
ape
T
ext
Ove
rlay
Speech
M
usi
c
M
onolo
gue
37
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China) X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
Groups and Features
University of Bremen (D):
Indoor/outdoor classifier based on colour distributions, input into a neural network
Outd
oors
In
doors
F
ace
People
Cit
ysca
pe
La
ndsc
ape
T
ext
Ove
rlay
Speech
M
usi
c
M
onolo
gue
38
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China) X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
Groups and Features
Univ. Md./INSA/Univ. Oulu (US):
Used text overlay detector from INSA de Lyon
Outd
oors
In
doors
F
ace
People
Cit
ysca
pe
La
ndsc
ape
T
ext
Ove
rlay
Speech
M
usi
c
M
onolo
gue
39
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China) X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
Groups and Features
Univ. Oulu/VTT (Fin):
Edge detection gradients for city/landscape, skin detector for faces, speech/music from low-level audio analysis
Outd
oors
In
doors
F
ace
People
Cit
ysca
pe
La
ndsc
ape
T
ext
Ove
rlay
Speech
M
usi
c
M
onolo
gue
40
4. The Search Task
– Task is similar to text analogue … – topics are formatted descriptions of an information need – task is to return up to 100 shots that meet the need
– Test data: 40.12 hours (14,524 common shots)– Features and/or ASR donated by CLIPS, DCU, IBM,
Mediamill and MSRA– NIST assessors
– judged top 50 shots from each submitted result set– subsequent full judgements showed only minor variations
in performance
– Used trec_eval to calculate measures
41
Search Topics
– Topics (25) multimedia, created by NIST– 22 had video examples (avg 2.7 each), 8 had
image (avg 1.9 each)– Requested shots with specific/generic:
– People: George Washington; football players– Things: Golden Gate Bridge; sailboats– Locations: ---; overhead views of cities– Activities : ---; rocket taking off– Combinations of the above:
• People spending leisure time at the beach• Locomotive approaching the viewer• Microscopic views of living cells
42
Search Types: Interactive and Manual
43
Manual runs: Top 10 (of 27)
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Recall
Pre
cisi
on
Prous Science
IBM-2
CMU_ MANUAL1
IBM-3
LL10_ T
CLIPS+ASR
Fudan_ Search_ Sys4
CLIPS+ASR+X
ICMKM-2
UMDMqtrec
44
Interactive runs top 10 (of 13)
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Recall
Pre
cisi
on
CMUInfInt1
DCUTrec11B.1
DCUTrec11C.2
CMU_ INTERACTIVE_ 2
UnivO_ MT5
IBM-4
DCUTrec11B.3
DCUTrec11C.4
UMDIqtrec
MSRA.Q-Video.2a
Prous Science
IBM-2
CMU_MANUAL1
45
Mean AvgP vs mean elapsed time
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0
5
10
15
20
25
30
IuVf1CMUInfIn
t1CMU_IN
TERACTIVE
DCUTrec11
B.
1DCUTrec
11B.
3DCUTrec
11C.
2DCUTrec
11C.4
IBM-4
ICMKM-1
MSRA.Q-
Video.1
MSRA.Q-V
ideo.2.
A
UMDIqtrec
UNivO_M
T5
Mean average precision
Mean elapsed time (mins.)
Wide variation in elapsed time.Not the dominant factor in effectiveness
46
Search: Unique relevant shots from each run
010203040506070
Interactive runs contributed most as expected.
47
Distribution of relevant shots
Top vs bottom of halves of result sets
1
51
101
151
201
251
301
351
Topic
Re
lev
an
t sh
ots
Bottomhalf
Tophalf
Not many additional relevant found in bottom half of result sets except for topics with a lot already.
48
Max/median AvgP by topic - interactive
0
0.2
0.4
0.6
0.8
1
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
Topic
Maximum
Median
0
50
100
150
200
250
300
350
Relevantshots
0
10
20
30
40
50
60
70
Relevantvideos
49
0
0.2
0.4
0.6
0.8
1
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
Topic
Maximum
Median
Why better median performance on topics 76, 84, 90, 97 ?
No single, simple explanation… e.g., topics with more relevant shots or videos, or those with example(s) from the search test collection aren’t necessarily “easier”
0
50
100
150
200
250
300
350
Relevantshots
0
10
20
30
40
50
60
70
Relevantvideos
Max/median AvgP by topic - manual
50
Relevant shots by file id (topics 75-87)
16
111621263136414651566166717681869196
101106111116121126131136141146151156161166171
1 6
11
16
21
26
31
36
41
46
51
56
61
66
71
76
81
86
91
96
10
1
10
6
11
1
11
6
12
1
12
6
13
1
13
6
14
1
14
6
15
1
15
6
16
1
16
6
17
1
Shot number in file
Vid
eo
file
id
75767778798081828384858687
Relevant shots are adjacent in many but not all cases … Local browsing among video shots can be important
51
Relevant shots by file id (topics 88-99)
16
111621263136414651566166717681869196
101106111116121126131136141146151156161166171176
1 6
11
16
21
26
31
36
41
46
51
56
61
66
71
76
81
86
91
96
10
1
10
6
11
1
11
6
12
1
12
6
13
1
13
6
14
1
14
6
15
1
15
6
16
1
16
6
17
1
Shot number in file
Vid
eo
file
id
888990919293949596979899
52
The Groups and Searching
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China) X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
53
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China) X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
The Groups and Searching
Carnegie Mellon University (US):
Modified Informedia to include TREC features and multiple image search engines with an experienced user for interactive search; manual search combined ASR, OCR and image matching with pseudo relevance feedback and query expansion
54
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China) X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
The Groups and Searching
CLIPS-IMAG (Fr):
3 manual runs using (a) donated features, (b) LIMSI transcript (c) both
55
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China) X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
The Groups and Searching
CWI Amsterdam (NL):
Presentation to follow
56
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China) X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
The Groups and Searching
Dublin City University (Irl):
Incorporated a donated ASR transcript, 7 donated features + 3 of their own features into the Fischlar video retrieval system, and ran interactive search with 12 users.
57
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China) X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
The Groups and Searching
Fudan University (China):
3 manual runs combining similarities from face recognition (for face topics), text recognition, image similarity and ASR transcript; used their own + donated features, LIMSI transcript, and some topics only.
58
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China) X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
The Groups and Searching
IBM Research (US):
Presentation to follow
59
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China) X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
The Groups and Searching
Imperial College (UK):
Query matched against 1 keyframe per shot using colour features and thumbnail placement relevance feedback for interactive search
60
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China) X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
The Groups and Searching
Indiana University (US):
Interactive search using ViewFinder video IR system
61
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China) X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
The Groups and Searching
Microsoft Research Asia (China):
used M/S Q-Video system including the 10 semantic features and 9 other low-level visual features, plus ASR; automatic shot matching and ranking with user varying the weights for different features.
62
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China) X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
The Groups and Searching
Proust Research (Spain):
Presentation to follow - one manual run which realised best or near-best performance for 18 of 25 topics !
63
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China) X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
The Groups and Searching
U.Md./INSA/U. Oulu (US):
Used donated ASR and OCR text, 8 donated features and colour correlogram matching of image vs. keyframe for manual runs with multi- dimensional browsing tool for interactive runs
64
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China) X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
The Groups and Searching
Univ. Oulu/VTT (Fin):
Also used donated ASR and OCR text, 8 donated features and their VIRE system with multi-modal indexing based on SOMs, and 8 users using an interactive video navigation tool
65
Groups doing the “Full Monty”
Carnegie Mellon U. (US)
CLIPS-IMAG (Fr) X
CWI Amsterdam (NL)
Dublin City University (Irl)
Fudan Univ. (China) X
IBM Research (US) X
Imperial College London (UK) X
Indiana University (US)
Institut Eurecom (Fr)
Mediamill/U Amsterdam (NL)
Microsoft Research Asia (China)X
National Univ. Singapore (Sing.) X
Prous Science (Esp)
RMIT University (Aus) X
Univ. Bremen (D) X
U. Maryland/INSA/U. Oulu (US)
Univ. Oulu/VTT (Fin)
Feature Search
1 2 3 4 5 6 7 8 9 10 Int. Man.
X X X X X X X X X X X
X X X X X
X
X X X X
X X X X X X X X X X X
X X X X X X X X X X X X
X X
X
X X X X X X X
X X X X X X X X X X
X X X X X X X X X X X X
X
X X
X X X
X X X X X X X
Shot Bound
66
5. Conclusions
– This track has grown significantly… data, groups, tasks, measures, complexity
– Donated features enabled many sites to take part and greatly enriched the progress .. this cannot be overstated … very collegiate and beneficial all-round
– Common shot definition – implications for measurement need closer look – seems it was successful
– The search task is becoming increasingly interactive, and we could do with guidance here
– Evaluation framework has settled down – should be repeated on new data with only minor adjustments
– Need more data (especially for feature extraction), more topics – looking at 120 hours of news video from 1998
– Need to encourage progress on manual/automatic processing – how? focus evaluation more?
– Probably ready to become one-day pre-TREC workshop with report-out/poster at TREC
67