deep
er
IBM
T.J
. Wat
son
Res
earc
h C
ente
r
©C
opyr
ight
IBM
Cor
pora
tion
2006
Inte
llige
nt In
form
atio
n M
anag
emen
t Dep
t.IB
M T
hom
as J
. Wat
son
Res
earc
h C
ente
r
Con
tact
: Pau
l Nat
sev
<nat
sev@
us.ib
m.c
om>
IBM
Mar
vel f
or T
REC
VID
06
Aut
omat
ic S
earc
h
This
mat
eria
l is
base
d up
on w
ork
fund
ed in
par
t by
the
U.S
. Gov
ernm
ent.
Any
opi
nion
s, fi
ndin
gs a
nd c
oncl
usio
ns o
r re
com
men
datio
ns e
xpre
ssed
in th
is m
ater
ial a
re th
ose
of th
e au
thor
(s) a
nd d
o no
t nec
essa
rily
refle
ct th
e vi
ews
of th
e U
.S.
Gov
ernm
ent.
Nov
embe
r 14t
h , 20
06G
aith
ersb
urg,
MD
IBM
Mar
vel f
or T
RE
CV
ID06
Aut
omat
ic S
earc
h | N
ov. 1
4th,
200
6
IBM
T.J
. Wat
son
Res
earc
h C
ente
r
©C
opyr
ight
IBM
Cor
pora
tion
2006
Ack
now
ledg
men
tsIB
M R
esea
rch
Inte
lligen
t Inf
orm
atio
n M
anag
emen
t Tea
m-
Apo
stol
(Pau
l) N
atse
v-
Jele
naTe
sic
-Jo
hn R
. Sm
ith-
Lexi
ngX
ie-
Milin
dN
apha
de-
Mur
ray
Cam
pbel
l-
Quo
c-B
aoN
guen
-S
hahr
amEb
adol
lahi
-A
lex
Hau
bold
(Col
umbi
a U
.)-
Dhi
rajJ
oshi
(Pen
n. S
tate
)-
Joac
him
Sei
dl(U
. Kla
genf
urt,
Aus
tria)
IBM
Res
earc
h K
now
ledg
e S
truct
ures
Gro
up (P
IQU
AN
T-II
Q&
A s
yste
m)
-Je
nnife
r Chu
-Car
roll,
Joh
n P
rage
r, P
ablo
Dub
oue
Col
umbi
a U
nive
rsity
(Sto
ry b
ound
arie
s)-
Lynd
on K
enne
dy, W
inst
on H
su, S
hih-
Fu C
hang
Nat
iona
l Uni
vers
ity o
f Sin
gapo
re (P
hras
e-al
igne
d an
d sp
eake
r-alig
ned
trans
crip
ts)
-S
hi-Y
ong
Neo
, Tat
-Sen
gC
hua
IBM
Mar
vel f
or T
RE
CV
ID06
Aut
omat
ic S
earc
h | N
ov. 1
4th,
200
6
IBM
T.J
. Wat
son
Res
earc
h C
ente
r
©C
opyr
ight
IBM
Cor
pora
tion
2006
Out
line
Sys
tem
Ove
rvie
wTe
xt-B
ased
Ret
rieva
l V
isua
l Con
tent
-Bas
ed R
etrie
val
Con
cept
Mod
el-B
ased
Ret
rieva
lM
ulti-
mod
al F
usio
nR
esul
ts a
nd C
oncl
usio
ns
IBM
Mar
vel f
or T
RE
CV
ID06
Aut
omat
ic S
earc
h | N
ov. 1
4th,
200
6
IBM
T.J
. Wat
son
Res
earc
h C
ente
r
©C
opyr
ight
IBM
Cor
pora
tion
2006
IBM
Res
earc
h A
utom
atic
Sea
rch
Syst
em O
verv
iew
App
roac
hes:
1.Te
xt-b
ased
: sto
ry-b
ased
retri
eval
with
au
tom
atic
que
ry re
finem
ent/r
e-ra
nkin
g
2.V
isua
l-bas
ed: l
ight
-wei
ght l
earn
ing
(dis
crim
inat
ive
and
near
est n
eigh
bor
mod
elin
g) w
ith s
mar
t sam
plin
g
3.M
odel
-bas
ed: a
utom
atic
que
ry-to
-m
odel
map
ping
bas
ed o
n qu
ery
topi
c te
xt o
r vis
ual e
xam
ples
4.Fu
sion
:•
Que
ry in
depe
nden
t•
Que
ry-c
lass
-dep
ende
nt w
ith s
oft o
r ha
rd c
lass
mem
bers
hip
Visu
al q
uery
exa
mpl
es
Fusi
on
Text
ual q
uery
topi
c
“Fin
d sh
ots
of a
n ai
rpla
ne ta
king
off”
VISU
AL
TEXT
Mul
ti-m
odal
resu
lts(T
e xt +
Vis
ual +
Mod
els)
4
Que
ry
Text
-B
ased
R
etrie
val
Mod
el-
Bas
ed
Ret
rieva
l
Visu
al-
Bas
ed
Ret
rieva
l
12
3
IBM
Mar
vel f
or T
RE
CV
ID06
Aut
omat
ic S
earc
h | N
ov. 1
4th,
200
6
IBM
T.J
. Wat
son
Res
earc
h C
ente
r
©C
opyr
ight
IBM
Cor
pora
tion
2006
IBM
Res
earc
h Te
xt R
etrie
val S
yste
mC
orpu
s In
dexi
ngS
hot-l
evel
AS
R/M
T do
cum
ents
S
tory
-leve
l AS
R/M
T do
cum
ents
Bot
h al
igne
d at
phr
ase
leve
l
Que
ry a
naly
sis:
Toke
niza
tion,
phr
asin
g, s
tem
min
gP
art-o
f-spe
ech
tagg
ing
& fi
lterin
g
Que
ry re
finem
ent:
Pse
udo-
Rel
evan
ce F
eedb
ack
Que
ry e
xecu
tion
and
fusi
onIB
M S
eman
tic S
earc
h E
ngin
e (J
uru)
Usi
ng T
F*ID
F-ba
sed
retri
eval
Fusi
on o
f sho
t-an
d st
ory-
base
d re
sults
Perf
orm
ance
(MA
P):
Sho
ts:
0.
041
Sto
ries:
0.
032
Fuse
d:
0.0
52 (o
ur b
est u
ni-m
odal
run)
See
Vol
kmer
& N
atse
v(IC
ME
200
6)
Text
ual q
uery
topi
c“F
ind
shot
s of
an
airp
lane
taki
ng o
ff”
Que
ry a
naly
sis
Sho
t-bas
ed q
uery
re
finem
ent“a
irpla
ne”,
“ta
ke o
ff”
Fina
l tex
t-bas
edra
nkin
g of
sho
ts
Sho
t-lev
el fu
sion
and
re-ra
nkin
g
Sto
ry-b
ased
que
ry
refin
emen
t
Sho
t-bas
ed re
fined
qu
ery
exec
utio
nS
tory
-bas
ed re
fined
qu
ery
exec
utio
nra
nked
sh
ots
Que
ry
1
“cra
sh”
“pilo
t”
rank
ed
stor
ies
Sho
t-lev
el s
core
pr
opag
atio
n &
rank
ing
rank
ed
shot
s
IBM
Mar
vel f
or T
RE
CV
ID06
Aut
omat
ic S
earc
h | N
ov. 1
4th,
200
6
IBM
T.J
. Wat
son
Res
earc
h C
ente
r
©C
opyr
ight
IBM
Cor
pora
tion
2006
IBM
Res
earc
h Vi
sual
Ret
rieva
l Sys
tem
Unb
alan
ced
lear
ning
from
m
ultim
edia
dat
aC
onte
nt-b
ased
do
wn-
sam
plin
g of
te
st s
et to
cre
ate
nega
tive
exam
ples
-Fu
sion
of t
wo
light
-w
eigh
t lea
rnin
g te
chni
ques
: k-N
N
and
SVM
Con
tent
-bas
ed
over
-sam
plin
g of
vi
sual
que
ry to
cr
eate
pos
itive
ex
ampl
es-
SVM
lear
ning
-M
AP
: 0.0
212
See
Nat
sev
et a
l. (A
CM
MM
200
5)
Visu
al q
uery
exa
mpl
es (a
irpla
ne ta
ke-o
ff)
Extr
act v
isua
l fea
ture
s
Fusi
on
Visu
al R
un
MEC
BR
Sele
ctio
n
Ato
mic
CB
R
OR
Fus
ion
SVM
SVM
run
for e
ach
bag
AN
D F
usio
n
3
Con
tent
-bas
ed o
ver-
sam
plin
g of
vis
ual q
uery
exa
mpl
es
Extr
act v
isua
l fea
ture
s
SVM
lear
ning
Fusi
on
Con
tent
-bas
ed d
own-
sam
plin
g of
test
set
IBM
Mar
vel f
or T
RE
CV
ID06
Aut
omat
ic S
earc
h | N
ov. 1
4th,
200
6
IBM
T.J
. Wat
son
Res
earc
h C
ente
r
©C
opyr
ight
IBM
Cor
pora
tion
2006
IBM
Res
earc
h M
odel
-Bas
ed R
etrie
val S
yste
mA
ppro
ache
s:
1.Q
uery
text
-bas
ed: a
utom
atic
ally
map
qu
ery
topi
c te
xt to
con
cept
mod
els
•Le
xica
l (W
ordN
etsi
mila
rity)
•Le
xica
l (le
xico
n m
appi
ng)
•Le
xica
l (di
rect
wor
d m
atch
)•
Sta
tistic
al (c
o-oc
curr
ence
-bas
ed)
•S
tatis
tical
(pse
udo-
RF)
2.Q
uery
exa
mpl
e-ba
sed :
aut
omat
ical
ly
map
que
ry e
xam
ples
to c
once
pt m
odel
s•
Con
tent
-bas
ed (m
odel
vec
tors
)•
Con
tent
-bas
ed (f
eatu
re s
elec
tion)
3.Fu
sion
:•
Que
ry in
depe
nden
t (M
AP
0.0
45)
•Q
uery
-cla
ss-d
epen
dent
Visu
al q
uery
exa
mpl
es
Fusi
on
Text
ual q
uery
topi
c
“Fin
d sh
ots
of a
n ai
rpla
ne ta
king
off”
Que
ry E
xam
ple
Ana
lysi
sQ
uery
Tex
t Ana
lysi
s
Fina
l mod
el-b
ased
re
trie
val r
esul
ts
Que
ry
Lexi
cal/s
tatis
tical
se
man
tic q
uery
re
finem
ent (
quer
y-to
-mod
el m
appi
ng)
Con
tent
-bas
ed
sem
antic
que
ry
refin
emen
t (qu
ery-
to-m
odel
map
ping
)
2
“airp
lane
”“t
ake
off”
mod
el
vect
ors
airp
lane
, he
licop
ter
airp
lane
, ou
tdoo
r, m
ilita
ry
IBM
Mar
vel f
or T
RE
CV
ID06
Aut
omat
ic S
earc
h | N
ov. 1
4th,
200
6
IBM
T.J
. Wat
son
Res
earc
h C
ente
r
©C
opyr
ight
IBM
Cor
pora
tion
2006
Bro
adca
st N
ews
Prog
ram
C
ateg
ory
Wea
ther
Ent
erta
inm
ent
Spo
rts
Loca
tion
Offi
ce
Mee
ting
Stu
dio
Out
door
Roa
dS
kyS
now
Urb
anW
ater
scap
eM
ount
ain
Des
ert
Peop
le
Cro
wd
Face
Per
son
Rol
es
Obj
ects
Flag
-US
Ani
mal
Com
pute
rV
ehic
leA
irpla
neC
arB
oat/S
hip
Bus
Bui
ldin
gV
eget
atio
n
Cou
rt
Gov
ernm
ent
Lead
er
Cor
pora
te
Lead
er
Pol
ice/
Sec
urity
Pris
oner
Milit
ary
Truc
k
Activ
ities
&
Even
ts
Peo
ple
Wal
k/R
unM
arch
Exp
losi
on/F
ireD
isas
ter
Gra
phic
s
Map
sC
harts
Obs
erva
tions
–Vis
ual c
once
pts
can
help
refin
e qu
ery
topi
cs
–Vis
ual c
onte
xt c
an d
isam
bigu
ate
wor
d se
nses
Idea –L
ever
age
auto
mat
ic v
isua
l con
cept
det
ecto
rs
for s
eman
tic m
odel
-bas
ed q
uery
refin
emen
t
Sem
antic
Con
cept
Lex
icon
–Hie
rarc
hy o
f 39
LSC
OM
-lite
conc
epts
–Sta
tistic
al m
odel
s ba
sed
on v
isua
l fe
atur
es a
nd m
achi
ne le
arni
ng
–Eac
h co
ncep
t mod
el d
escr
ibed
by
seve
ral w
ords
/phr
ases
from
Wor
dNet
Visu
al C
once
ptR
epre
sent
ativ
e w
ords
/phr
ases
Airp
lane
jet,
airc
raft,
car
rier,
war
plan
e
Nat
ural
dis
aste
rdi
sast
er, e
arth
quak
e, fi
re, f
lam
e,
flood
, hur
rican
e, to
rnad
o,
tsun
ami
Spo
rtssp
ort,
base
ball,
bas
ketb
all,
socc
er, t
enni
s, c
ricke
t, fo
otba
ll,
hock
ey, g
olf,
gam
e, m
atch
US
flag
Am
eric
an fl
ag, s
tars
and
stri
pes
Mod
el-B
ased
Ret
rieva
l—Se
man
tic L
exic
on2
©20
05 IB
M C
orpo
ratio
n
IBM
T.J
. Wat
son
Res
earc
h C
ente
r
MA
RVE
L Vi
deo
Sear
ch
Mod
el-B
ased
Ret
rieva
l—Le
xica
l (W
ordN
et)
App
roac
hTe
xtua
l que
ry to
pic
“Fin
d sh
ots
of a
n ai
rpla
ne ta
king
off”
Que
ry A
naly
sis
Que
ry R
efin
emen
t(Q
uery
-to-C
once
pt
Map
ping
& W
eigh
ting)
Con
cept
Mod
els
Wor
dNet
“airp
lane
”,
“tak
e of
f”
Out
door
s (1
.0),
Sky
(0.8
), M
ilita
ry (0
.6)
Mod
el-b
ased
rank
ingR
epos
itory
Que
ry E
xecu
tion
(Con
cept
-Bas
ed
Ret
rieva
l)
1.Q
uery
ana
lysi
sTo
kens
, ste
ms,
phr
ases
Sto
p w
ord
rem
oval
2.Q
uery
refin
emen
tA
utom
atic
map
ping
of q
uery
text
to
con
cept
mod
els
& w
eigh
tsLe
xica
l app
roac
h ba
sed
on
Wor
dNet
Lesk
sim
ilarit
y
3.Q
uery
exe
cutio
nC
once
pt-b
ased
retri
eval
usi
ng
stat
istic
al c
once
pt m
odel
sW
eigh
ted
aver
agin
g m
odel
fusi
on
MA
P: 0
.029
See
Hau
bold
et a
l. (IC
ME
200
6)
Que
ry Con
cept
Lexi
con
Toke
nize
r
2
©20
05 IB
M C
orpo
ratio
n
IBM
T.J
. Wat
son
Res
earc
h C
ente
r
MA
RVE
L Vi
deo
Sear
ch
Lexi
cal Q
uery
Ref
inem
ent (
Que
ry-to
-Con
cept
Map
ping
)S
eman
tic R
elat
edne
ss–W
ordN
etLe
sksi
mila
rity
bet
wee
n C
once
pt S
ynon
yms
and
Que
ry T
erm
s
–max
(Les
k)
betw
een
Con
cept
and
Que
ry T
erm
det
erm
ines
clo
sest
rela
tedn
ess
–Nor
mal
ized
sum
ov
er q
uery
term
s es
tabl
ishe
s Q
uery
Con
cept
Wei
ght v
ecto
r
Syno
nym
sC
once
pts
C0
Cro
wd
S 0,0
S 0,1
… S 0,m
S n,m
Cn
T 0: p
eopl
eT k
: sig
nQ
uery
Ter
ms
Σ/k
Que
ryC
once
ptW
eigh
ts
…
. . .
Que
ry: “
peop
le w
ith b
anne
rs o
r sig
ns”
T 0: p
eopl
epe
ople
-mar
chin
g: 4
737,
crow
d: 4
737,
…
T 1: b
anne
rpe
ople
-mar
chin
g: 2
16,
US
flag:
151
,…
T 2: s
ign
build
ing:
536
1,w
ater
scap
e/fr
ont:
531,
…
Que
ry-to
-con
cept
map
ping
& w
eigh
ts:
Bui
ldin
g: 1
879.
7Pe
ople
/mar
chin
g: 1
685.
7C
row
d: 1
625.
3 …
. . .
. . .
W0
Wn
max
max
2
©20
05 IB
M C
orpo
ratio
n
IBM
T.J
. Wat
son
Res
earc
h C
ente
r
MA
RVE
L Vi
deo
Sear
ch
Que
ry E
xecu
tion
(Mod
el-B
ased
Ret
rieva
l of S
hots
)
Fusi
on o
f wei
ght v
ecto
rs–Q
uery
con
cept
wei
ght v
ecto
r
–Sho
t con
cept
con
fiden
ce v
ecto
rFi
nal s
hot s
core
–Σ(Π)b
etw
een
quer
y an
d sh
ot c
once
pt v
ecto
rs
–Sor
ted
shot
sco
res:
rank
ed li
st o
f res
ults
Que
ry <
W0
, W
1,
W2
, W
3,
…
, W
k >
Shot
0<
C0
, C
1,
C2
, C
3,
…
, C
k >
Σ(Π):
S 0Sc
ore
Que
ry <
W0
, W
1,
W2
, W
3,
…
, W
k >
Shot
n<
C0
, C
1,
C2
, C
3,
…
, C
k >
Σ(Π):
S nSc
ore
. . .
2
©20
05 IB
M C
orpo
ratio
n
IBM
T.J
. Wat
son
Res
earc
h C
ente
r
MA
RVE
L Vi
deo
Sear
ch
Mod
el-B
ased
Ret
rieva
l—Le
xico
n M
appi
ng
App
roac
hTe
xtua
l que
ry to
pic
“Fin
d sh
ots
of a
n ai
rpla
ne ta
king
off”
Sem
antic
Q
uery
Ana
lysi
s(IB
M P
IQU
AN
T-II)
Rul
e-ba
sed
le
xico
n m
appi
ng
VEH
ICLE
:CA
TEG
OR
YFL
IGH
T:C
ATE
GO
RY
Airp
lane
Mod
el-b
ased
rank
ing
Con
cept
Mod
els
Rep
osito
ry
Que
ry E
xecu
tion
(Con
cept
-Bas
ed
Ret
rieva
l)
1.S
eman
tic q
uery
ana
lysi
sS
eman
tic a
nnot
atio
ns o
f 100
+ na
med
and
unn
amed
ent
ities
in
text
(peo
ple,
pla
ces,
eve
nts,
etc
.)IB
M P
IQU
AN
T-II
Q&
A e
ngin
e
2.R
ule-
base
d le
xico
n m
appi
ngM
anua
lly d
efin
ed ru
les
for
map
ping
text
ann
otat
ion
type
s to
LS
CO
M-li
tevi
sual
con
cept
s
3.Q
uery
exe
cutio
nC
once
pt-b
ased
retri
eval
usi
ng
stat
istic
al c
once
pt m
odel
sW
eigh
ted
aver
agin
g m
odel
fusi
on
Per
form
ance
:M
AP
: 0.0
18Q
RY
196
AP
: 0.2
51
Que
ry
2
Con
cept
pat
tern
sPO
S Ta
gger
Dee
p Pa
rser
Text
and
Vis
ual
Con
cept
Lex
ica
IBM
Mar
vel f
or T
RE
CV
ID06
Aut
omat
ic S
earc
h | N
ov. 1
4th,
200
6
IBM
T.J
. Wat
son
Res
earc
h C
ente
r
©C
opyr
ight
IBM
Cor
pora
tion
2006
Mod
el-B
ased
Ret
rieva
l—C
onte
nt-B
ased
App
roac
h
Sem
antic
mod
el v
ecto
r fea
ture
-39
-dim
ensi
onal
vec
tor o
f con
fiden
ces
-D
imen
sion
s co
rres
pond
to L
SC
OM
-lite
Lear
ning
tech
niqu
e-
Fusi
on o
f tw
o lig
ht-w
eigh
t lea
rnin
g te
chni
ques
: k-N
N a
nd S
VM
Sam
plin
g te
chni
que:
-S
ampl
e ps
eudo
-neg
ativ
e ex
ampl
es
usin
g co
here
nt c
lust
er c
entro
ids
Bes
t ind
ivid
ual m
odel
-bas
ed a
ppro
ach
-M
AP
: 0.0
34
-Q
RY
195
AP
: 0.5
974
See
Nat
sev
et a
l. (A
CM
MM
200
5)
Visu
al q
uery
exa
mpl
es (a
irpla
ne ta
ke-o
ff)
Extr
act m
odel
vec
tors
Extr
act v
isua
l fea
ture
s
Mod
el-b
ased
rank
ing
2
Fusi
on
MEC
BR
Sele
ctio
n
Ato
mic
CB
R
OR
Fus
ion
SVM
Con
tent
-bas
ed
dow
n-sa
mpl
ing
SVM
run
for e
ach
bag
AN
D F
usio
n
IBM
Mar
vel f
or T
RE
CV
ID06
Aut
omat
ic S
earc
h | N
ov. 1
4th,
200
6
IBM
T.J
. Wat
son
Res
earc
h C
ente
r
©C
opyr
ight
IBM
Cor
pora
tion
2006
Mul
ti-m
odal
Fus
ion
The
prob
lem
-Te
xt-,
visu
al-a
nd m
odel
-bas
ed re
triev
al a
re e
ach
good
at f
indi
ng c
erta
in th
ings
Text
: nam
ed p
eopl
e, o
ther
nam
ed e
ntity
Vis
ual:
sem
antic
sce
nes
cons
iste
nt in
col
or o
r lay
out,
e.g.
, spo
rts, w
eath
erC
once
pt M
odel
s: n
on-s
peci
fic q
uerie
s, e
.g. p
rote
st, b
oat,
fire
-A
vera
ging
the
retri
eval
mod
els
help
s [IB
M T
RE
CV
ID 2
005]
-Q
uery
-cla
ss o
r que
ry-c
lust
er d
epen
dent
fusi
on h
elps
mor
e [C
MU
, NU
S, C
olum
bia]
Our
app
roac
h-
Wei
ghte
d lin
ear c
ombi
natio
n fo
r fus
ion
-S
eman
tic q
uery
ana
lysi
sfo
r que
ry m
atch
ing
(bas
ed o
n P
IQU
AN
T-II
Q&
A te
chno
logy
)-
Har
d vs
. sof
t que
ry c
lass
es-
Trai
ning
Que
ries:
TR
ECVI
D20
05
sem
antic
qu
ery
anal
ysis
wei
ghte
d fu
sion
text
visu
al
mod
el
…se
arch
for
wei
ghts
test
que
ry
train
ing
quer
ies
quer
y m
atch
ing
4
IBM
Mar
vel f
or T
RE
CV
ID06
Aut
omat
ic S
earc
h | N
ov. 1
4th,
200
6
IBM
T.J
. Wat
son
Res
earc
h C
ente
r
©C
opyr
ight
IBM
Cor
pora
tion
2006
Que
ry D
epen
dent
Fus
ion
Com
pone
nts
Sem
antic
que
ry a
naly
sis
179
Sadd
am
Hus
sein
with
at l
east
on
e ot
her p
erso
n’s
face
at l
east
par
tially
vi
sibl
e.
Per
son:
NA
ME
34 4
7 P
erso
n:N
AM
E41
47
Per
son:
CA
TEG
OR
Y73
78
Bod
yPar
t:UN
KW
N82
85
Sem
antic
ann
otat
ion
(IBM
PIQ
UA
INT-
II en
gine
)
IBM
PIQ
UA
NT-
II se
man
tic a
nnot
atio
n en
gine
and
type
sys
tem
-D
esig
ned
for i
ntel
ligen
ce q
uest
ion-
answ
erin
g-
Def
ined
mor
e th
an 1
00 s
eman
tic e
ntiti
es o
ver t
ext
(Nam
ed/g
ener
ic) p
erso
n, o
bjec
t, ev
ent,
loca
tion,
etc
.
Que
ry c
lass
/ qu
ery-
com
pone
nt m
appi
ng
inpu
t que
ryou
tput
sem
antic
ann
otat
ions
(a) 4
que
ry c
lass
es:
Nam
ed P
erso
n, U
nnam
ed P
erso
n, S
ports
, Oth
ers
(b) 1
0 so
ft qu
ery
com
pone
nts:
Nam
ed P
erso
n, U
nam
edP
erso
n, S
ports
, Nam
ed E
ntity
, E
vent
, Sce
ne, O
bjec
t, Ve
hicl
e, W
eapo
n , O
ther
s
Per
son:
CA
TEG
OR
Y
Per
son:
NO
MIN
AL
Bod
yPar
t:*
Clo
thin
g:*
Unn
amed
P
erso
n
IBM
Mar
vel f
or T
RE
CV
ID06
Aut
omat
ic S
earc
h | N
ov. 1
4th,
200
6
IBM
T.J
. Wat
son
Res
earc
h C
ente
r
©C
opyr
ight
IBM
Cor
pora
tion
2006
196 snow 195 soccer goalpost 194 Condoleeza Rice 193 smokestacks 192 kiss on the cheek 191 adult and child 190 person and 10 books 189 four suited people w. flag 188 burning with flames 187 helicopters in flight 186 a natural scene 185 reading a newspaper 184 people with computer display 183 water boat-ship 182 soldiers or police 181 Bush, Jr. walking 180 uniformed people in formation179 Saddam Hussein +1 178 Dick Cheney 177 daytime protest w. building 176 escorting a prisoner 175 leaving or entering a vehicle174 tall buildings 173 emergency vehicles
Eve
ntN
amed
Entit
yN
amed
Pers
onO
bjec
tS
cene
Spo
rtsU
nam
edPe
rson
Veh
icle
Viol
ence
Oth
ers
172 office setting171 soccer goal170 tall building169 military vehicles168 road with cars167 airplane taking off166 palm trees.165 basketball164 ship or boat.163 meeting162 entering or leaving a building.161 people & banners/signs.160 on fire with flames159 George W. Bush and vehicle158 helicopter in flight.157 shaking hands156 tennis players155 map of Iraq154 Mahmoud Abbas153 Tony Blair152 Hu Jintao151 Omar Karami150 Iyad Allawi149 Condoleeza Rice
3 0 8 0 7 3 11 7 1 0
3 1 4 2 7 1 12 5 3 1
qry0
5qr
y06
coun
ts
Que
ry C
ompo
nent
s C
over
age
(200
5-06
topi
cs)
0506
IBM
Mar
vel f
or T
RE
CV
ID06
Aut
omat
ic S
earc
h | N
ov. 1
4th,
200
6
IBM
T.J
. Wat
son
Res
earc
h C
ente
r
©C
opyr
ight
IBM
Cor
pora
tion
2006
Mul
ti-m
odal
Fus
ion
Res
ults
0
0.050.1
0.150.2
0.25
MA
PEm
erge
ncy
vehi
cles
Tall
build
ings
Peop
le &
vehi
cles
Pris
oner
&so
ldie
rsPr
otes
t &bu
ildin
gDi
ck C
hene
ySa
ddam
Hus
sein
Peop
le in
unifo
rmG
eorg
e W
.B
ush
Milit
ary/
polic
eBo
ats/
ship
sCo
mpu
ter
Dis
play
Peop
le &
New
spap
er
text
visu
al
mod
el
Qin
d
Qcl
ass
Qco
mp
00.10.20.30.40.50.60.7
Peop
le &
New
spap
erNa
ture s
cene
sHe
licop
ter in
flight
Fire
Peop
le in
suits
Peop
le &
book
sAd
ults &
child
ren
Gree
ting b
ykis
sCh
imne
ys &
smok
eCo
ndole
eza
Rice
Socc
ergo
alpos
tsSn
ow sc
enes
text
visua
l
mode
l
Qind
Qclas
s
Qcom
p
IBM
Mar
vel f
or T
RE
CV
ID06
Aut
omat
ic S
earc
h | N
ov. 1
4th,
200
6
IBM
T.J
. Wat
son
Res
earc
h C
ente
r
©C
opyr
ight
IBM
Cor
pora
tion
2006
Mul
ti-m
odal
Fus
ion
Res
ults
(Col
or-c
oded
Per
form
ance
)
196 snow 194 Condoleeza Rice 193 smokestacks 192 kiss on the cheek 191 adult and child 190 person and 10 books 189 four suited people w. flag 188 burning with flames 187 helicopters in flight 186 a natural scene 185 reading a newspaper 184 people w/ computer display 183 water boat-ship 182 soldiers or police 181 Bush, Jr. walking 180 uniformed people 179 Saddam Hussein +1 178 Dick Cheney 177 daytime protest w. building 176 escorting a prisoner 175 leaving or entering a vehicle174 tall buildings 173 emergency vehiclesMAP
Text
Vis
ual
Mod
el
Qin
d
Qcl
ass
Qco
mp
IBM
Mar
vel f
or T
RE
CV
ID06
Aut
omat
ic S
earc
h | N
ov. 1
4th,
200
6
IBM
T.J
. Wat
son
Res
earc
h C
ente
r
©C
opyr
ight
IBM
Cor
pora
tion
2006
Mul
ti-m
odal
Fus
ion
Res
ults
(Rel
ativ
e Pe
rfor
man
ce)
-150
-100-50050100
150
200
AVG
emerg
ency
vehic
lestal
l buil
dings
peop
le & ve
hicles
esco
rting a
priso
ner
protes
t & bu
ilding
s Dick C
heney
Sadda
m Hus
sein
+1
peop
le in
formati
onBus
h, Jr.
walkin
gso
ldiers
or po
lice
water b
oat-s
hip compu
ter di
splay
readin
g a ne
wspap
er a natu
ral sc
ene
helic
opter
s in f
light
burni
ng w
ith fla
mes
four s
uited
peop
le w. fl
ag
perso
n and
10 bo
oks ad
ult an
d chil
dkis
s on t
he che
ek smok
estac
ksCon
dolee
za R
ice socc
er go
alpost
snow
Rel
ativ
e im
prov
emen
t (%
) -
quer
y-cl
ass
fusi
on v
s. q
uery
-inde
pend
ent f
usio
nO
bser
vatio
ns-
Con
cept
-rel
ated
que
ries
impr
oved
the
mos
t:“ta
ll bu
ildin
g”, “
pris
oner
”, “h
elic
opte
rs in
flig
ht”,
“soc
cer”
-N
amed
-ent
ity q
uerie
s im
prov
ed s
light
ly:
“Dic
k C
hene
y”, “
Sad
dam
Hus
sein
”, “B
ush
Jr.”
-G
ener
ic p
eopl
e ca
tego
ry d
eter
iora
ted
the
mos
t: “p
eopl
e in
form
atio
n”, “
at c
ompu
ter d
ispl
ay”,
“w/ n
ewsp
aper
”, “w
/ boo
ks”
IBM
Mar
vel f
or T
RE
CV
ID06
Aut
omat
ic S
earc
h | N
ov. 1
4th,
200
6
IBM
T.J
. Wat
son
Res
earc
h C
ente
r
©C
opyr
ight
IBM
Cor
pora
tion
2006
TREC
VID
06 A
utom
atic
and
Man
ual S
earc
h(A
ggre
gate
d Pe
rfom
ance
of I
BM
vs.
Oth
ers)
0.00
0.02
0.04
0.06
0.08
0.10
13
57
911
1315
1719
2123
2527
2931
3335
3739
4143
4547
4951
5355
5759
6163
6567
6971
7375
7779
8183
8587
Sub
mitt
ed R
uns
Mean Average Precision
IBM
Aut
omat
ic S
earc
h O
ther
Aut
omat
ic S
earc
h O
ther
Man
ual S
earc
h
Aut
omat
ic/M
anua
l Sea
rch
Ove
rall
Perf
orm
ance
(Mea
n A
P)
Mul
ti-m
odal
fusi
on d
oubl
es b
asel
ine
perfo
rman
ce!
Mod
el-b
ased
retri
eval
and
re-r
anki
ng in
stru
men
tal i
n pe
rform
ance
gai
nVi
sual
retri
eval
not
muc
h of
a fa
ctor
this
yea
r
IBM
Offi
cial
Run
s:Te
xt (b
asel
ine)
:
0.04
1Te
xt (s
tory
-bas
ed):
0.05
2
Mul
timod
al F
usio
n:Q
uery
inde
pend
ent:
0.07
6Q
uery
cla
sses
(sof
t):0.
086
Que
ry c
lass
es (h
ard)
:0.
087
IBM
Con
stitu
ent R
uns:
Mod
el-b
ased
Run
:
0.
045
Vis
ual-b
ased
Run
:
0.
021
IBM
Run
s
IBM
Mar
vel f
or T
RE
CV
ID06
Aut
omat
ic S
earc
h | N
ov. 1
4th,
200
6
IBM
T.J
. Wat
son
Res
earc
h C
ente
r
©C
opyr
ight
IBM
Cor
pora
tion
2006
TREC
VID
06 A
utom
atic
and
Man
ual S
earc
h (P
er-T
opic
Ave
rage
Pre
cisi
on o
f Bes
t IB
M v
s. B
est O
vera
ll)
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Mean AP
Emergenc
y veh
icles
Tall bu
ilding
s
People
& vehicle
s
Prisone
r & so
ldiers
Protes
t & bu
ilding
Dick C
heney
Sadda
m Hus
sein
People
in un
iform
George
W. B
ush
Military/
polic
eBoa
ts/sh
ips
Computer D
isplay
People
& New
spape
r
Nature sc
enes
Helicopte
r in fli
ghtFire
People
in su
its
People
& book
s
Adults
& ch
ildren
Greeting b
y kiss
Chimney
s & sm
oke
Condole
eza R
ice
Socce
r goalp
osts
Snow sc
enes
Average Precision
Bes
t IB
MB
est O
vera
ll
IBM
vs.
Oth
ers
Per-
Topi
c A
naly
sis
(Ave
rage
Pre
cisi
on)
Top
perfo
rman
ce o
n 6
out o
f the
24
quer
y to
pics
IBM
Mar
vel f
or T
RE
CV
ID06
Aut
omat
ic S
earc
h | N
ov. 1
4th,
200
6
IBM
T.J
. Wat
son
Res
earc
h C
ente
r
©C
opyr
ight
IBM
Cor
pora
tion
2006
TREC
VID
06 A
utom
atic
and
Man
ual S
earc
h (P
er-T
opic
Pre
cisi
on @
100
of B
est I
BM
vs.
Bes
t Ove
rall)
0.0
10.0
20.0
30.0
40.0
50.0
60.0
70.0
80.0
90.0
100.
0
Mean P@
100
Emergenc
y veh
icles
Tall bu
ilding
s
People
& vehicle
s
Prisone
r & so
ldiers
Protes
t & bu
ilding
Dick C
heney
Sadda
m Hus
sein
People
in un
iform
George
W. B
ush
Military/
polic
eBoa
ts/sh
ips
Computer D
isplay
People
& New
spape
r
Nature sc
enes
Helicopte
r in fli
ghtFire
People
in su
its
People
& book
s
Adults
& ch
ildren
Greeting b
y kiss
Chimney
s & sm
oke
Condole
eza R
ice
Socce
r goalp
osts
Snow sc
enes
Precision @ 100 (%)
Bes
t IB
MB
est O
vera
ll
IBM
vs.
Oth
ers
Per-
Topi
c A
naly
sis
(Pre
cisi
on a
t 100
)
Pers
on-X
Visu
alM
odel
s
Top
perfo
rman
ce o
n 6
out o
f the
24
quer
y to
pics
IBM
Mar
vel f
or T
RE
CV
ID06
Aut
omat
ic S
earc
h | N
ov. 1
4th,
200
6
IBM
T.J
. Wat
son
Res
earc
h C
ente
r
©C
opyr
ight
IBM
Cor
pora
tion
2006
Obs
erva
tions
(200
5)
Vis
ual r
etrie
val 2
x be
tter t
han
spee
ch re
triev
al th
is y
ear
-D
ue to
few
er s
ports
topi
cs a
nd fe
wer
nea
r-du
plic
ates
Con
cept
mod
els
help
ed s
igni
fican
tly (>
50%
gai
n ov
er b
asel
ine)
-N
o do
min
atio
n by
any
sin
gle
quer
y cl
ass
(e.g
., P
erso
n-X
, Spo
rts, e
tc.)
Aut
omat
ic s
earc
h on
par
with
man
ual s
earc
h!-
Due
to v
ery
few
man
ual s
ubm
issi
ons
Sear
ch s
yste
ms
need
to b
e co
mpr
ehen
sive
to b
e ef
fect
ive!
!!
Pro
posa
ls-
Con
side
r inc
reas
ing
# to
pics
to re
duce
ske
w o
n ag
greg
ate
mea
sure
s-
Con
side
r sta
ndar
dizi
ng q
uery
cla
sses
(Per
son-
X, S
ports
, etc
.)
2006
3x w
orse
100%
better
than
![They Are Not Equally Reliable: Semantic Event Search Using ... · (NIST) initiated the zero-example search (0Ex for short) in TRECVID 2013 [1] and 2014 [2]. Promising progress [43,](https://static.documents.pub/doc/80x56/5f36b36f0385e402d22b235c/they-are-not-equally-reliable-semantic-event-search-using-nist-initiated.jpg)
