© Arjen P. de Vries 2001 1

Arjen P. de Vriesarjen@acm.org

Fascinating Relationships between

Media and Text

© Arjen P. de Vries 2001 2

From Video to a List of Images…

…

© Arjen P. de Vries 2001 3

Query image

© Arjen P. de Vries 2001 4

Feature Space Indexing

N-dimensionalspace

Feature extraction

Ranking

Display

© Arjen P. de Vries 2001 5

Complicating Factors

• What are Good Feature Models?

• What are Good Ranking Functions?

• Subjective Queries

© Arjen P. de Vries 2001 6

Mirror Approach (1/2)

• Gather User’s Knowledge– Introduce semi-automatic processes for

selection and combination of feature models

• Local Information– Relevance feedback from a user

• Global Information– Thesauri constructed from all users

© Arjen P. de Vries 2001 7

Mirror Approach (2/2)

N-dimensionalspace

Feature extraction

Clustering Ranking

Concepts Display

Thesauri

© Arjen P. de Vries 2001 8

Visual Thesaurus

Correct cluster

representing‘Tree’, ‘Forest’

Glcm_47

‘Incoherent’ cluster

Fractal_23

Mis-labeled cluster

Gabor_20

© Arjen P. de Vries 2001 9

Mirror Results

Thesaurus Only

After Feedback

© Arjen P. de Vries 2001 10

‘Explaining’ the Results

• Paivio’s dual coding theory conjectures that the human brain processes textual terms (logogens) as well as image terms (imagens)

• Term frequency and inverse document frequency intuitively express the amount of surprise of observing the event represented by the term

• Also matches similar music: grunge, house, …

• Even works for predicting avalanches!

© Arjen P. de Vries 2001 11

Query Articulation Problem

N-dimensionalspace

Feature extraction

How to articulate the query?

© Arjen P. de Vries 2001 12

What is the query semantics ?

© Arjen P. de Vries 2001 13

More than Sub-Image Search

• Irrelevant Background

• Relevant Colors

• Distinguishing Shapes

• …

© Arjen P. de Vries 2001 14

Irrelevant Background

Query Result

© Arjen P. de Vries 2001 15

Known Item

© Arjen P. de Vries 2001 16

Query

© Arjen P. de Vries 2001 17

Results

…

© Arjen P. de Vries 2001 18

Query

© Arjen P. de Vries 2001 19

Results

…

© Arjen P. de Vries 2001 20

Problem

• Video TREC query images are quite different from the (low-quality) video data in the collection

• The current translation model from collection objects to query objects is incorrect

• Figure out better ways, possibly using:– Color thesaurus, transitions between colors– Explicit background and foreground models– …

© Arjen P. de Vries 2001 21

Summarizing

Don’t give up…

But…Stay Realistic!