EURECOM @ SAVA2015:Visual Features

for Multimedia Search

Maria Eskevich, Benoit Huet

Sept 14, 2015

Motivation System overview Results

Motivation

QUERY

Textualdescription

Textual descriptionof visual content

Videocollection

TranscriptTextual terms for

extracted visual concepts

Connection?Connection?

Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 2 / 8

Motivation System overview Results

Motivation

QUERY

Textualdescription

Textual descriptionof visual content

Videocollection

TranscriptTextual terms for

extracted visual concepts

Connection?Connection?

Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 2 / 8

Motivation System overview Results

Motivation

QUERY

Textualdescription

Textual descriptionof visual content

Videocollection

TranscriptTextual terms for

extracted visual concepts

Connection?Connection?

Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 2 / 8

Motivation System overview Results

Motivation

QUERY

Textualdescription

Textual descriptionof visual content

Videocollection

TranscriptTextual terms for

extracted visual concepts

Connection?Connection?

Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 2 / 8

Motivation System overview Results

System overview

QUERY

Textualdescription

Textual descriptionof visual content

ResultsegmentsRerankedresultsegments

Terrier 4.0IR System

Videocollection

TranscriptTextual terms for

extracted visual concepts

Indexingof

segments

Informationrequest

Confidence scorecalculation

Confidence scoresfor visual concepts

in shots correspondingto retrieved segments

Run type: visualRun type: text

Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 3 / 8

Motivation System overview Results

System overview

QUERY

Textualdescription

Textual descriptionof visual content

ResultsegmentsRerankedresultsegments

Terrier 4.0IR System

Videocollection

TranscriptTextual terms for

extracted visual concepts

Indexingof

segments

Informationrequest

Confidence scorecalculation

Confidence scoresfor visual concepts

in shots correspondingto retrieved segments

Run type: visualRun type: text

Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 3 / 8

Motivation System overview Results

System overview

QUERY

Textualdescription

Textual descriptionof visual content

ResultsegmentsRerankedresultsegments

Terrier 4.0IR System

Videocollection

TranscriptTextual terms for

extracted visual concepts

Indexingof

segments

Informationrequest

Confidence scorecalculation

Confidence scoresfor visual concepts

in shots correspondingto retrieved segments

Run type: visualRun type: text

Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 3 / 8

Motivation System overview Results

System overview

QUERY

Textualdescription

Textual descriptionof visual content

Resultsegments

Rerankedresultsegments

Terrier 4.0IR System

Videocollection

TranscriptTextual terms for

extracted visual concepts

Indexingof

segments

Informationrequest

Confidence scorecalculation

Confidence scoresfor visual concepts

in shots correspondingto retrieved segments

Run type: visualRun type: text

Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 3 / 8

Motivation System overview Results

System overview

QUERY

Textualdescription

Textual descriptionof visual content

Resultsegments

Rerankedresultsegments

Terrier 4.0IR System

Videocollection

TranscriptTextual terms for

extracted visual concepts

Indexingof

segments

Informationrequest

Confidence scorecalculation

Confidence scoresfor visual concepts

in shots correspondingto retrieved segments

Run type: visual

Run type: text

Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 3 / 8

Motivation System overview Results

System overview

QUERY

Textualdescription

Textual descriptionof visual content

Resultsegments

Rerankedresultsegments

Terrier 4.0IR System

Videocollection

TranscriptTextual terms for

extracted visual concepts

Indexingof

segments

Informationrequest

Confidence scorecalculation

Confidence scoresfor visual concepts

in shots correspondingto retrieved segments

Run type: visual

Run type: text

Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 3 / 8

Motivation System overview Results

System overview

QUERY

Textualdescription

Textual descriptionof visual content

Resultsegments

Rerankedresultsegments

Terrier 4.0IR System

Videocollection

TranscriptTextual terms for

extracted visual concepts

Indexingof

segments

Informationrequest

Confidence scorecalculation

Confidence scoresfor visual concepts

in shots correspondingto retrieved segments

Run type: visual

Run type: text

Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 3 / 8

Motivation System overview Results

System overview

QUERY

Textualdescription

Textual descriptionof visual content

Resultsegments

Rerankedresultsegments

Terrier 4.0IR System

Videocollection

TranscriptTextual terms for

extracted visual concepts

Indexingof

segments

Informationrequest

Confidence scorecalculation

Confidence scoresfor visual concepts

in shots correspondingto retrieved segments

Run type: visual

Run type: text

Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 3 / 8

Motivation System overview Results

Confidence score calculation

Calculate new confidence scores for theoverlapping terms from query information andvisual concepts extracted from the shotscorresponding to the retrieved video segments.

Query Information

Query terms(text/visual):ConfScore = 1.0

Word2Vec terms:ConfScore =Word2VecVectorDistance

w1 .. wn w1 .. wm

Informationbased onVisual

ConceptsExtraction

forRetrievedSegments

Visual Conceptsterms:

ConfScore=Visual Concepts

Extraction ConfScore

w1∑

ConfScoresComposition

NumOfOverlappingTerms ∗ 0.6

∑ConfScoresComposition

NumOfOverlappingTerms ∗ 0.2..wk

Word2Vec terms:ConfScore=

Visual ConceptsExtraction ConfScore

*Word2Vec Vector

Distance

w1∑

ConfScoresComposition

NumOfOverlappingTerms ∗ 0.2 -..wm

Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 4 / 8

Motivation System overview Results

Evaluation Results: Precision @5,10,20

Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 5 / 8

Motivation System overview Results

Evaluation Results: MAP metrics

Query Visual MAP MAP_bin MAP_tol MAiSPfields conceptstext none 0.5511 0.3529 0.3089 0.3431text Oxford 0.3196 0.2739 0.2053 0.2978visual Oxford 0.3368 0.2958 0.2293 0.3092text Leuven 0.3227 0.2801 0.2187 0.2958visual Leuven 0.3394 0.2970 0.2222 0.3117text CERTH 0.2295 0.2027 0.1554 0.1983visual CERTH 0.2624 0.2375 0.1822 0.2380

Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 6 / 8

Motivation System overview Results

Evaluation Results: MAP metrics

Query Visual MAP MAP_bin MAP_tol MAiSPfields conceptstext none 0.5511 0.3529 0.3089 0.3431text Oxford 0.3196 0.2739 0.2053 0.2978visual Oxford 0.3368 0.2958 0.2293 0.3092text Leuven 0.3227 0.2801 0.2187 0.2958visual Leuven 0.3394 0.2970 0.2222 0.3117text CERTH 0.2295 0.2027 0.1554 0.1983visual CERTH 0.2624 0.2375 0.1822 0.2380

Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 6 / 8

Motivation System overview Results

Results Discussion and Future Work

We achieve higher scores when establishing theconnection between visual concepts in thesegments and visual query definition, overtextual fields.

We have shown that the visual concepts definedfor the other task and extracted for thiscollection can be transferred to be used inthis task.

We did not outperform the baseline yet.Further tuning of the confidence scorescalculation and segments reranking is planned.

Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 7 / 8

Motivation System overview Results

Thank you for your attention!

Questions?

Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 8 / 8