TRECVID-2012 Semantic Indexing task: Overview

Georges Quénot

Laboratoire d'Informatique de Grenoble

George Awad

Dakota Consulting, Inc

also with Franck Thollard, Bahjat Safadi (LIG) and Stéphane Ayache (LIF) and support from the Quaero Programme

Outline

Task summary

Evaluation details Inferred average precision

Participants

Evaluation results Pool analysis

Results per category

Results per concept

Significance tests per category

Global Observations

Issues

Semantic Indexing task (1)

Goal: Automatic assignment of semantic tags to video segments (shots)

Secondary goals: Encourage generic (scalable) methods for detector development.

Semantic annotation is important for filtering, categorization, browsing, searching, and browsing.

Participants submitted three types of runs:

Full run Includes results for 346 concepts, from which NIST and Quaero evaluated 46.

Lite run Includes results for 50 concepts, subset of the above 346, 15 evaluated.

Pair run Includes results for 10 concept pairs, all evaluated. *NEW*

TRECVID 2012 SIN video data

Test set (IACC.1.C): 200 hrs, with durations between 10 seconds and 3.5 minutes.

Development set (IACC.1.A, IACC.1.B & IACC.1.tv10.training): 600 hrs, with durations between 10 seconds to just longer than 3.5 minutes.

Total shots: (Much more than in previous TRECVID years, no composite shots)

Development: 403,800

Test: 145,634

Common annotation for 346 concepts coordinated by LIG/LIF/Quaero

Semantic Indexing task (2)

Selection of the 346 target concepts

Include all the TRECVID “high level features” from 2005 to 2010 to favor cross-collection experiments

Plus a selection of LSCOM concepts so that:

we end up with a number of generic-specific relations among them for promoting research on methods for indexing many concepts and using ontology relations between them

we cover a number of potential subtasks, e.g. “persons” or “actions” (not really formalized)

It is also expected that these concepts will be useful for the content-based (known item) search task.

Set of relations provided:

427 “implies” relations, e.g. “Actor implies Person”

559 “excludes” relations, e.g. “Daytime_Outdoor excludes Nighttime”

Semantic Indexing task (3)

NIST evaluated 20 concepts + 5 concept pairs and Quaero evaluated 26 concepts + 5 concept pairs.

Six training types were allowed

A - used only IACC training data

B - used only non-IACC training data

C - used both IACC and non-IACC TRECVID (S&V and/or Broadcast news) training data

D - used both IACC and non-IACC non-TRECVID training data

E – used only training data collected automatically using only the concepts’ name and definition *NEW*

F – used only training data collected automatically using a query built manually from the concepts’ name and definition *NEW*

Datasets comparison

TV2007

TV2008=

TV2007 + New

TV2009 =

TV2008 + New

TV2010

TV2011=

TV2010 + New

TV2012=

TV2011 + New

Dataset length (hours)

~100 ~200 ~380 ~400 ~600 ~800

Master shots

36,262 72,028 133,412 266,473 403,800 549,434

Unique program

titles 47 77 184 N/A N/A N/A

Number of runs for each training type REGULAR FULL RUNS (51 runs) A B C D E F

Only IACC data 47

Only non-IACC data 0

Both IACC and non-IACC TRECVID data 0

Both IACC and non-IACC non-TRECVID data 3

used only training data collected automatically using only the concepts’ name and definition

0

used only training data collected automatically using a query built manually from concepts’ name and definition

1

LIGHT RUNS (91 runs) A B C D E F

Only IACC data 83

Only non-IACC data 0

Both IACC and non-IACC TRECVID data 0

Both IACC and non-IACC non-TRECVID data 4

used only training data collected automatically using only the concepts’ name and definition

1

used only training data collected automatically using a query built manually from concepts’ name and definition

3

Number of runs for each training type PAIR RUNS (16 runs) A B C D E F

Only IACC data 14

Only non-IACC data 0

Both IACC and non-IACC TRECVID data 0

Both IACC and non-IACC non-TRECVID data 1

used only training data collected automatically using only the concepts’ name and definition

0

used only training data collected automatically using a query built manually from concepts’ name and definition

1

Total Runs (107) 97 0 0 5 1 4

90% 5% 1% 4%

56 concepts evaluated 3 Airplane

4 Airplane_Flying

9 Basketball

13 Bicycling

15 Boat_Ship

16 Boy

17 Bridges

25 Chair

31 Computers

51 Female_Person*

54 Girl

56 Government_Leader

57 Greeting

63 Highway

71 Instrumental_Musician

72 Kitchen

74 Landscape

75 Male_Person*

77 Meeting

80 Motorcycle

84 Nighttime*

85 Office

95 Press_Conference

99 Roadway_Junction

101 Scene_Text*

105 Singing*

107 Sitting_down*

112 Stadium

116 Teenagers

120 Throwing

-The 7 marked with “*” are a subset of those tested in 2011

128 Walking_Running* 155 Apartments

163 Baby

198 Civilian_Person

199 Clearing

254 Fields

267 Forest

274 George_Bush

276 Glasses

297 Hill

321 Lakes

338 Man_Wearing_A_Suit

342 Military_Airplane

359 Oceans

434 Skier

440 Soldiers

901 Beach + Mountain 904 Bird + Waterscape_waterfront 907 Person + underwater

902 Old_people + Flags 905 Dog + Indoor 908 Table + Telephone

903 Animal + Snow 906 Driver + Female_Human_face 909 Two_People + Vegetation

910 Car + Bicycle

Evaluation

Each feature assumed to be binary: absent or present for each master reference shot

Task: Find shots that contain a certain feature, rank them according to confidence measure, submit the top 2000

NIST sampled ranked pools and judged top results from all submissions

Evaluated performance effectiveness by calculating the inferred average precision of each feature result

Compared runs in terms of mean inferred average precision across the: 46 feature results for full runs

15 feature results for lite runs

10 feature results for concept-pairs runs

Inferred average precision (infAP)

Developed* by Emine Yilmaz and Javed A. Aslam at Northeastern University

Estimates average precision surprisingly well using a surprisingly small sample of judgments from the usual submission pools

This means that more features can be judged with same annotation effort

Experiments on previous TRECVID years feature submissions confirmed quality of the estimate in terms of actual scores and system ranking

* J.A. Aslam, V. Pavlu and E. Yilmaz, Statistical Method for System Evaluation Using Incomplete Judgments

Proceedings of the 29th ACM SIGIR Conference, Seattle, 2006.

2012: mean extended Inferred average precision (xinfAP)

3 pools were created for each concept and sampled as:

Top pool (ranks 1-200) sampled at 100%

Bottom pool (ranks 201-2000) sampled at 10%

Judgment process: one assessor per concept, watched complete shot while listening to the audio.

infAP was calculated using the judged and unjudged pool by sample_eval

56 concepts 282949 total judgments

35361 total hits 17739 Hits at ranks (1-100)

9783 Hits at ranks (101-200) 7839 Hits at ranks (201-2000)

2012 : 25 Finishers

PicSOM Aalto U.

INF Carnegie Mellon U.

CEALIST CEA

VIREO City U. of Hong Kong

ECL_Liris Ecole Centrale de Lyon, Universit de Lyon

EURECOM EURECOM - Multimedia Communications

VideoSense EURECOM VideoSense Consortium

FIU_UM Florida International U. U. of Miami

FTRDBJ France Telecom Orange Labs (Beijing)

kobe_muroran Kobe U., Muroran Institute of Technology

IBM IBM T. J. Watson Research Center

ITI_CERTH Informatics and Telematics Institute (Centre for Research and Technology)

Quaero INRIA, IRIT, LIG, U. Karlsruhe

ECNU Institute of Computer Applications, East China Normal U.

JRS.VUT JOANNEUM RESEARCH Forschungsgesellschaft mbH Vienna U. of Technology

IRIM IRIM - Indexation et Recherche d'Information Multimédia GDR-ISIS

NII National Institute of Informatics

NHKSTRL NHK Science and Technical Research Laboratories

ntt NTT Cyber Space Laboratories School of Software, Dalian U. of Technology

IRC_Fuzhou School of Mathematics and Computer Science Fuzhou U.

stanford Stanford U.

TokyoTechCanon Tokyo Institute of Technology and Canon

MediaMill U. of Amsterdam

UEC U. of Electro-Communications

GIM U. of Extremadura

2012 : 25/52 Finishers

Task finishers Participants

2012 25 52

2011 28 56

2010 39 69

2009 42 70

2008 43 64

2007 32 54

2006 30 54

2005 22 42

2004 12 33

Participation

and

finishing

declined!

Why?

0

2000

4000

6000

8000

10000

12000

14000

16000

18000

20000

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45

Scene_Text

Landscape Glasses

Frequency of hits varies by feature

1%**

**from total test shots

10 Female_person

18 Male_person

21 Night_time

25 Scene_Text

26 Singing

27 Sitting_down

31 Walking_running

2011

common

features

Male_person

Walking_running

Civilian_person

Female_person

Man_wearing_suit

True shots contributed uniquely by team

Team No. of Shots

Team No. of shots

CEA 664 Qu 10

VIR 469

FIU 464

IBM 427

UEC 271

UvA 209

nii 156

ITI 127

NHK 99

FTR 63

Tok 146

Pic 46

IRI 37

CMU 16

Full runs Lite runs

Team No. of Shots

Team No. of shots

Fud 363 Vid 32 CEA 218 Kob 31 nii 211 NTT 26

FIU 190 FTR 25 GIM 132 NHK 23

UvA 119 Ecl 20

JRS 103 ITI 18

IBM 95 IRI 10

Tok 79 CMU 5

UEC 71 Pic 4

VIR 71 ECN 2

sta 37

Eur 33

Less unique

shots

compared

to TV2011

Baseline run by NIST

A median baseline run is created for each run type and training category.

Basic idea:

For each feature, find the median rank of each submitted shot calculated across all submitted runs in that run type and training category.

The final shot median rank value is weighted by the ratio of all submitted runs to number of runs that submitted that shot:

dXnsSubmitteNumberOfRu

rOfRunsTotalNumberankMedianShotX rankMedian *__

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

A_T

oky

oTe

chC

ano

n3

_brn

A

_To

kyo

Tech

Can

on

2_b

rn

A_T

oky

oTe

chC

ano

n1

_brn

A

_To

kyo

Tech

Can

on

4

A_U

vA.S

hel

do

n

A_U

vA.R

aj

A_U

vA.L

eon

ard

A

_-Q

uae

ro1

A

_nis

t.b

asel

ine.

med

ian

A

_-Q

uae

ro4

A

_-Q

uae

ro3

A

_-Q

uae

ro2

A

_IR

IM1

A

_IR

IM3

A

_Pic

SOM

_1

A_P

icSO

M_2

A

_IR

IM2

A

_Pic

SOM

_3

A_I

RIM

4

A_n

ii.K

itty

-AF1

A

_FTR

DB

J-SI

N-1

A

_CM

U4

A

_CM

U3

A

_CM

U2

A

_CM

U1

A

_Pic

SOM

_4

A_F

TRD

BJ-

SIN

-2

A_n

ii.K

itty

-AF2

A

_VIR

EO.B

asel

ine

A_I

BM

A

_IB

M

A_I

TI_C

ERTH

A

_ITI

_CER

TH

A_I

TI_C

ERTH

A

_UEC

1

A_I

TI_C

ERTH

A

_CEA

LIST

A

_NH

KST

RL1

A

_NH

KST

RL3

A

_CEA

LIST

A

_NH

KST

RL2

A

_NH

KST

RL4

A

_FIU

-UM

-1-b

rn

A_C

EALI

ST

A_F

IU-U

M-2

A

_FIU

-UM

-4

A_F

IU-U

M-3

-brn

A

_CEA

LIST

Category A results (Full runs)

Med

ian

= 0

.20

2

Mea

n In

fAP.

0.165 0.17

0.175 0.18

0.185 0.19

0.195 0.2

D_V

IREO

.Yo

uTu

be_

ASV

M

D_V

IREO

.Sem

anti

c_Fi

eld

_ASV

M

D_n

ist.

bas

elin

e.m

edia

n

D_V

IREO

.SP

_ASV

M

Category D results (Full runs) M

ean

InfA

P.

Med

ian

0

.18

0

Note: Category F has only 1 run (F_VIREO.Semantic_Pooling ) with score = 0.048

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

A_k

ob

e_m

uro

_l1

8

A_T

oky

oTe

chC

ano

n3

_brn

A

_ko

be_

mu

ro_l

6

A_U

vA.L

eon

ard

A

_ko

be_

mu

ro_r

18

A

_To

kyo

Tech

Can

on

4

A_-

Qu

aero

2

A_-

Qu

aero

3

A_n

ii.K

itty

-AF1

A

_sta

nfo

rd2

A

_Pic

SOM

_1

A_C

MU

3

A_C

MU

2

A_P

icSO

M_3

A

_IR

IM3

A_F

TRD

BJ-

SIN

-1

A_E

CN

U_1

A

_NT

T_D

UT_

1

A_E

CN

U_2

A

_Eu

reco

m_u

plo

ader

A

_EC

NU

_3

A_n

ist.

bas

elin

e.m

edia

n

A_V

IREO

.Bas

elin

e A

_NT

T_D

UT_

3

A_E

CN

U_4

A

_Eu

reco

m_V

ideo

Sen

se_

…

A_e

cl_l

iris

_kn

icks

A

_IB

M

A_I

TI_C

ERTH

A

_UEC

1

A_C

EALI

ST

A_N

HK

STR

L1

A_N

HK

STR

L4

A_V

ideo

sen

se_R

UN

2

A_C

EALI

ST

A_V

ideo

sen

se_R

UN

1

A_F

IU-U

M-4

A

_Vid

eose

nse

_RU

N3

A

_JR

SVU

T1

A_G

IM_R

un

3

A_C

EALI

ST

A_F

ud

aSys

Category A results (Lite runs) M

ean

InfA

P.

Med

ian

= 0

.21

2

0

0.05

0.1

0.15

0.2

0.25

0.3

D_U

vA.H

ow

ard

D_n

ist.

bas

elin

e.m

edia

n

D_V

IREO

.Yo

uTu

be_

ASV

M

D_V

IREO

.SP

_ASV

M

D_V

IREO

.Sem

anti

c_Fi

eld

_ASV

M

Category D results (Lite runs) M

ean

InfA

P.

Med

ian

= 0

.20

7

0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08

F_V

IREO

.Sem

anti

c_Po

olin

g

F_U

vA.B

ern

adet

te

F_n

ist.

bas

elin

e.m

edia

n

F_U

vA.P

enn

y

Category F results (Lite runs) M

ean I

nfA

P. M

edia

n =

0.0

54

Note: Category E has only 1 run (E_nii.Kitty-EL4 ) with score = 0.044

0

0.2

0.4

0.6

0.8

1

1.2

3

4

9

13

1

5

16

1

7

25

3

1

51

5

4

56

5

7

63

7

1

72

7

4

75

7

7

80

8

4

85

9

5

99

1

01

1

05

1

07

1

12

1

16

1

20

1

28

1

55

1

63

1

98

1

99

2

54

2

67

2

74

2

76

2

97

3

21

3

38

3

42

3

59

4

34

4

40

Median

10

9

8

7

6

5

4

3

2

1

Top 10 InfAP scores by feature (Full runs) In

f A

P.

3 Airplane

4 Airplane_flying

9 Basketball

13 Bicycling

15 Boat_ship

16 Boy

17 Bridges

25 Chair

31 Computers

51 Female_ person

54 Girl

56 Government_

Leader

57 Greeting

63 Highway

71 Instrumental_

Musician

72 Kitchen

74 Landscape

75 Male_ person

77 Meeting

80 Motorcycle

84 Nighttime

85 Office

95 Press_

Conference

99 Roadway_ Junction

101 Scene_Text

105 Singing

107 Sitting_down

112 Stadium

116 Teenagers

120 Throwing

128 Walking_ Running

155 Apartments

163 Baby

198 Civilian_Person

199 Clearing

254 Fields

267 Forest

274 George_Bush

276 Glasses

297 Hill

321 Lakes

338 Man_Wearing_

A_Suit

342 Military_Airplane

359 Oceans

434 Skier

440 Soldiers

Top 10 InfAP scores for 15 common features

(Lite AND Full runs)

In

fAP.

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

4 13 15 31 51 71 74 75 84 101 105 107 112 120 128

Median

10

9

8

7

6

5

4

3

2

1

4 Airplane

13 Bicycling

15 Boat_ship

31 Computers

51 Feamale_

Person

71 Instrumental_

Musician

74 Landscape

75 Male_person

112 Stadium

120 Throwing

128 Walking_ Running

84 Nighttime

101 Scene_text

105 Singing

107 Sitting_down

Features

Run name (mean infAP)

F_A_TokyoTechCanon3_brn_3 0.321 F_A_TokyoTechCanon2_brn_2 0.321 F_A_TokyoTechCanon1_brn_1 0.321 F_A_TokyoTechCanon4_4 0.298 F_A_UvA.Sheldon_1 0.297 F_A_UvA.Raj_2 0.289 F_A_UvA.Leonard_4 0.288 F_A_-Quaero1_1 0.269 F_A_-Quaero4_4 0.254 F_A_-Quaero3_3 0.254

Statistical significant differences among top 10 A-category full runs (using randomization test, p < 0.05)

Statistical significant differences among top 10 A-category full runs (using randomization test, p < 0.05) (2)

A_TokyoTechCanon2_brn_2

A_UvA.Raj_2

F_A_-Quaero1_1

F_A_-Quaero3_3

F_A_-Quaero4_4

A_UvA.Sheldon_1

F_A_-Quaero1_1

F_A_-Quaero3_3

F_A_-Quaero4_4

A_UvA.Leonard_4

F_A_-Quaero1_1

F_A_-Quaero3_3

F_A_-Quaero4_4

A_TokyoTechCanon4_4

F_A_-Quaero1_1

F_A_-Quaero3_3

F_A_-Quaero4_4

A_TokyoTechCanon1_brn_1

A_UvA.Raj_2

F_A_-Quaero1_1

F_A_-Quaero3_3

F_A_-Quaero4_4

A_UvA.Sheldon_1

F_A_-Quaero1_1

F_A_-Quaero3_3

F_A_-Quaero4_4

A_UvA.Leonard_4

F_A_-Quaero1_1

F_A_-Quaero3_3

F_A_-Quaero4_4

A_TokyoTechCanon4_4

F_A_-Quaero1_1

F_A_-Quaero3_3

F_A_-Quaero4_4

A_TokyoTechCanon3_brn_3

A_UvA.Raj_2

F_A_-Quaero1_1

F_A_-Quaero3_3

F_A_-Quaero4_4

A_UvA.Sheldon_1

F_A_-Quaero1_1

F_A_-Quaero3_3

F_A_-Quaero4_4

A_UvA.Leonard_4

F_A_-Quaero1_1

F_A_-Quaero3_3

F_A_-Quaero4_4

A_TokyoTechCanon4_4

F_A_-Quaero1_1

F_A_-Quaero3_3

F_A_-Quaero4_4

Statistical significant differences among top 10 D-category full runs (using randomization test, p < 0.05)

Run name (mean infAP)

F_D_VIREO.Semantic_Field_ASVM_5 0.180

F_D_VIREO.YouTube_ASVM_3 0.180

F_D_VIREO.SP_ASVM_4 0.179

No Significant

difference

Run name (mean infAP) L_A_kobe_muro_l18_3 0.358 L_A_TokyoTechCanon1_brn_1 0.355 L_A_TokyoTechCanon3_brn_3 0.354 L_A_TokyoTechCanon2_brn_2 0.353 L_A_kobe_muro_l6_1 0.348 L_A_UvA.Sheldon_1 0.346 L_A_UvA.Leonard_4 0.342 L_A_UvA.Raj_2 0.338 L_A_kobe_muro_r18_2 0.323 L_A_kobe_muro_l5_4 0.320

Statistical significant differences among top 10 A-category lite runs (using randomization test, p < 0.05)

A_kobe_muro_l18_3

L_A_kobe_muro_l6_1

L_A_kobe_muro_l5_4

L_A_kobe_muro_r18_2

Statistical significant differences among top D-category lite runs (using randomization test, p < 0.05)

Run name (mean infAP) L_D_UvA.Howard_3 0.282

L_D_VIREO.Semantic_Field_ASVM_5 0.207

L_D_VIREO.SP_ASVM_4 0.207

L_D_VIREO.YouTube_ASVM_3 0.207

L_D_UvA.Howard_3

L_D_VIREO.Semantic_Field_ASVM_5

L_D_VIREO.SP_ASVM_4

L_D_VIREO.YouTube_ASVM_3

Statistical significant differences among top F-category lite runs (using randomization test, p < 0.05)

Run name (mean infAP) L_F_VIREO.Semantic_Pooling_1 0.072

L_F_UvA.Bernadette_5 0.054

L_F_UvA.Penny_7 0.044

No Significant

difference

0

100

200

300

400

500

600

700

800

900

1000

1 2 3 4 5 6 7 8 9 10

Frequency of hits for concept pairs

1 Beach

+ Mountain

2 Old_people

+ Flags

3 Animal

+ Snow

4 Bird

+ Waterscape_

waterfront

5 Dog

+ Indoor

6 Driver

+ Female_human_

Face

7 Person

+ Underwater

8 Table

+ Telephone

9 Two_people

+ Vegetation

10 Car +

Bicycle

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

Category A results (Concept Pairs)

Med

ian

= 0

.04

1

Mea

n In

fAP.

Only 1 D and 1 F runs

Run name (mean infAP) P_A_TokyoTechCanon6_brn_6 0.076 P_A_FTRDBJ-SIN-3_3 0.071 P_A_baseline-firstconcept_3 0.056 P_A_baseline-combine-mul_1 0.056 P_A_baseline-combine-sum_2 0.054 P_A_CMU6_1 0.048 P_A_TokyoTechCanon5_brn_5 0.044 P_A_baseline-secondconcept_4 0.043 P_A_CMU5_2 0.039 P_A_FTRDBJ-SIN-4_4 0.032

Statistical significant differences among top 10 A-category Concept Pairs runs (using randomization test, p < 0.05)

A_TokyoTechCanon6_brn_6

A_CMU5_2

A_FTRDBJ-SIN-4_4

A_TokyoTechCanon5_brn_5

A_FTRDBJ-SIN-4_4

A_FTRDBJ-SIN-3_3

A_baseline-secondconcept_4

Site experiments include: focus on robustness, merging many different representations use of spatial pyramids improved bag of word approaches Fisher/super-vectors, VLADs, VLATs sophisticated fusion strategies (IRIM presentation to follow) combination of low and intermediate/high features analysis of more than one keyf rame per shot audio analysis using temporal context information use of metadata (Eurecom presentation to follow) machine learning: automatic evaluation of modeling strategies consideration of scalability issues

Some participation on the concept pair task (see Mediamill presentation to follow)

Still no improvement using external training data

Observations

2:10 - 2:30, Eurecom - Multimedia Communications (EURECOM)

2:30 - 2:50, IBM Research (IBM)

2:50 - 3:10, Kobe University; Muroran Insitute of Technology (kobe_muroran)

3:10 - 3:30, Break with refreshments served in the NIST West Square Cafeteria

3:30 - 3:50, Indexation et Recherche d'Information Multimédia GDR-ISIS (IRIM)

3:50 - 4:20, University of Amsterdam (MediaMill)

4:20 - 4:40, Discussion

4:50 p.m. NIST bus to Holiday Inn, Gaithersburg

Presentations to follow

Less participation again

Poll last year: Task becoming too big?

No new increase except for the development set.

Not enough novelty? Concept pair and “no annotation”.

US Aladdin program / MED task competition?

“Too much time was spent on extracting features but more effort should be on developing new frameworks and learning methods”, “Provide more auxiliary infor-mation, such as speech recognition results, or others”: IRIM initiative: sharing descriptors, classifier outputs and more

(see IRIM’s presentation to follow)

too late and too few for 2012 but ready for 2013 and more.

Maybe the number is hidden in joint participations?

SIN 2013

Globally keep the task similar and of similar scale

Further explore the “concept pair” and “no annotation” variants

Common training data for the “no annotation” variant is likely to be delivered LIG (F type)

Sharing of data proposed by IRIM

Possible method for measuring progress over years

New subtask about concept localization under consideration → annotation issue

Collaborative annotation available much earlier (end of February)

Feedback welcome

Sharing of data for TRECVID SIN

Organized by the IRIM groups of CNRS GRD ISIS.

IRIM proposes its data sharing organization for the TRECVID SIN task. This comprises:

a wiki with read-write access for all

a data repository with read access for all and currently a write access only via one of the organizers

a small set of simple file formats

a (quite) simple directory structure

Shared data mostly consist in descriptors and classification scores.

Rewarding principle (same as for other contributions)

share and be cited and evaluated

use freely and cite

Sharing of data for TRECVID SIN

Wiki (access with tv12 active participant login/password): http://mrim.imag.fr/trecvid/wiki

http://mrim.imag.fr/trecvid/wiki/doku.php?id=sin_2012_task

Associated data for SIN 2012 (access with IACC collection login/password): http://mrim.imag.fr/trecvid/sin12

Related actions: Sharing of low-level descriptors by CMU for TRECVID 2003-2004

Mediamill challenge (101 concepts) using TRECVID 2005 data

Sharing of detection scores by CU-Vireo on TRECVID 2008-2010 data

Possible extension to other TRECVID tasks, e.g. MED.