PLACING TASK 2015

Jaeyoung Choi (ICSI, Berkeley / TU Delft, Netherlands) Claudia Hauff (TU Delft, Netherlands)

Olivier Van Laere (Blueshift Labs, San Francisco) Bart Thomee (Yahoo Labs, San Francisco)

Sept 15th, 2015 Wurzen, Germany

TASK CHANGES

• two new sub-tasks• locale-based sub-task• mobility-based sub-task

• organizer baselines

• live leaderboard

LOCALE-BASED SUBTASK

• given a video or a photo, place it as accurately as possible on a map

LOCALE-BASED SUBTASK

• participants were given a geographic hierarchy of locales across the world• principally based on the GADM administrative boundaries• supplemented with neighborhood data from ClickThatHood

• participants choose the locale in the hierarchy where they most confidently believe a photo/video was taken

• or, if sufficiently confident, they pick a longitude and latitude coordinate

GADMDATABASE OF GLOBAL ADMINISTRATIVE AREAS

http://www.gadm.org/

CLICK THAT ‘HOOD!

click-that-hood.com

HIERARCHY DATA EXAMPLE

• United States@CA|Calif.|California@San Francisco@South of Market• ‘@‘ separates each layer in the hierarchy • ‘|’ separates any alternative name spellings

MOBILITY-BASED SUBTASK

• participants were given a sequence of photos captured by a user within a certain city• not all of the photos within the sequence were geo-tagged

• participants were asked to estimate the location of all photos with missing geo-coordinates

TASK DATASET

• drawn from the Yahoo Flickr Creative Commons 100 Million (YFCC100M) dataset

training testing#photos #videos #photos #videos

locale-based 4,672,382 22,767 931,573 18,316

mobility-based 148,349 0 33,026 0

PRECOMPUTED FEATURES• textual metadata

• as included in YFCC100M• visual features

• LIRE, GIST, SIFT• audio features

• MFCC, Pitch (Kaldi, SAcC)

paul bica

ORGANIZER BASELINES

• two open-source baselines were provided, one for each subtask• locale-based baseline - LM-based using textual metadata [Van

Laere et al. 2013] • mobility-based baseline - extrapolates location from temporal

neighbors [Hauff et al. SIGIR 2012]

https://github.com/ovlaere/placing-text/blob/mediaeval2015/README_placing2015.mdhttps://github.com/chauff/ImageLocationEstimation

LIVE LEADERBOARD• participants could submit runs and view their relative

standing toward others• evaluated on a development set, approx. 30% of test set• considered to be useful by participants

TASK EVALUATION

• locale-based subtask• hierarchical distance between ground truth locale and

predicted locale in the place hierarchy• geographic distance between ground truth coordinate and

predicted coordinate • mobility-based subtask

• geographic distance-based metric

• custom formula was used for the hierarchy-based metric• Karney’s formula was used for the distance-based metric

TASK EVALUATION

RUNS• run 1

• only provided textual metadata may be used• run 2

• only provided visual & aural features may be used• run 3

• only provided textual metadata, visual features and the visual & aural features may be used

• run 4–5• everything is allowed, except for crawling the exact items

contained in the test set, or any items by a test user taken within 24 hours before the first and after the last timestamp of a photo sequence in the mobility test set

PARTICIPANTS STATISTICSlocale mobility

run1 run2 run3 run4/5 run1 run2 run3 run4/5

CERTH/CEA LIST O O O O

RECOD O O O O

ImCube O O O O O O O OJKU_

Satellite O O O

Geo_ML O

TUDelft O (late)

LOCALE RESULTS - RUN 1cu

mula

tive

perc

enta

ge

0

15

30

45

60

75

geographic distance from groundtruth

1m 100m 10km 1000km

CERTH/CEA LIST ImCubeRECOD Geo_MLBaseline

cum

ulativ

e pe

rcen

tage

0

15

30

45

60

75

hierarchical distance from groundtruth

1m 100m 10km 1000km

CERTH/CEA LIST ImCubeRECOD Geo_MLBaseline

LOCALE RESULTS - RUN 2cu

mula

tive

perc

enta

ge

0

8

16

24

32

40

geographic distance from groundtruth

1m 100m 10km 1000km

CERTH/CEA LIST ImCubeRECOD TUD*

cum

ulativ

e pe

rcen

tage

0

8

16

24

32

40

hierarchical distance from groundtruth

1m 100m 10km 1000km

CERTH/CEA LIST ImCubeRECOD TUD*

LOCALE RESULTS - RUN 3cu

mula

tive

perc

enta

ge

0

15

30

45

60

75

geographic distance from groundtruth

1m 100m 10km 1000km

CERTH/CEA LIST ImCubeRECOD Baseline

cum

ulativ

e pe

rcen

tage

0

15

30

45

60

75

hierarchical distance from groundtruth

1m 100m 10km 1000km

CERTH/CEA LIST ImCubeRECOD Baseline

LOCALE RESULTS - RUN 4/5cu

mula

tive

perc

enta

ge

0

15

30

45

60

75

geographic distance from groundtruth

1m 100m 10km 1000km

CERTH/CEA LIST ImCubeRECOD Baseline

cum

ulativ

e pe

rcen

tage

0

15

30

45

60

75

hierarchical distance from groundtruth

1m 100m 10km 1000km

CERTH/CEA LIST ImCubeRECOD Baseline

MOBILITY RESULTS - RUN 1+2cu

mula

tive

perc

enta

ge

0

20

40

60

80

100

geographic distance from groundtruth

1m 100m 10km 1000km

ImCube JKU_Satellite

cum

ulativ

e pe

rcen

tage

0

20

40

60

80

100

geographic distance from groundtruth

1m 100m 10km 1000km

ImCube JKU_Satellite

MOBILITY RESULTS - RUN 3+4cu

mula

tive

perc

enta

ge

0

20

40

60

80

100

geographic distance from groundtruth

1m 100m 10km 1000km

ImCube JKU_Satellite

cum

ulativ

e pe

rcen

tage

0

20

40

60

80

100

geographic distance from groundtruth

1m 100m 10km 1000km

ImCube

CONCLUSIONS

• no visible trend; many different approaches• pre-processing, language model, clustering, GP, etc.

• limited time available for participants• leaderboard, features were helpful

• local semantic features from visual cue to be added