What’s happening in iCLEF? (the iCLEF Flickr Challenge)

transcript

What’s happening in iCLEF?(the iCLEF Flickr Challenge)Julio Gonzalo (UNED),

Paul Clough (U. Sheffield), Jussi Karlgren (SICS), Javier Artiles (UNED), Víctor Peinado (UNED)

iCLEF 2007/8Lessons learntiCLEF 2006 Good news FlickrRetrospective

A piece of good news for CLEF!

People have started to hyperlink MT versions of web pages!

The original URL

The world is in danger

The world needs Multilingual Information Access research/solutions

…(that do care about users)

Retrospective: goals for iCLEF 2006 Find realistic scenarios where multilingual search

arises naturally multilingual web communities growing around(mostly) non-textual items (ebay, Flickr)Explore Web 2.0 “new text” challenges: folksonomies,

blogs, wikis, etc. Reduce the cost of entry for new participants.

Provide a default multilingual search interface to Flickr. Improve the cost-effectiveness of the traditional

experiment design. Let participants work on their own experimental designs.

Flickr

Flickr advantages

Naturally multilingual Web 2.0 challenges (folksonomies) Replicability (unlike general web image

search facilities)

iCLEF 2006

Topics Ad hoc: find as many

photographs of (different) european parliaments as possible.

Creative: find five illustrations for this article about saffron in Italy.

Visual: What is the name of the beach where this crab is lying on?

Methodology Participants must

propose their own evaluation measures and experiment design.

Experiments

UNED focused on user’s behaviour How users deal with native/passive/unknown languages? Do they actually use CLIR facilities when available?

SICS focused on user’s perceptions Satisfaction (all tasks) Completeness (creative,ad-hoc) Quality (creative)

Sheffield also measured search effectiveness How many facets were retrieved (creative, ad-hoc) Was the image found? (visual)

Example result: choice of target languages along time.

Achievements

Found a natural CLIR task Flickr works for CLIR testing New perspectives for experiment design. Moved from bilingual to multilingual search

problems.

Stick to Flickr in 2007

Why participation was low?

Kept on recruiting users and forcing them to search whatever we asked for. Experiment design still too costly. User populations still too small.

Cost of entry still too high Building a Flickr interface / adapting the provided interface

was not trivial. Experiment design too open for newcomers.

Schedule collapses with standard tracks. A problem for CLEF groups potentially interested.

How to improve?

Kept on recruiting users and forcing them to search whatever we asked for. Experiment design still too costly. User populations still too small.

Cost of entry still too high Building a Flickr interface / adapting the provided interface

was not trivial. Experiment design too open for newcomers.

Schedule collapses with standard tracks. A problem for CLEF groups potentially interested.

Move to Flickr/Web users Make search task a game

Provide experiment design Participants ≈ Searchers

Adjust iCLEF calendar

iCLEF 2007 (i) Target search task

海 (sea, wave)

Find this image Clear goal for the user Clearly defined measures of success (appropriate

for a “Hall of Fame” experience) Invokes different searching strategies

iCLEF 2007 (ii) Evaluation aspects User’s behaviour (Observational/Log Analysis)

Search strategies with native/passive/unknown languages User’s perceptions

Usefulness Difficulty Clarity, etc.

Search effectiveness Success rate and its correlation with user profiles, search

strategies, etc.

iCLEF 2007 (iii) CL search interfaceSingle web interface for all experiments.

Available for: Groups willing to recruit users and conduct their

own experiments. Any web user wishing to enter the iCLEF Hall of

Fame (by finding more images than anyone else!). Groups willing to experience CL searching from

the user’s perspective: who is the best CLEF team searching cross-language?

Full experiments

iCLEF 2007/2008 schedule

End of October: Feedback from you! on the multilingual search interface experiment design evaluation measures

Mid-December: Guidelines + interface goes public Groups start their own experiments. People can start competing in the search challenge. Organization starts collecting logs from Flickr users

Mid-April: Flickr/Web user Logs released to interested groups Thematic session in Chorus Conference on Multimedia

Search and Access (hosted by Yahoo!). Paper submission: aligned with CLEF 2008

Start testing the interface now!

http://bender.lsi.uned.es:8080/iCLEF2007

Feedback to julio@lsi.uned.es

What’s happening in iCLEF? (the iCLEF Flickr Challenge)

Documents