Welcome to CLEF 2009

Carol PetersISTI-CNR, Pisa, Italy

Happy 10th Birthday

CLEF 2009 Workshop30 September – 2 October, Corfu, Greece

CLEF Objectives

Stimulate the development of multilingual IR systems (for European languages !)

Create a CLIR/MLIA community Construct publicly available test-suites

BY Conducting annual evaluation campaigns

Designing tracks/tasks to meet emerging needs and to stimulate research in the”right” direction

Objective: truly multilingual/multimedia systems

CLEF 2009 Workshop30 September – 2 October, Corfu, Greece

Evolution of CLEFCLEF 2000 Tracks

• mono-, bi- & multilingual text doc retrieval (Ad Hoc)• mono- and cross-language information on structured scientific data (Domain-Specific)

CLEF 2001 New

• interactive cross-language retrieval (iCLEF)

CLEF 2002 New

•cross-language spoken document retrieval (CL-SR)

CLEF 2003 New

• multiple language question answering (QA@CLEF) • cross-language retrieval in image collections (ImageCLEF)

CLEF 2005 New

• multilingual retrieval of Web documents (WebCLEF)• cross-language geographical retrieval (GeoCLEF)

CLEF 2008New

• cross-language video retrieval (VideoCLEF)• multilingual information filtering (INFILE@CLEF)

CLEF 2009New

• intellectual property (CLEF-IP)• log file analysis (LogCLEF)• large-scale grid experiments (Grid@CLEF)

CLEF Tracks: 2000 - 2009

Multilingual textual document retrieval (Ad Hoc) Interactive cross-language retrieval (iCLEF) Multiple language question answering (QA@CLEF) Cross-language retrieval in image collections

(ImageCLEF) Multilingual information filtering (INFILE@CLEF) Cross-language video retrieval (VideoCLEF) Intellectual property (CLEF-IP) – New this year Log file analysis (LogCLEF) – New this year MorphoChallenge (in collaboration with Pascal NoE)

CLEF 2009 Tracks

Pilot: Grid Experiments (Grid@CLEF)

CLEF 2009 Workshop30 September – 2 October, Corfu, Greece

CLEF2009 Coordination

Adaptive Informatics Research Centre, Helsinki University of Technology, Finland

Berlin School of Library and Information Science, Humboldt-Universität zu Berlin

Business Information Systems, U. Applied Sciences Western Switzerland, Sierre, Switzerland

CEA LIST, France Centre for Evaluation of Human Language & Multimodal

Communication Technologies (CELCT), Italy Centruum vor Wiskunde en Informatica, Amsterdam, NL Computer Science Department, U. Basque Country, Spain Computer Vision and Multimedia Lab, U. Geneva, CH Data Base Research Group, U. Tehran, Iran Dept. of Computer Science and Information Systems, U.

Limerick, Ireland Dept. of Information Engineering, U. Padua, Italy Dept. of Information Science, U. Hildesheim, Germany Dept. of Information Studies, U. Sheffield, UK Dept. of Medical Informatics and Clinical Epidemiology,

Oregon Health and Science U., USA Dept. Medical Informatics, U. Hospitals and University of

Geneva, Switzerland Department of Medical Informatics, Aachen University of

Technology (RWTH), Germany

Evaluations and Language Resources Distribution Agency, Paris, France

GERiiCO, Université de Lille, France Idiap Research Institute, Switzerland Information Retrieval Facility (IRF), Austria Laboratoire d'Informatique pour la Mécanique et les

Sciences de l'Ingénieur (LIMSI), Orsay, France U. Nacional de Educación a Distancia, Madrid, Spain Linguateca, Sintef ICT, Norway Linguistic Modelling Lab., Bulgarian Acad Sci Matrixware Information Services, Austria Mediamatics, Delft University of Technology, The

Netherlands Mitre Corporation, USA NIST, USA NLE Lab., Universidad Politènica de Valencia, Spain Research Institute for Artificial Intelligence, Romanian

Academy, Romania Romanian Institute for Computer Science, Romania Royal Institute of Technology (KTH), Stockholm,

Sweden School of Computing, DCU, Ireland Swedish Institute of Computer Science, Sweden TALP Research Center, Universitat Politécnica de

Catalunya, Barcelona, Spain.

CLEF is coordinated by the Istituto di Scienza e Tecnologie dell'Informazione, Consiglio Nazionale delle Ricerche, PisaThe following Institutions are contributing to the organisation of the different tracks of the CLEF 2008 campaign:

CLEF 2009 Workshop30 September – 2 October, Corfu, Greece

CLEF Steering Committee

Maristella Agosti, U.Padove, Italy Martin Braschler, Zurich, Switzerland Amedeo Cappelli, ISTI-CNR & CELCT, Italy Hsin-Hsi Chen, National Taiwan U., Taipei, Taiwan Khalid Choukri, ELRA/ELDA, Paris, France Paul Clough, University of Sheffield, UK Thomas Deselaers, RWTH Aachen University,

Germany Giorgio Di Nunzio, U. Padova, Italy David A. Evans, Clairvoyance Corporation, USA Nicola Ferro, U. Padova, Italy Christian Fluhr, CEA-LIST, Fontenay-aux-Roses,

France Norbert Fuhr, University of Duisburg, Germany Frederic C. Gey, U.C. Berkeley, USA Julio Gonzalo, LSI-UNED, Madrid, Spain Donna Harman, NIST, USA Gareth Jones, Dublin City University, Ireland Franciska de Jong, University of Twente,

Netherlands Noriko Kando, NII, Tokyo, Japan Jussi Karlgren, SICS, Sweden Michael Kluck, German Institute for International and

Security Affairs, Berlin, Germany

Natalia Loukachevitch, Moscow State University, Russia Bernardo Magnini, ITC-irst, Trento, Italy Thomas Mandl, U. Hildesheim, Germany Paul McNamee, Johns Hopkins University, USA Henning Müller, University & University Hospitals of

Geneva, Switzerland Douglas W. Oard, University of Maryland, USA Anselmo Peňas, LSI-UNED, Madrid, Spain Vivien Petras, Humboldt University, Berlin, Germany Maarten de Rijke, University of Amsterdam,

Netherlands Diana Santos, Linguateca, Sintef, Oslo, Norway Jacques Savoy, University of Neuchatel, Switzerland Peter Schäuble, Eurospider Information Technologies,

Switzerland Richard Sutcliffe, University of Limerick, Ireland Hans Uszkoreit, German Research Center for Artificial

Intelligence (DFKI), Germany Felisa Verdejo, LSI-UNED, Madrid, Spain José Luis Vicedo, University of Alicante, Spain Ellen Voorhees, NIST, USA Christa Womser-Hacker, University of Hildesheim,

Germany

CLEF 2009 Workshop30 September – 2 October, Corfu, Greece

CLEF2009:Track Coordinators

Ad Hoc: Abolfazl AleAhmad, Hadi Amiri, Eneko Agirre, Giorgio Di Nunzio, Nicola Ferro, Nicolas Moreau, Arantxa Otegi, Vivien Petras

iCLEF: Paul Clough, Julio Gonzalo, Jussi Karlgren QA@CLEF: Iñaki Alegria, Davide Buscaldi, Luís Miguel Cabral, Pere R. Comas,

Corina Forascu, Pamela Forner, Olivier Galibert, Danilo Giampiccolo, Nicolas Moreau, Djamel Mostefa, Petya Osenova, Anselmo Peñas, Álvaro Rodrigo, Sophie Rosset, Paolo Rosso, Diana Santos, Richard Sutcliff and Jordi Turmo

ImageCLEF: Brian Bakke, Steven Bedrick, Barbara Caputo, Paul Clough, Peter Dunker, Thomas Deselaers, Thomas Deserno, Ivan Eggel, Mark Oliver Güld, William Hersh, Patric Jensfelt, Charles E. Kahn Jr., Jana Kludas, Jayashree Kalpathy–Cramer, Henning Müller, Stefanie Nowak, Monica Lestari Paramita, Andrzej Pronobis, Saïd Radhouani, Mark Sanderson, Tatiana Tommasi, Theodora Tsikrika and Petra Welter

VideoCLEF: Gareth J.F. Jones, Martha Larson and Eamonn Newman INFILE: Romaric Besançon, Stéphane Chaudiron, Khalid Choukri, Meriama Laïb,

Djamel Mostefa and Ismaïl Timimi CLEF-IP: Florina Piroi, Giovanna Roda, John Tait and Veronika Zenz LogCLEF: Maristella Agosti, Giorgio Di Nunzio, Christine Doran, Inderjeet Mani,

Thomas Mandl, Julia Maria Schulz and Alexander Yeh Grid@CLEF: Nicola Ferro and Donna Harman MorphoChallenge: Graeme W. Blackwood, William Byrne Mikko Kurimo, Ville T.

Turunen and Sami Virpioja

CLEF 2009 Workshop30 September – 2 October, Corfu, Greece

CLEF 2009: Participating Groups

CLEF 2009 Workshop30 September – 2 October, Corfu, Greece

CLEF: Trend in Participation

CLEF2009: Europe = 81(69); N. America = 18(12); Asia = 16(15), S. America = 1(3), Africa = 1(1)

CLEF 2000 - 2009 Participation

0

10

20

30

40

50

60

70

80

90

100

110

120

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

Oceania

Africa

South Am erica

North Am erica

As ia

Europe

CLEF 2009 Workshop30 September – 2 October, Corfu,

Greece

CLEF2009:Participants per Track

Ad Hoc: 28(26) iCLEF: 6(6) QA@CLEF: 25(29) ImageCLEF: 62 (42) INFILE: 5(1)

VideoCLEF: 7(5) MorphoChallenge:

9(6)

CLEF-1P: 15 (new) LogCLEF: 6 (new) Grid@CLEF: 2 (pilot)

CLEF 2009 Workshop30 September – 2 October, Corfu, Greece

Participation by Track

CLEF 2009 Tracks

AdHoc

iCLEF

QA@CLEF

ImageCLEF

VideoClef

INFILE

MorphoChal lenge

CLEF-IP

LogCLEF GridCLEF

CLEF 2009 Workshop30 September – 2 October, Corfu, Greece

CLEF 2000 – 2009Participation per Track

CLEF 2000 - 2009 Tracks

0

10

20

30

40

50

60

70

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009Years

Par

ticip

atin

g G

roup

s

AdHoc

DomSpec

iCLEF

CL-SR

QA@CLEF

ImageCLEF

WebClef

GeoClef

VideoClef

INFILE

MorphoChallenge

CLEF-IP

LogCLEF

GridCLEF

CLEF 2009 Workshop30 September – 2 October, Corfu, Greece

CLEF Achievements Creation of very active multidisciplinary international

research community Investigation of core issues in MLIA

development of multiple language processing tools creation of linguistic resources implementation of appropriate cross-language retrieval models

and algorithms for different tasks and languages; Creation of important reusable test collections and

resources in diverse media for a large number of European languages

Significant and quantifiable improvements in the performance of MLIA systems

CLEF 2009 Workshop30 September – 2 October, Corfu, Greece

Test Collections2000 News documents in 4 languages GIRT German Social Science database

2009 A subset of CLEF multilingual corpus of news documents (Ad Hoc WSD-Robust) Hamshahri Persian newspaper corpus (Ad Hoc Persian task) TEL Library catalog records plus log files (Ad Hoc TEL task and LogCLEF) Flickr web-based image database (iCLEF) ResPubliQA document collection, (QAatCLEF: ResPubliQA) Transcripts of European parliamentary sessions & French news broadcasts (QAST) BELGAPICTURE image collection (ImageCLEFPhoto) Multilingual collections of Wikipedia documents and images (ImageCLEFwiki) Articles and images from Radiology and Radiography; IRMA collection for medical image annotation (ImageCLEFmed and medAnnotation) Dutch and English documentary television programs (VideoCLEF) Agence France Press (AFPnewswire stories in Arabic, French and English (INFILE) Patent documents from the European Patent Office (CLEF-IP)

Outline of Talk

Why, How, What

DIRECT@work in CLEF

CLEF 2009 Workshop30 September – 2 October, Corfu, Greece

CLEF Publications

9 CLEF Post-workshop Proceedings, Springer LNCS

Nearly 1000 Working Notes Reports on-line Google Scholar > 15,000 publications with

CLEF – cross-language information retrieval

CLEF 2009 Workshop30 September – 2 October, Corfu, Greece

CLEF 2008 Proceedings

Evaluating Systems for Multilingual and Multimodal Information Access 9th Workshop of the Cross-Language Evaluation Forum, CLEF 2008, Aarhus, Denmark, September 17-19, 2008, Revised Selected Papers Series: Lecture Notes in Computer Science , Vol. 5706Peters, C.; Deselaers, T., Ferro, N., Gonzalo, J., Jones, G.J.F., Kurimo, M., Mandl, Th.; Peñas, A.; Petras, V (Eds.)

CLEF 2009 Workshop30 September – 2 October, Corfu, Greece

Many, many thanks to Danilo Giampiccolo, CELCT

CLEF & TrebleCLEF

CLEF 2008 & 2009 is an activity of the TrebleCLEF Coordination Action under the Seventh Framework Programme of the European Commission.

TrebleCLEF organises a set of dissemination activities in the multilingual information access field.

CLEF 2009 Workshop30 September – 2 October, Corfu, Greece

Treble-CLEF

The CLEF research results have led to development of a new generation of multilingual retrieval system prototypes

BUT lack of technology transfer

Treble-CLEF extends the CLEF activity by: continuing to promote MLIA R&D via evaluation campaigns; providing a consistent training activity: tutorials, workshops,

summer school; producing best practice guidelines for system implementation; providing resources to encourage the multilingual system

development.

www.trebleclef.eu

CLEF 2009 Workshop30 September – 2 October, Corfu, Greece

Approach Evaluation

test collections and laboratory evaluation

user evaluation log analysis

Best Practices & Guidelines system-oriented aspects of

MLIA applications collaborative user studies user-oriented aspects of MLIA

interfaces

Dissemination and Training tutorials workshops summer school

CLEF 2009 Workshop30 September – 2 October, Corfu, Greece

Treble-CLEF Events

Workshop on Novel Methodologies for Evaluation in Information Retrieval, ECIR’08, Glasgow, Scotland

Workshop on Best Practices for the Development of Multilingual Information Access Systems, Segovia, Spain, June 08

Workshop on Best Practices for System Developers: Bringing Multilingual Information Access to Operational Systems, Winterthur, Switzerland, October 2008

Workshop on Best Practices in Query Log Analysis, London, May 2009

Summer School on Multilingual information Access, Pisa, June 2009

MLIA Technology Day – Dissemination of Results of Best Practices Workshops, 8 December 2009

CLEF 2009 Workshop30 September – 2 October, Corfu, Greece

CLEF 2010

Can CLEF continue without external funding for the central coordination?

??????

Participate in the discussion on Friday afternoon

CLEF 2009 Workshop30 September – 2 October, Corfu, Greece

Before I Forget

1.Posters: Put up between 8.00 & 13.30 Thursday morning

Pool Room 2: ImageCLEF PostersPool Room 1: All other Posters & Demos

2.USB StickECDL Conference ProceedingsAll Workshop Proceedings

3.Social Dinner - TonightHotel restaurant – 8 pm (Badge)

CLEF 2002Rome

CLEF 2006Alicante

CLEF 2005Vienna

CLEF 2004Bath

CLEF 2003Trondheim

CLEF 2008Aarhus

CLEF 2001Darmstadt

CLEF 2007Budapest

CLEF 2000

Lisbon

CLEF2009Corfu

CLEF2010?????

Come to CLEF – and see Europe!

CLEF 2009

Thank you for your attention

and

ENJOY THE Workshop !

CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008