Date post: | 01-Jan-2016 |
Category: |
Documents |
Upload: | maxwell-hill |
View: | 20 times |
Download: | 0 times |
CEA LIST ELDA Univ. Lille 3 - Geriico 101/10/09
CLEF@
1
INFILE
Overview of the INFILE Overview of the INFILE track at CLEF 2009track at CLEF 2009
multilingual INformation FILtering Evaluationmultilingual INformation FILtering Evaluation
Romaric Besançon (1), Djamel Mostefa, Olivier Hamon, Khalid Choukri (2), Stéphane Chaudiron,Ismaïl Timimi (3)
(1) (2) (3)
CEA LIST ELDA Univ. Lille 3 - Geriico 201/10/09
CLEF@
2
INFILE
Presentation of the INFILE track
Information Filtering EvaluationFilter documents from a document stream according to
long-term information needs (user profiles)
Second edition of the INFILE track in CLEF1 participant in 2008use same data in 2009
CEA LIST ELDA Univ. Lille 3 - Geriico 301/10/09
CLEF@
3
INFILE
Presentation of the INFILE track
Mutlilingual
English, French, Arabic for both documents and topics
Two tasks
batch filteringthe whole corpus is given to the participants, which
must return a list of filtered documents for each topic
adaptive filteringdocuments are provided to the participants one at a
time through an interactive procedure, with possible automated feedback to adapt the filtering system
closer to real usage in a context of competitive intelligence
CEA LIST ELDA Univ. Lille 3 - Geriico 401/10/09
CLEF@
4
INFILE
Document Collection
Built from a corpus of news from the AFP (Agence France Presse)
almost 1.5 million news in French, English and Arabic
For the information filtering task:
100 000 documents to filter, in each language NewsML format
standard XML format for news (IPTC)
CEA LIST ELDA Univ. Lille 3 - Geriico 501/10/09
CLEF@
5
INFILE
Document example
document identifier
keywords
headline
CEA LIST ELDA Univ. Lille 3 - Geriico 601/10/09
CLEF@
6
INFILE
Document example
IPTC category
AFP category
content
CEA LIST ELDA Univ. Lille 3 - Geriico 701/10/09
CLEF@
7
INFILE
Topics
50 interest profiles
20 profiles in the domain of science and technology
developped by CI professionals from French institutes INIST, ARIS, Oto Research, Digiport
30 profiles of general interest Profiles developed in French/English Translated into Arabic
CEA LIST ELDA Univ. Lille 3 - Geriico 801/10/09
CLEF@
8
INFILE
Topics
Each profile contains 5 fields:
title: a few words description
description: a one-sentence description
narrative: a longer description of what is considered a relevant document
keywords: a set of key words, key phrases or named entities
sample: a sample of relevant document (one paragraph)
Participants may use any subset of the fields for their filtering
CEA LIST ELDA Univ. Lille 3 - Geriico 1001/10/09
CLEF@
10
INFILE
Some topic examples
101102107113115118119127129
Fight against doping in sportsport economyElectronic votingDigital DivideThe free museumsRising oil pricesthe subprimes crisisthe crisis in DarfurThe FARC rebelion
131132136137138140143144149
E-government stakesWireless network and healthAir pollution and air qualityFight against climate changeDrugs and biotechnologyFruits and vegetables intakes and cancer preventionAvian influenzaNanotechnologies and nanosciencesScientific research in Arctic
in general domain
in scientific information domain
CEA LIST ELDA Univ. Lille 3 - Geriico 1101/10/09
CLEF@
11
INFILE
Constitution of the corpus
Same corpus as INFILE@CLEF 2008
With simulated feedback, we need the ground truth before the campaign
To build the corpus of documents to filter:find relevant documents for the profiles in the original
corpususe a pooling technique with results of IR tools
4 IR engines (Lucene, Indri, Zettair and CEA search engine), on several query fields combinations
iterative pooling using Mixture-of-Experts model
CEA LIST ELDA Univ. Lille 3 - Geriico 1201/10/09
CLEF@
12
INFILE
Constitution of the corpus (2)
keep all documents assessed
documents returned by IR systems by judged not relevant form a set of difficult documents
choose random documents (noise)
collection
retrieved
assessed
relevant
test collection
random
CEA LIST ELDA Univ. Lille 3 - Geriico 1301/10/09
CLEF@
13
INFILE
Corpus1
01
10
21
03
10
41
05
10
61
07
10
81
09
11
01
11
11
21
13
11
41
15
11
61
17
11
81
19
12
01
21
12
21
23
12
41
25
12
61
27
12
81
29
13
01
31
13
21
33
13
41
35
13
61
37
13
81
39
14
01
41
14
21
43
14
41
45
14
61
47
14
81
49
15
0
0
50
100
150
200engfreara
ara7312 7886 51241597 2421 1195
31,94 48,42 23,928,45 47,82 23,08
[0,107] [0,202] [0,101]
eng frenumber of documents assessednumber of relevant documentsavg number of relevant docs / topicstd deviation on number of relevant docs / topic[min,max] number of relevant docs / topics
Number of relevant documents for each topic, in each language
CEA LIST ELDA Univ. Lille 3 - Geriico 1401/10/09
CLEF@
14
INFILE
Tasks
Batch filtering (02/04 - 30/05)documents and topics available to participantsreturn list of filtered documents per topic (unordered)
Adaptive filtering (03/06 - 10/07)topics available to participantsdocuments available one at a time (one pass test)
interactive protocol using a client-server architecture (webservice communication)
new document available only if previous one has been filtered
available simulated user feedbackfor adapatationlimited number of feedbacks (200)
CEA LIST ELDA Univ. Lille 3 - Geriico 1501/10/09
CLEF@
15
INFILE
Evaluation metrics
Standard precision / recall / F-measure Utility (from TREC filtering tracks)
per profile and averaged on all profiles adaptivity: evolution curve (values computed each
10000 documents)
two experimental measuresoriginality
number of relevant documents a system uniquely retrieves
anticipationinverse rank of first relevant document detected
CEA LIST ELDA Univ. Lille 3 - Geriico 1601/10/09
CLEF@
16
INFILE
INFILE Participants
9 registered 5 submitted runs
batch filtering
3 participants, 12 runs interactive filtering
2 participants, 3 runs27
countryIMAG Institut Informatique et Mathématiques Appliquées de Grenoble FranceSINAIUAIC
société CADEGE FranceUOWD
team name institute
University of Jaen SpainUniversitatea Alexandru Ioan Cuza of IASI Romania
HossurTechUniversity of Wollongong (Comp.Sci & Engineering) Dubai
CEA LIST ELDA Univ. Lille 3 - Geriico 1701/10/09
CLEF@
17
INFILE
INFILE results
Repartition of runs by task and languages
arafre
eng
eng
ara
fre
batchadaptive
CEA LIST ELDA Univ. Lille 3 - Geriico 1801/10/09
CLEF@
18
INFILE
INFILE results – monolingual batch filtering
F-scoreIMAG IMAG_1 1597 413 0,26 0,30 0,21 0,21UAIC 1597 1267 0,09 0,66 0,13 0,05UAIC 1597 1331 0,06 0,69 0,09 0,03UAIC 1597 1331 0,06 0,69 0,09 0,03UAIC 1597 1507 0,06 0,82 0,09 0,03IMAG IMAG_2 1597 109 0,13 0,09 0,07 0,16IMAG IMAG_3 1597 66 0,16 0,06 0,07 0,22SINAI 1597 940 0,02 0,50 0,04 0,00SINAI 1597 196 0,01 0,08 0,01 0,13
monolingual englishteam run num_rel num_rel_ret precision recall Utility
uaic_4uaic_1uaic_2uaic_3
topics_1googlenews_2
CEA LIST ELDA Univ. Lille 3 - Geriico 1901/10/09
CLEF@
19
INFILE
INFILE results – crosslingual / adaptive filtering
team run num_rel num_rel_ret precision recall F-score UtilityUAIC uaic_4 2421 1120 0,09 0,44 0,12 0,05UAIC uaic_3 2421 1905 0,06 0,75 0,10 0,03UAIC uaic_2 2421 1614 0,06 0,67 0,09 0,02
team run num_rel num_rel_ret precision recall F-score UtilityHossurTech 4 2421 790 0,05 0,31 0,06 0,05
team run num_rel num_rel_ret precision recall F-score UtilityHossurTech 1 1597 819 0,10 0,45 0,10 0,07
crosslingual english / french
monolingual french
crosslingual french / english
57% best mono90% same team mono
crosslingual better than monolingual
CEA LIST ELDA Univ. Lille 3 - Geriico 2001/10/09
CLEF@
20
INFILE
INFILE results – anticipation/originality
team run recall anticipation originality originality(best)IMAG IMAG_1 0,30 0,43 1 4UAIC uaic_4 0,66 0,73 4UAIC uaic_1 0,69 0,75 0UAIC uaic_2 0,69 0,75 0UAIC uaic_3 0,82 0,86 93 267IMAG IMAG_2 0,09 0,22 0IMAG IMAG_3 0,06 0,14 0SINAI topics_1 0,50 0,57 9 9SINAI googlenews_2 0,08 0,10 15UOWD base 0,01 0,05 0 0HossurTech hossurtech_1 0,45 0,59 18 20
team run recall anticipation originality originality(best)UAIC uaic_4 0,44 0,58 0UAIC uaic_3 0,75 0,83 82 1292UAIC uaic_2 0,67 0,76 0HossurTech hossurtech_4 0,31 0,53 177 177
english target language
french target language
strongly correlated with recall
too few pariticipants
CEA LIST ELDA Univ. Lille 3 - Geriico 2101/10/09
CLEF@
21
INFILE
Approaches
Filteringadapted Information Retrieval tools (Lucene)SVM classifier with external ressources (GoogleNews)textual similarity measures with thresholds reasoning model (human plausible reasoning)
Adaptationadaptation of selection thresholdsuser feedback as parameter in reasoning model
Crosslingualbilingual dictionariesmachine translation
CEA LIST ELDA Univ. Lille 3 - Geriico 2201/10/09
CLEF@
22
INFILE
Conclusion and after…
Increasing participation, reasonable result, but not enough…
Currently, no INFILE track planned for next year
interest in multilingual filtering ?2/3 runs on monolingual Englishnot enough participants for crosslingual to have
comparative results
no funding INFILE evaluation kit will be made available
corpus of documents / topics / relevance assessments tools for the interactive adaptive filtering proceduretools for the evaluationdistributed by ELDA