Guillaume Cabanac [email protected]

Musings at the Crossroads ofMusings at the Crossroads ofDigital Libraries, Information Retrieval, Digital Libraries, Information Retrieval,

and Scientometricsand Scientometrics

http://bit.ly/rguCabanac2012http://bit.ly/rguCabanac2012

Guillaume [email protected]

March 28th, 2012

Outline of these Musings

2

Musings at the Crossroads of DL, IR, and SCIM

Guillaume Cabanac

Digital LibrariesDigital Libraries Collective annotations Social validation of discussion threads Organization-based document similarity

Information RetrievalInformation Retrieval The tie-breaking bias in IR evaluation Geographic IR Effectiveness of query operators

ScientometricsScientometrics Recommendation based on topics and social clues Landscape of research in Information Systems

The submission-date bias in peer-reviewed conferences

3


Guillaume Cabanac






4


Guillaume Cabanac


Question DL-1

How to transpose paper-based annotations into digital documents?

IRIRDLDL

SCIMSCIM

Guillaume Cabanac, Max Chevalier, Claude Chrisment, Christine Julien. “Collective annotation: Perspectives for information retrieval improvement.” RIAO’07 : Proceedings of the 8th conference on Information Retrieval and its Applications, pages 529–548. CID, may 2007.

5

Characteristics of paper annotation Secular activity: older than 4 centuries Numerous applicative contexts: theology, science, literature … Personal use: “active reading” (Adler & van Doren, 1972)

Collective use: review process, opinion exchange …

From Individual Paper-based Annotation …

US students

(Marshall, 1998)

1541

Annotated bible

(Lortsch, 1910)

Fermat’s last theorem

(Kleiner, 2000)

Annotations from Blake, Keats…

(Jackson, 2001)

Les Misérables

Victor Hugo

1630 1790 1830 1881 1998


Guillaume Cabanac

6

… to Collective Digital Annotations

author

87%

reader13%

1993 2005

ComMentor … iMarkup … Yawas … Amaya …

> 20 annotation systems(Cabanac et al., 2005)

Web servers (Ovsiannikov et al., 1999)

Annotation server

a discussion thread

Hard to share ‘lost’

hardcopy


Guillaume Cabanac

7

W3C Annotea / Amaya (Kahan et al., 2002)

Digital Document Annotation: Examples

a reader’s comment

discussionthread

Arakne, featuring “fluid annotations” (Bouvin et al., 2002)


Guillaume Cabanac

8

Collective Annotations Reviewed 64 systems designed during 1989–2008

Collective Annotation Objective data

Owner, creation date Anchoring point within the document. Granularity: all doc, words…

Subjective information Comments, various marks: stars, underlined text… Annotation types: support/refutation, question… Visibility: public, private, group…

Purpose-oriented annotation categories

Annotation remark

Annotation reminder

Annotation argumentation

Personal Annotation Space


Guillaume Cabanac

9


Guillaume Cabanac


Question DL-2

How to measure the social validity ofa statement according to the

argumentative discussion it sparked off?

IRIRDLDL

SCIMSCIM

Guillaume Cabanac, Max Chevalier, Claude Chrisment, Christine Julien. “Social validation of collective annotations : Definition and experiment.” Journal of the American Society for Information Science and Technology, 61(2):271–287, feb. 2010, Wiley. DOI:10.1002/asi.21255

10

Scalability issue

Which annotationsshould I read?

Social validation = degree of consensus of the group

Social Validation

Social Validation of Argumentative Debates


Guillaume Cabanac

11

Social Validation of Argumentative Debates

BeforeAnnotation magma

AfterFiltered display

Informing readers about how validated each annotation is


Guillaume Cabanac

12

Overview

Two proposed algorithms Empirical Recursive Scoring Algorithm (Cabanac et al., 2005)

Bipolar Argumentation Framework Extension based on Artificial Intelligence research works (Cayrol & Lagasquie-Schiex, 2005)

Social Validation Algorithms

validity

0socially neutral

– 1 socially refuted

1socially confirmed

case 1case 2case 3 case 4

A

B

A

B


Guillaume Cabanac

13

Example

Computing the social validity of a debated annotation

Social Validation AlgorithmMusings at the Crossroads of DL, IR, and SCIM

Guillaume Cabanac

14

Validation with a User-study

Design

Corpus: 13 discussion threads= 222 annotations + answers

Task of a participant Label opinion type Infer overall opinion

Volunteer subjects

53

119

Aim: social validation vs human perception of consensus


Guillaume Cabanac

15

Q1 Do people agree when labeling opinions? Kappa coefficient (Fleiss, 1971; Fleiss et al., 2003)

Inter-rater agreement among n > 2 raters

Weak agreement, with variability subjective task

Experimenting the Social Validation of Debates

Debate Id

Fair to good

Poor


Guillaume Cabanac

Val

ue o

f K

appa

agreement

16

Q2 How well SV approximates HP? HP = Human Perception of consensus SV = Social Validation algorithm

1. Test whether PH and VS are different (p < 0.05) Student’s paired t-test: (p = 0,20) > ( = 0,05)

2. Correlate HP et SV Pearson’s coefficient of correlation r

r(HP, SV) = 0.48 shows a weak correlation

Experimenting the Social Validation of Debates

HP – SV

Density y = p(HP – SV)

example: HP = SV for 24 % of all cases


Guillaume Cabanac

Den

sity

17


Guillaume Cabanac


Question DL-3

How to harness a quiescent capital present in any community:

its documents?

IRIRDLDL

SCIMSCIM

Guillaume Cabanac, Max Chevalier, Claude Chrisment, Christine Julien. “Organization of digital resources as an original facet for exploring the quiescent information capital of a community.” International Journal on Digital Libraries, 11(4):239–261, dec. 2010, Springer. DOI:10.1007/s00799-011-0076-6

18

Personal Documents Filtered, validated, organized information…

… relevant to activities in the organization

Paradox: profitable, but under-exploited Reason 1 – folders and files are private

Reason 2 – manual sharing

Reason 3 – automated sharing

Consequences People resort to resources available outside of the community Weak ROI why would we have to look outside when it’s already there?

Documents as a Quiescent WealthMusings at the Crossroads of DL, IR, and SCIM

Guillaume Cabanac

19

Mapping the documents of the community SOM [Kohonen, 2001] Umap [Triviumsoft] TreeMap [Fekete & Plaisant, 2001]…

Limitations Find the documents with same topicssame topics as D Find documents that colleagues useuse with D

concept of usage: grouping documentsgrouping documents ⇆ keeping stuff in commonkeeping stuff in common

How to Benefit from Documents in a Community?Musings at the Crossroads of DL, IR, and SCIM

Guillaume Cabanac

20

Organization-based similarities inter-folder

inter-document

inter-user


Guillaume CabanacHow to Benefit from Documents in a Community?

21

Purpose: Offering a global view of … people and their documents

Based on document contents Based on document usage/organization

Requirement: non-intrusiveness and confidentiality

OperationalOperational needs Find documents

With related materials With complementary materials

Seeking people ⇆ seeking documents

ManagerialManagerial needs Visualize the global/individual activity Work position required documents

How to Help People to Discover/Find/Use Documents?

community


Guillaume Cabanac

22

4 views = {documents, people} {group, unit}

1. Group of documents Main topics Usage groups

2. A single document Who to liaise with? What to read?

3. Group of people Community of interest Community of use

4. A single people Interests Similar users (potential help)

Proposed System: Static AspectMusings at the Crossroads of DL, IR, and SCIM

Guillaume Cabanac

23


Guillaume Cabanac






24


Guillaume Cabanac

Question IR-1

Is document tie-breaking affecting the evaluation of

Information Retrieval systems?

IRIRDLDL

SCIMSCIM


Guillaume Cabanac, Gilles Hubert, Mohand Boughanem, Claude Chrisment. “Tie-breaking Bias : Effect of an Uncontrolled Parameter on Information Retrieval Evaluation.” M. Agosti, N. Ferro, C. Peters, M. de Rijke, and A. F. Smeaton (Eds.) CLEF’10 : Proceedings of the 1st Conference on Multilingual and Multimodal Information Access Evaluation, volume 6360 de LNCS, pages 112–123. Springer, sep. 2010. DOI:10.1007/978-3-642-15998-5_13

25

Measuring the Effectiveness of IR systems User-centered vs. System-focused [Spärck Jones & Willett,

1997]

Evaluation campaigns 1958 Cranfield, UK 1992 TREC (Text Retrieval Conference), USA 1999 NTCIR (NII Test Collection for IR Systems), Japan 2001 CLEF (Cross-Language Evaluation Forum), Europe …

“Cranfield” methodology Task Test collection

Corpus Topics Qrels

Measures : MAP, P@X ... using trec_eval

[Voorhees, 2007]


Guillaume Cabanac

26

Runs are Reordered Prior to Their EvaluationQrels = qid, iter, docno, rel Run = qid, iter, docno, rank, sim,

run_id

Reordering by trec_evalqid asc, sim desc, docno desc

Effectiveness measure = f (intrinsic_quality, )MAP, P@X, MRR…


Guillaume Cabanac

27

Consequences of Run Reordering Measures of effectiveness for an IRS s

RR(s,t) 1/rank of the 1st relevant document, for topic t

P(s,t,d) precision at document d, for topic t

AP(s,t) average precision for topic t

MAP(s) mean average precision

Tie-breaking bias

Is the Wall Street Journal collection more relevant than Associated Press?

Problem 1comparing 2 systemsAP(s1, t) vs. AP(s2, t)

Problem 2 comparing 2 topicsAP(s, t1) vs. AP(s, t2)

ChrisChris

EllenEllen

Sensitive to document

rank


Guillaume Cabanac

28

What we Learnt: Beware of Tie-breaking for AP Poor effect on MAP, larger effect on AP

Measure bounds APRealistic APConventionnal APOptimistic

Failure analysis for the ranking process Error bar = element of chance potential for improvement

padre1, adhoc’94


Guillaume Cabanac

29


Guillaume Cabanac

Question IR-2

How to retrieve documents matching keywords and

spatiotemporal constraints?

IRIRDLDL

SCIMSCIM


Damien Palacio, Guillaume Cabanac, Christian Sallaberry, Gilles Hubert. “On the evaluation of geographic information retrieval systems: Evaluation framework and case study.” International Journal on Digital Libraries, 11(2):91–109, june 2010, Springer. DOI:10.1007/s00799-011-0070-z

30

Geographic Information Retrieval Query = “Road trip around Aberdeen summer 1982”

Search engines Topic term {road, trip, Aberdeen, summer}

spatial {AberdeenCity, AberdeenCounty…} Geographic temporal [21-JUN-1982 .. 22-SEP-1982]

term {road, trip, Aberdeen, summer}

1/6 queries = geographic queries Excite (Sanderson et al., 2004) AOL (Gan et al., 2008) Yahoo! (Jones et al., 2008)

Current issue worth studying


Guillaume Cabanac

31

The Internals of a Geographic IR System 3 dimensions to process

Topical, spatial, temporal

1 index per dimension Topic bag of words, stemming, weighting, comparing with VSM… Spatial spatial entity detection, spatial relation resolution… Temporal temporal entity detection…

Query processing with sequential filtering e.g., priority to theme, then filtering according to other dimensions

Issue: effectiveness of GIRSs vs state-of-the-art IRSs?

Hypothesis: GIRSs better than state-of-the-art IRSs


Guillaume Cabanac

32

Case Study: the PIV GIR System Indexing: one index per dimension

Topical = Terrier IRS Spatial = tiling Temporal = tiling

Retrieval Identification of the 3 dimensions in the query Routing towards each index Combination of results with CombMNZ [Fox & Shaw, 1993; Lee 1997]


Guillaume Cabanac

33

Case Study: the PIV GIR System Principle of CombMNZ and Borda Count


Guillaume Cabanac

34

Case Study: the PIV GIR System Gain in effectiveness


Guillaume Cabanac

35


Guillaume Cabanac

Question IR-3

Do operators in search queries improve the effectiveness of search results?

IRIRDLDL

SCIMSCIM


Gilles Hubert, Guillaume Cabanac, Christian Sallaberry, Damien Palacio. “Query Operators Shown Beneficial for Improving Search Results.” S. Gradmann, F. Borri, C. Meghini, H. Schuldt (Eds.) TPDL’11 : Proceedings of the 1st International Conference on Theory and Practice of Digital Libraries, volume 6966 de LNCS, pages 118–129. Springer, sep. 2011. DOI:10.1007/978-3-642-24469-8_14.

Various Operators Quotation marks, Must appear (+), boosting operator (^),

Boolean operators, proximity operators…

36

Information need

“I’m looking for research projects funded in the DL domain”

Regular query Query with operators

Search Engines Offer Query Operators


Guillaume Cabanac

Our Research Questions

37


Guillaume Cabanac

38

Our Methodology in a Nutshell

Regular query V1: Query variant with operators

V3V2

V4VN. . .


Guillaume Cabanac

39

Effectiveness of Query Operators TREC-7 per Topic Analysis: Boxplots

‘+’ and ‘^’


Guillaume Cabanac

40

Effectiveness of Query Operators Per Topic Analysis: Box plot

AP of TREC’s regular query

Query variant highest AP

32Topics

AP (

Avera

ge P

reci

sion)

0.2

0.1

0.3

0.4

Query variant lowest AP


Guillaume Cabanac

41

Effectiveness of Query Operators TREC-7 Per Topic Analysis

‘+’ and ‘^’

MAP = 0.1554

MAP ┬ = 0.2099+35.1%


Guillaume Cabanac

42


Guillaume Cabanac






43


Guillaume Cabanac

Question SCIM-1

How to recommend researchers according to their research topics

and social clues?

IRIRDLDL

SCIMSCIM



Guillaume Cabanac. “Accuracy of inter-researcher similarity measures based on topical and social clues.” Scientometrics, 87(3):597–620, june 2011, Springer. DOI:10.1007/s11192-011-0358-1

44

Recommendation of Literature (McNee et al., 2006)

Collaborative filtering Principle: mining the preferencespreferences of researchers

those who liked this paper also liked…

Snowball effect / fad Innovation? Relevance of theme?

Cognitive filtering Principle: mining the contentscontents of articles

profile of resources (researcher, articles) citation graph

Hybrid approach

????


Guillaume Cabanac

45

Foundations: Similarity Measures Under Study

Model Coauthors graph authors auteurs Venues graph authors conferences / journals

Social similarities Inverse degree of separation length of the shortest path Strength of the tie number of shortest paths Shared conferences number of shared conference editions

Thematic similarity Cosine on Vector Space Model di = (wi

1, … , win)

built on titles (doc / researcher)


Guillaume Cabanac

46

Computing Similarities with Social Clues Task of literature review

Requirement topical relevance Preference social proximity (meetings, project…)

re-rank topical results with social clues

Combination with CombMNZ (Fox & Shaw, 1993)

Final result: list of recommended researchers

CombMNZ

Degree of separation

Strength of ties

Shared conferences

Social list

Topical list

CombMNZ TS listTS list


Guillaume Cabanac

47

Evaluation Design Comparison of recommendations and researchers’ perception

Q1 : Effectiveness of topical (only) recommendations? Q2 : Gain due to integrating social clues?

IR experiments: Cranfield paradigm (TREC…) Does the search engine retrieve relevant documents?

Doc relevant?

assessor

relevance judgments{0, 1} binary[0, N] gradual

qrels

trec_eval

Effectiveness measuresMean Average PrecisionNormalized Discounted Cumulative Gain

topic S1 S2

1 0.5687 0.6521

… … …

50 0.7124 0.7512

avg 0.6421 0.7215

improvement +12.3 % significativity p < 0.05 (paired t-test)

search engine xinput

topic

corpus


Guillaume Cabanac

48

Evaluating Recommendations

doc relevant ?

assessor

relevance judgments{0, 1} binary[0, N] gradual

qrels

trec_eval

Effectiveness measures Mean Average PrecisionNormalized Discounted Cumulative Gain

topic S1 S2

1 0.5687 0.6521

… … …

50 0.7124 0.7512

avg 0.6421 0.7215

improvement +12.3 % significativity p < 0.05 (paired t-test)

search engine xinput

topic

corpus

name of a researcher

researcher

« With whom would you like to chat for improving your research? »

recommender system

topical topical + social

#subjects

Top 25


Guillaume Cabanac

49

Experiment Features

Data dblp.xml (713 MB = 1.3M publications for 811,787 researchers) Subjects 90 researchers-contacts contacted by mail

74 researchers began to fill the questionnaire. 71 completed it

Interface for assessing recommendations


Guillaume Cabanac

50

Experiments: Profile of the Participants Experience of the 71 subjects Mdn = 13 years

74

Productivity of the 71 subjects Mdn = 15 publications


Guillaume Cabanac

Num

ber

of p

arti

cipa

nts

Num

ber

of p

arti

cipa

nts

Seniority

Number of publications

51

Empirical Validation of our Hypothesis Strong baseline effective approach based on VSM

+8.49 % = significant improvement (p < 0.05 ; n = 70)

of topical recommendations by social clues

0,5

0,6

0,7

0,8

0,9

1

global < 15 publis >= 15 publis < 13 ans >= 13 ans

Thématique Thématique + Social

productivity experience

+8,49 %+8,49 % +10,39 %+10,39 % +7,03 %+7,03 % +6,50 %+6,50 % +10,22 %+10,22 %

ND

CG


Guillaume Cabanac

Topical Topical + social

yearsyears

52


Guillaume Cabanac

Question SCIM-2

What is the landscape of research in Information Systems from the perspective of gatekeepers?

IRIRDLDL

SCIMSCIM



Guillaume Cabanac. “Shaping the landscape of research in Information Systems from the perspective of editorial boards : A scientometric study of 77 leading journals.” Journal of the American Society for Information Science and Technology, 63, to appear in 2012, Wiley. DOI:10.1002/asi.22609

53

Landscape of Research in Information Systems The gatekeepers of science


Guillaume Cabanac

54

Landscape of Research in Information Systems The 77 core peer-reviewed IS journals in the WoS


Guillaume Cabanac

55

Landscape of Research in Information Systems Exploratory data analysis


Guillaume Cabanac

56

Landscape of Research in Information Systems Exploratory data analysis


Guillaume Cabanac

57

Landscape of Research in Information Systems Topical map of the IS field


Guillaume Cabanac

58

Landscape of Research in Information Systems Most influential

gatekeepers


Guillaume Cabanac

59

Landscape of Research in Information Systems Number of gatekeepers per country


Guillaume Cabanac

60

Landscape of Research in Information Systems Geographic and gender diversity


Guillaume Cabanac

61


Guillaume Cabanac

Question SCIM-3

What if submission date influenced the acceptance of conference papers?

IRIRDLDL

SCIMSCIM



Guillaume Cabanac. “What if submission date influenced the acceptance of conference papers?” Submitted to the Journal of the American Society for Information Science and Technology, Wiley.

62

Conferences Affected by a Submission-Date bias? Peer-review


Guillaume Cabanac

63

The Submission-Date bias Dataset from the ConfMaster conference management system


Guillaume Cabanac

64

The Submission-Date bias Influence of submission date on bids


Guillaume Cabanac

65

The Submission-Date bias Influence of submission date on average marks


Guillaume Cabanac

Conclusion

66


Guillaume Cabanac





Thank you

http://www.irit.fr/~Guillaume.Cabanachttp://www.irit.fr/~Guillaume.Cabanac

Twitter: @tafanorTwitter: @tafanor

Date post:	10-Jan-2016
Category:	Documents
Upload:	billie
View:	51 times
Download:	4 times