EERQI Final Conference,

March 15-16, 2011.

Bruxelles

This project is funded by the Socioeconomic Sciences and Humanities Section.

Automatic Semantic Analysis

Ágnes Sándor

Xerox Research Centre Europe

Agnes.Sandor@xrce.xerox.com

Objectives

1. Investigate the possibilities of automating the discovery of semantic

indicators of quality:

• Highlight salient sentences to provide reading assistance to the peer-

reviewer and to enhance the search engine

• Genre analysis to refine indicators relevant to special sub-domains of

educational research

2. Rethink the method of using citations as quality indicators:

• Citation analysis: typing citations according to the author’s motivation

of citing

1.Semantic indicators of

quality

Status of semantic analysis in EERQI

Intrinsic evaluation criteria:

• based on evidence provided

by the meaning of the texts

• the evaluation of the evidence

is carried out by peers

→ the result is directly related

to the text

nnnnnnnnnnnnnnnnnn

• for small-scale evaluation

Semantic analysis

Extrinsic evaluation criteria:

• based on evidence provided

by usage statistics

• evaluation relies on statistics

nnnnnnnnnnnnnnnnnnnnnnnn

→ the result is indirectly

related to the text e.g.

citation index, impact factor

• for large-scale evaluation

EERQI semantic analysis

IS MEANT TO detect salient sentences as evidence indicating research quality.

IS NOT MEANT TOprovide the semantic representation of the entire research article – that is not feasible. The research article itself is the best representation of the research it describes. This is why it is not a database or a PowerPoint presentation.

EERQI semantic analysis

IS MEANT TO

assist the evaluator by automatically providing

evidence of discourse conveying quality

IS NOT MEANT TO

substitute of the evaluator

"the paper makes a ‘significant’contribution"

67/100

EERQI semantic analysis

detect salient sentences as evidence

indicating research quality

EERQI semantic analysis

Rigour

Significance

Originality

Integrity

Style

detect salient sentences as evidence

indicating research quality

Test of the applicability of the criteria

Rigour

Significance

Originality

Integrity

Style

1. Textual evidence for

quality criteria is not

shared by peers

2. The unit of textual

evidence is not

necessarily the

sentence

3. The match between

types of criteria and

textual evidence is not

clear

→ What can we propose?

Our approach

Highlight sentences that are conceptually salient

Assist the reader for faster and more effective

comprehension

Salient sentences/1

summary

Summary sentences

This paper considers the cultural issues

associated with the use of flexible

delivery in offshore international education.

The purpose of this study was to examine

differences in the perceptions of internships

between college students studying travel

and tourism and practitioners in the field .

Thomas Kuhn’s theory of progress in science: scientific

activity regarded as problem-solving

→ sentences describing research issues:

Salient sentences/2

Many commentators have mistaken this technological possibility for a dominant

social reality, failing to take account of the factors that limit cross-border study

(for example, Knight 1994; West 1998).

Students’ experience of learning in virtual environments is a critical area of

concern for educators throughout the world (HIS 1995) but little research has

been conducted into the cultural bases of such experiences.

The first issue I want to raise is the practical limitations on the flexibility of the

institution.

Highlighting for reading assistance

Summary

Research issue

Summary +

Research issue

Words triggering

detection

evaluation

Rigour

Significance Originality

Integrity Style

Xerox Incremental Parser

Comparative study

→Highlighting does not allow to evaluate according to all the criteria

→Results are different according to genre: more kinds of expressions should be considered

→Highlighting makes it possible to rapidly filter out bad quality

DOMAIN

THEORETICAL EMPIRICAL

SUMMARY

(error-rate)

RES. ISSUE

(error-rate)

SUMMARY

(error-rate)

RES. ISSUE

(error-rate)

sociology 4%

(4%)

17%

(2%)

10%

(16%)

15%

(19%)

psychology 3%

(21%)

21%

(61%)

7%

(7%)

11%

(50%)

history 3%

(24%)

17%

(12%)

- -

First number: the percentage of the sentence type out of all the sentences

Second number: the error rate out of the detected sentences

→Theoretical articles: the proportion of research issue sentences detected

is significantly higher than that of summary sentences

→The best performance comes from theoretical sociology articles

Genre analysis

Peer-review test

Please briefly summarize the reviewed article in your own words. In the

summary, please state the main subject of the reviewed article, the

research problem as well as the main conclusions/results/open

questions, etc.

Peer-reviewer summary Automatically highlighted sentence

This article considers the

development of educational

policy that has affected

primary and secondary

education for the past two

decades in the Republic of

Ireland.

… this article provides a

detailed and critical analysis

of the evolution of

accountability policy and

practice in Irish primary and

post - primary education

during the past 20 years.

Peer-review test

Please briefly summarize the reviewed article in your own words. In the

summary, please state the main subject of the reviewed article, the

research problem as well as the main conclusions/results/open

questions, etc.

Peer-reviewer summary Automatically highlighted sentence

This article considers the

development of educational

policy that has affected

primary and secondary

education for the past two

decades in the Republic of

Ireland.

… this article provides a

detailed and critical analysis

of the evolution of

accountability policy and

practice in Irish primary and

post - primary education

during the past 20 years.

Results for English (144 summaries)

• To what extent are peer-reviewer summaries based on sentence

extraction?

84% of the human summary sentences correspond to a sentence from

the article

• To what extent do the peer-reviewer sentences correspond to our

definition of salient sentence?

56% of the corresponding sentences fulfill the criteria of salient

sentences

• What is the performance of XIP for the detection of salient sentences?

68% of the salient sentences are detected by XIP

• Do the salient sentences cover the topics mentioned in the summary?

In average 4 times more occurrences of the nouns from the human

summary sentences in the highlighted sentences

Use in the search engine

relevant content

enhance

document

search

quality

judgment

highlighting salient sentences

English

French

German

Swedish

Xerox Incremental

Parser (XIP)

reading

support

formalisation of salient

sentences

Wrap-up

2. Citation analysis

Citation types

ARGUMENTATION:

The term of diversity is still used rather ambiguously in debates on multiculturalism, identity politics , anti -

discrimination policies and educational contexts (BREWSTER et al. 2002 ; VERTOVEC/WESSENDORF 2004).

EVIDENCE:

As WOOD's (2003) critical assessment of the U.S. diversity debate proves, since 1978, different levels of court

rulings establishing affirmative action and equal employment opportunity schemes in public institutions,

organizations and businesses have forced both public and private actors to introduce diversity mechanisms into

their particular organizational contexts.

IMPORTANCE:

Tenenbaum and colleagues (2001) recently empirically defined and examined key features of constructivist

learning environments and their incorporation into two different learning environments (on-campus and distance

learning).

QUALIFICATION:

There is, as one would expect, much to applaud in Pring's approach - eclectic yet robustly coherent, circumspect

yet insistently principled.

SURPRISE:

The second set of concerns, the silences and absences that merit more substantial attention, cluster round the

surprisingly uneven presence of democratic theory within the intellectual narrative of the paper.

Using citations as quality indicators

In collaboration with Fredrik Åström

model of scholarly communication

Exact sciences

Humanities

Differences of the

interpretation of the

role of citations in

quality assessment

Social Sciences

Citation in scholarly communication

EVIDENCE

ARGUMENTATION

IMPORTANCE

QUALIFICATION

SURPRISE?