James Christie

Automated Marking of Essays for Content

Earliest recorded use is some

2,000 years BC

by the Chinese

for their Administrators!

How long have essays been used for assessment?

Essay Definition 1 of 2

… requires a response composed by the examinee, usually in the form of one or more sentence, of a nature that no single response or pattern of responses can be listed as correct, and the accuracy and quality of which can be judged subjectively only by one skilled or informed in the subject, …

Essay Definition 2 of 2

… but even an expert cannot usually classify a response as categorically right or wrong. Rather, there are different degrees of quality or merit which can be recognized. …

attributed to

Stalnaker, 1951

Possible criteria for automated essay

marking• Ease of creating a scoring schema• Ability to score on various mark regimes• Ease of identification on non-scoring elements• Ease of modification should scoring error(s) occur• Consistent and reproducible scoring• Acceptability of results to human markers,

essayists, …• Defensibility• Accuracy and precision• Coachability avoidance• Cost

Model Essay

The cat sat on the mat.

Marking Schema

Mark Item

3 cat satmat

[max:3]

Content Data Structure

1 z a a 0 2 3 4 0 3 * 3 0 n 3

2 z f cat 1 0 0 0 0 0 * 0 1 n 0

3 z f sat 1 0 0 0 0 0 * 0 1 n 0

4 z f mat 1 0 0 0 0 0 * 0 1 n 0

Essay Set

• ALPHA– The cat sat on the mat.

• BRAVO– The cat sat on the floor.

• CHARLIE– The dog lay on the floor.

• MODEL– The cat sat on the mat.

Process interface

What LEVEL of Diagnostics to use [0 ... 3] : 0

What ESSAY SET to use .................... : catmat

Enter SCHEMA to use ...................... : catmat

ALPHA.EXT . BRAVO.EXT . CHARLIE.EXT . MODEL.EXT .

Started on Thursday, February 06 2003 at 15:59:05

Finished on Thursday, February 06 2003 at 15:59:06

Schema ReportEssay set ............. catmat

Schema Report using ... catmat

Entities 1 - 4 :

Entity ID : 1 2 3 4Entity Type : a f f fPart ID's : z

Essay Name : ALPHA.EXT: y y y y BRAVO.EXT: _ y y _ CHARLIE.EXT: _ _ _ _ MODEL.EXT: y y y y

Content Report

Essay set .............. catmat

Content Report using ... catmat

Essay Name : Words Sentences Usage[%] Coverage[%]Part: z[ 3] Mark[ 3] %[100]

ALPHA.EXT: 6 1 50.00 100.00 3 3 100.00

BRAVO.EXT: 6 1 33.33 50.00 0 0 0.00

CHARLIE.EXT: 6 1 0.00 0.00 0 0 0.00

MODEL.EXT: 6 1 50.00 100.00 3 3 100.00

Started on Thursday, February 06 2003 at 15:59:05

Finished on Thursday, February 06 2003 at 15:59:06

Marked 4 file(s): scanned 4 file(s)

Marking Performance

Essay Set

First v Second Markers

Human v SEAR

A 0.704** / 0.700** 0.594** / 0.596**

B 0.810** / 0.740** 0.404** / 0.376**

C 0.164 / 0.277 0.302* / 0.394**

D N/A 0.238* / 0.336**

Pearson / Spearman

Significance **= 0.01 *= 0.05

Future work [style]

• obtain marked essays for style marking

– plain ASCII essays• using a common set of metrics

– word-processed essays• using a common set of metrics• augmented with word-

processing based metrics

Future work [content]

• maximise use of active and passive voices

• cope with spelling (and grammar) errors• increased coverage of Bloom’s

Taxonomy• include non-textual feature(s)• develop

– better feedback to the essayist– better feedback to the examiner– plagiarism detection mechanism(s)

If manual marking equalsDa Vinci’s Helical Screw,

then does SEAR equal the first powered flight?

Is the future equal to the ISS?

Is this the future [for style and content]?

English The cat sat on the mat.

Italian Il gatto era seduto sullo zerbino.

Greek I gata ekatse ston kanape.

Russian Koshka sidit na matrase.

French Le chat s’est assis sur le tapis.

German Die Katze sass auf dem Teppich.

Dutch De kat zat op de mat.

Spanish El gato se sent’s en la alfombra.

etc

James R Christie

The Robert Gordon UniversityFaculty of Design and Technology

School of ComputingRoom B23b St Andrew Street, Aberdeen

‘Phone +44 [0] 1224 262725Fax +44 [0] 1224 262727

E-mail j.christie@rgu.ac.ukURL

http://www.comp.rgu.ac.uk/staff/jrc