ICCS Main Survey Marker Training The scoring guides Day 1 Hamburg, July 2008.

ICCS Main SurveyMarker Training

The scoring guides

Day 1

Hamburg, July 2008

ICCS Marker Training, HamburgJuly 2008

Since the Field-Trial

• Reduction of OER items from 8 to 6– CI2MBO1 (awful)– CI2VSO1 (data acceptable BUT concerns

about the overlap between cognition and values tipped this item out ahead of one of the two items dealing with money paid to the unemployed)

• Text revision to three items– CI2ETO1 (ambiguity removed)– CI2WFO1 and CI2WFO2 (“unemployed”

explicated to prevent students providing examples of circumstances that might lead to unemployment)


Purpose of Marker Training

To develop a common, shared understanding of the purpose of each item and the corresponding item codes for scoring student responses.


The Open-response item scoring guides

The only true common element across countries.

Crucial aspects:– the descriptions of acceptable responses for

each score category– the scored sample responses

The main purposes of the initial day(+) of the seminar are to confirm:

1. The conceptual description of each response category

2. The score (and justification for this score) allocated to each sample response


The Open-response item scoring guides

Item-by-item, small group discussions to answer the questions:

1. Do the conceptual categories for each score make sense, is there anything “missing” ? ALL GROUPS

2. Are you confident that sample response represents the score AND can you explain to others why it does? Please record comments for those sample responses that you feel need to be moved or are confusing. BY RESPONSE CATEGORY (for some items)

ANDPlease flag any responses that you would like to have

included in a scoring training sheet.Please take notes from your group on a scoring guide

ICCS Main SurveyMarker Training

Marking Operations

Hamburg, July 2008

“Lamb’s lettuce” is vegetarian (apparently it refers to a salad eaten often by sheep).


Recruiting Markers

• Preferable to use people with teaching experience (at the Grade level if possible)

• Preferable to use people who know and understand CCE

• Current teachers can be used (as part of professional development)

• Need to be decisive, cooperarative (know when to speak and when not to) and sympathetic to assessment (can be a danger with teachers).


Leading Markers

• Leading markers can be used primarily to resolve marking problems – e.g. ‘What should I give this?’… in the trial marking the leading marker may well be you.


Training Procedures

1. Brief introduction of project including reference to assessment framework

2. Item-by-item training with discussion of item and scoring guide followed by practice marking of student responses – make sure that the scoring system is clear for each item.


Considerations when using the guides

• Spelling, quality of expression, grammar and punctuation are not assessed.

• It does not matter where students write responses.

• HOWEVER – Civics and Citizenship Education provides an unusual set of challenges when considering the language of student responses.



• In many countries students have not had extensive formal classroom experience of CCE.

• Many, many students do not have the explicit CCE vocabulary to express conceptual understandings efficiently.

• One significant challenge in assessing written student responses is to ‘unpack’ student attempts to express conceptual understandings using their ordinary (i.e. non-technical) language.



• This challenge has two dangers that can affect marker reliability.

1. Some markers can mistakenly believe that because students do not have the language skill to express answers succinctly that all answers are correct.

2. Some markers believe that unless students use appropriate CCE language they cannot be credited with providing correct responses.



• BOTH misconceptions must be dealt with in the marker training.

• Student responses can only be scored on the basis of what they have written – markers should not try to infer student responses on the basis of what has not been written.

• To this end, the students’ choice of the words they use MUST be given careful attention.



• Many students provide ‘feel-good’ responses (expressions of value rather than understanding) that can be very well expressed but still without substance.

• Many students provide complex re-wordings of the question stem without answering the question – these can be deceptive as often they are eloquent.


Item Codes

• For the open-response items the permissible codes will be:

• 0,1,2 – conventional score categories

• 9 – Missing [the item has remained untouched at all by the student]

• 8 – Not scored or out of range


Not Scored or out of range items• Codes/scores should be written by markers

in the box to the left of the ‘Q’.• Not scored or out of range items will be

picked up only at data entry.• Minimise/avoid not scored (and out of

range) items by having markers re-check each booklet for the scores after it is marked.

• One easy way is to count the number of open-ended response items per booklet (e.g. 3) and then just count the scores in boxes for that booklet.


Not scored or out of range• If a not scored or out of range item still

makes it through to data entry then the code 8 should be entered by the data entry person (unless someone is there to resolve the problem at that moment) but NOT recorded on the booklet.

• You will need a system to resolve problems at the data entry level. The data entry people to contact a person who can read/hear the response and provide a score when there is a not scored or out of range score code.

• Individual systems will need to meet local circumstances (i.e. do what works for you).


Reliability Marking/Quality Control

• Double marking of booklets as outlined by DPC

• Reliability marking provides only post-hoc information, it is more important that issues be resolved and monitored during the marking process.

• Outstanding marker training• Leading marker to act as final

adjudicator of queries (e.g. ‘What score do I give this?’


Reliability Marking/Quality Control

Additional thoughts/possibilities• Pre score some real student responses in

your own language and use them as additional training scores…

• and/or as “warm-up” or “control” responses. (Good especially given the rotation of items through booklets).

• Have markers “flag” interesting, difficult to score responses and use them as a basis for ongoing discussions of the marking.


Item Code Question Number by Booklet Number

B1 B2 B3 B4 B5 B6 B7

CI2PDO1 5 27 22

CI2BIO1 14 4 32

CI2ETO1 15 5 32

CI2RRO1 28 18 8

CI2WFO2 29 20 9

CI2WFO1 26 16 5


Questions or comments?

[email protected]

Date post:	01-Jan-2016
Category:	Documents
Upload:	junior-richard
View:	213 times
Download:	0 times

ICCS Main Survey Marker Training The scoring guides Day 1 Hamburg, July 2008.

Documents