Inter-rater Reliability Training and Consensus Building

Peggy Muehlenkamp December 2012

Agenda● Background Building● Inter-rater Reliability/Consensus

Building● Scoring

○ plan○ rubric○ procedure

College/CareerReady

Writing on Demand

Inter-rater Reliability

Scoring vs. GradingScore(s): The number(s) or letter(s) assigned to an assessment via the process of measurement.

Grade(s): The number(s) or letter(s) reported at the end of a set period of time as a summary statement of evaluations made of students.

Transforming Classroom Grading, Robert Marzano

Holistic vs. Analytic

Procedure

● Papers are mixed: 6-8● 2 Readers will score each paper● Scores are averaged to determine overall

score

ExampleReader 1: 75 Score: MReader 2: 72 Score: T

Overall Score: _____

Ground Rules● Professional discussions

● Stay focused on task

● Refrain from conversations

● All scoring takes place in this room.

Scoring Tips● Know your Biases

● Let go of the Number Labels

● Take Comfort in the 2nd

Reader

Scoring Process

● Record Reader # on Paper

● Read Paper - quick, but thorough

● Score - overall impression

1st Decision

Example Prompt

Essay #1● clear understanding of the writing task ● demonstrates effective skill in responding to the

prompt.● takes a position and offers a critical context for

discussing the issue● essay addresses complexity by fully responding

to several counterarguments ● development of ideas is logical and specific

● the writer’s concise style and critical insight result in an essay that convincingly supports the main ideas as well as implies connection to further issues

● organization of the essay is clear● introduction and conclusion are clear and effective● essay demonstrates a good command of language,

using a variety of sentence constructions as well as varied and precise word choice

● few errors to distract the reader

Essay #1 (cont.)

Essay #2ACT Score: 2

● weak response to the task

● takes a position on the issue with a qualification

● thinly developed - very little explanation to

support each position

● some organization is evident

Essay #2 (cont.)

● supporting ideas are not grouped

● few transitions

● introduction and conclusion are not developed

● language use is simple

● little variety in sentence structure

● errors are sometimes distracting, but do not affect

understanding

Essay #3ACT Score: 1

● shows little skill in responding to the writing task

● takes a position, but fails to support with reasons

● minimal development of ideas

● some supporting claims are not understandable

Essay #3 (cont.)

● no discernable organization○ minimal introductory statement/no conclusion○ ideas are not logically grouped○ no transitions

● simple sentence structure and word choice

● language usage errors are distracting and make

some portions difficult to understand

Essay #4ACT Score: 2

● inconsistent skill in responding to task● takes a position/no development of counter-

argument● thinly developed - little explanation to support

second idea● some organization is evident

Essay #4 (cont.)

● discernible introduction

● conclusion is minimal

● simple transitions

● some variety in sentence structure

● appropriate word choice

Essay #5ACT Score: 4

● adequate response to the writing task● takes a position on the issue● shows recognition of complexity by

acknowledging counterargument● does not provide response to counter arguement● adequate development of ideas

○ ideas are developed with some specific reason and detail○ examples support ideas

Essay #5 (cont.)

● organization is clear

● somewhat developed introduction and conclusion

● obvious transitions throughout essay

● some logical sequencing of ideas is evident

● language use shows some sentence variety

● appropriate word choice

● errors do not impede understanding

Norming

Are We in Agreement?