Evaluating and improving student achievement based

on pre-set criteria

Brian Whalley(b.whalley@sheffield.ac.uk)(For more information and resources see:

http://brianbox.net/pedagogies/criteria/criteria.html)

Various things in the notes for each slide plus a few references at the end

We’re going to look at

• Assessment• Feedback• Supporting students

and, by the way,Talk about a variety of aspects of

education - whatever you want!(But we shall need to use a bit of ‘jargon’)

What is it like to be a student?(with apologies to Chiroptera and Thomas Nagel)

Please discuss with a neighbour

What you mean by:1. Feedback2. Feedforward

What do (you think) students mean (ie think we mean) by these terms?

Outcomes(Constructive) alignment

Alignment thus … Learners construct meaning from what they do to learn. [This concept derives from cognitive psychology and constructivist theory, and recognizes] the importance of linking new material to concepts and experiences in the learner's memory, and extrapolation to possible future scenarios via the abstraction of basic principles through reflection. The teacher makes a deliberate alignment between the planned learning activities and the learning outcomes. This is a conscious effort to provide the learner with a clearly specified objective, a well designed learning activity or activities that are appropriate for the task, and well designed assessment criteria for giving feedback to the learner.

Feedback

• Should be incorporated into the assessment

• Students learn most by ‘feedback’– Geoff Petty

So we need to bring feedback into the alignment process

when to give feedback

When can you? When should you?When do you?Do others in your department agree

on when you do?Do others in your department give

‘proper’ feedback anyway (module scheme)?

The Ten Principles of Assessment and Feedback

• 1. Help clarify what good performance is • via aims, criteria, standards

• 2. Encourage ‘time and effort’ on challenging learning tasks.

• 3. Deliver high quality feedback information that helps learners self-correct.

• 4. Encourage positive motivational beliefs and self-esteem.

• 5. Encourage interaction and dialogue around learning (peer and teacher student)

(Nicol 2007)

10 principles continued

• 6. Facilitate the development of self-assessment and reflection in learning.

• 7. Give learners choice in assessment – content and processes.

• 8. Involve students in decision-making about assessment policy and practice.

• 9. Support the development of learning communities.

• 10. Help teachers adapt teaching to student needs.

How do we use exams (and coursework)

to deliver these?

It’s all about assessment …’Students can escape bad teaching but they cannot escape bad assessment’’

(David Boud, 1995)

And thus about Feedback

and a bit about Feedforward

and so we need a plan ….

So we’ll need to clarify things a bit and explain

something about jargon

Hmmm…..

I think we do this too often

is the underpinning concept behind the requirements for programme specification, declarations of Intended Learning Outcomes (ILOs) and assessment criteria, and the use of criterion based assessment.

Constructive Alignment

Towards a better learning system(‘Abolish feedback’)

It depends what you mean by……• Information as ‘thing’ or ‘concept’• Feedback as ‘thing’ or ‘concept’

– We are not explicit in this, students do not often/always know what we mean?

– And do we know….?Avoid:Positive feedback = goodNegative feedback = badFeedforward = information in advance

Feedbackin a mechanical control system

The centrifugal governor

Is used to keep shaft speeds constant - but, for students - we want to increase the ‘rate of learning/education’, call this ‘attainment’

Feedback and Feedforward in a control system sense

Do ‘exams’ provide a reference input (or even output?)A MCQ might do this. Criterion Referencing might too.

Educational ‘ramp up’To raise the level of ‘attainment’We need controlled input, ramping up.

– Output, achievement, results

attainment

EeEe Marks, no remarks for

Exam events

Ese Ee Delayed marks and remarks forEssay event

TTask set with CriterionReferenced assessment

Ee endstart

‘Attainment’ Ee

Delayed marks and remarks for Exam event, EeEe

Can I be more constructive? Bringing feedback into modules

AT

MR AT

MRTask Activity Marks Remarks

A

MR REe

MAT

MR

Gradient = ‘learning rate’

Two Activity c/work

Two Activity c/work + exam

How do we increase the gradient?

The ‘connoisseur approach’ to assessment

They want ‘the higheststandards’ - achievements

How do competitors know they have achieved?feedback

What do you (think you) mean by:

• Standards• Criteria• RubricRemarks: ‘You did quite well …’ (to

an American) = ‘You did very well’‘Marks’ and ‘Remarks’ (my usage)

Moral: students and tutors need to make sure that the terms they use are understood by everyone concerned; this includes terms like ‘feedback’

‘Marks’ and ‘Remarks’

• Marks - are just that, for a piece of work – (This begs the Q of what the work is….)

• Remarks - the comments that go with it

• The trick is to get appropriate remarks to go with the marks, as quickly as possible

[Preflights and After Action Reviews help – Increases the frequency of control ‘shoves’– Helps avoid ‘sticking points’ (Troublesome knowledge etc)

– Increases student-tutor contact]

A quick look at ‘Standards’

• UPPER SECOND • Students are able to demonstrate the

following, with respect to the criteria relevant to their discipline:

o a thorough understanding of key concepts and knowledge, and of a range of supporting evidence;

o informed consideration of key issues and interpretation of evidence;

• ability to integrate material from a variety of sources, and to deploy established techniques of analysis and enquiry, accurately and effectively;

etc

University of Sheffield, 29_Assessment.doc

Also ‘Grade Descriptors’

• These are from the tutor’s point of view.• How do students know how to achieve

designated grades?• How does ‘feedback’ help with this

process?• How are standards maintained? Eg by

various markers?‘I know a 2:1 when I see it; understanding criteria for

degree classification in franchised university programmes’ Ecclestone, K. (2001) Journal of Further and

Higher Education 25: 301-313

Assessment criteria

are descriptions of what the learner has to do in order to demonstrate that the learning outcomes have been achieved. [threshold standard; minimum expectation to achieve credit]

Dumbing down?

Actually, you are coaching(mentoring, providing scaffolding etc)

• By providing exemplars of how to achieve the standards

• By showing students their attainments (or not)

• To show how students can better achieve and

• Gain confidence in their work (and achievements)

‘Everyone’ needs trainers

Umm, sometimes it works ……

Teamwork

Feed forward

It is not judgmental. The negative connotations of past failures are banished There is no such thing as failure just Feedforward. It is much easier to deliver. People are less defensive when discussing future performanceFeedforward is taken less personally, provokes less resistance.

Marshall Goldsmith

Charting

• DIDET project

• Allison Littlejohn– Flexible learning

Undergraduate-postgraduate-PhD and the wider community

charting Supporting employers andemployment using another 3Cs

Slides from:Littlejohn, JISCPresentation, 2008

Using criterion referenced assessment

• Avoid the ‘connoisseur’ approach• Show students what is needed to accomplish tasks• Provide remarks with the marks• Marks and remarks go hand in hand• Give a sense of achievement• Thus, criteria need to be assignment specific and so• Link all this back to JITT (Just in Time

Teaching/Preflights) and AAR (After Action Reviews)– Providing proper feedback and feedforward

Examples etc:In general:1. Determine the Task and set the Activity

2. Set and refine your Objectives and Learning Outcomes

3. Set the Assessment Criteria (4 - 7 are usually enough)

4. Note potential problems, ‘troublesome knowledge’ etc and provide assistance with this (eg ‘preflights*’ before the main activity)

5. Students do the activity, write it up, paying attention to the criteria

6. You mark it according to the criteria (if you have helpers this makes it easier for them as the criteria are easily defined and adhered to)

7. Record marks for each criterion in a spreadsheet row (and add a few comments in a cell), use these to calculate the task’s mark profile (more feedback)

8. E-mail out the row of marks and remarks, separately, to each student, in one go using a nifty-tifty bit of php scripting§.

Learning outcomesAt the end of the module students will be expected to be able to:･ Plan and structure a major piece of work･ Use referencing appropriate to a published paperAssessment Task• In week 9 you are required to submit a plan for the dissertation.

The plan should include the following:･ a summary of the main argument of the dissertation (200 words);･ an indication of the structure, section headings and outline of the

content of each section (500 words);･ indication of, and full references for, texts consulted and texts to

be discussed in the dissertation.

Assessment Criteria Writing your criteria

1. Consider the learning outcomes being tested. 2. Consider the assignment you wish to set.3. Brainstorm requirements for, or attributes of, successful

performance in the assignment. 4. If necessary, specify the level of difficulty and the particular

context for the assignment. 5. Focus on what is essential and categorise the requirements

into clear criteria. 6. Check that criteria are assessable (valid and reliable) and

that they are clearly worded and unambiguous (ask students to explain what they think you mean).

(based on an example in Gosling and Moon, 2002) ･

Marking and criteria(an example from the dissertation preparation case)

Evidence of study/reading in preparation of the plan (20%)

Articulation of a clear idea/theme to be discussed in the dissertation(30%)

Design of a workable and defensible structure for the dissertation (30%)

Identification of texts to which the dissertation will refer using the Harvard referencing system (20%)

Although the criteria here are quite broad it at least shows how much effort should be expended on each section, e.g. don’t overload it with a literature review.

Practical (poster) assessmentTake this poster;

how might it be assessed?‘hmm, a low 2.1, perhaps 61%?’

This ‘connoisseur approach’ is also used for:

Not very a good assessment method is it?Malcolm Knowles was doing better in 1975 with a few criteria and a Likert scale for each.

Essays; essay exams; dissertations; projects

(The connoisseur approach does not do well in fulfilling the good practice of Gibbs, Nicol and McFarlane-Dick.)

Criteria in the mark scheme for Experiment 1 of the practical 1 from the previous slide

The table below shows marks achieved for each of 10 criteria of an exercise.Each student gets an individual e-mail with the marks for each components and some remarks.

Problemspecified

Manipulation of

system

Problemsolution

CluesGuidance

Help

Provision ofFeedback on

solutions

RulesSystem

ExamplesProcedures

Task Task Task

ResourcesSupport Support

Assessment

T Tacit knowledge required

S Sticking point(s) likely

TS

A temporal sequence describing a rule-based learning design, In Oliver et al. 2007. Describing ICT-based learning designs that promote quality learning outcomes

R R

Setting out a learning design shows not only how the task should be set up but where criteria for assessment can be identified.

A final thought:adding the veg

We have lots of ideasTechnology is helping greatly

butWhat we need to do is keep reviewing the teaching-learning process:‘ …… the time, energy and resources students devote to activities designed to enhance learning at university.’

(Kerri-Lee Krause, 2005)

A few extra references to get the basic concepts

• University of Plymouth Assessment handbook (www.plymouth.ac.uk/files/extranet/docs/TLD/Assessment guidelines March09.pdf)

• University of Bath, ‘How to use assessment criteria’ (http://www.bath.ac.uk/learningandteaching/themes/assessment-criteria/)

• Stevens, D. D., & Levi, A. J. (2005) Introduction to Rubrics, (Sterling, VA: Stylus).