Enhancing students’ involvement in the assessment process via

exemplars.

Prof. Kay Sambell, Northumbria University.

Higher Education Academy 6th Annual Conference: ‘Shaping the Future: exploring impacts and changes to the student

learning experience.’ Univ Hertfordshire, 22-23 June 2010.

Project assumes it is important to help students appreciate the rules of engagement in university

assessment practices. • Context: Assessment of Assessment for Learning

(AfL) – Assessment reform movement seeks to involve

students actively and explicitly so students no longer passive recipients or helpless victims of university assessment practices.

2

Rationale for student involvement– Long term goals

• Society demands more than passive graduates who have simply complied with an assessment regime: – ‘assessment for the longer term’, ‘sustainable assessment,’

‘future-oriented assessment’ (Knight & Yorke, 2002; Carless et al, 2007; Boud & Falchicov, 2006, Boud & Associates, 2010; The Weston Manor Group Assessment Manifesto, 2008)

• Foster evaluative expertise and self-monitoring (Sadler, 1989)

• ‘Assessment as learning’ i.e. Student active engagement in assessing own learning essential to self-regulation (Earl, 2003)

– Shorter term goals• Involvement also a means of helping students internalize a

discipline’s expectations/notions of quality (Gibbs, 2009)• Fairness (Struyven, 2005)

3

4

Indispensable conditions for self regulation (Sadler 1989)

• the student comes to hold a concept of quality roughly similar to that held by the teacher...

• is able to monitor continuously the quality of what is being produced during the act of production itself..

• ...and has a repertoire of alternative moves or strategies from which to draw

(knows the standard or goal)

(recognises the ‘gap’ between their work and the standard)

(can take action to fill the ‘gap’)

Social constructivist assessment process models (Rust et al, 2005)

• Not enough to ‘publish’ criteria/guidelines etc. (Price et al, 2006)– ‘Transparency’ seems to have become muddled with the idea of

‘writing things down’ (Orr, 2007: 646).

• Instead, assessment viewed as a social practice which students (and staff) need, like anything else, to learn by doing/participation.– It’s difficult to communicate the meaning of assessment criteria

• They often include words such as ‘analytical’, ‘critical’ or ‘satisfactory’ which are relatively meaningless unless you have a framework in which to understand them (Rust et al, 2005).

5

• Dominant model has been to develop ‘social constructivist’ pre-assessment intervention workshops

• (Rust et al, 2005; Norton et al, 2002; Harrington et al,2006; Orsmond et al, 2002; Bloxham & West, 2007)

– Designed to engage students actively with assessment criteria and standards.

– Dialogue vital to convey ‘tacit knowledge.’

6

Further challenges include...

• Academics view criteria via disciplinary/discoursal lenses – The ‘building blocks’ with which disciplinary

knowledge/the domain itself is constructed vary. – As subject experts, academics are familiar with

the discourses of their particular knowledge communities, which are visible in the highly situated context of their academic communities.

7

The challenges of ‘indeterminancy’ with pre-set criteria.

• Further, the appraisal of complex works (high-order cognitive abilities, lateral thinking, divergent responses, innovative activities) can rarely be boiled down to a set of measures (criteria) that a non-expert can employ to arrive at the ‘correct’ appraisal– Criteria are ‘indeterminate’ (Sadler, 2009) .

8

Optional workshops can help, but....

• Students don’t always attend.• May see them as ‘remedial.’• May focus on strategic-related approaches (e.g.

‘essay-planning explained clearly’) rather than deep-related approaches (e.g. ‘gave insight into [Psychology]’). (Harrington et al, 2006)

• May use criteria mechanistically (as things to cover/put in in order to get marks, rather than expressing understanding of the topic).

(Bloxham & West, 2007)

9

Our research question:

How far and in what ways can exemplars, embedded in subject teaching, offer first-year students a way of ‘seeing’ the knowledge-constructing enterprise within a situated domain which underpins judgements of quality that a subject-expert makes?

10

In our case study exemplars used to enhance staff-student dialogue about assessment and

learning practices by encouraging students to...

– develop a shared understanding of the tacit knowledge and implicit rules of engagement that experts employ when assessing student work, which includes complex subject knowledge, situated in the context of the discipline (Orr, 2007; Bloxham, 2009).

– actively make, in a scaffolded and supportive learning environment, evaluative judgments about the relative merit of concrete examples of student writing

– participate in assessment in a generative way (that is, providing, rather than simply receiving feedback)

11

What Are Exemplars?

• Exemplars are ‘key examples chosen so as to be typical of designated levels of quality or competence. The exemplars are not standards themselves but are indicative of them. ..they specify standards implicitly.’

(Sadler, 1989 cited in Handley et al, 2008, p44).

12

Constructing effective exemplars

• Handley et al (2008: 44) suggest that exemplars– may be complete assignments or excerpts.– may be authentic pieces of student work, or may

be (re)constructed by staff (so as to illustrate specific pedagogic points in as transparent manner as possible).

– may be annotated with feedback to • help students understand what tutors look for.• help students build their self assessment skills .

13

Exemplars carefully chosen by lecturers to..

• act as ‘pre-emptive formative assessment’ (Carless, 2007)– ‘Engineer in’ the essence of the discipline.– Act as ‘instructional scaffolding’ by trying to

circumvent conceptual mis-steps or incomplete understandings before they occur and interfere with how students tackle assessment• I.e. teachers use their experience and expertise to

select common mistakes novices make, to bring them to visibility.

14

Pedagogic Process

Preparation before workshop• Students prepared short piece of writing on a key concept in

subject domain(not more than 1 side A4)Phase 1• Brought this to session, where they were given 4 exemplars

on this topic to read and discuss.• Students asked to work in small groups and place exemplars in rank

order.Phase 2• Tutors revealed and discussed rationale for their rankings.Phase 3• Students asked to reflect and generate feedback & reflect on

how would change own work.

15

General findings: students’ perspectives.

• “I think seeing it just makes you understand it more. Like, someone can stand there and say, 'You shouldn't do this and that' but until you've actually seen it then you don't know what that looks like.”

• “I was just writing how I thought it should be done, just in my own head. But now I know what they are looking for and what I should be, how I should be writing, so yes, I think it's been helpful.”

• “It helps, because you read it and think ‘Oh, they’ve put that, and that didn’t sound right, or that looks right’, and so it helps you listen out for your own writing.”

16

Surface approaches.

• “I thought some of them immediately looked a bit like they weren't going to be quite right. The one that had bullet points in it. I was a bit, 'Well that's a bit strange for an essay…”

• “There were a couple where a lot of it started with 'I' and 'My' and that's just immediately, when you look at, well, when we looked at those two they kind of jumped out as, 'Oh-oh, this might not be great!’”

17

Attempts to see differently in the light of lecturers’ dialogue.

• “What we thought was best or worst was different to what they thought!”

• “When I first read number 4, I thought it was really good, I liked that she said what she thought. But then like, I myself, I went back and read it again and it totally doesn't follow the question or anything”.

18

Adjusting views of the rules of engagement in university assessment

practices.• Different sightings of criteria/standards

– ‘At school you got told what you needed to put in your essays. You'd get criteria… Well, not like these criteria…more than that: exactly what you need and what you don't need to put in.’

• Different sightings of effective literacy practices– “Actually, at school, they might help you too much and it might

be their words you're writing rather than your own. I think you kind of have to just take that initiative.”

– “I think they want us to draw on a lot of different viewpoints and not just think, 'Right, this is what I think, this is what I'm going to write about' or 'This is what I've read about in this one book so that's going to be my point.‘”

19

Thanks for listening!

• For further information about the Rules of Engagement project please contact the project director kay.sambell@northumbria.ac.uk

• or see the project pages under development on the ESCalate website: http://escalate.ac.uk/6488

20