Analysing feedback

transcript

Rola Ajjawi & Karen Barton Centre for Medical Education

Gwyneth Hughes, Institute of Education, London.

If you wanted to make sure your students did NOT receive and act on your feedback what would you do?

Overview of session

• JISC project introductions

• Why analyse feedback?

• 2 Tools for analysing feedback from the IOE and Dundee

• Applying each tool at the IOE and Dundee

• Resulting feedback profiles and audits

• Benefits of feedback profiling and some challenges

Project Introductions

• Assessment Careers-Institute of Education http://www.ioe.ac.uk/assessmentcareers

• http://youtu.be/VSaGbPoXPh0

• Interactive Assessment and Collaboration via Technology (InterACT) http://blog.dundee.ac.uk/interact/

• http://www.youtube.com/watch?v=S5bBFEbXDD0

Why analyse feedback?

Which is the main reason why you might analyse feedback? A Widely inconsistent practice in feedback B Lack of learner engagement with and understanding of

feedback C High teacher effort - low efficiency D Transmitted feedback creates dependency on teacher E NSS scores Result of the poll:

Feedback categories

• Giving praise (P1)

• Recognising progress or ipsative feedback (P2)

• Criticisms

• Correction of errors (C1)

• Factual criticisms (C2)

• Criticism of approach (C3)

• Giving advice

• Specific (current assignment)(A1)

• General (current assignment) (A2)

• For future assignments (A3)

• Clarifications (Q)

• Unclassified statements (O)

http://assessmentcareers.jiscinvolve.org/wp/about/

1. Focus of feedback?

• Content

• Style, format and language

• Sources and references

2. Level of Feedback (Hattie & Timperley 2007)

• Task

• Process

• Self-regulation

• Self

3. Depth of feedback (Brown & Glover 2006)

• Identified

• Categorised/Described

• Corrected/Exemplified

• Explained

• Future oriented

http://blog.dundee.ac.uk/interact/

IOE Dundee

• Which categories are most helpful for student learning?

• Criticisms

• Giving advice

• Specific (A1)

• General (A2)

IOE Dundee

• Content

• Task

• Process

• Self-regulation

• Self

• Identified

• Explained

• Future oriented

• Criticisms

• Giving advice

• Specific (A1)

• General (A2)

IOE Dundee

• Does the context matter?

• Content

• Task

• Process

• Self-regulation

• Self

• Identified

• Explained

• Future oriented

Coding framework

• The score is the number of times a

classification appears in the feedback • The default unit for analysis was the

sentence • Where a sentence contains clauses that make

distinct points, it was split into separate clauses, each of which was classified separately.

• Neutral comments that for example describe the piece of work, but do not make any judgement are unclassified

IOE & Dundee

Dundee: Coding Examples -ve +ve

Identified Poor Good

Categorised / Described

In which country? This is a good way of structuring a session

Corrected / Exemplified

This is not accurate – you may choose to read this article rather than rely on personal experience alone to support your teaching …

A safe, supportive and challenging environment is important to get open and active dialogue and discussion

Explained One of the problems with this type of format is that the respondent can easily look up the correct answers

The pilot will allow you to see if the items are understood and the data is what you are looking for

Future-orientated Do you encourage self and peer assessment? What would be the advantages and potential dangers of peer assessment in inter-professional learning?

Agree – how easy is this in a self-orientated patient-centred health provider?

Samples tested

• Formative and summative feedback for modules on 4 postgraduate programmes in Education and Psychology (171 samples)

• Recorded the total number in each category and the average per script

• Ranked the categories to obtain a feedback profile at programme level as well as an aggregate profile of the 4 programmes.

Dundee IOE

• Access database built to provide a simple interface for data entry which can be easily exported for further data manipulation

• Random 5% sample of all summative assignments marked in 2011 (by assignment and tutor) was selected (140 samples)

• Assignments retrieved from Blackboard

• Recorded the total number in each category and average per tutor

Feedback analysis All assessments (n=171)

No of occurrences Per script Rank

P1 Praise 725 4.2 1

P2 Progress 4 0.0

C1 Criticism 382 2.2 3

C2 263 1.5 4

C3 215 1.3 5

A1 Advice 474 2.8 2

A2 182 1.1 7

A3 28 0.2

Q Clarification 215 1.3 5

O 66 0.4

Results from the IOE

Results from the poll:

Vote on whether or not this is expected

P1 Praise 725 4.2 1

P2 Progress 4 0.0

C2 263 1.5 4

C3 215 1.3 5

A1 Advice 474 2.8 2

A2 182 1.1 7

A3 28 0.2

O 66 0.4

Results from the IOE

P1 Praise 725 4.2 1

P2 Progress 4 0.0

C2 263 1.5 4

C3 215 1.3 5

A1 Advice 474 2.8 2

A2 182 1.1 7

A3 28 0.2

O 66 0.4

P1 Praise 725 4.2 1

P2 Progress 4 0.0

C2 263 1.5 4

C3 215 1.3 5

A1 Advice 474 2.8 2

A2 182 1.1 7

For future

assignments 28 0.2

O 66 0.4

Vote: Is this the profile you would want?

P1 Praise 725 4.2 1

P2 Progress 4 0.0

C2 263 1.5 4

C3 215 1.3 5

A1 Advice 474 2.8 2

A2 182 1.1 7

For future

assignments 28 0.2

O 66 0.4

Dundee: Feedback audit results

• 140 assignments assessed

– Main focus of feedback was content related (95%)

– Main level of feedback was on the task (72%)

– Positive and negative feedback was equal (51% vs. 49%)

• Confirmed the wide variety in quantity and nature of feedback provided by tutors

Focus of feedback

Level of feedback

• Is this the profile you would want?

• As programme leader or director would you expect this diversity?

Benefits of doing the audit

• Raises awareness of quantity and quality of feedback provided individually and across the centre

• Opportunity for faculty development

• Objective measure of changes in feedback culture

Enablers and Challenges

• Success factors – Workshop

– Easy to use

• Challenges – Categories overlap

– Shift from grading to feedback

Success factors • Discussion of codes • Double coding and negotiation

of codes – establish inter-rater reliability

• Having examples of codes Challenges • Difficulty in interpretation of

codes and feedback • Workload

What are other factors in other institutions? How could feedback profiling be implemented without funding?

IOE Dundee

Why analyse feedback?

Which is the main reason why you might analyse feedback? A Widely inconsistent practice in feedback B Lack of learner engagement with and understanding of

feedback C High teacher effort - low efficiency D Transmitted feedback creates dependency on teacher E NSS scores

Any other questions?

References

1. Brown, E. & Glover, C. (2006) Evaluating written feedback. in: B. C. & K. Klegg (Eds) Innovative assessment in higher education. London, Routledge), 81-91.

2. Hattie, J. & Timperley, H. (2007) The Power of Feedback. Review of Educational Research, 77, 81-112.

3. Hughes, G. (2011) Aiming for Personal Best: a Case for Introducing Ipsative Assessment in Higher Education Studies in Higher Education 36 (3): 353 – 367.

4. Orsmond, P. & Merry, S. (2011) Feedback alignment: effective and ineffective links between tutors’ and students’ understanding of coursework feedback. Assessment & Evaluation in Higher Education. 36(2): 125-126.