27
Rola Ajjawi & Karen Barton Centre for Medical Education Gwyneth Hughes, Institute of Education, London.
Rola Ajjawi & Karen Barton Centre for Medical Education
Gwyneth Hughes, Institute of Education, London.
If you wanted to make sure your students did NOT receive and act on your feedback what would you do?
Overview of session
• JISC project introductions
• Why analyse feedback?
• 2 Tools for analysing feedback from the IOE and Dundee
• Applying each tool at the IOE and Dundee
• Resulting feedback profiles and audits
• Benefits of feedback profiling and some challenges
Project Introductions
• Assessment Careers-Institute of Education http://www.ioe.ac.uk/assessmentcareers
• http://youtu.be/VSaGbPoXPh0
• Interactive Assessment and Collaboration via Technology (InterACT) http://blog.dundee.ac.uk/interact/
• http://www.youtube.com/watch?v=S5bBFEbXDD0
Why analyse feedback?
Which is the main reason why you might analyse feedback? A Widely inconsistent practice in feedback B Lack of learner engagement with and understanding of
feedback C High teacher effort - low efficiency D Transmitted feedback creates dependency on teacher E NSS scores Result of the poll:
Feedback categories
• Giving praise (P1)
• Recognising progress or ipsative feedback (P2)
• Criticisms
• Correction of errors (C1)
• Factual criticisms (C2)
• Criticism of approach (C3)
• Giving advice
• Specific (current assignment)(A1)
• General (current assignment) (A2)
• For future assignments (A3)
• Clarifications (Q)
• Unclassified statements (O)
http://assessmentcareers.jiscinvolve.org/wp/about/
1. Focus of feedback?
• Content
• Style, format and language
• Sources and references
2. Level of Feedback (Hattie & Timperley 2007)
• Task
• Process
• Self-regulation
• Self
3. Depth of feedback (Brown & Glover 2006)
• Identified
• Categorised/Described
• Corrected/Exemplified
• Explained
• Future oriented
http://blog.dundee.ac.uk/interact/
IOE Dundee
• Which categories are most helpful for student learning?
• Giving praise (P1)
• Recognising progress or ipsative feedback (P2)
• Criticisms
• Correction of errors (C1)
• Factual criticisms (C2)
• Criticism of approach (C3)
• Giving advice
• Specific (A1)
• General (A2)
• For future assignments (A3)
• Clarifications (Q)
• Unclassified statements (O)
http://assessmentcareers.jiscinvolve.org/wp/about/
IOE Dundee
1. Focus of feedback?
• Content
• Style, format and language
• Sources and references
2. Level of Feedback (Hattie & Timperley 2007)
• Task
• Process
• Self-regulation
• Self
3. Depth of feedback (Brown & Glover 2006)
• Identified
• Categorised/Described
• Corrected/Exemplified
• Explained
• Future oriented
http://blog.dundee.ac.uk/interact/
• Giving praise (P1)
• Recognising progress or ipsative feedback (P2)
• Criticisms
• Correction of errors (C1)
• Factual criticisms (C2)
• Criticism of approach (C3)
• Giving advice
• Specific (A1)
• General (A2)
• For future assignments (A3)
• Clarifications (Q)
• Unclassified statements (O)
http://assessmentcareers.jiscinvolve.org/wp/about/
IOE Dundee
• Does the context matter?
1. Focus of feedback?
• Content
• Style, format and language
• Sources and references
2. Level of Feedback (Hattie & Timperley 2007)
• Task
• Process
• Self-regulation
• Self
3. Depth of feedback (Brown & Glover 2006)
• Identified
• Categorised/Described
• Corrected/Exemplified
• Explained
• Future oriented
http://blog.dundee.ac.uk/interact/
Coding framework
• The score is the number of times a
classification appears in the feedback • The default unit for analysis was the
sentence • Where a sentence contains clauses that make
distinct points, it was split into separate clauses, each of which was classified separately.
• Neutral comments that for example describe the piece of work, but do not make any judgement are unclassified
IOE & Dundee
Dundee: Coding Examples -ve +ve
Identified Poor Good
Categorised / Described
In which country? This is a good way of structuring a session
Corrected / Exemplified
This is not accurate – you may choose to read this article rather than rely on personal experience alone to support your teaching …
A safe, supportive and challenging environment is important to get open and active dialogue and discussion
Explained One of the problems with this type of format is that the respondent can easily look up the correct answers
The pilot will allow you to see if the items are understood and the data is what you are looking for
Future-orientated Do you encourage self and peer assessment? What would be the advantages and potential dangers of peer assessment in inter-professional learning?
Agree – how easy is this in a self-orientated patient-centred health provider?
Samples tested
• Formative and summative feedback for modules on 4 postgraduate programmes in Education and Psychology (171 samples)
• Recorded the total number in each category and the average per script
• Ranked the categories to obtain a feedback profile at programme level as well as an aggregate profile of the 4 programmes.
Dundee IOE
• Access database built to provide a simple interface for data entry which can be easily exported for further data manipulation
• Random 5% sample of all summative assignments marked in 2011 (by assignment and tutor) was selected (140 samples)
• Assignments retrieved from Blackboard
• Recorded the total number in each category and average per tutor
Feedback analysis All assessments (n=171)
No of occurrences Per script Rank
P1 Praise 725 4.2 1
P2 Progress 4 0.0
C1 Criticism 382 2.2 3
C2 263 1.5 4
C3 215 1.3 5
A1 Advice 474 2.8 2
A2 182 1.1 7
A3 28 0.2
Q Clarification 215 1.3 5
O 66 0.4
Results from the IOE
Results from the poll:
Vote on whether or not this is expected
No of occurrences Per script Rank
P1 Praise 725 4.2 1
P2 Progress 4 0.0
C1 Criticism 382 2.2 3
C2 263 1.5 4
C3 215 1.3 5
A1 Advice 474 2.8 2
A2 182 1.1 7
A3 28 0.2
Q Clarification 215 1.3 5
O 66 0.4
Results from the IOE
Feedback analysis All assessments (n=171)
No of occurrences Per script Rank
P1 Praise 725 4.2 1
P2 Progress 4 0.0
C1 Criticism 382 2.2 3
C2 263 1.5 4
C3 215 1.3 5
A1 Advice 474 2.8 2
A2 182 1.1 7
A3 28 0.2
Q Clarification 215 1.3 5
O 66 0.4
Feedback analysis All assessments (n=171)
No of occurrences Per script Rank
P1 Praise 725 4.2 1
P2 Progress 4 0.0
C1 Criticism 382 2.2 3
C2 263 1.5 4
C3 215 1.3 5
A1 Advice 474 2.8 2
A2 182 1.1 7
A3
For future
assignments 28 0.2
Q Clarification 215 1.3 5
O 66 0.4
Vote: Is this the profile you would want?
No of occurrences Per script Rank
P1 Praise 725 4.2 1
P2 Progress 4 0.0
C1 Criticism 382 2.2 3
C2 263 1.5 4
C3 215 1.3 5
A1 Advice 474 2.8 2
A2 182 1.1 7
A3
For future
assignments 28 0.2
Q Clarification 215 1.3 5
O 66 0.4
Dundee: Feedback audit results
• 140 assignments assessed
– Main focus of feedback was content related (95%)
– Main level of feedback was on the task (72%)
– Positive and negative feedback was equal (51% vs. 49%)
• Confirmed the wide variety in quantity and nature of feedback provided by tutors
Focus of feedback
Level of feedback
• Is this the profile you would want?
• As programme leader or director would you expect this diversity?
Benefits of doing the audit
• Raises awareness of quantity and quality of feedback provided individually and across the centre
• Opportunity for faculty development
• Objective measure of changes in feedback culture
Enablers and Challenges
• Success factors – Workshop
– Easy to use
• Challenges – Categories overlap
– Shift from grading to feedback
Success factors • Discussion of codes • Double coding and negotiation
of codes – establish inter-rater reliability
• Having examples of codes Challenges • Difficulty in interpretation of
codes and feedback • Workload
What are other factors in other institutions? How could feedback profiling be implemented without funding?
IOE Dundee
Why analyse feedback?
Which is the main reason why you might analyse feedback? A Widely inconsistent practice in feedback B Lack of learner engagement with and understanding of
feedback C High teacher effort - low efficiency D Transmitted feedback creates dependency on teacher E NSS scores
Any other questions?
References
1. Brown, E. & Glover, C. (2006) Evaluating written feedback. in: B. C. & K. Klegg (Eds) Innovative assessment in higher education. London, Routledge), 81-91.
2. Hattie, J. & Timperley, H. (2007) The Power of Feedback. Review of Educational Research, 77, 81-112.
3. Hughes, G. (2011) Aiming for Personal Best: a Case for Introducing Ipsative Assessment in Higher Education Studies in Higher Education 36 (3): 353 – 367.
4. Orsmond, P. & Merry, S. (2011) Feedback alignment: effective and ineffective links between tutors’ and students’ understanding of coursework feedback. Assessment & Evaluation in Higher Education. 36(2): 125-126.
