Measuring the impact of instant high quality
feedback.
Stephen Nutbrown: [email protected] Beesley: [email protected]
Colin Higgins: [email protected]
University of Nottingham & Nottingham Trent University
BackgroundoAutomated assessment
oWhy do we assess work? Assessment of Learning vs Assessment for learning
oExample - teaching programming – a practical subject, assessment for learning is vital.
oFeedback core part of assessment for learning
OverviewoStudy of 141 Computer Science students
oAccording to the NSS, relative to other questions, students unhappy with their feedback.
oWhat is good feedback?
oIntroduction to TMA (Automated marking system) to produce feedback in line with ‘good feedback’.
oMeasure student performance – same exercise + multiple submissions
oMeasure student performance on following exercises without multiple subs
Aim: Improve feedback, measure the resultoHelp student’s learn!
oGeneral properties of good feedback :o Informative (How to improve!) & specifico Reliable & Consistento Clearly communicatedo Timelyo The assessment (and feedback) should be useful for teachers
TMA – A brief introduction
(The Marker’s Apprentice)
Electronic framework for combining automated, semi-automated, manual marking.
Mark Scheme broken into many parts, a tool for each part which may be:
Automated: e.g Spell checks, word counts, grammar checks for reports
functionality tests + convention tests for programming.
Semi-automated (help a human marker):
e.g Selecting options from a rubric or bank of feedback, each with actions on how to improve.
Manual (entirely up the marker):
e.g Free text entry
Instant?oIf all of the tools for an assignment are automated, feedback can be given instantly.
oAlso allows for marking of drafts, or using a subset of tools for a “pre-submission” to give an idea of how the student is doing.
Use in Computer ScienceoSeveral thousand submissions so far at University of Nottingham UK and University of Nottingham China.
oA range of automated tools have been developed which work with TMA for the assessment of programming.
A typical programming exercise
Overall
Does it work?X% of total grade + feedback
Is the source code of good quality?Y% of grade + feedback
Input
Output?
Rule based conventions tooloOne of several tools – but this is the one we will focus on today.
o127 rules, based on PMD (http://pmd.sourceforge.net)
oSearches for particular patterns.
oCan identify common problems and their exact line number.
oFor each violation marks deducted & feedback giveno Difficulties due to repeat violations, weightings, difficulty.
Tool example - conventions
Extremely difficult to manually mark – imagine thousands of lines of code each submission, for 140+ students.Bad code, example with common issues Good code, example without common issues int a = 99;int b = 10;int c; if(a < b) c = b-a;else if (a > b) c = a-b;else c = 0;
int first = 99;int second = 10;int calculatedDifference; if (first < second) { calculatedDifference = second – first;} else if (first > second) { calculatedDifference = first - second;} else { calculatedDifference = 0;}
Linking back to good feedback
Good example (How it should be done)
Bad example (Help identify the problem)
Reason
Hyperlink back to learning resource (Lecture slides)
Feedback given instantly
Perfectly consistent
Feedback sessions to discuss – keep communication open
Used in combination with functionality tests.
Screenshots - overview
Further help
Programming is just an example. The same is possible for reports, or other assignment types.
Programming – Coursework 0
First measurement:
oDoes not count towards final grade, as many submissions as they wish. getting started with Java.
oInstant feedback (automated) makes feedback on drafts possible.
oAssessed using functionality and conventions tools from before.
oMeasure performance between submissions
Measuring improvements within one exercise
CW0, does not count towards the final grade. 346 Submissions from 60 unique students (Avg. 5.7 each)
0102030405060708090
60.877.2 First submission
Last Submission
First Submission, Last Submission
Ave
rage
Gra
de
Results raise some questions
oDid they read the feedback and engage with it?
oDid they just mechanically fix problems?
oDid they learn anything?
Should fix line 7. See lecture 2
Submission 1
Well done
Submission n
Nothing
Survey of students (35 responses)
Did you read the instant feedback? Yes (100%)
Did the feedback highlight areas which can be improved? Yes (91%) No (9%)
Did you improve the quality of your submission based on your feedback? Yes (77%) No (23%)
Do you feel the feedback assisted in your learning and will help you in future work? Yes (68%) No (32%)
Coursework 1oCounts towards final grade (100%!)
oSame coursework set 2 years ago.
oMade up of 3 parts:o Part 1 – Ensures every student has received specific, instant feedback at least
once from TMA using functionality and convention tools.. Allowed 2 submissions.
o Part 2 – No pre-submissions. Compare results to previous years(This part had a focus on their testing of their own work)
o Part 3 – 1 pre-submission [a subset of tests] containing very few tests, for peace of mind.
Avg. Number of convention mistakes
05
101520253035
Previous Year (118)This Year(141)
Cohort
Avg
con
venti
on v
iola
tions
35.4% difference
Useful for teachers
Classe
s
General
If Stat
ementsLo
ops
Meth
ods
Objects
Strings
Exceptions
0
20
40
60
80
100
Average grade for each section
AnalysisoSame course material
oSame lecturer
oSame assignment
oImproved feedback. Specific, instant, clearly communicated, timely, useful for teachers too.
ConclusionsoThe feedback generally helped students to learn (assessment for learning)
oThe feedback, as it was specific and broken down into sections, is useful for teachers.
oThe assessment technique is vitally important to student experience and has a huge impact on student learning
oSimilar findings in China, not formally analysed.
General: For other disciplines
This study strongly highlights the importance of good feedback and well-considered assessment techniques. Students created submissions of much higher quality than before.
TMA almost forces good feedback practises, even for semi-automated tools
◦ Require action◦ Split up mark scheme – specific◦ Saves time
To take awayoFeedback is extremely important, considering all of the guidelines is a good start, even if it isn’t automated.
oAutomated assessment may help, but is not a replacement for human interaction.
oCan you do anything to improve the turnaround time or detail of your feedback?
oTMA will be made generally available soon, if you would like a demo, please feel free to ask.
