Post on 29-Mar-2020
transcript
Deakin University CRICOS Provider Code: 00113B
Susie Macfarlane, Dr Dominique Condo & Dr Claire Margerison
Assessment for learning not marks: Designing standards-based rubrics for students’ learning and feedback literacy
Deakin University CRICOS Provider Code: 00113B
Background context
Deakin University CRICOS Provider Code: 00113B
Issues: Unit / course context
1. Students weren’t looking at rubric
2. Students complaining about marks, but were not
focussed on learning or feedback, as focussed on marks
3. There is no clear statement of the standard we expect
students to achieve
4. Students didn’t use feedback to improve their work
Deakin University CRICOS Provider Code: 00113B
Literature:Impact of
rubrics, grades and feedback on
learning
Deakin University CRICOS Provider Code: 00113B
Q1. How does the assessment support students’ understanding of the standard?
Do grades tell us what a student can or can’t do?
Does a pass mark represent the expected standard?
Boud, D. (2017). Standards-based assessment for an era of increasing transparency. In Carless, D., Bridges, S., Chan, C. and Glofcheski, R. (Eds). Scaling up assessment for learning in higher education. Dordrecht: Springer, 19-31.
Deakin University CRICOS Provider Code: 00113B
Marks (out of 100) provide false precision about standard of work
“If only three or four categories of difference can be ascertained, then this is all that can be
legitimately reported. Any finer grain of reporting is spurious and cannot be justified.”
(Boud, 2017)
Q1. How does the assessment support students’ understanding of the standard?
Deakin University CRICOS Provider Code: 00113B
Q2. How does the assessment support learning?
Marks provide very little information the student can
use to improve
Students need rich information
Boud, D. (2017). Standards-based assessment for an era of increasing transparency. In Carless, D., Bridges, S., Chan, C. and Glofcheski, R. (Eds). Scaling up assessment for learning in higher education. Dordrecht: Springer, 19-31.
Deakin University CRICOS Provider Code: 00113B
Who sets the standard?
Do students engage with and trying to understand the standard?
Who evaluates the work?
Who provides feedback?
Is feedback a one-way flow of information, or a dialogue?
Do students use the feedback?
Do we know if students’ work improved?
Q3. How does the assessment support students’ active participation?
Deakin University CRICOS Provider Code: 00113B
Definition:
A rubric must have:
• evaluative criteria• quality definitions for those criteria at particular levels, and • a scoring strategy.
Dawson, 2015
Popham, 1997
Deakin University CRICOS Provider Code: 00113B
Rubrics:
• Make expectations and criteria explicit
• Facilitate feedback and self-assessment
• Support students’ self-regulated learning
• Students like rubrics and find them useful
(Jonsson and Svingby, 2007)
(Jonsson and Svingby, 2007)
Jonsson and Pandero, 2017
Reddy and Adrade, 2017
Deakin University CRICOS Provider Code: 00113B
Aims
Aims
Students engage with feedback
Focus students and markers on assessment criteria (standards and
capabilities)(rather than marks)
Students evaluate work against the expected
standard
Feedback is information
about the gap
between the actual level and the reference level
which is used to alter the gap in some way.
(Ramaprasad 1983 p.4, cited in Boud and Molloy, 2013)
Feedback definition:
Aims
Students use feedback to improve their work
Students and markers evaluate work against the expected standard
Focus students and markers on assessment criteria (standards and
capabilities)(rather than marks)
Deakin University CRICOS Provider Code: 00113B
Rubric design
Aim
Focus on standards and capabilities
(rather than marks)
Rewrote assessment criteria: authentic cognitive skills or task that can be evidencedRewrote the rubric with 3 levels, one at the expected standard, clear descriptorsFramed language around development and learning
Strategies
Expected standard
Positive statement of what the student can do
Deakin University CRICOS Provider Code: 00113B
ExcellentProfessionalSophisticatedStrong
GoodAdequate
PoorWeak
Unpacking notions of quality
Deakin University CRICOS Provider Code: 00113B
ClarityAccuracyPrecision
Quality descriptors - cognitive
RelevanceBreadthDepth
LogicSignificanceFairness
FeasibilitySustainabilityEffectiveness Necessity
Discipline or context specific understandings of quality
Elder & Paul (2010) Foundation for Critical thinking
SufficiencyCompleteness ValidityRationality
ConsistencyAuthenticityEfficiency
Fluent, accurate task performanceRespectful client interaction
Correct clinical reasoningAmount of assistance required
Application to new contexts
Specific, contextualised and authentic quality
indicators
Aim
Focus on standards and capabilities
(rather than marks)
Students and markers evaluate work against the expected standard
Students submit a draft and evaluate it using the rubric. Markers also provide feedback.Removed marks from the rubric students used to self evaluate
Strategies
Rewrote assessment criteria: authentic cognitive skills or task that can be evidencedRewrote the rubric with 3 levels, one at the expected standard, clear descriptorsFramed language around development and learning
Engaging students in the rubric.
Deakin University CRICOS Provider Code: 00113B
Students create the rubric
Students critique or contribute to the rubric
Students use the rubric to evaluate their own or others’ work
Students use the rubric to evaluate an example
Teacher uses rubric to evaluate an example
Teacher shows high and low quality work
Teacher explains rubric
Nicol (2010) Assess. & Eval. in HE
Student engagement with rubricFr
om lo
wes
t to
high
est l
evel
of p
artic
ipat
ion
Goals
Rewrote assessment criteria: authentic cognitive skills or task that can be evidencedRewrote the rubric with 3 levels, one at the expected standard, clear descriptorsFramed language around development and learning
Aim
Students use feedback to improve their work
Focus on standards and capabilities
(rather than marks)
Students self review and receive feedback
on draft
Strategies
Students submit a draft and evaluate it using the rubric. Markers also provide feedback on draft.Removed marks from the rubric students used to self evaluate
Feedback Mark 2
Students submit a draft and evaluate it using the rubric. Markers also provide feedback on draft.
Rewrote assessment criteria: authentic cognitive skills or task that can be evidencedRewrote the rubric with 3 levels, one at the expected standard, clear descriptorsFramed language around development and learning
Aim
Students use feedback to improve their work
Students resubmit assignment explaining their use of feedbackPD of Unit ChairsTrained markers to provide feedback on the standard student has achieved on Assessment Criteria
Focus on standards and capabilities
(rather than marks)
Students self review and receive feedback
on draft
Strategies
Removed marks from the rubric students used to self evaluate
Goals
Deakin University CRICOS Provider Code: 00113B
Impact and Outcomes
Unit Chair perspective
No remark requests since new rubric
Feedback students can use: 50% on draft
Students’ self assessment
Alignment - Markers provide feedback on students’ self evaluation
Confidence as assessor
Deakin University CRICOS Provider Code: 00113B
Response from markers
“As a marker, I felt that self assessed rubrics were very helpful for providing meaningful and individualised feedback.”
“Knowing the level the student felt they were at definitely helped direct feedback and helped to justify any discrepancies between marks given and the students perceived level.”
“Being able to provide feedback based not only on the work the student produced, but also on their planned direction (self identified areas to improve) I believe meant constructive feedback was easier to provide.”
Deakin University CRICOS Provider Code: 00113B
Outcomes: self evaluation
Asking students to self-mark helps fine tune their abilities to review and improve their work and helps me provide more individualised feedback. Self-marking uniquely enables me to provide feedback on how accurately students perceive their own work, for example a student may think that they should focus on citing more references when really they need to provide more details about the references they identify.
By helping students review their own work more accurately they are better able to improve their work independently.
I can also better provide feedback on the skills individual students find the most difficult and want help with e.g. if a student identified that they found it difficult to source and discuss peer reviewed literature I can provide specific feedback about how they did or didn't achieve this and further recommend how to improve. At the end of this process if a student improved on a weakness it is easier for an assessor like me to identify this and commend them on their hard work and show that the effort paid off.
Deakin University CRICOS Provider Code: 00113B
Response from markers
“I think the rubric is easy to use and provide feedback with.It ensures more consistency in the assessment/feedback process and I think makes the expectations clear to the marker and student alike.
However it can sometimes be a little too simple and there is room for interpretation within each of the criteria within the rubric.”
Deakin University CRICOS Provider Code: 00113B
1. Replaced exams with other assessment
2. Made a common structure for rubric
3. Implemented students’ self assessment using the rubric
Course wide changes in M.Diet
Deakin University CRICOS Provider Code: 00113B
Discussion
Deakin University CRICOS Provider Code: 00113B
References
Ajjawi & Boud, D (2018) Examining the nature and effects of feedback dialogue, Assessment & Evaluation in Higher Education, 43:7, 1106-1119, DOI: 10.1080/02602938.2018.1434128
Barton, K, Schofield, S, McAleer, S, Ajjawi, R. (2016) Translating evidence-based guidelines to improve feedback practices: the interact case study, BMC Medical Education, 16: 53
Bearman M and Ajjawi R (2018) From “Seeing Through” to “Seeing With”: Assessment Criteria and the Myths of Transparency. Frontiers in Education, 3: 96, doi: 10.3389/feduc.2018.00096
Boud, D. (2017). Standards-based assessment for an era of increasing transparency. In Carless, D., Bridges, S., Chan, C. and Glofcheski, R. (Eds). Scaling up assessment for learning in higher education. Dordrecht: Springer, 19-31.
Boud, D & Molloy, E, (2013) Feedback in Higher and Professional Education: Understanding it and doing it well, Routledge, UK.
Carless, D (2006) Differing perceptions in the feedback process, Studies in Higher Education, 31(2) 219 – 233
Dawson, P. (2015). Assessment rubrics: towards clearer and more replicable design, research and practice. Assess. Eval. High. Educ. 42, 347–360. doi: 10.1080/02602938.2015.1111294
Jönsson A., Panadero E. (2017) The Use and Design of Rubrics to Support Assessment for Learning. In: Carless D., Bridges S., Chan C., Glofcheski R. (eds) Scaling up Assessment for Learning in Higher Education. The Enabling Power of Assessment, vol 5. Springer, Singapore
Hattie, J and Timperley, H (2007) The Power of Feedback, Review of Educational Research, 77: 81-112
Deakin University CRICOS Provider Code: 00113B
References
Hounsell, D (2015) Flipping feedback, WISE Assessment Briefing #12, CETL, Available at https://t.co/cq1cek57ui
Nicol, D. (2010). From monologue to dialogue: Improving written feedback processes in mass higher education. Assessment and Evaluation in Higher Education, 35(5), 501-517. doi:10.1080/02602931003786559
Reddy, Y. M., and Andrade, H. (2010). A review of rubric use in higher education. Assessment and Evaluation in Higher Education, 35, 435–448. doi: 10.1080/02602930902862859
Sadler, D. Royce(2009) 'Grade integrity and the representation of academic achievement', Studies in Higher Education, 34: 7, 807 — 826
Key elements of feedbacks
Hattie & Timperley (2007). The power of feedback
Where am I going?
How am I going?
Where to next?
ULOsTask
Rubrics
Rubrics Self evaluation
DraftsPeer feedback
Feedback responseFeedback is used
Deakin University CRICOS Provider Code: 00113B
Programmatic assessment
“Marks and grades may for convenience be aggregated across a programme, but such a process says little about how a student meets the outcomes for a programme as a whole.”
Boud, D. (2017). Standards-based assessment for an era of increasing transparency. In Carless, D., Bridges, S., Chan, C. and Glofcheski, R. (Eds). Scaling up assessment for learning in higher education. Dordrecht: Springer, 19-31.
Deakin University CRICOS Provider Code: 00113B
False rigour
“…an explicit model produces outputs (grades) that appear to have
been substantially ‘validated’ through careful attention to all the
steps.
However, the model itself is characterized by indeterminacy, that is
inherently weak. … its implementation creates a veil of rigour that
makes it difficult for learners to question either the process or the
outcome”
Sadler (2009, p.5)
2015 SENS Rubric enhancement project
RUBRIC PRINCIPLES
1. Assessment Criteria that clearly express and allow judgement of the key learning outcomes of the unit, and are have a clear and obvious to connection to ULOs, CLOs and GLOs
2. Clearly describe the observable, measurable learning outcomes the students demonstrate, not content topics
3. Focus the assessment criteria on authentic graduate capabilities and knowledge, not the specifics of a particular assessment task
4. Specifically describe the different levels of performance for each criteria, to give students a clearer understanding of the characteristics of a good performance and what they need to do to improve next time
5. Write the Quality Descriptors to clearly communicate the authentic characteristics of a good performance in the world (e.g. using the Paul and Elder (2010) Universal intellectual standards of critical thinking: clarity, accuracy, precision, relevance, depth, breadth, significance, logic, fairness)
6. Performance level labels refer to the level of achievement, and clearly indicate the level students are expected to achieve.
Deakin University CRICOS Provider Code: 00113B
Issues: M.Diet
Students: High achieving, competitive & marks focussed
Staff: Small course team, willingness to make changes and interested in continuous improvement cycle
Everyone in team, willingness to come to a shared understanding
Deakin University CRICOS Provider Code: 00113B
Course wide standards based assessment
• Introduced competency based assessment across whole course
• No marks are given
• Students collect portfolio of evidence against the DAA competencies
• use portfolio in their mock interviews
Added in transferable skills / GLOs