The Shift in Learning & Development www.paradigm.edu.au
Enhancing Assessment Practice
ACT Assessors’
Professional Development Day
Presented by Greg Weis
Friday, June 11, 2010
The Shift in Learning & Development www.paradigm.edu.au
Assessment
How would you define the term “assessment”?
Definition: “Assessment in training is about measuring an individual’s ability to see if they have reached the stated outcomes of a training program…Assessment is based upon ‘the gathering and judging of evidence about the performance of individuals’.” [italics and bold added]
Tovey, M., & Lawlor, D., (2008). Training in Australia. (3rd Edition).
The Shift in Learning & Development www.paradigm.edu.au
Assessment
What implications for assessors?
“…high levels of professional expertise are needed to decide how to develop competence and how to make holistic, context-sensitive judgements about whether it has been achieved.” [italics and bold added]
Schofield and McDonald (2004, p. 4)
The Shift in Learning & Development www.paradigm.edu.au
Assessment
Figure 1
Yes No
Indirect Life’s experiences
Reports
Records
Purposes Certification
RPL Extent of learning
Types Diagnostic Formative
Summative Methods
Knowledge Skills
Attitudes
Assessment
Qualification or
Statement of Attainment
issued
Learning
Direct Teacher facilitated Learner instigated
Judgement
Competent?
Interpretation
Evidence
N.B. For the sake of illustration, The ‘Evidence Cycle’ part of the above diagram, has been shown as a separate process to assessment, whereas in reality it is actually a part of the assessment process.
The Shift in Learning & Development www.paradigm.edu.au
Quality of Assessment Practices
http://www.nqc.tvetaustralia.com.au/nqc_publications
The Shift in Learning & Development www.paradigm.edu.au
Assessment Moderation
What do we mean by the term “moderation”?
NQC definition: “Moderation is the process of bringing assessment judgements and standards into alignment. It is a process that ensures the same standards are applied to all assessment results within the same Unit(s) of Competency. It is an active process in the sense that adjustments to assessor judgements are made to overcome differences in the difficulty of the tool and/or the severity of judgements.” National Quality Council, (2009). A Code of Professional Practice for Assessment Validation and Moderation. Page 6 & 7.
The Shift in Learning & Development www.paradigm.edu.au
Assessment Moderation
What does “moderation” mean in plain English? Purpose / Definition:
• Allows for adjustments to be made before the final determination of a result.
• To determine, based on the evidence before another qualified assessor, that the assessment judgement is sound. In other words, another qualified assessor would have reached the same conclusion or judgement had he/she be presented with the same evidence.
The Shift in Learning & Development www.paradigm.edu.au
Assessment Moderation
Why?
1. Best Practice (We work for a Quality RTO)
2. Promoted by NQC (National Quality Council)
3. Benefits students
4. Benefits assessors
The Shift in Learning & Development www.paradigm.edu.au
Assessment Moderation
Reliability
Inter-rater reliability: Consistency of judgement by different assessors, using the same assessment tool and procedure.
Intra-rater reliability: Consistency of judgement across time and location, by the same assessor using the same assessment tool.
Definitions courteous of Gillis, S., & Griffin, P., (2008). Competency Assessment. In J. Athanasou (Ed.), Adult Education and Training.
The Shift in Learning & Development www.paradigm.edu.au
Assessment Moderation
That the original assessor and the moderating assessor need to reach consensus on the final determination of the assessment result.
Implications? • Competent / Not Yet Competent • Graded assessment – TAFE. Pass, Credit, Distinction.
What does “Inter-rater” reliability mean in plain English?
The Shift in Learning & Development www.paradigm.edu.au
Assessment Validation
What do we mean by the term “validation”?
NQC definition: “Validation…involves checking that the assessment tool produced valid, reliable, sufficient, current and authentic evidence to enable reasonable judgements to be made as to whether the requirements of the relevant aspects of the Training Package or accredited course had been met. It includes reviewing and making recommendations for future improvements to the assessment tool, process and/or outcomes.” National Quality Council, (2009). A Code of Professional Practice for Assessment Validation and Moderation. Page 6. [bold added]
The Shift in Learning & Development www.paradigm.edu.au
Assessment Validation
What does “validation” mean in plain English? Purpose / Definition:
To check that the assessment tool ‘mirrors’ the unit of competency.
This includes, as a minimum:
• Elements & Performance Criteria
• Required Skills & Knowledge; Attitude of the student
• The Critical Aspects of Assessment / Competency
In short…It assesses what it claims to assess
The Shift in Learning & Development www.paradigm.edu.au
Assessment Validation
Construct Validity
Criterion related Validity
Face Validity
Consequential Validity
Content Validity
Five ‘facets’ of Validity
Face Validity: The extent to which the assessment task resembles real workplace activities.
Content Validity: How well the content of the assessment task reflects the required knowledge and skills specified in the Unit of Competency. Definitions courteous of Gillis, S., & Griffin, P., (2008). Competency Assessment. In J. Athanasou (Ed.), Adult Education and Training.
Validity
The Shift in Learning & Development www.paradigm.edu.au
Assessment Validation
“Our project was based on…[a] broader approach to validity which sees it as the extent to which the interpretation and use of an assessment outcome can be supported by evidence.” [Italics in original] Thomson, Saunders and Foyster (2001, p. 10)
Validity
“Validity is the main assessment game––some would say the only game.” Thomson, Saunders and Foyster (2001, p. 10)
The Shift in Learning & Development www.paradigm.edu.au
Assessment Validation
https://vetinfonet-staging.det.wa.edu.au/progdev/resources.aspx#Assessing%20competence
The Shift in Learning & Development www.paradigm.edu.au
Assessment Validation
How can you assist your RTO with Content & Face validity? Feedback on assessment tasks
• Know the Units of Competency inside-out. Evidence Guide.
• Trial different assessment methods
• Realistic scenarios for assessment
• Role plays
• Know the workplace task/s – make assessments realistic.
• Realistic scenarios for assessment
• Role plays
The Shift in Learning & Development www.paradigm.edu.au
Assessors’ Competencies
Implementation of AQTF 2010 circa 04/2011
(a) have the necessary training and assessment competencies as determined by the National Quality Council or its successors, and
1.4 Training and assessment is delivered by trainers and assessors who:
(b) have the relevant vocational competencies at least to the level being delivered or assessed, and (c) can demonstrate current industry skills directly relevant to the training/assessment being undertaken and (d) continue to develop their VET knowledge and skills as well as their industry currency and trainer/assessor competence.
The Shift in Learning & Development www.paradigm.edu.au
Assessor’s Competencies
Implications for assessors?
• May need to work “voluntarily”
• May have to attend DET workshops on assessment practice
Demonstrate currency
• Professional reading
VET Knowledge & Skills
• Get back into the workplace
• Industry workshops / shows
• Professional reading • May have to attend DET workshops on assessment practice • Professional reading
The Shift in Learning & Development www.paradigm.edu.au
In Conclusion…
Assessment is the ‘pointy end’ learning
It’s all about outcomes for the students!
Assessment is on the ‘radar’ of the regulators
Remember…
• Learning & Development professionals hone and enhance their assessment practice
Questions?
The Shift in Learning & Development www.paradigm.edu.au
References Gillis, S., & Griffin, P., (2008). Competency Assessment. In J. Athanasou (Ed.), Adult Education and Training. Terrigal, NSW: David Barlow Publishing.
National Quality Council, (2009a). A Code of Professional Practice for Assessment Validation and Moderation. Melbourne, VIC: TVET Australia.
National Quality Council, (2009b). Implementation Guide: Validation and Moderation. Melbourne, VIC: TVET Australia.
National Quality Council, (2009c). Guide for Developing Assessment Tools. Melbourne, VIC: TVET Australia.
Schofield, K., & McDonald, R., (2004). High Level Review of Training Packages: Working Paper 7: Supporting quality teaching, learning and assessment. Brisbane: ANTA.
Thomson, P., Saunders, J., & Foyster, J., (2001). Improving the validity of competency-based assessment. Adelaide: NCVER.
Tovey, M., & Lawlor, D., (2008). Training in Australia. (3rd Edition). Frenchs Forest, NSW: Pearson Education Australia.