44
TVET Australia Assessment, validation and moderation A power point presentation developed by the NQC to support information sessions on assessment, validation and moderation
TVET Australia
Assessment, validation and moderation
A power point presentation developed by the NQC to support information sessions on assessment, validation and moderation
Contact
NQC Secretariat TVET AustraliaLevel 22/ 390 St Kilda Road Melbourne Vic 3004Telephone: +61 3 9832 8100Email: [email protected] Web: www.nqc.tvetaustralia.com.au
Disclaimer
This work has been produced on behalf of the National Quality Council with funding provided through the Australian Government Department of Education, Employment and Workplace Relations and state and territory governments. The views expressed in this work are not necessarily those of the Australian Government or state and territory governments
Acknowledgement
This presentation was designed to support the interactive information sessions that formed part of the NQC’s communication and dissemination strategy: NQC products: validation and communication. Reports and materials which focus on validation and moderation may be downloaded from the NQC website athttp:www.nqwc.tvetaustralia.com.au/nqc_publications
This work was produced for the National Quality Council by
Andrea Bateman, Quorum QA Australia Pty LtdChloe Dyson, Quorum QA Australia Pty Ltd
Quality of assessment
Setting the Scene
Concerns about the quality of assessments and comparability of standards across the VET sector
OECD (2008) Reviews of VET (Australia) NQC (2008) Industry Expectations of VET Service Skills SA (2010) VETiS Project
Today’s workshop
Assessment Developing Assessment Tools Competency Mapping Simulated Assessment
Engaging industry Assessment Quality Management Framework
Validation and moderation System considerations Diverse settings
NQC resources
• Guide for Developing Assessment Tools• Assessment Facts Sheets
• Simulated Assessment• Making Assessment Decisions• Peer Assessment and Feedback• Quality Assuring Assessment Tools• Assessor Partnerships• Systematic validation
• Assessor Guide: Validation and Moderation
Session 1: What is assessment?
Purposeful process of systematically gathering, interpreting, recording and communicating to stakeholders, information on student performance.
Assessment Purposes
Evaluative designed to provide information to evaluate
institutions and curriculum/standards – primary purpose is accountability
Diagnostic Produce information about the candidate’s
learning Formative
Produce evidence concerning how and where improvements in learning and competency acquisition are required
Summative Used to certify or recognise candidate
achievement or potential
Assessment Purposes Assessment for learning occurs when teachers use
inferences about student progress to inform their teaching (formative)
Assessment as learning occurs when students reflect on and monitor their progress to inform their future learning goals (formative)
Assessment of learning occurs when teachers use evidence of student learning to make judgements on student achievement against goals and standards (summative)
http://www.education.vic.gov.au/studentlearning/assessment/preptoyear10/default.htm
Session 2: Developing Assessment Tools
NQC Products• Guide for Developing Assessment Tools• Assessment Facts Sheets
• Simulated Assessment• Making Assessment Decisions• Peer Assessment and Feedback• Quality Assuring Assessment Tools
• Assessor Guide: Validation and Moderation
http://www.nqc.tvetaustralia.com.au/nqc_publications
ReliabilityValidity
Assessment toolValidation Moderation
Impact
Changes to definitions within the NQC publications AQTF 2010 User Guide documentation; and the Training Package Development Handbook
Key Stages – developing assessment toolsKey Stages – developing assessment tools
identify and describe the purposes for the assessment
identify the assessment information that can be used as evidence of competence/learning
identify a range of possible methods that might be used to collect assessment information
define the contexts for interpreting assessment information in ways that are meaningful for both assessor and candidate
determine the decision making rules
define procedures for coding and recording assessment information
identify stakeholders in the assessment and define their reporting needs.
Essential Characteristics - Assessment Tool
An assessment tool includes the following components:
The context and conditions for the assessment The tasks to be administered to the candidate An outline of the evidence to be gathered from the
candidate The evidence criteria used to judge the quality of
performance (i.e., the assessment decision making rules); as well as the
The administration, recording and reporting requirements.
Ideal Characteristics
The context Competency mapping The information to be provided to the candidate The evidence to be collected from the candidate Decision making rules Range and conditions Materials/resources required Assessor intervention Reasonable adjustments Validity evidence Reliability evidence Recording requirements Reporting Requirements
Competency Mapping
The components of the Unit(s) of Competency that the tool should cover should be described. This could be as simple as a mapping exercise between the components within a task (eg each structured interview question) and components within a Unit or cluster of Units of Competency. The mapping will help determine the sufficiency of the evidence to be collected as well as the content validity.
Advice regarding competency mapping can be found in the NQC Assessor Guide: Validation and Moderation
Competency Mapping
Component of Unit(s) of Competency BSBITU201A Produce simple word processed documents
Methods Elements Required Skill and Knowledge
Evidence Guide
1 Practical demonstration Elements 1.1 RS3, RK3
2 Third party report Elements 1 – 3 RS1-5, RK 1-4 CA1, CA2
3 Workplace documents Elements 1-3 especially 1.1, 2, 3
RS1-5, RK 1-4 CA2
Workplace Project: 14 tasks to be performed by the candidate
TAEASS401A Plan Assessment Activities and Processes
1.
Specify
2.
Inte
gra
te
3.
chara
cte
ristic
s
4.
Purp
ose
5.
info
rmatio
n
6.
meth
ods
7.
tools
8.
Reso
urc
es
9.
Role
s
10.
Appeals
11.
Tim
elin
es
12.
Reaso
nable
adju
stm
ent
13.
feedback
14.
Record
ing
Elements
Determine Assessment Approach
Prepare the Assessment Plan
Develop Assessment Instruments
Required skills
Cognitive Interpersonal skills
Research and evaluation skills
Communication skills
Interpersonal skills
Competency Mapping: Steps in the process
Step 1: Unpack the unit of competency to identify its critical components.
Step 2: For each assessment method, list the tasks to be performed by the candidate.
Step 3: For each assessment method, map the critical components of the unit to each assessment task.
Refer to NQC Assessor Guide: Validation and Moderation
Level of specificity in mapping – Risk Assessment
Risk can be determined by consideration of: Safety (eg potential danger to clients from
an incorrect judgement) Purpose and use of the outcomes (eg
selection purposes) Human capacity (eg level of expertise and
experience of the assessors) Contextual (eg changes in technology,
workplace processes, legislation, licensing requirements and/or training packages)
Decision Making Rules
The rules to be used to: Check the quality of the evidence (i.e.
the rules of evidence) Judge how well the candidate
performed on the task according to the standard expected
Synthesise evidence from multiple sources to make an overall judgement
Additional advice – refer to Fact Sheet 2
Reasonable Adjustments
This section of the assessment tool should describe the guidelines for making reasonable adjustments to the way in which evidence of performance is gathered without altering the expected performance standards (as outlined in the decision making rules).
Simulated assessment
For the purposes of assessment, a simulated workplace is one in which all of the required skills are performed with respect to the provision of paid services to an employer or the public can be demonstrated as though the business was actually operating.
In order to be valid and reliable, the simulation must closely resemble what occurs in a real work environment.
The simulated workplace should involve a range of activities that reflect real work experience. It should allow the performance of all of the required skills and demonstration of the required knowledge.
Ref: AQTF definition (refer to Activity Handout), Assessment Fact Sheet 1
Activity 1: Engaging Industry
In your groups discuss what input employers (you might wish to specify a vocational area) could provide to develop valid assessment tools and processes.
For the following scenarios, note down 2/3
questions you could ask employers and how the responses will inform the development or review of assessment tools and/or processes.Relevant NQC support materials:
•Industry Enterprise & RTO Partnership•Assessment Fact Sheets: Assessor Partnerships•Assessor Guide: Validation and Moderation
Activity 2: Self Assessment
In groups of 3, review the assessment tool using the self assessment checklist from the NQC (2009) Implementation Guide (Template A.1, p. 45).
Identify any gaps in the tool?
Discuss the pros and cons of including such additional information within the tool?
Tool Review
Has clear, documented evidence of the procedures for collecting, synthesising, judging and recording outcomes (i.e., to help improve the consistency of assessments across assessors [inter-rater reliability]).
Has evidence of content validity (i.e., whether the assessment task(s) as a whole, represents the full range of knowledge and skills specified within the Unit(s) of competency.
Reflect work-based contexts, specific enterprise language and job-tasks and meets industry requirements (i.e., face validity).
Adheres to the literacy and numeracy requirements of the Unit(s) of Competency (construct validity).
Has been designed to assess a variety of evidence over time and contexts (predictive validity).
Has been designed to minimise the influence of extraneous factors (i.e., factors that are not related to the unit of competency) on candidate performance (construct validity).
Tool Review
Has clear decision making rules to ensure consistency of judgements across assessors (inter-rater reliability) as well as consistency of judgements within an assessor (intra-rater reliability).
Has a clear instruction on how to synthesise multiple sources of evidence to make an overall judgement of performance (inter-rater reliability).
Has evidence that the principles of fairness and flexibility have been adhered to.
Has been designed to produce sufficient, current and authentic evidence.
Is appropriate in terms of the level of difficulty of the task(s) to be performed in relation to the skills and knowledge specified within the relevant unit(s) of Competency.
Has outlined appropriate reasonable adjustments that could be made to the gathering of assessment evidence for specific individuals and/or groups.
Has adhered to the relevant organisation assessment policy.
Quality Checks
Panel Pilot Trial
Refer to Fact Sheet 4, Quality assuring assessment tools
Session 3: Assessment Quality Management
NQC Products• Code of Professional Practice: Validation &
Moderation• Implementation Guide: Validation and Moderation• Assessment Facts Sheets
• Quality Assuring Assessment Tools• Systematic Validation• Assessor Partnerships
• Assessor Guide: Validation and Moderation
http://www.nqc.tvetaustralia.com.au/nqc_publications
Validation
Validation is a quality review process. It involves checking that the assessment tool produced valid, reliable, sufficient, current and authentic evidence to enable reasonable judgements to be made as to whether the requirements of the relevant aspects of the Training Package or accredited course had been met. It includes reviewing and making recommendations for future improvements to the assessment tool, process and/or outcomes.
NQC Implementation Guide: Validation and Moderation 2009
Outcomes of validation
Recommendations for future improvements
Context and conditions for the assessment Task/s to be administered to the candidates Administration instructions Criteria used for judging the quality of performance
(e.g. the decision making rules, evidence requirements etc)
Guidelines for making reasonable adjustments to the way in which the evidence of performance was gathered to ensure that the expected standard of performance specified within the Unit(s) of Competency has not been altered
Recording and reporting requirements.
Moderation
Moderation is the process of bringing assessment judgements and standards into alignment. It is a process that ensures the same standards are applied to all assessment results within the same Unit(s) of Competency. It is an active process in the sense that adjustments to assessor judgements are made to overcome differences in the difficulty of the tool and/or the severity of judgements.
NQC Implementation Guide: Validation and Moderation 2009
Outcomes of moderation
Recommendations for future improvement and adjustments to assessor judgements (if required) and
Recommendations for improvement to the assessment tools
Adjusting the results of a specific cohort of candidates prior to the finalisation of results and
Requesting copies of final candidate assessment results in accordance with recommended actions.
Validation vs Moderation
Features Validation Moderation
Assessment QualityManagement Type
Quality Review Quality Control
Primary Purpose Continuous improvement Bring judgements and standards into alignment.
Timing On-going Prior to the finalisation of candidate results
Focus Assessment tools; andCandidate Evidence (includingassessor judgements) (desirable only)
Assessment tools, and; Candidate Evidence, including assessor judgements (mandatory)
Type of Approaches
Assessor PartnershipsConsensus MeetingsExternal (validators or panels)
Consensus MeetingsExternal (moderators or panels)Statistical
Outcomes Recommendations for futureimprovements
Recommendations for improvements; andAdjustments to assessor judgements (if required)
Types of Approaches - Statistical
Limited to moderation
Yet to be pursued at the national level in VET
Requires some form of common assessment task at the national level
Adjusts level and spread of RTO based assessments to match the level and spread of the same candidates scores on a common assessment task
Maintains RTO-based rank ordering but brings the distribution of scores across groups of candidates into alignment
Strength
Strongest form of quality control
Weakness
Lacks face validity, may have limited content validity
Types of Approaches - External
Types
Site Visit Versus
Central Agency
Strengths
Offer authoritative interpretations of standards
Improve consistency of standards across locations by identifying local bias and/or misconceptions (if any)
Educative
Weakness
Expensive
Less control than statistical
Types of Approaches – Assessor Partnerships
Validation only
Informal, self-managed, collegial
Small group of assessors
May involve:
Sharing, discussing and/or reviewing one another’s tools and/or judgements
Benefit
Low costs, personally empowering, non-threatening
May be easily organised
Weakness
Potential to reinforce misconceptions and mistakes
Ref: Implementation Guide, Assessment Fact Sheet 5
Types of Approaches - Consensus
Typically involves reviewing their own & colleagues assessment tools and judgements as a group
Can occur within and/or across organisations
Strength Professional development, networking, promotes
collegiality and sharing
Weakness Less quality control than external and statistical
approaches as they can also be influenced by local values and expectations
Requires a culture of sharing
Systematic Validation (consensus)
Indicators Yes/No
Action
Is there a plan for assessment validation (including validation of RPL assessment) in place?
Does your plan:•Determine the sample of units of competency to be validated over a set period of time•Provide dates for proposed validation activities•Include details about who will participate in assessment validation, including the Chair of consensus panels, if relevant•Include a strategy to ensure that all relevant staff are involved•Identify what processes and materials will be used for implementing and recording the outcomes of assessment validation
Does your RTO have terms of reference in place to guide the work of consensus panels?Does your RTO have validation materials (policy, procedure, forms) in place that cause participants to engage effectively in validation?Does your RTO have a process for monitoring the action taken as a result of validation?Does your RTO have a process and plan in place for reviewing the effectiveness of assessment validation? Ref: Assessor
Guide
System considerations
What is the most appropriate approach to validation?
Condition Suggested approachWhenever my RTO conducts internal validation few opportunities for improvement arise
Consider including external representation on your validation panel
Our assessors are contractors and cannot come to validation consensus meetings because my RTO can’t afford to pay for their time and some are located interstate
Consider establishing assessor validation partnerships at your local level, but ensure that improvements identified are recorded and fed back to other assessors and formalised
Our RTO conducts high risk units related to licensing, where the licensing authority has mandated the use of assessment tools it provides
Consider consensus moderation, ideally with external representation on your panel.
Our RTO is new and assessors do not have a lot of experience
Consider inviting an external person with expertise in assessment tool design to validation consensus meetings
Quality Assurance (Input approach)
Quality Control (Outcome approach)
Quality Review (Retrospective approach)
Examples include: Industry competency standards
as the benchmarks for assessment
National assessment principles
Minimum qualifications for
assessors (i.e., TAE40110)
Development of a Professional Code of Practice
Standardisation of reporting
formats
Assessment Guidelines and Policy Documents
Benchmark examples of varying
levels of performances
Assessment tool banks
Common assessment tasks
Exemplar assessment tools
Panelling, Piloting and/or Trialling of assessment tools.
Professional development
programs/workshops for assessors
Examples include: Moderation in which adjustments to
assessor judgements are made to overcome differences in the difficulty of the assessment tool and/or severity of the judgement.
Examples Include: Monitoring and auditing of registered
training organisations Review and validation of assessment
tools, processes and outcomes to identify future improvements.
Follow-up surveys with key
stakeholders (e.g., student destination surveys, employer feedback on how well the assessment outcomes predicted workplace performance).
Assessment Quality Management
Quality management in diverse settings
Identified barriers: Structural (i.e., the organizational and resource
aspects) – financial, variations of definitions across key documents
Process (i.e., the practices and activities that take place) – rolling enrolments, partnering arrangements, workloads
Personal factors (i.e., the attitudinal, assessment literacy and expectations of the key players).
Strategies deployed by RTOsRefer to Handout – Quality management processes
in diverse settings.
Activity 3: Assessment Quality Management
Activity 4: Assessment Quality ManApproach Strategies
implemented
Quality assurance
Quality control
Quality management
Ref: Implementation Guide
Principal authorAssociate Professor Shelley Gillis
Deputy DirectorWork-based Education Research Centre
Victoria University
Email: [email protected]: 0432 756 638
Andrea BatemanDirector
Education ConsultantQuorum QA Australia Pty Ltd
Email: [email protected]
Phone: 0418 585 754
Chloe DysonDirector
Education ConsultantQuorum QA Australia Pty Ltd
Email: [email protected]: 0408124825
