Quality of assessment practices

Rob StowellLearning Australia

Background

In late 2008 the NQC commissioned Learning Australia to undertake a scoping study to: • review what constitutes and contributes to quality

assessment and how perceptions or understandings of this may have changed in the last five years from the point of view of key stakeholders, including industry.

• identify the key issues impacting on the integrity and quality of assessment outcomes.

• analyse the breadth of existing materials available to support the VET workforce in determining quality assessment outcomes.

• determine what the critical components of quality assessment are.

Structure of report

Two stage project that built on Investigations into industry expectations of vocational education and training assessment - approved by NQC in mid 2008.

Stage 1 report comprises two volumes:• Volume one describes the outcomes of the scoping study

on the quality of assessment practices. • Volume two provides a listing of assessment support

materials Stage 2 report provides:

• review of assessment materials identified in stage 1.• a strategy for addressing the gaps, and establishes a

plan for the dissemination and implementation of any new, updated or existing materials for supporting quality assessment.

Copies of the reports

http://www.nqc.tvetaustralia.com.au/nqc_publications

Quality assessment

Clear and well understood benchmarks for assessment.

Principles of validity, reliability, fairness and flexibility.

Assessment decisions based on evidence which comply with the rules of evidence.

Quality assured - the AQTF2007 Essential Standards for Registration.

An integral part of learning and training process and the outcomes of assessment are actually used to inform the development of programs, modify training delivery to meet the needs of learners and improve learning. In quality assessment practices, evidence and feedback are used to identify where learners are in their learning, what they need to do and how best to achieve this.

Quality judgements.

Key recommendations in scoping

study Clear benchmarks for assessment Status and capability of assessors Improve assessment quality assurance by supporting the

effective implementation of the current AQTF standards High quality, industry endorsed model assessment support

materials Build assessment expertise within the VET sector Improve capability of assessors to make quality assessment

judgements Ensure that assessment is an integral part of the learning

and training process.

Changing emphasis in assessment

NowPast Future

Outcomes focused

Learner centered

Producing learners

Competence

Authenticity

Developing capacity to

make judgements

Learning and assessment skills beyond

formal training

Knowledge focused

Teacher centered

Testing students

Educational measurement

Activity - Complete the cartoon

Assessment and learning

Assessment may be seen as a summative process which has little or no connection to the processes of training and teaching.

Assessment evidence used for certification purposes and may not be used to identify learners’ achievements, diagnose their strengths and weaknesses and inform the design of learning plans that assist learners to achieve competency.

Summative or ‘assessment of learning’ is important but there is increasing interest in ‘assessment for learning’ or and its role in promoting learner achievement.

Assessment for learning

Assessment for learning focuses on the gap between where learners are in their learning, and where they need to be – the desired goal ie: competent performance. It is based on the idea that learners will improve most if they understand:

the competency to be achieved, where they are in relation to the required level of

performance how they can achieve the required level of

performance.

Refining formative assessment

Paul Black and Dylan Wiliam, Inside the Black Box - 1998 Refined the term ‘formative assessment’ by emphasising

that assessment is only formative when: it is an integral part of the learning and teaching

process; and assessment evidence is actually used to modify

teaching to meet the needs of learners and improve learning.

The Black Box: findings

Black and Wiliam’s research indicates that improving learning through assessment depends on five deceptively simple factors:

1. Providing effective feedback to students.

2. Learners’ active involvement in their own learning.

3. Adjusting teaching to take account of the results of assessment.

4. Recognising the profound influence of assessment on learners’ motivation and self-esteem - both crucial influences on learning.

5. Ensuring learners assess themselves and understand how to improve.

Performance range in control

groups

Range of typical performance enhancements in ‘assessment for learning’ groups

Performance range in Assessment for Learning

groups

Source: Black and Wiliam (1998)

Average effect size: 25% shift in performance compared to control groups after 2.5 years, and a reduced ‘spread’ in the performance range.

Low

Low

High

High

Evidence – Black and Wiliam

Research on assessment for learning in VET

Lack of a research base Policy and practice in formative assessment driven by

concerns about learner motivation, engagement and inclusion.

Connection between formative assessment and learner performance not adequately explored in VET.

Some recent work • Davies and Ecclestone 2008• Boud UTS – www. assessment futures.com.

Assessment for learning principles

Assessment for learning is part of effective planning for VET delivery focuses on how learners’ learn is central to training delivery involves high quality interactions between VET trainers and

learners is a key professional skill of Vet practitioners takes account of the importance of learner motivation assumes that learners understand the criteria on which they are

assessed assumes that learners receive constructive guidance about how

to improve develops learners’ capacity for self assessment so that they

become reflective and self managing recognises the full range of achievements of all learners

Five key strategies …

Sharing LearningExpectations

Questioning

Clarifying and sharing learning intentions and criteria for success

Engineering effective discussion, questions, and learning tasks

Feedback Moving learners forward with feedback

Self AssessmentActivating learners as the owners of their own learning

Peer AssessmentActivating learners as instructional resources for one another

…and one big idea.

Use evidence about learning to adapt instruction to meet learner needs.

Find x.

3 c

ms

4 cms

x

So, how do you do it?

The procedure is actually quite simple. First you arrange things into different groups. Of course, one pile may be sufficient, depending on how much there is to do.If you have to go somewhere else due to lack of facilities, that is the next step.Otherwise you are pretty well set. It is important not to overdo things – that is, it is better to do too few things at once than too many.In the short run, this might not seem important, but complications can easily arise. A mistake can be expensive as well.After the procedure is completed, one arranges the materials into different groups again. Then they can be put into their appropriate places.Eventually they’ll be used once more and the whole cycle will have to berepeated. However, that is a part of life.So, how do you do it?

Washing clothes

What do these words in the text refer to?

Different groups Somewhere else Facilities Complications Mistake Appropriate places

Key question

What do Pythagoras and clothes washing have to tell us?

Skills of VET practitioners

Provide learners with constructive guidance about how to improve. This may involve:

Ensuring that learners know what competent performance is;

Pinpointing learners’ strengths and advising on how to develop them;

Being clear and constructive about any weaknesses and how they might be addressed; and

Providing opportunities for learners to improve upon their work.

Develop learners' capacity for self-assessment so that become reflective and self-managing learners.

Obtain information about learner progress and use it in planning at individual, group and program levels.

Design and implement learning activities that promote high quality interactions with learners that feature thoughtful questions, careful listening and reflective responses.

Plan assessment, observe learning, analyse and interpret evidence of learning, give feedback to learners and support learners in self-assessment.

Negotiate and manage flexible assessment options including self assessment, peer assessment and collaborative assessment.

Ensure that learners understand the criteria on which they are assessed so that they know what it is they are trying to achieve.

Activity – Rate VET Practitioner skills

Powerful in promoting quality learning and assessment

Difficult to implement

Easy to implement

Not so powerful in promoting quality learning and assessment

1 2

3 4

Implementing assessment for learning VET practitioners ‘know’ most of this already So the problem is not a lack of knowledge The problem is how to do assessment for learning Telling VET practitioners what to do doesn’t work - there

is a need for action research to develop practitioners skills to implement assessment for learning

Experience alone is not enough - if it were, then the most experienced teachers would be the best teachers - we know that’s not true (Hanushek, 2005)

People need to reflect on their experiences in systematic ways that build their accessible knowledge base, learn from mistakes, etc. (Bransford, Brown & Cocking, 1999)

Further information

Publications:Inside the Black Box, Raising Standards Through Classroom Assessment (Paul Black and Dylan Wiliam, 1998)

Straitjacket or springboard for sustainable learning? The implications of formative assessment practices in vocational learning cultures in The Curriculum Journal, Volume 19, No. 2 June, 2008

Quality Assurance Agency for Higher Education, Integrative Assessment : Balancing assessment of and assessment for learning, QAA 2007

Formative Assessment in Action: Weaving the Elements Together  (Shirley Clarke, 2005)

Websites:www.aaia.org.uk

www.assessmentfutures.com

www.qca.org.uk

www.slamnet.org.uk

http://arg.educ.cam.ac.uk

http://cms.curriculum.edu.au/assessment/

http://www.assessment-reform-group.org/

http://www.tki.org.nz/r/assessment/atol_online/tutorials_e.php