Post on 30-Oct-2014
description
transcript
Assessment of Higher Education Learning Outcomes
Hamish Coatescoatesh@acer.edu.au
acer.edu.au/highereducation
Rationales for change
Building assessment collaboration
s
Building capacity and improvement
Rationales for change
What is the “best” university in your country? How do
you know?
95%
65%
An experiment
Score distributions for eight groups (institutions, faculties, subjects,
etc.)
Perf
orm
an
ce
Minimum standard
“Standards”, not “standardisation”
Generalising the principles of good assessment will
encourage diversification and
excellence
Little data
Commitment to
effectiveness
UniversalElite Mass
Happiness data
Effectiveness data
Plan
Act
Evaluate
Improve
Hunch
Data for leadership
Enhanced management, evaluation and quality monitoring – to do more, better, with less
Higher education growing in significance and scale: clear rationales for increasing output and ensuring quality outcomes
Huge cost and competitive pressures – evidence-based management helps expand provision with quality
Internationalisation pervades all facets of teaching and learning, and graduate outcomes – international perspectives are vital
Shaping complexitie
s
Further reliance on flawed, simplistic rankings will constrain growth and prosperity – robust multidimensional perspectives required
Building assessment collaboration
s
AHELO at a
glance
Are valid and reliable cross-cultural/-institutional/-linguistic comparisons of HE learning outcomes scientifically and practically feasible?
Conceived ~2005, announced 06, scoping 06-09, work 10-13
An international feasibility study – not a ‘pilot’
For students near end of ‘bachelor degree’, also faculty, national managers, institutional coordinators, policymakers, expert advisors, etc.
Design, develop, validate and evaluate frameworks, tests, infrastructure and processes
HEI reports (not for systems), along with international reports and materials, and recommendations for a main study
>300 individuals involved directly
~270 HEIs
Up to 30,000 leaders, faculty, students
Many thousands of public stakeholders
~$30m total cost
‘Strands’ of testing
Contextual dimension
Generic Skills Economics Engineering
Student Context Instrument
Faculty Context Instrument
Institution Context Instrument
National Context Information
• Outcome specification
• Document analysis• Consultation• Synthesis, review
Framework creation
• Gather existing materials
• Item workshops• Technical review• Framework mapping• Adaptation,
translation• Verification
Item creation
• Qualitative testing
• Quantitative testing
• OperationalisationInstrument validation
Framework and item
development Tuning AHELO / QAA frameworks Curriculum documents Accreditation systems Discipline research
Authentic, hybrid item types ‘Above content’ reasoning Focus on ‘common core’
AHELO world map
Linguistically diverse…
Arabic Dutch English Finish Flemish Italian Japanese Korean Norwegian Russian Slovak Spanish
+ Abu Dhabi…
Building capacity and improvement
Change horizons
Assessment collaboration
s
Institutional
positioning
Faculty improvemen
t
Policy research
Learner feedback
Stakeholder
engagement
System monitoring
Building communities
Building awareness
Types ofsharing
Assessment collaboration
sBuild frameworks
Produce items
Select and use
Analyse and report
Definitions Tasks Processes Results
How assessment collaboratio
ns work
Pre
para
tion Establish
coordinating centreTranslate and validate instrumentsEngage institutionsTrain institution coordinatorsPrepare for testing
Assessm
en
t
Select studentsAdminister secure testScore responsesVerify and provide data
Rep
ort
ing
Prepare multilevel benchmarking reportsDistribute reports to institutionsInterpretation for monitoring and improvement
Assessment of Higher Education Learning Outcomes
Hamish Coatescoatesh@acer.edu.au
acer.edu.au/highereducation