Assessing higher education learning outcomes globally

Professor Hamish Coateshamishc@unimelb.edu.au

Study at a glance

Are valid and reliable comparisons of learning outcomes scientifically and practically feasible?

Conceived ~2005, announced 06, scoping 06-09, work 10-13

An international feasibility study – not a ‘pilot’

Testing 23000 ‘bachelor degree’ students, with data from 5000 faculty, 25 national managers, 250 institutional coordinators, 17 policymakers, and hundreds of advisors and stakeholders

Designed, developed, validated and evaluated frameworks, tests, infrastructure and processes, etc.

HEI reports (not for systems), along with international reports and materials, and recommendations for a main study

Linguistically diverse…

Arabic Dutch EnglishFinishFlemish Italian Japanese

KoreanNorwegian Russian SlovakSpanish

Is it feasible to develop frameworks and instruments to test discipline-specific learning outcomes?

Test fields: Economics, Engineering

Contextual Dimension

Generic Skills Economics Engineering

Student

Faculty

Institution

National

Instrument architecture

• Outcome specification

• Document analysis• Consultation• Synthesis, review

Framework creation

• Gather existing materials

• Item workshops• Technical review• Framework mapping• Adaptation,

translation• Verification

Item creation

• Qualitative testing

• Quantitative testing

• OperationalisationInstrument validation

Framework and item

development Tuning AHELO / QAA frameworks Curriculum documents Accreditation systems Discipline research

Authentic, hybrid item types ‘Above content’ reasoning

Source version

Translation 1

Translation 2

Final check

National review

Reconciliation

Verification: verifier (linguist) and domain

specialist

Designed to maintain cross-national comparability of assessment materials

A holistic, robust and flexible approach, linked with item production and validation

Adaptations managed as a continuous processNative speakers of target languagetrained to detect specific pitfalls

Economists or engineers who are speakers of target language

Translation, adaptation

and verification

Psychometric reviews

3

6

14

8

17

19

19

6

22

3

14

3

8

25

22

19

22

28

14

22

28

33

6

11

53

36

31

36

25

36

42

28

22

44

31

42

33

28

31

33

33

17

25

39

28

17

40

39

0% 20% 40% 60% 80% 100%

The task made me apply knowledge and skills in real-world ways

The task required me to apply capability gained in my program

The task challenged me to think

The assessment materials were easy to understand

The task was relevant to future professional practice

The materials stimulated my interest in the task

There was good linkage between the questions in each task

The task was relevant to my program of study

The task assessed an appropriate range of knowledge and skills

The task was relevant to the content being assessed

The time available was sufficient for me to complete this task

The task covered topics relevant to my program

Per cent

Strongly disagree

Disagree

Undecided

Agree

Strongly agree

Student feedback for a task

Project implementati

on

Leadership: International project management and supporting national teams

Operationalisation: Preparing tests and context instruments for secure online delivery, and training makers

Sampling: Engaging institutions, sampling faculty and students, and quality assurance

Assessment: Supporting national training, managing testing, and managing marking quality

Reporting: Compiling data products, psychometric and statistical analysis, and system, institution and stakeholder reports

Evaluating: Scientific and practical feasibility, recommendations for full-scale study

0

10

20

30

40

50

60

70

80

90

100

50 150 250 350 450 550 650 750 850 950

Stud

ents

(cum

ulat

ive

per c

ent)

Generic Skills Score

This institution Mean of all institutions All institutions

Country communicatio

ns

Defined engagement cycle established to support systems, institutions and students… but not faculty

Designed sustainable business models for such work… but other models

Assessment frameworks and test instruments developed to support multidimensional test/context instrumentation

Established methods for test design, development, translation/adaptation and validation

Defined operational workflow and quality control procedures required to support global testing

Forming awareness of how such work is positioned globally

Broad insights

Change horizons

AHELO

Institutional

positioning

Faculty improvemen

t

Policy research

Learner feedback

Stakeholder

engagement

System monitoring

Assessing higher education learning outcomes globally

Professor Hamish Coateshamishc@unimelb.edu.au