Localisation of Assessment:

Theory & Practice

Barry O’Sullivan

Validity & Validation

A Validation Model

Localisation in Theory

Localisation in Practice

1

Overview

2

Language Testing = Validity

the title that never was

3

A theory of validity is only of practical value if it is translatable into a coherent

theory or ‘model’ of validation which can then be operationalised through a set of

validation procedures. O’Sullivan & Weir, 2011

We cannot understand assessment if

we do not understand validity

4

THE SOCIO-COGNITIVE MODEL

focusing on the test-taker

5

PERFORMANCE Task Language

Target Language

Task Performance Parameters

Social Factors

Personal Characteristics

Cognitive Characteristics

TEST TASK

TEST TAKER

6

Machine or Human

Answer Key or Rating Scale

Selection & Training Consistency

Accuracy

Interpretation

Scoring Systems

7

Test Taker Test Task [including examiner]

Scoring

8

Weir (2005)

9

10

Weir (2005)

?

11

O’Sullivan (2011)

12

LOCALISATION - THEORY

refocusing on the test-taker

12

The process of ensuring that all predicted construct-irrelevant

variance related to the test taking population is controlled for

in the design and delivery of a test

13

Parameters related to the test population are known and fixed

Inferences or decisions are limited to a specific local context

Testing a specific population within a specific context in order to make

decisions that will apply specifically to that context

14

Neutral

Linguistically

Visually

Conceptually

No curriculum No local language No cultural references

No local references No cultural references

No complex cognitive No local references No cultural references

General Purpose Examinations

15

Localised Examinations

Test Population

Specific

Linguistically

Visually

Conceptually

Curriculum Local language Cultural references

Local references Cultural references

Possible complex cognitive Local references Cultural references

16

LOCALISATION - PRACTICE

the effect on the test

17

Exaver [Mexico] - 2005 - Word-lists

Task design

Aptis [BC - UK] - 2012 -

Test Concept Localised to:

Domain Culture

Age L1

18

Scoring

System

Appropriate Content

Appropriate Task

Appropriate Language

Appropriate Scoring

Appropriate Topic

Test

Taker

Test

Task

19

Case Study [India] Population 15 & 17 yrs Secondary School Similar Culture Similar Education Mixed ability Mixed gender Mixed L1

Localisation Language Reviewed [inc. instructions] Reading Topics Listening Topics Writing Topics/Tasks Speaking Topics Speaking Images

20

Speaking Task

Images

21

Cost Change means cost in terms of external input, training, quality assurance, systems development, delivery and monitoring

People Identification, training and nurturing of local expertise [assessment; technology; management; administration etc.]

Resource Technical – hardware and software + skills; financial; political/social

22

THE FUTURE

where to next?

23

Fuller theoretical definition

Broader understanding of how it works in practice

Operational checklists

Research on impact of localisation

23

barry.o’sullivan@britishcouncil.org