Post on 14-Jun-2018
transcript
Localisation of Assessment:
Theory & Practice
Barry O’Sullivan
Validity & Validation
A Validation Model
Localisation in Theory
Localisation in Practice
1
Overview
2
Language Testing = Validity
the title that never was
3
A theory of validity is only of practical value if it is translatable into a coherent
theory or ‘model’ of validation which can then be operationalised through a set of
validation procedures. O’Sullivan & Weir, 2011
We cannot understand assessment if
we do not understand validity
4
THE SOCIO-COGNITIVE MODEL
focusing on the test-taker
5
PERFORMANCE Task Language
Target Language
Task Performance Parameters
Social Factors
Personal Characteristics
Cognitive Characteristics
TEST TASK
TEST TAKER
6
Machine or Human
Answer Key or Rating Scale
Selection & Training Consistency
Accuracy
Interpretation
Scoring Systems
7
Test Taker Test Task [including examiner]
Scoring
8
Weir (2005)
9
10
Weir (2005)
?
11
O’Sullivan (2011)
12
LOCALISATION - THEORY
refocusing on the test-taker
12
The process of ensuring that all predicted construct-irrelevant
variance related to the test taking population is controlled for
in the design and delivery of a test
13
Parameters related to the test population are known and fixed
Inferences or decisions are limited to a specific local context
Testing a specific population within a specific context in order to make
decisions that will apply specifically to that context
14
Neutral
Linguistically
Visually
Conceptually
No curriculum No local language No cultural references
No local references No cultural references
No complex cognitive No local references No cultural references
General Purpose Examinations
15
Localised Examinations
Test Population
Specific
Linguistically
Visually
Conceptually
Curriculum Local language Cultural references
Local references Cultural references
Possible complex cognitive Local references Cultural references
16
LOCALISATION - PRACTICE
the effect on the test
17
Exaver [Mexico] - 2005 - Word-lists
Task design
Aptis [BC - UK] - 2012 -
Test Concept Localised to:
Domain Culture
Age L1
18
Scoring
System
Appropriate Content
Appropriate Task
Appropriate Language
Appropriate Scoring
Appropriate Topic
Test
Taker
Test
Task
19
Case Study [India] Population 15 & 17 yrs Secondary School Similar Culture Similar Education Mixed ability Mixed gender Mixed L1
Localisation Language Reviewed [inc. instructions] Reading Topics Listening Topics Writing Topics/Tasks Speaking Topics Speaking Images
20
Speaking Task
Images
21
Cost Change means cost in terms of external input, training, quality assurance, systems development, delivery and monitoring
People Identification, training and nurturing of local expertise [assessment; technology; management; administration etc.]
Resource Technical – hardware and software + skills; financial; political/social
22
THE FUTURE
where to next?
23
Fuller theoretical definition
Broader understanding of how it works in practice
Operational checklists
Research on impact of localisation
23
barry.o’sullivan@britishcouncil.org