25
Developing the APTIS Testing System Professor Barry O’Sullivan Senior Advisor, English Language Assessment
Developing the APTIS Testing System
Professor Barry O’SullivanSenior Advisor, English Language Assessment
Overview
Localisation
Developing theories
Computerisation
Conclusions
DEVELOPING THEORIESconceptualising the test
System Design
Thinking about Test Performance
PERFORMANCETask Language
Target Language
Task Performance Parameters
Social Factors
Personal Characteristics
Cognitive Characteristics
Then There’s Scoring…
Machine or Human
Answer Key or Rating Scale
Selection & Training Consistency
Accuracy
Interpretation
Validation Theory
Decisions, Decisions…
Why are we testing?
How Many Decisions?
TEST SCORE
Task Language•Vocabulary &
Grammar•Genre, Mode, etc.
Target Language•Vocabulary &
Grammar•Function, Mode etc.
Task Performance Parameters•Time & Support•Knowing Assessment
Criteria•Physical Conditions, etc.
Social Factors•Interlocutor [who;
number]•Relative Age, gender
etc.•Familiarity & Status•Etc.
Personal Characteristics•Physical•Psychological•Experiential
Cognitive Characteristics•Maturity•Ability•Etc.
Scoring System•Raters & Rating•Scales & Keys•Analysis & Grading•Etc.
Why all the Fuss?
Curriculumexplicitimplicit
Deliveryroom design
materialsteachers
Assessmentjudgemental
developmental
Decisions
FOCUS ON READINGtheories of language progression
The Reading Model
Word/Sentence
Paragraph
Text [within]
Text [external]
Inter Text [full text level]
Inter Text [paragraph level]
A
B
C
How it Works
Introducing
CORE[Grammar & Vocabulary]
Speaking
Writing
Listening
Reading
LOCALISATIONthe british council approach
Theory to Practice – Test Design
Scoring System
Appropriate Content
Appropriate Task
Appropriate Language
Appropriate Scoring
Appropriate Topic
Test Taker
Test Task
LOCALISATION
Defining Localisation
The process of ensuring that all construct-irrelevant
variance related to the test taking population is
controlled for in the design and delivery of a test
Testing a well defined population within a well defined context in order to make decisions that will apply only to that context
Localisation – Case Study
Curriculum
Delivery Assessment
Case Study - Approach
Context
Population 15 & 17 yrsSecondary SchoolMixed ability, GenderVarious L1 [multilingual]Similar CultureSimilar CurriculumSimilar Delivery
Localisation
Language Reviewed[inc. instructions]
Reading TopicsListening TopicsWriting Topics/TasksSpeaking TopicsSpeaking Images
COMPUTERISATIONthe growing importance of the machine
The Computer & Test Practice
Item DevelopmentSpecificationWritingTriallingBankingArchiving
Test ScoringRater TrainingTest Break UpScoringAnalysing/AwardingScore Reporting
Test TakingKnowledgeReceptive SkillsProductive Skills
ResearchLanguageData AnalysisTask PerformanceRating Performance
System ManagementBack OfficeReporting to Clients
Test DeliveryComputerTabletOther…
The Computer & Test Practice
Item DevelopmentSpecificationWritingTriallingBankingArchiving
CONCLUSIONSwhat’s the point?
The Message?
Every decision impacts on ALL stakeholders, though especially on the test taker
Without a clearly defined ABILITY MODEL the test is suspect
Without a clearly defined VALIDATION MODEL the test is suspect
It is close to impossible to deliver any test without technology
