Post on 22-Feb-2018
transcript
© UCLES 2013
Cambridge English Language Assessment: Principles of good practice for test development and quality management
Dr Angeliki Salamoura, Research and Validation GroupGraeme Harrison, Head of Assessment Services, Southern Cone & Andes
© UCLES 2013
Overview• Cambridge English Language Assessment:
who we are & what we offer• Principles of test development• Quality management:
Question Paper Production process• Cambridge English: Advanced (CAE) • Teaching Knowledge Test (TKT)
© UCLES 2013
Cambridge English Language
Assessment
University of Cambridge
Cambridge University
Press
Cambridge English
A not-for-profit university department
© UCLES 2013
Language Policy & strategy
Analysis & diagnostics
Curriculum reform
Materials & resources
Teacher Development
Assessment &
certification
Programme implementation
& evaluation
Cambridge English
educationalvalue chain
© UCLES 2013
Language Policy & strategy
Analysis & diagnostics
Curriculum reform
Materials & resources
Teacher Development
Assessment &
certification
Programme implementation
& evaluation
Cambridge English
educationalvalue chain
© UCLES 2013
English language
exams since 1913 One
of the largest dedicated research
teams
4.5 millioncandidates
per yearAccepted by
12,500organisations
globally
2,700 test centres in
130 countries
Over 50,000preparation
centres worldwide
Assessment and Certification
© UCLES 2013
© UCLES 2013
© UCLES 2013
© UCLES 2013
1913: The first Certificate of Proficiency in English (CPE)
The 1913 UCLES Regulations:
‘The Certificate of Proficiency in English is designed for Foreign Students who desire a satisfactory proof of their knowledge of the language with a view to teaching it in foreign schools.’
© UCLES 2013
1913: The first CPE
Candidates: 3 Exam centres: 2 Fee: £3 (≈ 2,500 CLP) Length (time):
© UCLES 2013
1913: The first CPE
Candidates: 3 Exam centres: 2 Fee: £3 (≈ 2,500 CLP) Length (time): 12 hours!
© UCLES 2013
1913: The first CPE
(ii) Oral: (f) Dictation: ½ hour.(g) Reading aloud and Conversation: ½ hour
© UCLES 2013
1913: The first CPE
(ii) Oral: (f) Dictation: ½ hour.(g) Reading aloud and Conversation: ½ hour
Pass rate: ?
© UCLES 2013
1913: The first CPE
(ii) Oral: (f) Dictation: ½ hour.(g) Reading aloud and Conversation: ½ hour
Pass rate: 0%
© UCLES 2013
1913: The first CPE
The 1913 examiner report on Phonetics:
‘None of the candidates [had] given the subject sufficient attention. The transcriptions were distinctly poor and suggested that the candidates had had but little practice…’
© UCLES 2013
Cambridge English exams and the CEFRCPEC2
1913
FCEB2
1939
CAEC1
1991
PETB1
1981
KETA2
1988
FlyersC2Movers
StartersBeg-A2
Effective Operational Proficiency
level
Council of EuropeVantage
level
Council of Europe
Threshold level
Council of Europe
Waystage level
Mastery level
Breakthrough Level
© UCLES 2013
Maintaining standards over time -an item banking approach using IRT
Probabilistic models:Estimating the probability that a candidate of known ability will
succeed on an item of known difficulty
Item Response models and Rasch analysis provide the statistical approach for building a measurement scale and doing item banking
© UCLES 2013
TEST
A test is a specific (and necessarily limited) sample of performance, from which users seek to make broader generalisations about the knowledge, skills, or abilities a person has.
© UCLES 2013
Test qualities…
Validity Reliability
Impact Practicality
© UCLES 2013
Balancing test qualities…
Validity Reliability
Impact Practicality
© UCLES 2013
The longer the test…
…the more it is.
…the less it is.
valid reliable practical
© UCLES 2013
A Fine Balance
© UCLES 2013
Multiple Choice Test
© UCLES 2013
Short Response
© UCLES 2013
Speaking Test
© UCLES 2013
Speaking Test
Describe and explain why…• Evaluate performance on a scale
Repeat the phrase you hear in the recording
• Accurately repeated = 1• Not accurately repeated = 0
© UCLES 2013
Balancing test qualities…
Validity Reliability
Impact Practicality
© UCLES 2013
How do we ensure reliability?
© UCLES 2013
Speaking and WritingRecruitment
Certification
Training
Standardisation &
RecertificationMonitoring
Examiner Administration
© UCLES 2013
How do we ensure test validity &
fitness for purpose?
© UCLES 2013
Socio-cognitive framework for test validation(Weir 2005)
© UCLES 2013
Socio-cognitive framework for test validation(Weir 2005)
How are the characteristics of
test takers catered for by this exam?
© UCLES 2013
Socio-cognitive framework for test validation(Weir 2005)
Are the characteristics of the test tasks and the test administration fair to the test takers?
© UCLES 2013
Socio-cognitive framework for test validation(Weir 2005)
Are the cognitive processes required to complete the
task appropriate?
© UCLES 2013
Socio-cognitive framework for test validation(Weir 2005)
How far can we depend on the scores of the test?
© UCLES 2013
Socio-cognitive framework for test validation(Weir 2005)
What effects does the test have on its various stakeholders?
© UCLES 2013
Socio-cognitive framework for test validation(Weir 2005)
What external evidence is there outside of the test
scores themselves that the test is doing a good job?
© UCLES 2013
Does the test measure the right things?
Research Notes:quarterly online publication
Studies in Language Testing:Examining Writing (2007)Examining Reading (2009) Examining Speaking (2011) Examining Listening (2013)
© UCLES 2013
© UCLES 2013
External Specialists
•Assessment Managers•Assessment Administrators•Operations Administrators •Validation Officers•Clerical Markers
• Chair (writing team leader)
• Item Writing Team• Content Vetters• Proofreaders• Examiners
Cambridge English Language Assessment
Stakeholders•Pretest centres•Pretest candidates
© UCLES 2013
The Recipe for Question Paper Production
Rejection
Editing
Pre-editing
Commissioning of material
Revision
© UCLES 2013
Pre-editing and Editing – the cake mix
ParticipantsAssessment ManagerChair (writing team
leader)Writers
AimsTopicTopicalityLevel of language Suitability for the task LengthFocus StyleAccuracy
© UCLES 2013
The Recipe for Question Paper Production
Rejection Pretesting
Editing
Pre-editing
Commissioning of material
Revision
© UCLES 2013
Aims of Pretesting – the taste test
• Generate statistical data• Difficulty• Discrimination
• Generate qualitative feedback• Check the quality and fairness of items and
tasks
© UCLES 2013
The Recipe for Question Paper Production
Rejection
Pretest Review
Pretesting & Trialling
Editing
Pre-editing
Commissioning of material
Revision
© UCLES 2013
Pretest Review – the proof of the pudding…
ParticipantsAssessment ManagerChair (writing team leader)Experienced writerAimsevaluate resultsfinalise key
© UCLES 2013
The Recipe for Question Paper Production
Test Construction
Pretest Review
Pretesting & Trialling
Editing
Pre-editing
Commissioning of material
© UCLES 2013
Test Construction
Participants: Assessment ManagerChair (writing team leader)Experienced writerAims:Construct question papersEnsure correct level of difficultyCoverage of skills and appropriate content
© UCLES 2013
The Recipe for Question Paper Production
Test Construction
Pretest Review
Pretesting & Trialling
Editing
Pre-editing
Commissioning of material
Paper checking before sign-off
© UCLES 2013
© UCLES 2013
What is Cambridge English: Advanced (CAE)?
Established and high-quality test of EnglishTaken by hundreds of thousands of students globallyFocused on the level needed for university study (CEFR C1)Proven validity and reliability
© UCLES 2013
Accepted by 3,000 organisations worldwide
Higher education
UK, Australia, North America
Immigration
UK Border AgencyDIAC - Australia
Employers
Accenture, Bayer,Ernst & Young,
Dell
Teachers
Mexico,Finland
State schools
Romania,Germany
© UCLES 2013
Why is CAE different?
CAETherm
om
eter test
More than just a snapshot of English skills
© UCLES 2013
Helps students improve their English
Positive impact
Students take a language improvement course
Focus on communicative language skills
Focus on grammar and vocabulary
© UCLES 2013
Test content
12345
Reading – 75 min
Writing - 90 min
Use of English – 60 min
Listening - 40 min
Speaking - 15 min
© UCLES 2013
Test content
1234
Reading and Use of English - 90 min
Writing – 90 min
Listening - 40 min
Speaking - 15 min
© UCLES 2013
Enhanced securityTest day photosStatistical analysis of results
before releaseOnline results verification
service
© UCLES 2013
Results and Certificate
© UCLES 2013
Why take Cambridge English: Advanced (CAE)?
Established – recognised worldwide
Positive impact – focus on language skills
High quality – test content; administration; results
© UCLES 2013
Teaching Knowledge Test (TKT)
• For both experienced and new teachers• Flexible, modular structure
© UCLES 2013
Module 2: Planning for language teaching
TKT - OverviewC
ore
Module 1: Background to language teaching
Module 3: Classroom management
TKT: Young Learners
TKT: Practical
TKT: CLIL
TKT: Knowledge about LanguageS
pecialist
© UCLES 2013
Summary
Quality Assurance through the QPP process
Cambridge English: Advanced for C1 certification
Teaching Knowledge Test for teacher development
© UCLES 2013
Further information– Learn more about Cambridge English teaching
qualifications – www.cambridgeenglish.org/sector/teaching
– For more information on Cambridge English: Advanced– http://www.cambridgeenglish.org/exams-and-
qualifications/advanced/
– Harrison.G@CambridgeEnglish.org
© UCLES 2013
Thank you!
Any questions?
© UCLES 2013
A range of fit for purpose exams
© UCLES 2013
1913: The first CPE
items about grammar & lexis:‘Give the past tense and past participle of… dividing them into strong and weak; add explanations:tell, wake, buy’
items on grammar & lexical usage:‘Embody each of the following words into a sentence in such a way as to show that you clearly apprehend its meaning:commence, comment, commend…’
© UCLES 2013
What can students do with CAE?
Listening/Speaking Reading Writing
CAN follow abstract argumentation, for example the balancing of alternatives and the drawing of a conclusion.
CAN read quickly enough to cope with the demands of an academic course.
CAN write an essay which shows ability to communicate, giving few difficulties for the reader.
Examples CEFR Level C1 abilities
© UCLES 2013
Typical abilities at CEFR Level C1
React appropriately in different cultural and social situations
Express yourself with a high level of fluency
Participate effectively in meetings or academic tutorials and seminars
Deal confidently with living in an English-speaking environment
Communicate effectively at managerial and professional level
Follow an academic course at university level
7
5
6
4
3
2
1
Carry out complex and challenging research
© UCLES 2013
What is the impact if students are below this level?
Increased requirement for supportStudents fail to reach academic potential
Dissatisfaction amongst home students
© UCLES 2013
Test content
12345
Reading - 1 hour 15 minutes
Writing - 1 hour 30 minutes
Use of English - 1 hour
Listening - 40 minutes
Speaking - 15 minutes
© UCLES 2013
ReadingPart 1 Part 2 Part 3 Part 4Multiple choice6 questions
Gapped text6 questions
Multiple choice7 questions
Multiple matching15 questions
© UCLES 2013
Reading text example
© UCLES 2013
WritingPart 1 Part 2
Compulsory questionWrite one of the following: article, report, proposal or letter.
180-220 words
1 task from choice of 5One of the following: article, competition entry, contribution to a longer piece, essay, information sheet, a letter, a proposal, a report, a review
220–260 words
© UCLES 2013
Writing – sample of students workA book review – The Pelican Brief
© UCLES 2013
Use of EnglishPart 1 Part 2 Part 3 Part 4 Part 5
Multiple-choicecloze12 questions
Open cloze15 questions
Word formation10 questions
Gapped sentences5 questions
Key word transformat-ions8 questions
© UCLES 2013
Use of English - example
© UCLES 2013
ListeningPart 1 Part 2 Part 3 Part 4
Multiple choice6 questions
Sentence completion8 questions
Multiple choice6 questions
Multiple Matching10 questions
© UCLES 2013
Listening
© UCLES 2013
SpeakingPart 1 Part 2 Part 3 Part 4
Conversation betweenthe interlocutor and each candidate (spoken questions)
3 minutes
Individual ‘long turn’ foreach candidate with brief response from the second candidate. In turn, the candidates are given three pictures to talk about
A 1-minute ‘long turn’ for each candidate,plus a 30-second response from the second candidate
A two-way conversation between the candidates. The candidates are given spoken instructions with written and visual stimuli, which are used in a decision-making task
4 Minutes
A discussion on topics related to the collaborative task (spoken questions)
4 minutes
© UCLES 2013
Speaking
http://www.cambridgeesol.org/assets/wmv/univ/c1-speaking.wmv