of 72
8/9/2019 118613 Assessing Speaking Presentation Slides
1/72
Assessing SpeakingAn approach grounded in theory
and practice
Evelina D. Galaczi
Jill Buggey
8/9/2019 118613 Assessing Speaking Presentation Slides
2/72
Who are you?A. Im a teacher.
B. Im the co-ordinator/director of studies.
C. Im a learner.
D. Im a parent.
E. Im a researcher.
8/9/2019 118613 Assessing Speaking Presentation Slides
3/72
Where are you?
Cambridge ESOL exam centres
8/9/2019 118613 Assessing Speaking Presentation Slides
4/72
Assessing SpeakingAn approach grounded in theory
and practice
Evelina D. Galaczi
Jill Buggey
8/9/2019 118613 Assessing Speaking Presentation Slides
5/72
5
Cambridge AssessmentA leading authority
Influencing thinking through publishing
and consultancy
Leading-edge and operational research onassessment in education
Not for profit
150 yearsexperience
Delivering world-class assessment
Part of Cambridge University
Delivering Assessment
Influencing Thinking
8/9/2019 118613 Assessing Speaking Presentation Slides
6/72
Our reach Los AngelesMexico City
So PauloBuenos Aires
St Petersburg
Moscow
Warsaw
Berlin
Cambridge
Paris
Madrid
Bologna
Naples
Athens
Istanbul
Dubai
Pune
DelhiChennai
Shanghai
Guangzhou
Beijing
Seoul
Taipei
Ho Chi Minh City
Kuala Lumpur
Sydney
Cambridge ESOL exam centres
Office, Staff or
Representative
2,700test centres
50,000preparation centres
4 mcandidates
130countries
18,000Speaking Examiners
700Team Leaders
43Regional Team Leaders
23Professional Support
Leaders
8/9/2019 118613 Assessing Speaking Presentation Slides
7/72
Our task in this session to overview different speaking test formats
to explore paired speaking tests in terms
of research and practice
to explore implications for the classroom
8/9/2019 118613 Assessing Speaking Presentation Slides
8/72
How can speaking ability be assessed?
Direct formats
Semi-direct formats
8/9/2019 118613 Assessing Speaking Presentation Slides
9/72
How can speaking ability be assessed?
Direct formats
interaction with a liveexaminer:
one-to-one
paired
group
scored by human raters
8/9/2019 118613 Assessing Speaking Presentation Slides
10/72
How can speaking ability be assessed?
Semi-direct formats
computer-based:test taker speech iselicited by a machine
scored by human ratersor machine scored
Direct formats
interaction with a liveexaminer:
one-to-one
paired
group
scored by human raters
8/9/2019 118613 Assessing Speaking Presentation Slides
11/72
Why direct assessment of speaking? Direct speaking tests focus on production
and interaction
Computer-based testing lacksco-constructionand interaction
8/9/2019 118613 Assessing Speaking Presentation Slides
12/72
No best way to test speaking each format has its unique advantages
and limitations
each format is suitable for differentpurposes
8/9/2019 118613 Assessing Speaking Presentation Slides
13/72
Traditional approach to speaking
testsThe direct individual speaking test format
Examiner Test taker
8/9/2019 118613 Assessing Speaking Presentation Slides
14/72
Issues with the individual speaking test
format
More restricted tasks and types of interaction Unequal rights between examiner and candidate
Unbalanced power distribution
Examiner Testtaker
8/9/2019 118613 Assessing Speaking Presentation Slides
15/72
Whats the solution?
8/9/2019 118613 Assessing Speaking Presentation Slides
16/72
What about a paired format?
Test Taker A
Testtaker B
Interviewer Assessor
Test Taker A
Interviewer
Test taker A
Interviewer
8/9/2019 118613 Assessing Speaking Presentation Slides
17/72
Positive
effects of
paired
tests
8/9/2019 118613 Assessing Speaking Presentation Slides
18/72
Positive
effects of
paired
tests
Reflect theclassroom
8/9/2019 118613 Assessing Speaking Presentation Slides
19/72
Positive
effects of
paired
tests
Reflect theclassroom
More pairwork
8/9/2019 118613 Assessing Speaking Presentation Slides
20/72
8/9/2019 118613 Assessing Speaking Presentation Slides
21/72
Positive
effects of
paired
tests
Reflect theclassroom
More pairwork
Broader rangeof language
Moreevidence oflanguage
proficiency
8/9/2019 118613 Assessing Speaking Presentation Slides
22/72
Positive
effects of
paired
tests
Reflect theclassroom
More pairwork
Broader rangeof language
Moreevidence oflanguage
proficiency
Allow moreinteractionalsymmetry
8/9/2019 118613 Assessing Speaking Presentation Slides
23/72
Potential
issuesReflect theclassroom
More pairwork
Broader rangeof language
Moreevidence oflanguage
proficiency
Allow moreinteractionalsymmetry
8/9/2019 118613 Assessing Speaking Presentation Slides
24/72
Potential
issuesReflect theclassroom
More pairwork
Broader rangeof language
Moreevidence oflanguage
proficiency
Allow moreinteractionalsymmetry
Complexity and
uncontrolled
variables
8/9/2019 118613 Assessing Speaking Presentation Slides
25/72
Potential
issuesReflect theclassroom
More pairwork
Broader rangeof language
Moreevidence oflanguage
proficiency
Allow moreinteractionalsymmetry
Complexity and
uncontrolled
variables
Individual
scores based
on jointconstruction
8/9/2019 118613 Assessing Speaking Presentation Slides
26/72
Is this a fair test?
What insights does the available
research provide?
8/9/2019 118613 Assessing Speaking Presentation Slides
27/72
True or false? In a paired format ...
6. There is more variety of speech functions
5. The effect of background variables is higher
3. There is more collaborative development of topics
2. The test taker has more opportunities for initiating
topics
4. There is more examiner talk
1. The test taker has more turns
8/9/2019 118613 Assessing Speaking Presentation Slides
28/72
True or false? In a paired format ...
6. There is more variety of speech functions
5. The effect of background variables is higher
3. There is more collaborative development of topics
2. The test taker has more opportunities for initiating
topics
4. There is more examiner talk
1. The test taker has more turns True
8/9/2019 118613 Assessing Speaking Presentation Slides
29/72
True or false? In a paired format ...
6. There is more variety of speech functions
5. The effect of background variables is higher
3. There is more collaborative development of topics
2. The test taker has more opportunities for initiating
topics
4. There is more examiner talk
1. The test taker has more turns True
True
8/9/2019 118613 Assessing Speaking Presentation Slides
30/72
True or false? In a paired format ...
6. There is more variety of speech functions
5. The effect of background variables is higher
3. There is more collaborative development of topics
2. The test taker has more opportunities for initiating
topics
4. There is more examiner talk
1. The test taker has more turns True
True
True
8/9/2019 118613 Assessing Speaking Presentation Slides
31/72
True or false? In a paired format ...
6. There is more variety of speech functions
5. The effect of background variables is higher
3. There is more collaborative development of topics
2. The test taker has more opportunities for initiating
topics
4. There is more examiner talk
1. The test taker has more turns True
True
True
False
8/9/2019 118613 Assessing Speaking Presentation Slides
32/72
True or false? In a paired format ...
6. There is more variety of speech functions
5. The effect of background variables is higher
3. There is more collaborative development of topics
2. The test taker has more opportunities for initiating
topics
4. There is more examiner talk
1. The test taker has more turns True
True
True
False
True
8/9/2019 118613 Assessing Speaking Presentation Slides
33/72
True or false? In a paired format ...
6. There is more variety of speech functions
5. The effect of background variables is higher
3. There is more collaborative development of topics
2. The test taker has more opportunities for initiating
topics
4. There is more examiner talk
1. The test taker has more turns True
True
True
False
True
True
8/9/2019 118613 Assessing Speaking Presentation Slides
34/72
Interaction in paired formats more complexcognitive and strategic
processes
8/9/2019 118613 Assessing Speaking Presentation Slides
35/72
8/9/2019 118613 Assessing Speaking Presentation Slides
36/72
Interaction in paired formats more complexcognitive and strategic
processes
which elicit a richer range of language
which allows for better inferences to be
made about a candidates proficiency in
wider real-life contexts
8/9/2019 118613 Assessing Speaking Presentation Slides
37/72
Effect of background variablesInterlocutor effect (OSullivan, 2002)The effect of background variables on performance from other
candidates and examiners
proficiency level personality
acquaintanceship
cultural background
age gender
status, etc.
Note: both in individual and paired tests
8/9/2019 118613 Assessing Speaking Presentation Slides
38/72
How influential are these variables?
Effect on discourse?
Effect on scores?
8/9/2019 118613 Assessing Speaking Presentation Slides
39/72
8/9/2019 118613 Assessing Speaking Presentation Slides
40/72
Effect of background variablesproficiency level
personality
acquaintanceship
some influence on conversation, but
impact is small
8/9/2019 118613 Assessing Speaking Presentation Slides
41/72
Effect of background variablesproficiency level
personality
acquaintanceship
some influence on conversation, but
impact is small
interplay between task type andpersonality
8/9/2019 118613 Assessing Speaking Presentation Slides
42/72
Effect of background variablesproficiency level
personality
acquaintanceship
some influence on conversation, but
impact is small
interplay between task type andpersonality
interplay between acquaintanceship,
gender and cultural norms
8/9/2019 118613 Assessing Speaking Presentation Slides
43/72
Variability related to different characteristics
of conversational partners is all that
happens in the real world. And so they arethings that we should be interested in
testing. (Swain, cited in Fox, 2004)
What should we do about it?
8/9/2019 118613 Assessing Speaking Presentation Slides
44/72
Paired speaking tests present
challenges, but mostly opportunities
(Brooks, 2008)
What should we do about it?
8/9/2019 118613 Assessing Speaking Presentation Slides
45/72
8/9/2019 118613 Assessing Speaking Presentation Slides
46/72
The Cambridge approach a range of task types, only one of which
is paired:
benefit from paired format
control possible limitations
use of analytical and global criteria andscales
8/9/2019 118613 Assessing Speaking Presentation Slides
47/72
The Cambridge approach a range of task types, only one of which
is paired:
benefit from paired format
control possible limitations
use of analytical and global criteria andscales
training and monitoring of examiners
8/9/2019 118613 Assessing Speaking Presentation Slides
48/72
Which of these Cambridge ESOL tests
do not use a paired format? Cambridge English: Proficiency (CPE)
Cambridge English: Advanced (CAE)
Cambridge English: First (FCE)
Cambridge English: Preliminary (PET)
Cambridge English: Key (KET)
Cambridge English: Young Learners (YLE)
IELTS Cambridge English: Business Certificates (BEC)
Why not?
8/9/2019 118613 Assessing Speaking Presentation Slides
49/72
Which of these Cambridge ESOL tests
do not use a paired format? Cambridge English: Proficiency (CPE)
Cambridge English: Advanced (CAE)
Cambridge English: First (FCE)
Cambridge English: Preliminary (PET)
Cambridge English: Key (KET)
Cambridge English: Young Learners (YLE)
IELTS Cambridge English: Business Certificates (BEC)
8/9/2019 118613 Assessing Speaking Presentation Slides
50/72
8/9/2019 118613 Assessing Speaking Presentation Slides
51/72
What does this mean for classrooms?
8/9/2019 118613 Assessing Speaking Presentation Slides
52/72
8/9/2019 118613 Assessing Speaking Presentation Slides
53/72
What does this mean for classrooms?
students need to be supported with
appropriate materials
suitable classroom practice to developcollaborative skills
8/9/2019 118613 Assessing Speaking Presentation Slides
54/72
What is meant by collaborative skills?
Collaborativeskills
8/9/2019 118613 Assessing Speaking Presentation Slides
55/72
Topic development
Collaborativeskills
What is meant by collaborative skills?
8/9/2019 118613 Assessing Speaking Presentation Slides
56/72
Topic development Turn-taking
Collaborativeskills
What is meant by collaborative skills?
8/9/2019 118613 Assessing Speaking Presentation Slides
57/72
Topic development Turn-taking
Interactive listening andlistener support
Collaborativeskills
What is meant by collaborative skills?
8/9/2019 118613 Assessing Speaking Presentation Slides
58/72
Topic development Turn-taking
Speaker nominationInteractive listening and
listener support
Collaborativeskills
What is meant by collaborative skills?
8/9/2019 118613 Assessing Speaking Presentation Slides
59/72
Cambridge English: First
Speaking Part 3 (collaborative task) B2 level
decision-making task based on a series of
pictures candidates talk together for about 3
minutes
8/9/2019 118613 Assessing Speaking Presentation Slides
60/72
Cambridge English: First
Speaking Part 3 (collaborative task)Id like you to imagine that a local caf wants to
attract more people. Here are some of the
suggestions theyre considering.
First, talk to each other about how successful
these suggestions might be.
Then decide which two would attract most
people.
8/9/2019 118613 Assessing Speaking Presentation Slides
61/72
8/9/2019 118613 Assessing Speaking Presentation Slides
62/72
Collaborative skillsextract 1
Ana Yeahthis one seems to be good, the terrace, but
maybe if you have a good weather
Giallo Yes, I dont know here in Cambridge
Ana Yeah, its difficult [both laugh]
Giallo ..and then if you. have just opened a caf or a bar
maybe its not agood idea to put some place , sorry,
some tables outside. I think its better to concentrate on
music orother ideas.
Ana Yeah, yeah .
8/9/2019 118613 Assessing Speaking Presentation Slides
63/72
Collaborative skillsextract 2Ana Yeah, yeah . What do you think about football
matches?
Giallo Depends on the target of the public that you are
interested in. I mean if you are interested in Ana Sports?
Giallo Yeah, if you are interested in attracting men its more
than good, its a perfect idea but I dont know. You can
tell! I mean would you go in a caf with a football match?Ana: Yeah, me not but most of the people I think maybe its
a good idea.
8/9/2019 118613 Assessing Speaking Presentation Slides
64/72
Collaborative skillsextract 3Ana But I think that the one that would attract... the most
people, I think, is the place where you can watch TV I
mean football matches or sports becauselots of people
are interested in that
Giallo Mmm
Ana and maybe the coffee free is also important
GialloYes
Ana Young people has not a lot of money, yeah
Giallo Yeah, I dont know about television. I think when you
begin, maybe its better to put like some comfortable
sofas becauseitsan example of your caf.
8/9/2019 118613 Assessing Speaking Presentation Slides
65/72
Examples of collaborative skills
Ana Giallo Both
agrees with
Giallo
initiates
discussion
turn-take well
asks Giallo
questions
invites Ana to
speak
dont interrupt
each other
develops the
topic by giving
reasons for her
decisions
moves
discussion
forward
listen and
respond to what
their partner
says
8/9/2019 118613 Assessing Speaking Presentation Slides
66/72
8/9/2019 118613 Assessing Speaking Presentation Slides
67/72
Developing collaborative skills
Areas to improve on How can your students
improve?
Turn takingInitiating discussion
Giving views and
opinions
Analyse clipsRecord your students
Practise useful
language
Raise awareness ofskills and language
needed
8/9/2019 118613 Assessing Speaking Presentation Slides
68/72
8/9/2019 118613 Assessing Speaking Presentation Slides
69/72
8/9/2019 118613 Assessing Speaking Presentation Slides
70/72
8/9/2019 118613 Assessing Speaking Presentation Slides
71/72
Summary of todays webinar
overview of different speaking test formats
exploration of paired speaking tests in
terms of research and practice implications for the classroom
resources available
8/9/2019 118613 Assessing Speaking Presentation Slides
72/72