L ANGUAGE T ESTING S ERVICES John H.A.L. de Jong 2004 1 The Role of the Common European Framework...

LANGUAGE TESTING SERVICES John H.A.L. de Jong 2004 1

The Role of the

Common European Framework

John H.A.L. de Jong

EALTA Conference, Kranjska Gora, Slovenia, May 14 – 16, 2004




Language

The Common Framework

Application

Overview


Bilingual Brain


Speaking: From intention to articulation

CONCEPTUALIZERMessage structureFORMULATORUtterance structureARTICULATORUtterance Levelt, 1989



languagestructures

declarative

discourse

declarative

discourse

languagestructures

DecodingSpokenturn

socialsocial

Encoding EncodingDecoding Spokenturn

Spokenturn

Person A Person B


Facility in Spoken Language is not Oral Proficiency

LanguageLanguage use

CultureWorld knowledge

–OP includes the ability to use the language in socially appropriate ways in a range of complex functions over a variety of topics.

–High OP implies adequate FSL–High FSL does not guarantee

high OP

OP(L2) f (OP(L1), FSL(L2), Culture(C2), Social (S2))FSL(L2) OP(L2) - OP(L1)


Some Basic Issues in the CEFSome of which are often forgotten


Types of Scores

Continuous infinite number of values (1, 2, 3, etc. But also 1.2, 1.53 1.6 etc.)

Discrete fixed set of fixed values

01

34

2


Defining stages on a continuous scale

Stage 1Stage 2

Stage 1 Stage 2 Stage 3 Stage 4

Defining stages on a continuum is arbitrary.

We can define different numbers of stages, according to our liking.

As long as we clearly define where a one stages ends and another begins.

Note: There is always a stage below the lowest defined stage

Below Stage 1

Below Stage 1


Qua

lity

Quantity

Potential Language Development:follows two basic dimensions


Dimensionality The quantity development is in fact multidimensional and quality can develop along each of the dimensions


the number of ...domains (school, work, home, etc.)

functions (ask, command, inquire, etc.) notions (north, south, table, mother, cat, eat, drink, etc.)

situations (meeting, e-mail, telephone, etc.)

locations (market, school, police station, etc.)

topics (weather, study, holidays, etc.)

roles (listener in audience, participant in discussion, etc)

... that a language user can deal with.

QUANTITY refers to:


Language use is effectiveleading to degree of precision

1) in understanding what is meant2) in expressing one’s meaningleading to

communication with least possible effort

Language use is efficient

the degree to which

QUALITY refers to:


Combining Q & Q

Note: there is always a level below the lowest defined level

A combination ofQuantity & Quality:

BreakthroughWaystage

Threshold

Vantage

Operational Proficiency

Mastery Mastery

definesLevels ofLanguageProficiency

Below Breakthrough

Independent

Co-operative

Basic functions

Fully independent

Fluent and Spontaneous

No pr blem

Note: It is essential to understand that in the CEF Mastery does NOT mean ‘native speaker’ level. Surely an illiterate native speaker is not even at Breakthrough for reading. And there are big differences among native speakers with respect to the other skills as well.

C1C1

B2B2

B1B1

AAI’m afraid

you are below A1A1


Simplify

A1A2

B1B2

C1C2

Below A1

A more simple system of level names (universal: no translation problems)


Broad & Narrow

ABC

A1.1A1.2

A2.2

A2.1

B1.1

B1.2

B2.1

B2.2

C1.1

C1.2

C2.1

C2.2

A1A2

B1B2

C1C2

Below A1

Now we can make level distinctions

and ones


Oral scale

C2C2 Conveys finer shades of meaning precisely and naturally.

C1C1 Shows fluent, spontaneous expression in clear, well-structured speech.

B2B2 Relates information and points of view clearly and without noticeable strain.

B1B1 Relates comprehensibly main points he/she wants to make on familiar matters

A2A2 Relates basic information on, e.g. work, background, family, free time etc. .

A1A1 Makes simple statements on personal details and very familiar topics.

EXAMPLE: Oral Interaction ScaleNote: a person at the border between level A2 and B1 will probably be able to do 80% of the tasks at level A2,

but can also do 50% of but can also do 50% of the tasks at level B2, etc, because the underlying scale is the tasks at level B2, etc, because the underlying scale is

continuous and the CEF descriptors have been IRT scaledcontinuous and the CEF descriptors have been IRT scaled..

1%1%

5%5%

15%15%.

50%50%

80%80%

95%95%

80%80%


CommunicativeStrategies

CommunicativeLanguage Competencies

Hierarchy of ScalesTo illustrate the hierarchical principle one branch of the hierarchy is worked out in detail to the right

Reception Production

Understanding

Note: this branch of the hierarchy is part of the quantity dimension: how much a

language learner can do

a native speakerConversation

InformalDiscussion

FormalDiscussion

Obtaining Goodsand Services

Interviewing &being interviewed

Spoken Written

Interaction Mediation

Overall language ProficiencyGlobal Scale

And at each further node in the hierarchy the CEF offers descriptive scales.

CommunicativeActivities

The Global scale is for Overall Language Proficiency.


The Quality Hierarchy

CommunicativeStrategies

GeneralLinguistic

VocabularyRange

Range

GrammaticalAccuracy

PhonologicalControl

VocabularyControl

OrthographicControl

Control

Linguistic Sociolinguistic Pragmatic

CommunicativeLanguage Competencies

CommunicativeActivities

Overall language ProficiencyGlobal ScaleThe same Global Scale

analyzed here into the constituent elements of the quality dimension: how well a learner can do.

And here too there is a scale at each node.


Automatic speaking tests

Integrated “listen then speak” items

Real-time processing of spoken linguistic forms

Overall score and diagnostic subscores• Sentence Mastery• Vocabulary• Fluency• Pronunciation


What is tested?

Facility in spoken English: • ability to track what is said, • extract meaning in real time, • formulate responses• produce responses at a conversational pace, that are

relevant and intelligible


Conversational Skill

hear utterance

get lexical items

extract structure

build semantic field

contextualize

infer demand (if any)

articulate response

construct response

select lexical items

build phrase structure

select register

decide on response

(After: Levelt, 1989)


Table 1: Overview of main data on human scoring with CEFSET-10 SST

Number of candidates 587 528Number of raters 7 4Total number of ratings 8464 2500Interrater reliability (ANOVA) .95 .94Rater agreement (cumulative)- Exact same level- 1 level difference or less- 2 levels difference or less- >2 levels difference or less

64%28%

6%2%

67%24%

6%3%

Native & Non-Native Overall CDF


Experiment: Score interpretation on CEF scalePhonePass SET 10 ~ CEF


CEF English vs Spanish

y = 0,5418x - 0,0315

R2 = 0,9949

y = -0,0024x3 + 0,0047x2 + 0,6334x - 0,1583

R2 = 1

-8

-6

-4

-2

0

2

4

6

8

-8 -6 -4 -2 0 2 4 6 8

English

Sp

anis

h

Linear regression

Polynomial regression

C2

B1

B2C1

A2

English and Spanish on CEF scale


-6.0

-4.0

-2.0

0.0

2.0

4.0

6.0

A1 A2 B1 B2 C1 C2

CoE Levels

Dif

fcu

lty E

sti

mate

s

Spanish

English

North, 2000

Figure 1: Estimates for CEF level Cut-off from North (2000) and from responses to open-ended tasks in Spanish and English

English and Spanish vs. CEF scale


So what isis the role of the CEF?

• Transparency - Tool for communication

• Comparability - Common yardstick

• Progress - Stimulates discussion

Date post:	21-Dec-2015
Category:	Documents
View:	214 times
Download:	0 times

L ANGUAGE T ESTING S ERVICES John H.A.L. de Jong 2004 1 The Role of the Common European Framework...

Documents