Using the CEFR in Catalonia

Neus Figuerasnfiguera@xtec.cat

The EOI system

●State funded language schools (+16)

●Two levels defined (aimed at B1 and B2)

● Published curricula

●13 different languages

●40,000 students

●16,000 certificate exams every year

●Standardised certificates since 1995

Issues to be solved in 2002-1

●Revise certificate examinations.

● Is examination difficulty equivalent across time?

● Is the lower certificate consistently easier than the higher certificate?

●Are the different certificates in the different languages comparable?

Issues to be solved in 2002-2

●Develop level specifications related to the CEFR.

●Revise existing curricula in relation to the CEFR.

●Link certificates to CEFR levels.

● Project design

● Empirical scale development

● Item banking (English)

● Manual procedures for linkage, (Specification Standardisation, Empirical validation)

● Defining and exemplifying A2

● Developing curriculum objectives for A2, B1 and B2

● Developing test specifications for A2, B1 and B2

Project Overview 2003-2007

Scale Development

Item BankingLinking tests

Changing curricula

CEFR

Challenges

●Where to start?●Involve teachers (and item writers).●Improve existing practice.●Bring in the ELP onto the project.●How? Need to count on experts.●Limited resources.

Scale development (all languages)

CEFR Methodology

Step 1: selecting level descriptors.

Step 2: translation into Catalan.

Step 3: mapping descriptors onto levels.

Step 4: developing and validating new scales.

Descriptors Teachers* Languages

● Reading 40 103 10● Listening 46 99 10● Speaking 80 92 12 ● Writing 53 89 12

● Grammar 34 81 12 ● Vocabulary 61 73 12

* Arabic, Basque, Catalan, Dutch, English (>40), French (>20), German (>15), Italian, Japanese, Portuguese, Russian, Spanish

Teachers involved

Lessons learnt from scale development

●Continued training/familiarisation is necessary.

●Appearances need to be checked empirically.

●Exact correspondence may not be possible, but is it desirable?.

●Linkage does not mean equivalence.

Item Banking (English)

MethodologyStep 1

● Booklet development

Booklet design for anchoring items (2003)

  S1 S2 S3 S4 S5 L1 L2 L3 A1 A2 A3

V1 X X       X          

V2   X X       X        

V3     X X       X      

V4       X X X          

V5 X       X   X        

V6           X   X X X  

V7             X X   X X

V8           X X   X   X

MethodologyStep 2● Data collection (260-784 students per item)● Analyses: CTT and IRT (Total surviving items :301)

Step 3● Standard setting procedures: - test centered - examinee centered - annual average pass rate● Setting (provisional) cut off scores at Elemental (B1) and

Aptitud (B2)

TCC of the eight versions(2003)

-3 -2 -1 0 1 2 3

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Theta

p1 p2 p3 p4 p5 p6 p7 p8

Using the Manual for linking exams (English, French, German)

Challenges in Specification

●How to map the examination? Different versions of the test? Specifications?

●How to tackle differences of coverage (subskills)?

●Who does what?

●Who checks it?

Challenges in Standardisation

●Reference “r” materials not ready until 2005.

●Differences across skills, across languages.

●Assessing task vs. item levels.

● No “linguistic competence” reference materials.

The proposal from Dutch CEFR project

●Training.●Describing texts and items according to

set parameters (reading and listening).●Estimating their CEFR levels.●Pretesting the items thus labelled.●Calibrating the items.●Standard-setting on the scale coming

from the calibration.●Assigning a psychometric level to the

items. ●Assigning a definitive level to the items.

Using the Council of Europe Item CD (German)

The German booklet

  Reading Listening

StudentsB1 B2

294 307

ItemsA2 C1

40 40

JudgesNative-non native

12 10

Institutions Dialang, Goethe, WBT, TestDAF, EOI

Dialang, Goethe, WBT, EOI

Inter-judge Consistency in assessing items

R15L22

Judges: Consistency with the Pre-Estimation

1 2 3 4 5 6 7 8 9 10 11 12 13

0,0

0,2

0,4

0,6

0,8

1,0

Mean

Reading: = 0.98

Pre-est.

r tot

Judges1 2 3 4 5 6 7 8 9 10 11 12 13

0,0

0,2

0,4

0,6

0,8

1,0

Mean

Listening: = 0.94

Pre-est.

r tot

Judges

Consistency pre-estimation with Empirical Results

Developing CEFR based curricula

Developing CEFR-based curricula

●Focusing on what students can do.●Drafting objectives. ●Defining content.●Defining assessment criteria. +●Methodological guidelines.

Conclusions so far

The CEFR IS a Bible ( but only in the widest sense of the word)

The Manual is no book of spells

There is no ONE holy Grail

●We learnt much more about our exams.

●It has been a competence building process.

●We have become less dogmatic.

●We know there is further work to do and room for improvement.

●Combining highly technical work with enthusiasm is crucial.

How do I know if my B1 is your B1?

This is my B1. What’s your B1 like?