Military Language Testing at the Military Language Testing at the National Defence University and National Defence University and

the Common European the Common European FrameworkFramework

BILC CONFERENCE 2006. BILC CONFERENCE 2006. BUDAPESTBUDAPEST

• General language examination in five General language examination in five languageslanguages

ARMA bilingual military examination inARMA bilingual military examination in nine languages nine languages

NATO STANAG 6001 monolingualNATO STANAG 6001 monolingual language examination language examination

Interviews with officers and NCOsInterviews with officers and NCOs

Hungarian officers and NCOs Hungarian officers and NCOs serving at the NATO headquarters serving at the NATO headquarters

Job descriptions of soldiers Job descriptions of soldiers serving in NATO positions serving in NATO positions

Observance of the regulations for the Observance of the regulations for the conduct of examinations;conduct of examinations;

Continuous training and observing ofContinuous training and observing of testers testers

Continuous pretesting and renewal ofContinuous pretesting and renewal of items; items;

NATO STANAG 6001

2100

605 709 717

1391

971

050

010

0015

0020

0025

00

2002 2003 2004 2005 2006Jan. -Apr.

2006

cand

idat

es

2002

0

50

100

150

200

L S R W L S R W L S R W

LEVEL 1 LEVEL 2 LEVEL 3

CAN

DID

ATES

/ R

ESU

LTS

CANDIDATES

RESULTS

2003

050

100150200250300350400450

L S R W L S R W L S R W

LEVEL 1 LEVEL 2 LEVEL 3

CAN

DID

ATES

/ R

ESU

LTS

CANDIDATES

RESULTS

2004

0

100

200

300

400

500

600

L S R W L S R W L S R W

LEVEL 1 LEVEL 2 LEVEL 3

CAN

DID

ATES

/ R

ESU

LTS

CANDIDATES

RESULTS

2005

0

100

200

300

400

500

600

700

L S R W L S R W L S R W

LEVEL 1 LEVEL 2 LEVEL 3

CAN

DID

ATES

/ R

ESU

LTS

CANDIDATES

RESULTS

2006 Jan. - Apr.

050

100150200250300350400450

L S R W L S R W L S R W

LEVEL 1 LEVEL 2 LEVEL 3

CA

ND

IDA

TES

/ R

ES

ULT

S

CANDIDATES

RESULTS

COMMON EUROPEAN COMMON EUROPEAN FRAMEWORKFRAMEWORK

ESL Exams in HungaryESL Exams in Hungary

• University Of Cambridge ESOLUniversity Of Cambridge ESOL• Trinity College London ESOL & GESETrinity College London ESOL & GESE• The European Language CertificatesThe European Language Certificates• Pitman QualificationsPitman Qualifications• Österreichisches Sprachdiplom Deutsch ÖSDÖsterreichisches Sprachdiplom Deutsch ÖSD• Goethe-Institut vizsgarendszereGoethe-Institut vizsgarendszere• LEXINFOLEXINFO• LCCIEB EFB angol üzleti szaknyelvLCCIEB EFB angol üzleti szaknyelv• EuroEuro• ECL nyelvvizsgaECL nyelvvizsga• Bibliai héberBibliai héber• Diplomáciai szaknyelvi vizsgaDiplomáciai szaknyelvi vizsga• Certifications de l´ Alliance française de ParisCertifications de l´ Alliance française de Paris

Linking Separate AssessmentsLinking Separate Assessments

• EquatingEquating

• CalibratingCalibrating

• Statistical moderationStatistical moderation

• BenchmarkingBenchmarking

• Social moderationSocial moderation

Linking ProcessLinking Process

• Selection of appropriate proceduresSelection of appropriate procedures

• Project planningProject planning

• Coordination of the participantsCoordination of the participants

• Application of the proceduresApplication of the procedures

• Recording of resultsRecording of results

Linking ProceduresLinking Procedures

• FamiliarisationFamiliarisation

• SpecificationSpecification

• StandardisationStandardisation

• Empirical ValidationEmpirical Validation

FamiliarisationFamiliarisation

• Discussion of the Common European Discussion of the Common European Framework levelsFramework levels

• Qualitative analysis of the Common Qualitative analysis of the Common European Framework levelsEuropean Framework levels

The Specification Stage

Manual:Manual:Relating Language Relating Language

Examinations to the Examinations to the Common European Common European

Framework of Reference for Framework of Reference for Languages: Learning, Languages: Learning, Teaching, AssessmentTeaching, Assessment

(CEF)(CEF)

SpecificationSpecification::

Description of the coverage of Description of the coverage of the examination in relation to the examination in relation to

the Common European the Common European FrameworkFramework

Content analysis procedures:

filling in a number of checklistsfilling in a number of checklists using relevant CEF descriptors to relate using relevant CEF descriptors to relate

the exam to the categories and levels of the exam to the categories and levels of the CEFthe CEF

The AThe Aims of the Specificationims of the Specification

To analyse the content in relation to To analyse the content in relation to the relevant categories of the CEFthe relevant categories of the CEF

To profile the examination in relation to To profile the examination in relation to

the relevant descriptor scales of the the relevant descriptor scales of the CEFCEF

To make a claim that an examination is To make a claim that an examination is related to a particular level of the CEFrelated to a particular level of the CEF

Phase 1: Phase 1: A general description of the A general description of the examination or test tasksexamination or test tasks

Phase 2: Phase 2: A detailed description of the A detailed description of the examination or test tasksexamination or test tasks

SStage 1tage 1General Examination Description General Examination Description

FormForm (A1) (A1)What is the overall aim?

What are the more specific objectives?

What are the principal domains?

Which communicative activities are tested?

What types of test tasks are used?

What information is published for candidates and teachers?

SStage 2tage 2Forms on Different Forms on Different Aspects of of

the Examinationthe Examination The test development process (Form A2)The test development process (Form A2) The marking schemes and scoring rules for The marking schemes and scoring rules for

different subtests (Form A3)different subtests (Form A3) The grading and standard-setting procedures The grading and standard-setting procedures

for different subtests (Form A4)for different subtests (Form A4) The reporting of results (Form A5)The reporting of results (Form A5) The analysis and review procedures (Form The analysis and review procedures (Form

A6)A6)

FormForm 3A: 3A: MarkingMarking

How are the tests marked?

Where are the tests marked?

What criteria are used to select the markers?

How is accuracy of marking promoted?

Are productive or integrated test tasks single or double rated?

If double rated, what procedures are used when differences between raters occur?

The Detailed Description of the The Detailed Description of the ExaminationExamination

Forms for the Communicative Forms for the Communicative Language ActivitiesLanguage Activities

Listening ComprehensionListening Comprehension Reading ComprehensionReading Comprehension Spoken InteractionSpoken Interaction Written InteractionWritten Interaction Spoken ProductionSpoken Production Written ProductionWritten Production Integrated Skill CombinationsIntegrated Skill Combinations Integrated SkillsIntegrated Skills Spoken MediationSpoken Mediation Written MediationWritten Mediation

Listening ComprehensionListening Comprehension

Which situations, content categories, domains are the test takers expected to show ability in?

Which communication themes are the test takers expected to be able to handle?

Which communicative tasks are the test takers expected to be able to handle?

What kind of communicative activities and strategies are the test takers expected to be able to handle?

After reading the scale for After reading the scale for Overall Listening Overall Listening

Comprehension, indicate and Comprehension, indicate and justify at which level(s) of the justify at which level(s) of the scale the subtest should be scale the subtest should be

situatedsituated

Forms for the Aspects of Forms for the Aspects of Communicative Language Communicative Language

CompetenceCompetence

ReceptionReception InteractionInteraction ProductionProduction MediationMediation

OutcomeOutcome

A A narrative description of the narrative description of the examinationexamination

CCompleted versions of the relevant ompleted versions of the relevant formsforms

A graphical presentation of the A graphical presentation of the relationship of the examination to the relationship of the examination to the CEF levelsCEF levels

Thank you for your attentionThank you for your attention