Francesco Gratton

2013

Testing in the time Testing in the time of crisisof crisis

BILC PROFESSIONAL SEMINARBILC PROFESSIONAL SEMINAR

Stockholm , October 14 - 17, 2013Stockholm , October 14 - 17, 2013

INNOVATIVE TEST DESIGNS AND FORMATSINNOVATIVE TEST DESIGNS AND FORMATS

Lt.Col. F. Lt.Col. F. GrattonGratton

Francesco Gratton

2013

Summary:Summary:

• Past situation (up to Sept 2013)Past situation (up to Sept 2013)

• New Course Of Actions adoptedNew Course Of Actions adopted

• A do-it-all softwareA do-it-all software

• ProposalsProposals

Francesco Gratton

2013

1: Multilevel test (level 1 to 4)

2: Multiple choice questions (60 for L & R)

3: No penalties for wrong answers

4: Duration: R 105’ / L 90’

5: Separate Sections (& levels)

6: # of correct answers multiplied for a coefficient

7: Potential use of “F” factor

RECEPTIVE SKILLS (Listening & Reading)RECEPTIVE SKILLS (Listening & Reading)

Stanag Proficiency Test 1.0 Stanag Proficiency Test 1.0

Francesco Gratton

2013

How STANAG levels were awarded

(Stanag Proficiency TestStanag Proficiency Test 1.0)

# of correct answers

fixed coefficient (1,66)

Multiplied by

RECEPTIVE SKILLSRECEPTIVE SKILLS

Francesco Gratton

2013

CORRECT % CORRECT % % % %

1 1,67 11 18,37 26 43,42 41 68,47 56 93,52

2 3,34 12 20,04 27 45,09 42 70,14 57 95,19

3 5,01 13 21,71 28 46,76 43 71,81 58 96,86

4 6,68 14 23,38 29 48,43 44 73,48 59 98,53

5 8,35 15 25,05 30 50,1 45 75,15 60 100,2

6 10,02 16 26,72 31 51,77 46 76,82

7 11,69 17 28,39 32 53,44 47 78,49

8 13,36 18 30,06 33 55,11 48 80,16

9 15,03 19 31,73 34 56,78 49 81,83

10 16,7 20 33,4 35 58,45 50 83,5

21 35,07 36 60,12 51 85,17

22 36,74 37 61,79 52 86,84

23 38,41 38 63,46 53 88,51

24 40,08 39 65,13 54 90,18

25 41,75 40 66,8 55 91,85

LEVEL 0 LEVEL 1

CORRECT % CORRECT % % % %

1 1,67 11 18,37 26 43,42 41 68,47 56 93,52

2 3,34 12 20,04 27 45,09 42 70,14 57 95,19

3 5,01 13 21,71 28 46,76 43 71,81 58 96,86

4 6,68 14 23,38 29 48,43 44 73,48 59 98,53

5 8,35 15 25,05 30 50,1 45 75,15 60 100,2

6 10,02 16 26,72 31 51,77 46 76,82

7 11,69 17 28,39 32 53,44 47 78,49

8 13,36 18 30,06 33 55,11 48 80,16

9 15,03 19 31,73 34 56,78 49 81,83

10 16,7 20 33,4 35 58,45 50 83,5

21 35,07 36 60,12 51 85,17

22 36,74 37 61,79 52 86,84

23 38,41 38 63,46 53 88,51

24 40,08 39 65,13 54 90,18

25 41,75 40 66,8 55 91,85

LEVEL 1

CORRECT % % CORRECT % % %

1 1,67 11 18,37 26 43,42 41 68,47 56 93,52

2 3,34 12 20,04 27 45,09 42 70,14 57 95,19

3 5,01 13 21,71 28 46,76 43 71,81 58 96,86

4 6,68 14 23,38 29 48,43 44 73,48 59 98,53

5 8,35 15 25,05 30 50,1 45 75,15 60 100,2

6 10,02 16 26,72 31 51,77 46 76,82

7 11,69 17 28,39 32 53,44 47 78,49

8 13,36 18 30,06 33 55,11 48 80,16

9 15,03 19 31,73 34 56,78 49 81,83

10 16,7 20 33,4 35 58,45 50 83,5

21 35,07 36 60,12 51 85,17

22 36,74 37 61,79 52 86,84

23 38,41 38 63,46 53 88,51

24 40,08 39 65,13 54 90,18

25 41,75 40 66,8 55 91,85

LEVEL 2

CORRECT % % % CORRECT % %

1 1,67 11 18,37 26 43,42 41 68,47 56 93,52

2 3,34 12 20,04 27 45,09 42 70,14 57 95,19

3 5,01 13 21,71 28 46,76 43 71,81 58 96,86

4 6,68 14 23,38 29 48,43 44 73,48 59 98,53

5 8,35 15 25,05 30 50,1 45 75,15 60 100,2

6 10,02 16 26,72 31 51,77 46 76,82

7 11,69 17 28,39 32 53,44 47 78,49

8 13,36 18 30,06 33 55,11 48 80,16

9 15,03 19 31,73 34 56,78 49 81,83

10 16,7 20 33,4 35 58,45 50 83,5

21 35,07 36 60,12 51 85,17

22 36,74 37 61,79 52 86,84

23 38,41 38 63,46 53 88,51

24 40,08 39 65,13 54 90,18

25 41,75 40 66,8 55 91,85

LEVEL 2 LEVEL 3

CORRECT % % % % CORRECT %

1 1,67 11 18,37 26 43,42 41 68,47 56 93,52

2 3,34 12 20,04 27 45,09 42 70,14 57 95,19

3 5,01 13 21,71 28 46,76 43 71,81 58 96,86

4 6,68 14 23,38 29 48,43 44 73,48 59 98,53

5 8,35 15 25,05 30 50,1 45 75,15 60 100,2

6 10,02 16 26,72 31 51,77 46 76,82

7 11,69 17 28,39 32 53,44 47 78,49

8 13,36 18 30,06 33 55,11 48 80,16

9 15,03 19 31,73 34 56,78 49 81,83

10 16,7 20 33,4 35 58,45 50 83,5

21 35,07 36 60,12 51 85,17

22 36,74 37 61,79 52 86,84

23 38,41 38 63,46 53 88,51

24 40,08 39 65,13 54 90,18

25 41,75 40 66,8 55 91,85

LEVEL 3 LEVEL 4

Francesco Gratton

2013

1: Functional language assessed in a global manner

2: Structured interview

3: Tailored to the candidate

4: Checks & probes

5: 1 to 2 role plays

SPEAKING (SPEAKING (holistically assessed holistically assessed ))

PRODUCTIVE SKILLSPRODUCTIVE SKILLSPRODUCTIVE SKILLSPRODUCTIVE SKILLSProficiency Test 1.0Proficiency Test 1.0Proficiency Test 1.0Proficiency Test 1.0

Francesco Gratton

2013

Three tasks (one for each level)

PRODUCTIVE SKILLSPRODUCTIVE SKILLSPRODUCTIVE SKILLSPRODUCTIVE SKILLSJFLT 1.0JFLT 1.0JFLT 1.0JFLT 1.0

WRITING (WRITING (holistically assessed holistically assessed ))

Francesco Gratton

2013

Summary:Summary:

• Past situationPast situation

• New COAs adoptedNew COAs adopted

• A do-it-all softwareA do-it-all software

• ProposalsProposals

Francesco Gratton

2013

•Specifications Specifications

•Cut-off score Cut-off score

• Joint DatabaseJoint Database

New COAs adoptedNew COAs adopted

Francesco Gratton

2013

Francesco Gratton

2013

• PURPOSEPURPOSE• PURPOSEPURPOSE

• ADMINISTRATION PROCEDURESADMINISTRATION PROCEDURES• ADMINISTRATION PROCEDURESADMINISTRATION PROCEDURES

• VALIDATION PROCEDURESVALIDATION PROCEDURES• VALIDATION PROCEDURESVALIDATION PROCEDURES

• TEST FORMATTEST FORMAT• TEST FORMATTEST FORMAT

• LEVELS OF LINGUISTIC KNOWLEDGELEVELS OF LINGUISTIC KNOWLEDGE• LEVELS OF LINGUISTIC KNOWLEDGELEVELS OF LINGUISTIC KNOWLEDGE

• TEST CONTENTTEST CONTENT• TEST CONTENTTEST CONTENT

Francesco Gratton

2013

1: Multilevel test (level 1 to 4)

2: Multiple choice questions (60 for L & R)

3: No penalties for wrong answers

4: Duration: R 105’ / L 90’

5: Separate Sections (& levels)

6: # of correct answers multiplied by a coefficient

7: Potential use of “F” factor

RECEPTIVE SKILLS (Listening & Reading)RECEPTIVE SKILLS (Listening & Reading)

Stanag Proficiency Test 1.0 Stanag Proficiency Test 1.0 TWAS HIGH

TIME!!!

Francesco Gratton

2013

Stanag Proficiency Test 2.0Stanag Proficiency Test 2.0New Key-factors New Key-factors

• Each section is a mini-test (L & R)

• Plus levels

• Percentages

• Each section is a mini-test (L & R)

• Plus levels

• Percentages

Francesco Gratton

2013

•Specifications Specifications

•Cut-off score Cut-off score

• Joint DatabaseJoint Database

New COAs adoptedNew COAs adopted

Francesco Gratton

2013

Section 1 –Stanag level 1 (questions from 1 to 15)

No.Correct answ.

Level awarded

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

0 0+ 1Section 2 –Stanag level 2 (questions from 16 to 30)

No.Correct answ.

Level awarded

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

1 1+ 2Section 3 –Stanag level 3 (questions from 31 to 45)

No.Correct answ.

Level awarded

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

2 2+ 3

Section 4 –Stanag level 4 (questions from 46 to 60)

No.Correct answ.

Level awarded

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

3 3+ 4

JFLT 2.0: RDS (Listening & Reading)

Francesco Gratton

2013

EXAMPLE Level 1 (15 questions) Level 2 (15 questions) Level 3 (15 questions)

Max score: 45

Level 1 (15 questions) Level 2 (15 questions) Level 3 (15 questions)

Max score: 45

All candidates answer correctly to 30 questionsAll candidates answer correctly to 30 questionsAll candidates answer correctly to 30 questionsAll candidates answer correctly to 30 questions

Old test Old test Old test Old test WithWithWithWith they would all get thethey would all get thethey would all get thethey would all get the same scoresame scoresame scoresame score

Francesco Gratton

2013

CANDIDATO CORRECT ANSWERS

LEVEL 1(15 Questions)

LEVEL 2(15 Questions)

LEVEL 3(15 Questions) FINAL LEVEL

BIANCHI 30 10 10 10 3ROSSI 30 11 10 9 2+VERDI 30 13 12 5 2GIALLI 30 15 8 7 1+ARANCIONI 30 15 6 9 1

Francesco Gratton

2013

•Specifications Specifications

•Cut-off score Cut-off score

• Joint DatabaseJoint Database

New COAs adoptedNew COAs adopted

Francesco Gratton

2013

NEW LISTENING & READING ITEMSNEW LISTENING & READING ITEMS

JDB (JOINT DATA BASE)JDB (JOINT DATA BASE) JOINT EFFORTJOINT EFFORT

HOW NOT TONOT TO MAKE JOINT EFFORTSHOW NOT TONOT TO MAKE JOINT EFFORTS

Francesco Gratton

2013

Test-writers involvedTest-writers involved

• Accustomed to military environmentAccustomed to military environment

• Language Testing Seminar Language Testing Seminar

• QualifiedQualified

• Norming SessionsNorming Sessions

Francesco Gratton

2013

THE JDB FLOW

THE JDB FLOW

CARABINIERICARABINIERIARMYARMY

AIR FORCE

AIR FORCE

NAVYNAVY

Francesco Gratton

2013

SEP OCT NOV DEC JAN FEB MAR APR MAJ JUN JUL2012 2013

1st

Group

• PREP

ARATION O

F

FIRST

BATCH O

F

ITEM

S

•SENT T

O OTH

ER

GROUP

Phase1

TEST

ERS M

EET

FOR 1°

REVISI

ON

Phase2

PRE-

TEST

ING

REVISI

ON TRIA

LLIN

#

50 IT

EMS

(-30

%)

Phase3

FINAL

MODIFI

CATION

APPROVAZIO

NE

Phase4

AUGWG GdL GdL GdL

2nd

Group

FEBRUARY 201320 NEW ITEMS.

INTO JDB

•PREP

ARATION O

F

FIRST

BATCH O

F

ITEM

S

•SENT T

O OTH

ER

GROUP

Phase1

TEST

ERS M

EET

FOR 1°

REVISI

ON

Phase2

Phase3

PRE-

TEST

ING

REVISI

ON TRIA

LLIN

#

50 IT

EMS

(-30

%)

Phase4

FINAL

MODIFI

CATION

APPROVAZIO

NE

•PREP

ARATION O

F

FIRST

BATCH O

F

ITEM

S

•SENT T

O OTH

ER

GROUP

Phase1

TEST

ERS M

EET

FOR 1°

REVISI

ON

Phase2

Phase3

PRE-

TEST

ING

REVISI

ON TRIA

LLIN

#

50 IT

EMS

(-30

%)

3rd

Group

4th

Group •REP

ERIM

ENTO

• PREP

ARAZIONE

# 12

0 TTIV

AZIONI

•INVIO

AD A

LTRO

GRUPPO

Phase1

MAJ 201340 NEW ITEMS.

INTO JDB

NOVEMBER 201380 NEW ITEMS.

INTO JDB

TIMINGSTIMINGS

AUGUST 201360 NEW ITEMS.

INTO JDB

Francesco Gratton

2013

Summary:Summary:

• Past situationPast situation

• COA (specs, JDB, cut-off score)COA (specs, JDB, cut-off score)

• A do-it-all softwareA do-it-all software

• ProposalsProposals

Francesco Gratton

2013

WHAT’S THE DIFFERENCE ?WHAT’S THE DIFFERENCE ?

Francesco Gratton

2013

PC-assessment-related terminologyPC-assessment-related terminologyTerm Definition Stakes

Assessment Any systematic method of obtaining evidence (through questions) for a purpose.

Quiz … measures for the purpose of providing feedback to the student. Low

Survey … to determine needs required to fulfill a defined purpose. Low

Test … measures knowledge for the purpose of informing the student on their current level

Medium

Exam … measures knowledge for the purpose of documenting the current level of knowledge High

Francesco Gratton

2013

the software is used for:the software is used for:

• Needs analysis:(surveys) • Placement test• Any training activity• Assessment:

– First level survey – Post-course– Pre-certification – CertificationCertification

Francesco Gratton

2013

Software’s system Software’s system Schema Schema

Software’s system Software’s system Schema Schema

Create questions

And organize them in

tests using a windows

based PC

Francesco Gratton

2013

Assessment ……Assessment ……

via Browser

QuestionManager

AssessmentManager

AssessmentDefinitions

Questions

… allows to choose:

• Time limits

• Feedback to test-taker

• Styles (Template)

• Jumps

• Question shuffling

• Instructions to test-

takers

… allows to choose:

• Time limits

• Feedback to test-taker

• Styles (Template)

• Jumps

• Question shuffling

• Instructions to test-

takers

On Windows PC

QuestionManager

AssessmentManager

Francesco Gratton

2013

AssessmentsAssessments

• … also created with

authoring manager by

selecting Qs previously

created

• Any question can be

chosen from the

database

• … also created with

authoring manager by

selecting Qs previously

created

• Any question can be

chosen from the

database

AssessmentDefinitions

Questions

su PC Windows

QuestionManager

AssessmentManager

via Browser

QuestionManager

AssessmentManager

Francesco Gratton

2013

Software’s system Software’s system Schema Schema

Software’s system Software’s system Schema Schema

Create questions

And organize them in

tests using a windows

based PC

Set security parameters,

schedule assessment and link to

other (Learning

Management Systems

assessment published using any browser, secure browsers, or a PC/MAC

Result reports, CIA, graphs, gimmicks, you name it …..

Francesco Gratton

2013

Types of Questions: Multiple ChoiceTypes of Questions: Multiple ChoiceTypes of Questions: Multiple ChoiceTypes of Questions: Multiple Choice

Francesco Gratton

2013

Likert Scale (for questionnaires) Likert Scale (for questionnaires)

Francesco Gratton

2013

Essay QuestionEssay Question

1. Candidate can write free text in the space provided

2. Testers will evaluate later

1. Candidate can write free text in the space provided

2. Testers will evaluate later

Francesco Gratton

2013

Francesco Gratton

2013

Summary:Summary:

• Past situationPast situation

• COA (specs, JDB, cut-off score)COA (specs, JDB, cut-off score)

• A do-it-all softwareA do-it-all software

• Proposals Proposals

Francesco Gratton

2013

• Wide projectWide project

• Sharing experience & Sharing experience &

capabilitiescapabilities

• Optimization of resourcesOptimization of resources

• No alternative to B.A.T.No alternative to B.A.T.

Testing in the time Testing in the time of crisisof crisis

Francesco Gratton

2013

Francesco Gratton

2013

A Bilateral-based CDB

A multilateral-based CDB

COMBINED DATABASE

or …

or …

Francesco Gratton

2013

Francesco Gratton

2013

FLOWFLOW

TIME SCHEDULETIME SCHEDULE

SPECSSPECS1 2

3

Francesco Gratton

2013

Francesco Gratton

2013

Thank youThank you

francesco.gratton@gmail.comfrancesco.gratton@gmail.com

canutesting@sclingue.esercitocanutesting@sclingue.esercito

.difesa.it.difesa.it