Three strategies for assessment in autonomous language learning

Joan Jamieson, Northern Arizona University, USA

&

Carol A. Chapelle, Iowa State University, USA

Three Strategies

Adaptivity

Feedback

Self-assessment

An Adaptive Strategy

Learner would benefit from more than one form of material

Computer should select appropriate form based on responses to questions

Overview of LEA

Interest and

Ability Finder

Beginning Reading

Beginning Listening

Beginning Writing

Intermediate Reading

Intermediate Listening

Intermediate Writing

Advanced Reading

Advanced Listening

Advanced Writing

Results &

Recommendations

Results &

Recommendations

Results &

Recommendations

The Interest Survey

Select test form

Select recommendations

Items on Interest Survey

Text Use Scores I would like to improve my ability to Used for selecting

business vs. general tests

Understand popular materials in writing

0/1

Understand written technical materials about my business

If 1 then select business reading test

0/1

I live in a place where Used for selecting ESL vs. EFL strategies (on recommendations page/ default = EFL)

Most people speak English If 1 then select ESL strategies

0/1

Example Strategies

Beginning Listening - ESL

Beginning Listening - EFL

Phone an information line for a business or community service. Listen to the recorded message. Write what it is about. Go to a fast-food restaurant or a store. Stand where you can hear a cashier. Write down two questions the cashier asks customers.

Listen to the news on an English-speaking radio station and write down words or numbers you hear. Find an Internet story site. Listen to and read an interesting story. .

A Feedback Strategy

Learner benefits from total scores

Learner might benefit more from part scores

Example Computing Total Score

Vocabulary items

i t e m

answer

correct

value of SCORE

initial value SCORE = 0

1

confident

1

SCORE = 0 + 1

2

cheerful

1

SCORE = 1 + 1

3

exhausted

0

SCORE = 2 + 0

4

in a bad mood

1

SCORE = 2 + 1

Read the dialog. Drag and drop the answers into the blanks. tense relaxed confident in a bad mood exhausted cheerful Amy: Did you meet Sandy, the new receptionist? Isn't she great? Steve: I met her yesterday. She's energetic and hard-working. She also seems very

(1) ___________ that she can do her job well. Amy: I hope so. The last receptionist, Bob, wasn't as (2) _________ as Sandy is. He almost

never smiled. He always seemed to be (3) ____________

and nervous about all the work he had to do.

Steve: You're right. Something always put him (4) _________ and he was mad for the whole day. [Answers: 1. confident 2. cheerful 3. tense 4. in a bad mood]

TOTAL=SCORE/NUMBER OF ITEMS*100

75% = 3 / 4 * 100

Part Scores Reflect Subskills

Tests are often made up of subskills

Each item can be coded according to subskill

Scores for subskills can be computed by including codes

Table of Specifications

Skill # of items % of items Listening

10 25

Vocabulary

4 10

Speaking

6 15

Grammar 10 25

Pronunciation 6 15

Reading 4 10

# of items 40 % of items 100

Tags for LEO Tests

TAG What the TAG means

L listeningLIN listening for informationLID listening for ideasG grammarG1 grammar point 1G2 grammar point 2G3 grammar point 3S speakingV vocabularyR readingP pronunciationP1 pronunciation point 1P2 pronunciation point 2

Tags in Script for Grammar Section

Josh: Hi Kim. How are you doing?

Kim: I'm great! I just finished my science project. <tag> <G, G2> (1)_________ </tag> going to the movies?

Josh: It's <tag> <G, G3> (2)________ </tag> early to go the movies. <tag> <G, G2> (3)________ </tag> we go out to lunch instead? We can have pizza at Pizza Bob's.

Kim: I'm not hungry <tag> <G, G3> (4)________ </tag> to eat pizza. <tag> <G, G2> (5)_______ </tag> go to the Salad Spot instead.

Josh: How would we get there? It's too far <tag> <G, G3> (6) _________ </tag> and I don't have a car.

Kim: We can take the bus!

walking Why don't too Let's not to walk Let's How about enough

Using Tags with System Variables

“score” yields percentage correct

score (tag) yields percentage correct for any items with a given “tag”

score (G2) yields percentage correct of 2nd point of grammar—expressions for suggesting

Combining Tags and System Variables

score (L | G | V | S | P | R)

n/m1= “rawscore(LIN) / tqw(LIN)”  n/m2= “rawscore(LID) / tqw(LID)” n/m3= “rawscore(G1) / tqw(G1)” n/m4= “rawscore(G2) / tqw(G2)”

n/m5= “rawscore(G3) / tqw(G3)”

Mock-up of Progress Report Screen

Progress Report LEO 3 Test

Learner’s name:Score: score (L | G | V | S | P | R)

Language area Number correct/number of items

Listening for information n/m1

Listening for ideas n/m2

Grammar (point1*) n/m3

Grammar (point2*) n/m4

Grammar (point3*) n/m5

Screen Shot of Progress Report

Using Tags to Report Scores

TAG What the TAG means Possible scores

L R V

C

listening reading vocabulary comprehension

Total Reading Total Reading Vocabulary Reading Comprehension Listening Total Listening Vocabulary Listening Comprehension Vocabulary—reading and

listening Comprehension—reading and

listening

A Self-Assessment Strategy

Learner may benefit by comparing his/her perspective of performance with score

Computer can collect self-confidence data along with performance data

Example of Self-Confidence Item

Was your answer correct? How sure are you? Click a circle below .

Completely Not sure at allsure

Superimposed Self-Assessment Item

Was your answer correct? How sure are you? Click in a circle for each answer.

1.

2.

3.

Completely Not sure sure at all

Computing Average Confidence (Tarone and Yule, 1989)

Circle clicked 5 4 3 2 1 total average confidence

correct answers 20 5 3 2 0 29 4.52incorrect answers 0 0 4 5 2 11 2.00  

(20*5)+(5*4)+(3*3)+(2*2)+(0*1)/29 = 4.52 

(4*3)+(5*2)+(2*0)/11 = 2.00

Tarone, E., & Yule, G. (1989). Focus on the language learner. Oxford, UK: Oxford University Press.

Computing Self-Monitoring Index

Derived by subtracting self-confidence rating on incorrect items from self-confidence rating on correct items:

4.52 – 2.00 = 2.52

Index ranges in value from 4 to - 4

Messages could be provided instead of numbers

Self-Assessment Superimposed onto Progress Report

Self-Assessment: You seem to be aware of your own ability. When you gave the correct answer, you were very sure you were correct. When you gave the wrong answer, you were not too sure you were correct.

Implementing Self-Assessment

Tag self-assessment items <SA>

Save value of “rawscore (SA)” separately for correct and incorrect items: IF ANSWER = 1 THEN SAOK = SAOK + rawscore (SA)

IF ANSWER = 0 THEN SANO = SANO + rawscore (SA)

Calculating Average Scores

AVGSAOK = SAOK / # CORRECT ITEMS

AVGSANO = SANO / # INCORRECT ITEMS

MONITORING INDEX = AVGSAOK-AVGSANO

Example of Computing Self-Assessment Scores

Item answer correct value of SCORE

value of SA value of SAOK value of SANO

initial value SCORE =0

initial value SA =0

initial value SAOK =0

initial value SANO =0

1

confident

1

SCORE = 0 + 1

5

SAOK = 0 + 5

2

cheerful

1

SCORE = 1 + 1

4

SAOK = 5 + 4

3

exhausted

0

SCORE = 2 + 0

1

SANO = 0 + 1

4

in a bad mood

1

SCORE = 2 + 1

4

SAOK = 9 + 4

AVGSAOK = SAOK / # CORRECT 4.3= 13 / 3

AVGSANO = SANO / # INCORRECT 1.0 = 1 / 1

TOTAL = SCORE / NUMBER OF ITEMS * 100

75% = 3 / 4 * 100

MONITORING INDEX = 4.3 – 1.0 = 3..3

Three Strategies for Individualizing Assessment

Adapting level, content, and recommendations based on learner’s responses

Additional feedback in the form of diagnostic scores

Self-assessment to heighten learner’s metacognitive awareness

Three strategies for assessment in autonomous language learning

Joan Jamieson, Northern Arizona University, USA

&

Carol A. Chapelle, Iowa State University, USA