Computerized Adaptive Testing: What is it and How Does it
Work?
Slide 2
Goals of this session Learn about Computerized Adaptive Testing
(CAT) Review Item Response Theory (IRT) Combining CAT with IRT Pros
and cons of CAT Answer questions
Slide 3
Not to be confused with Computerized Adaptive Testing: Not as
cute, but far fewer hairballs.
Slide 4
PART I Introduction to CAT
Slide 5
Motivation for Understanding CAT There are already operational
assessments that use CAT Some believe it will revolutionize
classroom testing in the future Interesting idea that speaks to
potential of computers to have new uses in education Item Response
Theory is all over testing now
Slide 6
OK, so what is CAT? A type of assessment where a question is
displayed on a monitor Students use mouse to select answer Computer
chooses next question based on previous responses Next question is
displayed on monitor, or else test ends
Slide 7
A graphical representation Questions chosen depend on prior
responses
Slide 8
Analogy: A Game of 20 Questions I am thinking of an object. You
have 20 yes-or-no questions to figure it out. Would you write out
all your questions ahead of time? 1) Is it an animal? 2) Is it a
vegetable? 3) Is it blue? 4) Is it red? 5) Is it bigger than a car?
6) Etc.
Slide 9
20 Questions, Continued Isnt it more effective to base your
next question on previous answers? 1) Is it an animal? NO. 2) Is it
a vegetable? YES. 3) Is it commonly found in a salad? YES. 4) Is it
green? NO. 5) Would Bugs Bunny eat it? YES.
Slide 10
Same principle used in CAT Computer keeps track of each
students pattern of responses so far As test progresses, learn more
about individual student Choose next question (item) to get maximal
info about that particular students level of ability Purpose of
assessment: Get best possible information about students
Slide 11
Some items are more informative than others? Sure! Some items
are easier than others: 2 + 2 vs. 54389 + 34697 Some items are more
relevant than others: 3 + 7 vs. Academy Awards question Some items
are better at discerning proficient students from those who need
improvement
Slide 12
Which is most informative? Suppose we have only 2 types of
students: Advanced and Beginning Use the test to classify each
student Which item below is the best for this purpose?
ItemP(Correct|Advanced)P(Correct|Beginning) 152% 275%34%
3100%0%
Slide 13
Item 3 is the best Item 1 is completely useless Item 2 gives
some information Item 3 is all you need!
ItemP(Correct|Advanced)P(Correct|Beginning) 152% 275%34%
3100%0%
Slide 14
But wait Wouldnt we choose Item 3 for ALL students? If so, why
customize a test for an individual student? Answer: For some
students, Item A is more informative. For others, Item B is more
informative.
Slide 15
When is one item more informative than another? Item A: 2 + 2
Item B: (34 + 68) / 2 If youve answered many difficult items
correctly, Item A is waste of time If youve answered many easy
items incorrectly, Item B is too hard Thus, give Item B to
high-performing students, Item A to low-performing students
Slide 16
Isnt that unfair? It seems like CAT penalizes students for
performing well at start If we give different items to different
students, how can we compare their performances? The above question
arises whether we use CAT or not Item Response Theory to the
rescue!
Slide 17
Summary of Part I CAT customizes assessment based on previous
responses, as in 20 Questions Certain items more informative than
others For some students, Item A is more informative; for others,
Item B is When give different items to different students, need way
to relate student performances (Item Response Theory)
Slide 18
PART II Review of Item Response Theory
Slide 19
Item Response Theory (IRT) Quantifies the relation between
examinees and test items For each item, gives probability of
correct response by ability level Provides a means for describing
characteristics of items, estimating ability of examinees Places
examinees on common scale when they have taken different items
Slide 20
The IRT Model: One item
Slide 21
Different items have different curves
Slide 22
Where did those curves come from? In IRT, ability is denoted by
Probability of a correct response is Each item has its own values
of a, b, and c. We know them from field testing a is the
discrimination: Related to the slope b is the difficulty: Harder
item, higher b c is the guessing parameter: Chance of lucky
guess
Slide 23
Effect of the a parameter All curves shown have equal b and c
parameters Larger a increases the slope in the middle
Slide 24
Effect of the b parameter All curves shown have equal a and c
parameters Larger b means harder item
Slide 25
Effect of the c parameter All curves shown have equal a and b
parameters c is the left asymptote
Slide 26
Wait a minute What do you mean by a student with an ability of
1.0? Does an ability of 0.0 mean that a student has NO ability?
What if my student has a reading ability of -1.2? What in the world
does that mean???
Slide 27
The ability scale Ability is on an arbitrary scale that just
happens to be centered around 0.0 We use arbitrary scales all the
time: Fahrenheit Celsius Decibels Nevertheless, need more
user-friendly reporting: scaled scores on conventional scale like
200-300
Slide 28
Giving a score for each student First assign an ability ()
value to each student (say, -4 to 4) Student is given the value of
that is most consistent with his/her responses The better he/she
does on the test, the higher the value of that he/she receives
Computer converts the score to a scaled score Report final
score!
Slide 29
Assigning scores Set of answers: (C,C,I,C,C,I,I,C,C,C,I,C,C) We
know which items were taken by each student: a, b, c parameters If
Student 1s items were harder than Student 2s, take into account
through item parameters Student 1: = 1.25, scaled score = 290
Student 2: = 0.65, scaled score = 268 Can compare students who took
different items!!!
Slide 30
Summary of Part II If you didnt get all that, dont worry Just
remember: In IRT, different items have different curves (depending
on a, b, c parameters) IRT allows us to give scores on the same
scale, even when students take different items These features
critical in CAT So how do we choose which items to give?
Slide 31
PART III Combining CAT with IRT
Slide 32
CAT Reminder CAT customizes assessment based on previous
responses For some students, Item A is more informative; for
others, Item B is With IRT, its OK to give different items to
different students
Slide 33
Which item would you choose next? PREVIOUS RESPONSES: 10 + 19 =
? Answered correctly. 27 + 38 = ? Answered incorrectly. 12 + 26 = ?
Answered incorrectly. POSSIBLE ITEMS TO GIVE NEXT: 18 + 9 = ? 13 +
17 = ? 14 + 20 = ?
Slide 34
Item selection to match ability/difficulty Want to give items
appropriate to ability 2 + 2 is not informative for high-performing
students; (34 + 68) / 2 is not informative for low- performing
students Student has taken 10 items, awaits 11th Classic approach:
Give item whose difficulty (b) is closest to current ability
estimate ()
Slide 35
Which item is better for = -1.2? Easier item Harder item
Slide 36
More complex item selection Previous method: Match difficulty
to ability This criterion only uses b parameter and Recall that a
parameter is related to slope, c is guessing parameter Shouldnt we
consider those when choosing next item?
Slide 37
Another item selection method Ideal item: High value of a;
value of b close to ; low value of c Fisher Information combines
these factors into a single number Choose item with highest Fisher
Info
Slide 38
Game: Which item would you choose? Suppose our current estimate
of is 0.6 Itemabc 10.8-0.20.25 21.10.40.15 31.02.20.18
Slide 39
Results If matching ability estimate (0.6) with difficulty, we
would give Item 2 If using Fisher Info, we would give Item 2
Itemabc 10.8-0.20.25 21.10.40.15 31.02.20.18
Slide 40
Round 2 Suppose our current estimate of is 0.7 Itemabc
11.30.90.20 21.10.60.22 30.8-0.10.10
Slide 41
Round 2 Results If matching ability estimate (0.7) with
difficulty, we would give Item 2 If using Fisher Info, we would
give Item 1 Itemabc 11.30.90.20 21.10.60.22 30.8-0.10.10
Slide 42
Summary of Part III Tailor items to be most informative about
individual students ability Do this by combining CAT with IRT One
method: Match difficulty with current estimate of Another method:
Take all parameters into account via Fisher Info
Slide 43
PART IV Practical Considerations
Slide 44
Problem: Content Balance In operational testing, must balance
content (e.g., math test of algebra, geometry, number sense) What
if all your most informative items come from the same content
strand? In practice, dozens of constraints for each CAT: Content,
topics, enemies list, etc. CAT solution: Pick most informative item
among those in play
Slide 45
Problem: Test security CAT administered on multiple occasions
Person A takes exam, memorizes items, tells Person B. Person B
takes exam, benefits from Person As information Different students,
different items; however, some items more popular than others CAT
solution: Limit the amount each item can be administered
Slide 46
CAT Pros Convenient administration Immediate scoring Items
maximally informative: Exams just as accurate, with shorter tests
Items at correct level: High-performing students not bored,
low-performing students not overwhelmed
Slide 47
CAT Cons Limited by technology Potential bias versus students
with less computer experience Content balance less exact than
paper-and- pencil testing Test security Expensive
Slide 48
Final summary Introduction to CAT: Benefits of giving different
items to different students Review of IRT Using IRT to select items
in a CAT Pros and cons of CAT
