1
An Investigation of The Response An Investigation of The Response Time for Maths Items in A Computer Time for Maths Items in A Computer
Adaptive Test Adaptive Test
C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK
Chris Wheadon and Qingping HeChris Wheadon and Qingping He
((CEM Centre, Durham University, UKCEM Centre, Durham University, UK))
2
Background
• The CEM Centre’s Baseline Assessments on Primary and Secondary School Students
• The CEM Baseline Tests for Secondary Schools (for students aged 11-18: three separate projects) Paper based, Efficiency, Cost
• Use of CTT and IRT in Computerised Testing and to Interpret Test Data
• To Develop A Computerised Test to Replace the Existing Paper-based Baseline Tests for the Three Secondary Projects
C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK
3
• CABT comprises a calibrated item bank and a question
display system
• The adaptive test contains an adaptive maths test and
an adaptive English vocabulary test
• Testing is delivered through the Web or from the
school’s local area network
CABT-The CEM Computer Adaptive Baseline Test
C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK
4
Establishing A Calibrated Item Bank
C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK
• Testing of items by school students from different year groups through the administration of a series of tests containing common items
• Calibration of items in each test using the Rasch model (items meeting the Rasch requirements are used)
• Setting of the reference test and equating of different tests using the Rasch model and common items
• Further new item calibration through the embedding of new items in the adaptive test
5
Developing The Question Display System - Conducting Adaptive Testing
C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK
• Realisation of adaptivity is achieved through the implementation in the question display system of the Rasch model for ability estimation (MLE method is used) and question selection
• Rules for stopping test: minimum number of question; maximum number of questions; and convergence value
• Variable starting difficulty for the first question for different year groups.
6
C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK
Preliminary Results from theAdaptive Maths Test
• There are over 500 items in the maths item bank
• Effort has been made to make the items as curriculum-free as possible
• Items are content-independent to each other
• Item types include MCQ, short free text entry questions and interactive questions
• Items cover a wide range of difficulties in order for the three secondary projects (for three different year groups) to use the same adaptive test as their baseline test
7
70
80
90
100
110
120
130
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Item Sequence Number
Sc
ale
d A
bili
ty
0
20
40
60
80
100
120
140
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Item Sequence Number
Tim
e T
aken
(s)
C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK
Individual Student: The Distribution of Ability and the Distribution of Item Response Time
8
80
85
90
95
100
105
110
115
120
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Item Sequence Number
Sc
ale
d A
bili
ty
0
20
40
60
80
100
120
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Item Sequence Number
Tim
e T
aken
(s)
C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK
Individual Student: The Distribution of Ability and the Distribution of Item Response Time
9
-8.00 -6.00 -4.00 -2.00 0.00 2.00 4.00 6.00
Item Difficulty (logits)
0
50
100
150
Re
sp
on
se
Tim
e (
s)
C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK
All Items: The Effect of Item Difficulty on Response Time
10
40
60
80
100
120
140
160
180
200
2202
39
9
67
2
22
99
58
22
8
64
1
61
2
47
5
23
60
22
65
62
6
63
9
64
9
22
66
26
9
62
4
65
4
57
3
21
24
22
81
49
3
23
34
21
22
23
52
Item Id
Re
sp
on
se
Tim
e (
s)
Year 7Year 10Year 12
C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK
A Selection of Items: The Effect of Age of Test Takers on Item Response Time
11
40
60
80
100
120
140
160
180
2359 67
0
2300 33
4
478
490 58 32
648
2360 64
9
2265 51
0
493
2270 64
1
733
228
504
2266 53
626 71
658 39
Item Id
Res
po
nse
Tim
e (s
)
Correct
Incorrect
C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK
A Selection of Items: The Relationship between Response Time for Correct Answers and Response Time for Incorrect Answers
12
Q.631 Understanding a simple algebraic relationship. Difficulty: -0.6 logits.
The table represents a relationship between x and y. What is the missing number in the table?
a) 9 b) 10 c) 11 d) 12 e) 13
X Y 2 5 3 7 4 ? 7 15
C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK
Individual Item: The Effect of Ability and Age on Response Time (Difficulty: -0.6 logits) – Easy Item
13
Correct Answers
3.1
3.2
3.3
3.4
3.5
3.6
3.7
3.8
3.9
<= -.50
-.49 --.20
-.19 -.10
.11 -.50
.51 -1.27
1.28 -2.65
2.66+
Ability Band (logits)
Sca
led
Res
po
nse
Tim
e
Year 7Year 10Year 12
Incorrect Answers
3
3.2
3.4
3.6
3.8
4
4.2
<= -1.05
-1.04- -.70
-.69 --.45
-.44 --.21
-.20 -.08
.09 -.79
.80+
Ability Band (logits)
Sca
led
Res
po
nse
Tim
e
Year 7Year 10Year 12
C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK
14
<= -.56 -.55 - -.28 -.27 - .05 .06 - .45 .46+
Ability Measure (Banded, logits)
2
3
4
5
6S
ca
led
Re
sp
on
se
Tim
e
C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK
Ability Bands within a Year Group for the Specific Item (Year 7)
15
C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK
Q .490 C om paring tw o fractions. D ifficu lty : 1 .1 log its .
C om pare the tw o exp ress ions:
E xpress ion A : 5
1817161514
E xpress ion B : 4
20191817
E xpress ion A is g rea te r than express ion B E xpress ion B is g rea te r than express ion A The express ions a re equa l
Individual Item: The Effect of Ability and Age on Response Time (Difficulty: 1.1 logits) – Medium-difficulty Item
16
Correct Answers
3.9
4
4.1
4.2
4.3
4.4
4.5
4.6
4.7
4.8
4.9
<=1.73
1.74 -1.98
1.99 -2.19
2.20 -2.41
2.42 -2.80
2.81 -3.64
3.65+
Ability Band (logits)
Sca
led
Res
po
nse
Tim
e
Year 7Year 10Year 12
Incorrect Answers
3
3.2
3.4
3.6
3.8
4
4.2
4.4
4.6
4.8
<=1.15
1.16 -1.45
1.46 -1.65
1.66 -1.82
1.83 -2.06
2.07 -2.44
2.45+
Ability Band (logits)
Sca
led
Res
po
nse
Tim
e
Year 7Year 10Year 12
C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK
17
<= 1.82 1.83 - 2.09 2.10 - 2.38 2.39 - 3.05 3.06+
Ability Measure (Banded, logits)
2
3
4
5
6
7
Sca
led
Re
sp
on
se
Tim
e
C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK
Ability Bands within a Year Group for the Specific Item (Year 10)
18
Q.2359 Reading a pie chart, working with fractions. Difficulty: 2.9 logits.
The pie chart represents the different colours of cars in Albert Street. If there are 144 cars in total, how many are blue (segment z)?
Free response answer.
1/6
1/9
2/3
z
C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK
Individual Item: The Effect of Ability and Age on Response Time (Difficulty: 2.9 logits) – Difficult Item
19
Correct Answers
4.2
4.4
4.6
4.8
5
5.2
5.4
5.6
<=2.14
2.15 -2.92
2.93 -3.49
3.50 -4.05
4.06 -4.80
4.81 -5.78
5.79+
Ability Band (logits)
Scale
d R
esp
on
se T
ime
Year 7Year 10Year 12
Incorrect Answers
3
3.5
4
4.5
5
5.5
<=.24
.25 -.99
1.00 -1.57
1.58 -2.05
2.06 -2.51
2.52 -3.29
3.30+
Ability Band (logits)
Scale
d R
esp
on
se T
ime
Year 7Year 10Year 12
C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK
20
<= 1.82 1.83 - 2.09 2.10 - 2.38 2.39 - 3.05 3.06+
Ability Measure (Banded, logits)
2
3
4
5
6
7
Sc
ale
d R
es
po
ns
e T
ime
C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK
Ability Bands within a Year Group for the Specific Item (Year 12)
21
Conclusions and Further Work
• Response time for all the maths items in the test generally increases with item difficulty but shows great variability
• Item difficulty levels and the age and ability of test takers have significant influence on item response time
• The information obtained in this study can be used for constructing more efficient tests
• Further work will involve investigating the relationship between CABT results and students’ subsequent academic performance
C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK