Determining a Hierarchy of Correctness Through Student Transitions on the FMCE Kyle J. Louis 1,2 , Bartholomew J. Ricci 1,3 , and Dr. Trevor I. Smith 1,2 Department of 1 Physics & Astronomy, 2 STEAM Education, and 3 Mathematics, Rowan University, Glassboro, NJ, 08028 Unified Ranking Hypotheses H 0 : The number of transitions from one answer choice to another is the same in both directions H a : More students transition in one direction between two answer choices than in the other Background Purpose: To determine a hierarchy of “correctness” on a commonly used multiple-choice assessment in introductory physics courses. • Data were collected from over 7,000 students • We omit J as an answer choice because it provides no understanding of a student’s understanding. References 1. R. K. Thornton and D. R. Sokoloff, Am. J. Phys. 66, 338 (1998). 2. T. I. Smith, K. A. Gray, K. J. Louis, B. J. Ricci, and N. J. Wright, PERC Proceedings, p. 380 (2017). 3. Y. Suh and D. M. Bolt, Psychometrika 75, 454 (2010). 4. R. D. Bock, Psychometrika 37, 29 (1972). 5. Q. McNemar, Psychometrika 12, 153 (1947). 6. A. H. Bowker, J. Am. Stat. Assoc. 48, 572 (1948). Acknowledgements We thank Sam McKagan and Ellie Sayre for providing access to data from PhysPort's Data Explorer. We also thank Kerry Gray, Nicholas Wright, Ian Griffin, and Ryan Moyer for their previous contributions as members of the research team. Time F o r c e + - 0 Time F o r c e + - 0 Time A B C F o r c e + - 0 F o r c e + - 0 Time F o r c e + - 0 Time F o r c e + - 0 Time D E F G F o r c e Time F o r c e + - 0 Time H + - 0 The car moves toward the right and is slowing down at a steady rate (constant acceleration). __18. Force and Motion Concept Evaluation [1] Assume that friction is so small that it can be ignored. A force is applied to the car. Choose the one force graph ( A through H ) that for each statement below which could allow the described motion of the car. 0 + are described below. Choose the letter (A Future Research • Determining the models for each answer choice via interviews • Synthesize results into a unified ranking system • Use additional analyses to rank responses with different assumptions • Assumption: Students are more likely to choose more sophisticated responses after instruction than before instruction. • Using the False Discovery Rate (FDR) correction, the adjusted p -value determines whether or not a transition is statistically significant [5,6] Q18 A B C D E F G H A 4 28 2 9 0 2 4 26 B 14 717 6 33 5 4 12 87 C 7 37 6 10 1 4 12 46 D 16 208 13 87 10 14 63 236 E 1 11 1 4 2 1 0 8 F 10 45 3 16 2 16 22 72 G 22 250 13 59 7 21 93 281 H 92 1420 52 227 26 72 281 1904 Posttest P r e t e s t Response Comparison Adjusted p -value Percent of Population B > H < 0.001 22.3% B > G < 0.001 3.9% B > D < 0.001 3.6% A > H < 0.001 1.7% B > F < 0.001 0.7% B > C < 0.001 0.6% E > H 0.01 0.5% A > G 0.002 0.4% Statistically Significant Transitions Response Comparison Adjusted p -value Percent of Population G = H 1 8.3% D = H 1 6.9% D = G 1 1.8% Statistically Insignificant Transitions • Assumption: Students who choose correct responses on most questions are more likely to choose more sophisticated incorrect answers than students who choose few correct responses [2] • 2-parameter nested logit model for a multiple choice test [3,4] • a : related to slope of the item response curve (IRC) • Discriminates student understanding • b : Difficulty of the question • Right shift means harder question • Left shift means easier question • Correctness of answer choices determined by higher a k value Question 18 Item Response Curve θ P(θ) 0.0 0.2 0.4 0.6 0.8 1.0 −3 −2 −1 0 1 2 3 A B C D E F G H Item Response Theory B > {H, G, D, F, C}; D = G = H A > {H, G}; E > H correct most common also negative slope best incorrect (constant magnitude) worst incorrect (maybe) McNemar-Bowker Chi-Square Test for Asymmetry Incorrect: P k (✓ )= ⇣ 1 - 1 1+e -a(✓ -b) ⌘ e a k ( ✓ -b k ) P i e a i ( ✓ -b i ) Correct: P (✓ )= 1 1+e -a(✓-b) B > A > D = G = H > C > F a k : 0 > -1.43 = -1.47 = -1.52 > -1.87 > -2.11 B > A > D = G = H
Transcript
Determining a Hierarchy of Correctness Through Student Transitions on the FMCEKyle J. Louis 1,2, Bartholomew J. Ricci 1,3, and Dr. Trevor I. Smith 1,2
Department of 1 Physics & Astronomy, 2 STEAM Education, and 3 Mathematics, Rowan University, Glassboro, NJ, 08028
Unified Ranking
HypothesesH0: The number of transitions from one answer choice to another is the same in both directions
Ha: More students transition in one direction between two answer choices than in the other
BackgroundPurpose: To determine a hierarchy of “correctness” on a commonly used multiple-choice assessment in introductory physics courses.• Data were collected from over 7,000 students• We omit J as an answer choice because it provides no
understanding of a student’s understanding.
References1. R. K. Thornton and D. R. Sokoloff, Am. J. Phys. 66, 338 (1998).2. T. I. Smith, K. A. Gray, K. J. Louis, B. J. Ricci, and N. J. Wright, PERC
Proceedings, p. 380 (2017).3. Y. Suh and D. M. Bolt, Psychometrika 75, 454 (2010).4. R. D. Bock, Psychometrika 37, 29 (1972).5. Q. McNemar, Psychometrika 12, 153 (1947). 6. A. H. Bowker, J. Am. Stat. Assoc. 48, 572 (1948).
AcknowledgementsWe thank Sam McKagan and Ellie Sayre for providing access to data from PhysPort's Data Explorer. We also thank Kerry Gray, Nicholas Wright, Ian Griffin, and Ryan Moyer for their previous contributions as members of the research team.
Time
F o r c e
+
-
0 Time
F o r c e
+
-
0 Time
F o r c e
+
-
0 Time
F o r c e
+
-
0 Time
F o r c e
+
-
0 Time
A B C D E F G F
o r c e
Time
F o r c e
+
-
0
Questions 14-21 refer to a toy car which can move to the right or left along a horizontal line (the positive part of the distance axis).
+0Assume that friction is so small that it can be ignored.
You may use a choice more than once or not at all. If you think that none is correct, answer choice .J
A force is applied to the car. Choose the one force graph ( through ) for each statement below which could allow the described motion of the car to continue.
A H
The car moves toward the right (away from the origin) with a steady (constant) velocity.
__14.
The car is at rest.__15.The car moves toward the right and is speeding up at a steady rate (constant acceleration).
__16.
The car moves toward the left (toward the origin) with a steady (constant) velocity.
__17.
The car moves toward the right and is slowing down at a steady rate (constant acceleration).
__18.
The car moves toward the left and is speeding up at a steady rate (constant acceleration).
__19.
The car moves toward the right, speeds up and then slows down.
__20.
The car was pushed toward the right and then released. Which graph describes the force after the car is released.
__21.
None of these graphs is correct.J
F o r c e
+
-
0 Time
H
+
-
0
Time
F o r c e
+
-
0 Time
F o r c e
+
-
0 Time
F o r c e
+
-
0 Time
F o r c e
+
-
0 Time
F o r c e
+
-
0 Time
A B C D E F G F
o r c e
Time
F o r c e
+
-
0
Questions 14-21 refer to a toy car which can move to the right or left along a horizontal line (the positive part of the distance axis).
+0Assume that friction is so small that it can be ignored.
You may use a choice more than once or not at all. If you think that none is correct, answer choice .J
A force is applied to the car. Choose the one force graph ( through ) for each statement below which could allow the described motion of the car to continue.
A H
The car moves toward the right (away from the origin) with a steady (constant) velocity.
__14.
The car is at rest.__15.The car moves toward the right and is speeding up at a steady rate (constant acceleration).
__16.
The car moves toward the left (toward the origin) with a steady (constant) velocity.
__17.
The car moves toward the right and is slowing down at a steady rate (constant acceleration).
__18.
The car moves toward the left and is speeding up at a steady rate (constant acceleration).
__19.
The car moves toward the right, speeds up and then slows down.
__20.
The car was pushed toward the right and then released. Which graph describes the force after the car is released.
__21.
None of these graphs is correct.J
F o r c e
+
-
0 Time
H
+
-
0
Time
F o r c e
+
-
0 Time
F o r c e
+
-
0 Time
F o r c e
+
-
0 Time
F o r c e
+
-
0 Time
F o r c e
+
-
0 Time
A B C D E F G F
o r c e
Time
F o r c e
+
-
0
Questions 14-21 refer to a toy car which can move to the right or left along a horizontal line (the positive part of the distance axis).
+0Assume that friction is so small that it can be ignored.
You may use a choice more than once or not at all. If you think that none is correct, answer choice .J
A force is applied to the car. Choose the one force graph ( through ) for each statement below which could allow the described motion of the car to continue.
A H
The car moves toward the right (away from the origin) with a steady (constant) velocity.
__14.
The car is at rest.__15.The car moves toward the right and is speeding up at a steady rate (constant acceleration).
__16.
The car moves toward the left (toward the origin) with a steady (constant) velocity.
__17.
The car moves toward the right and is slowing down at a steady rate (constant acceleration).
__18.
The car moves toward the left and is speeding up at a steady rate (constant acceleration).
__19.
The car moves toward the right, speeds up and then slows down.
__20.
The car was pushed toward the right and then released. Which graph describes the force after the car is released.
__21.
None of these graphs is correct.J
F o r c e
+
-
0 Time
H
+
-
0
Time
F o r c e
+
-
0 Time
F o r c e
+
-
0 Time
F o r c e
+
-
0 Time
F o r c e
+
-
0 Time
F o r c e
+
-
0 Time
A B C D E F G F
o r c e
Time
F o r c e
+
-
0
Questions 14-21 refer to a toy car which can move to the right or left along a horizontal line (the positive part of the distance axis).
+0Assume that friction is so small that it can be ignored.
You may use a choice more than once or not at all. If you think that none is correct, answer choice .J
A force is applied to the car. Choose the one force graph ( through ) for each statement below which could allow the described motion of the car to continue.
A H
The car moves toward the right (away from the origin) with a steady (constant) velocity.
__14.
The car is at rest.__15.The car moves toward the right and is speeding up at a steady rate (constant acceleration).
__16.
The car moves toward the left (toward the origin) with a steady (constant) velocity.
__17.
The car moves toward the right and is slowing down at a steady rate (constant acceleration).
__18.
The car moves toward the left and is speeding up at a steady rate (constant acceleration).
__19.
The car moves toward the right, speeds up and then slows down.
__20.
The car was pushed toward the right and then released. Which graph describes the force after the car is released.
__21.
None of these graphs is correct.J
F o r c e
+
-
0 Time
H
+
-
0
Force and Motion Concept Evaluation [1]Assume that friction is so small that it can be ignored. A force is applied to the car. Choose the one force graph (A through H) that for each statement below which could allow the described motion of the car.
Questions 22-26 refer to a toy car which can move to the right or left on a horizontal surface along a straight line (the + distance axis). The positive direction is to the right.
0 + Different motions of the car are described below. Choose the letter (A to G) of the acceleration-time graph which corresponds to the motion of the car described in each statement. You may use a choice more than once or not at all. If you think that none is correct, answer choice J.
E
F
G +
-
o
A c c e l
Time
+
-
o
A c c e l
Time
+
-
o
A c c e l
Time
A
B
C
D
+
-
o
A c c e l
Time
+
-
o
A c c e l
Time
+
-
o
A c c e l
Time
+
-
o
A c c e l
Time
None of these graphs is correct.J
_____22. The car moves toward the right (away from the origin), speeding up at a steady rate. _____23. The car moves toward the right, slowing down at a steady rate. _____24. The car moves toward the left (toward the origin) at a constant velocity. _____25. The car moves toward the left, speeding up at a steady rate. _____26. The car moves toward the right at a constant velocity.
Future Research• Determining the models for each answer choice via
interviews• Synthesize results into a unified ranking system• Use additional analyses to rank responses
with different assumptions
• Assumption: Students are more likely to choose more sophisticated responses after instruction than before instruction.
• Using the False Discovery Rate (FDR) correction, the adjusted p-value determines whether or not a transition is statistically significant [5,6]
Q18 A B C D E F G HA 4 28 2 9 0 2 4 26B 14 717 6 33 5 4 12 87C 7 37 6 10 1 4 12 46D 16 208 13 87 10 14 63 236E 1 11 1 4 2 1 0 8F 10 45 3 16 2 16 22 72G 22 250 13 59 7 21 93 281H 92 1420 52 227 26 72 281 1904
Posttest
Pretest
Response Comparison Adjusted p-value Percent of
PopulationB > H < 0.001 22.3%B > G < 0.001 3.9%B > D < 0.001 3.6%A > H < 0.001 1.7%B > F < 0.001 0.7%B > C < 0.001 0.6%E > H 0.01 0.5%A > G 0.002 0.4%
Statistically Significant Transitions
Response Comparison Adjusted p-value Percent of
PopulationG = H 1 8.3%D = H 1 6.9%D = G 1 1.8%
Statistically Insignificant Transitions
• Assumption: Students who choose correct responses on most questions are more likely to choose more sophisticated incorrect answers than students who choose few correct responses [2]
• 2-parameter nested logit model for a multiple choice test [3,4]
• a: related to slope of the item response curve (IRC)• Discriminates student understanding
• b: Difficulty of the question• Right shift means harder question• Left shift means easier question
• Correctness of answer choices determined by higher ak value
Question 18 Item Response CurveTrace lines for Question 18
θ
P(θ)
0.0
0.2
0.4
0.6
0.8
1.0
−3 −2 −1 0 1 2 3
A
B
C
D
E
F G
H
Item Response Theory
B > {H, G, D, F, C}; D = G = H A > {H, G}; E > H
correct
most common
also negative slope
best incorrect(constant magnitude)
worst incorrect (maybe)
McNemar-Bowker Chi-Square Test for Asymmetry
Incorrect: Pk(✓) =⇣1� 1
1+e�a(✓�b)
⌘eak(✓�bk)Pi
eai(✓�bi)
Correct: P (✓) = 11+e�a(✓�b)
B > A > D = G = H > C > Fak : 0 > �1.43 = �1.47 = �1.52 > �1.87 > �2.11
