Item Analysis
Parames Laosinchai11 July 2019
“Mathematics reveals its secrets only to those who approach it with pure love, for its own
beauty.”
2
Archimedes
Outline1) Distractor analysis
2) Difficulty and discriminability
3) Validity
3
Distractor Analysis Form kahoot.it to Excel What does the Excel sheet tell us?
From kahoot.it to Excel Students take the test Download the report Convert the answers Copy to Excel sheet
5
Place your screenshot here
Place your screenshot here
6
Answer keys The correct answer
for each item Row 2
Student answers From cell B4
Place your screenshot here
7
Answer count The number of
students who select that choice
The correct choice is highlighted
Place your screenshot here
8
Average rank On average, how
good are the students who select that choice?
Rank 1 has the highest total score
Place your screenshot here
9
Student scores The score of each
student for each item
What Does the Excel Sheet Tell Us?▫ How many students select each choice?
▫ Is the answer key selected most?▫ Which choice is not selected at all?
▫ How good are those students?▫ Who does a choice appeal to?▫ Should that be expected?
▫ Should the item be improved and how?▫ The stem, the key, or the distractors?
10
11
Metsämuuronen 2018
Difficulty and Discriminability Item difficulty Interpreting Item Difficulty Index Item discrimination Interpreting Item Discrimination Index
Item Difficulty▫ The knowledge/skill level required to answer an item
(or passing a test)▫ Usually measured by Item Difficulty Index
▫ The average score (0−1) of students for that item▫ Should be called Item Easiness Index▫ Acceptable range depends on the type of the item▫ For a dichotomous item (0 or full mark), it is the proportion of
correct answers
13
Place your screenshot here
14
Difficulty Item Difficulty Index The Difficulty Index
for the whole test is on the right (same row)
(Cronbach’s alpha is the measure of reliability)
Interpreting Item Difficulty Index
No choice True false 4 choices 5x5 pairs Difficulty
> 0.80 > 0.9 > 0.85 > 0.84 Too easy
0.61−0.80 0.81−0.90 0.71−0.85 0.69−0.84 Easy
0.40−0.60 0.70−0.80 0.55−0.70 0.52−0.68 Medium
0.20−0.39 0.60−0.69 0.40−0.54 0.36−0.51 Hard
< 0.20 < 0.60 < 0.40 < 0.36 Too hard
15
Item Discrimination▫ The effectiveness of an item at discriminating those
who know the content from those who do not (OR)▫ The extent to which the success on an item
corresponds to the success on the whole test OR▫ Usually measured by Item Discrimination Index
▫ The correlation between the score for an item and the score for all other items (corrected item-total correlation)
16
Place your screenshot here
17
Discriminability Corrected item-total
correlation The average
correlation for the whole test is on the right (same row)
18
Metsämuuronen 2018
Validity Eliciting expert’s opinions Index of Item-Objective Congruency
Eliciting Experts’ Opinions
Objective 1 Objective 2 Objective 3 Objective 4
Item 1
Item 2
Item 3
Item 4
Item 5
20
Eliciting Experts’ Opinions (cont.)▫ For each expert, rate whether an item assesses a
certain objective▫ Popular practice
▫ +1 means assess▫ 0 means unsure▫ -1 means do not assess
21
Index of Item-Objective Congruence (IIOC)▫ Average the group of relatively high-rated objectives▫ Average the group of relatively low-rated objectives▫ Half the difference is IIOC
22