Item Analysis

Parames Laosinchai11 July 2019

“Mathematics reveals its secrets only to those who approach it with pure love, for its own

beauty.”

2

Archimedes

Outline1) Distractor analysis

2) Difficulty and discriminability

3) Validity

3

Distractor Analysis Form kahoot.it to Excel What does the Excel sheet tell us?

From kahoot.it to Excel Students take the test Download the report Convert the answers Copy to Excel sheet

5

Place your screenshot here

Place your screenshot here

6

Answer keys The correct answer

for each item Row 2

Student answers From cell B4

Place your screenshot here

7

Answer count The number of

students who select that choice

The correct choice is highlighted

Place your screenshot here

8

Average rank On average, how

good are the students who select that choice?

Rank 1 has the highest total score

Place your screenshot here

9

Student scores The score of each

student for each item

What Does the Excel Sheet Tell Us?▫ How many students select each choice?

▫ Is the answer key selected most?▫ Which choice is not selected at all?

▫ How good are those students?▫ Who does a choice appeal to?▫ Should that be expected?

▫ Should the item be improved and how?▫ The stem, the key, or the distractors?

10

11

Metsämuuronen 2018

Difficulty and Discriminability Item difficulty Interpreting Item Difficulty Index Item discrimination Interpreting Item Discrimination Index

Item Difficulty▫ The knowledge/skill level required to answer an item

(or passing a test)▫ Usually measured by Item Difficulty Index

▫ The average score (0−1) of students for that item▫ Should be called Item Easiness Index▫ Acceptable range depends on the type of the item▫ For a dichotomous item (0 or full mark), it is the proportion of

correct answers

13

Place your screenshot here

14

Difficulty Item Difficulty Index The Difficulty Index

for the whole test is on the right (same row)

(Cronbach’s alpha is the measure of reliability)

Interpreting Item Difficulty Index

No choice True false 4 choices 5x5 pairs Difficulty

> 0.80 > 0.9 > 0.85 > 0.84 Too easy

0.61−0.80 0.81−0.90 0.71−0.85 0.69−0.84 Easy

0.40−0.60 0.70−0.80 0.55−0.70 0.52−0.68 Medium

0.20−0.39 0.60−0.69 0.40−0.54 0.36−0.51 Hard

< 0.20 < 0.60 < 0.40 < 0.36 Too hard

15

Item Discrimination▫ The effectiveness of an item at discriminating those

who know the content from those who do not (OR)▫ The extent to which the success on an item

corresponds to the success on the whole test OR▫ Usually measured by Item Discrimination Index

▫ The correlation between the score for an item and the score for all other items (corrected item-total correlation)

16

Place your screenshot here

17

Discriminability Corrected item-total

correlation The average

correlation for the whole test is on the right (same row)

18

Metsämuuronen 2018

Validity Eliciting expert’s opinions Index of Item-Objective Congruency

Eliciting Experts’ Opinions

Objective 1 Objective 2 Objective 3 Objective 4

Item 1

Item 2

Item 3

Item 4

Item 5

20

Eliciting Experts’ Opinions (cont.)▫ For each expert, rate whether an item assesses a

certain objective▫ Popular practice

▫ +1 means assess▫ 0 means unsure▫ -1 means do not assess

21

Index of Item-Objective Congruence (IIOC)▫ Average the group of relatively high-rated objectives▫ Average the group of relatively low-rated objectives▫ Half the difference is IIOC

22