Chapter Eleven: Improving Teacher-Developed

Assessments

Danielle LamarreDarcy ChurchJason HowardSarah Fearon

What are three sources of test-improvement judgments?

• Your own assessment instruments• Colleague judgments• Judgments of students

3 sources test-improvement judgments

Yourself• Adherence to item-specific guidelines and general item

writing commandments• Contribution to score-based inference• Accuracy of content• Absence of content lacunae• Fairness

Your colleagues• Colleagues review assessments based on the previously

discussed criteriaYour students

• Simply have the students complete the assessment first and then distribute a questionnaire; conduct this consecutively not simultaneously!

What is item difficulty?

The measurement of the percentage or proportion of test takers who answer the item correctly

How do you compute item difficulty index (level)?

• Often referred to as “p value” P = R/T• R= the number of students responding correctly

(right) to an item• T= the total number of students responding to the

item• P value can range from 0 to 1.00• For example a p value of .98 would signify an

item answered correctly by almost all students, Similarly, an item with a p value of 0.15 would be one that most students (85%) missed

What is the item discrimination index?

• How frequently an item is answered correctly by those who perform well on the total test

• Fundamentally, an item-discrimination index reflects the relationship between students’ responses for the total test and their responses to a particular item (page 278, 2014)

ITEM-DISCRIMINATION INDICES

Type of item Proportion of correct responses on total test

Positive Discriminator High scorers>Low scorers

Negative discriminator High scorers<Low scorers

Nondiscriminator High scorers=Low scorers

How do you compute the item discrimination index?

Procedure:1. Order the test papers from high to low score by total score

2. Divide the papers into a high group and a low group, with an equal number of papers in each group

3. Calculate a p value for each of the high and low groups

4. Subtract pl from ph to obtain each item’s discrimination index (D)

GUIDELINES FOR EVALUATION THE DISCRIMINATING EFFICIENCY OF ITEMS

Discrimination Index Item Evaluation0.40+ Very good items0.30 – 0.39 Reasonably good items, but

possibly subject to improvement

0.20 – 0.29 Marginal items, usually needing improvement

0 – 0.19 Poor items, to be rejected or improved by revision

Example • If 75% of the students in the top group answer an

item correctly, and 20% of the students in the bottom group answered the item correctly, what is the item discrimination index?

• D = 0.75 – 0.20 = 0.55

• Is this a good item?

• This is a good item

What is a distractor analysis?• Analyzing the wrong answers or distractors• Scrutiny of a student’s empirical performances

on the different answer options contained in multiple-choice and matching items

• Good distractors are highly effective with the lower performing group

What are the five review criteria teachers consider to improve assessment

procedures?

1. Adherence to item-specific guidelines and general item-writing commandments.

a) Follow the five general item-writing commandments listed in Chapter 6

b) Follow the item-specific guidelinesc) Fix any violations

2. Contribution to score-based inference.

What are the five review criteria teachers consider to improve assessment

procedures?

3. Accuracy of content. Be sure that the content of the assessment is accurate and relevant.

4. Absence of content lacunae. Make sure that all key elements of the content are present.

5. Fairness. Ensure that the assessment does not partake in any bias. Perform a bias review as a necessary precaution.

Reference

Popham, W.J. (2014). Classroom assessment: What teachers need to know (5th ed.). Boston, MA: Pearson Education Inc.