Date post: | 15-Dec-2015 |
Category: |
Documents |
Upload: | elaine-rayfield |
View: | 217 times |
Download: | 0 times |
AGENDA
Presentation and Discussion Item writing guidelinesExamples
Develop, Evaluate, and Revise Items
A test blueprint is a table of specifications that weights each objective according to how important it is or how much time is spent covering that objective, links objectives to test items and summarizes information
Essential for Competency Testing
Test blueprint
BLUEPRINT TIPS
- List objectives in a table
- Identify length of test
- Designate number of items per objective
-Evaluate the importance of each objective and
assign items accordingly
- Often assign one point per item
EXAMPLE OF BLUEPRINT
Objective Weighting Items
Objective 1 25% 10
Objective 2 25% 10
Objective 3 25% 10
Objective 4 25% 10
Some items are more appropriate when testing different kinds of knowledge; or when tapping into different kinds of cognitive processes.
We’ll focus on multiple choice items because recruiting faculty to score open-ended items would be difficult.
Type of Item Construction Scoring
True/False Difficult Easy
Matching Easy Easy
Completion Easy Difficult
Multiple Choice Difficult Easy
Essay Easy Difficult
Writing Items
Focus on a single problem when writing an item
Use new situations to assess applicationAvoids memorization exercisesAllows for synthesis and evaluation
Keep content of items independentstudents shouldn’t be able to use one item to
answer another, although a set of items may tap into a shared scenario
Avoid opinion-based items Address a mix of higher-order and
lower-order thinking skills.
Content
HIGHER ORDER THINKING Bloom Haladyna
Analysis
Synthesis
Application
Comprehension
Knowledge
Evaluation
Predicting
Evaluating
Defining
Recalling
Problem Solving
Higher Order Skills
Lower Order Skills
WRITING ITEMS TO ASSESS HIGHER ORDER THINKING (BASED ON HALADYNA)
Try item stems such as "If . . ., then what happens?", "What is the consequence of . . .?", or "What would happen if . . .?" (predicting)
Ask students to make a decision based on predetermined criteria, or to choose criteria to use in making a decision, or both (evaluating).
Require the student to use combinations of recalling, summarizing, predicting, and evaluating to solve problems.
Avoid excess words – be succinct Use specific, appropriate vocabulary Avoid bias (age, ethnicity, gender,
disabilities) Write stems and options in third person Underline or bold negative or other
important words Have others review your items
Style/format
The stem should clearly state the problem Place the main idea of the question in the stem, not the
item options Keep the stem as short as possible Don’t provide clues to correct answer in stem (e.g.,
grammatical clues) If the stem is a complete sentence, end with a period and
begin all response options with upper-case letters. If the stem is an incomplete sentence, begin all response
options with lower case letters. Use negative stems rarely
Writing the stem
Writing the distracters
Make sure there is only one correct answer for each item
Develop as many effective plausible options as possible, but three are sufficient (Rodriguez, 2005) It is better to have fewer options than to write BAD
options to meet some quota! Vary the location of the correct answer when
feasible (Flip a coin), or put options in logical order (e.g. chronological, numerical)
Avoid excessive use of negatives or double negatives
Keep options independent Keep options similar (in format)
Length and wording
DO use as distracters: common student misconceptions (perhaps from
open-ended responses from previous work) words that “ring a bell” or “sound official” responses that fool the student who has not
mastered the objective DO NOT use as distracters:
responses that are just as correct as the right answer implausible or silly distracters
Use “all of the above” and “none of the above” sparingly
Don’t use “always” or “never”
Don’t give clues to the right answer
Writing the distracters
ALL THE OPTION CHOICES SHOULD USE PARALLEL GRAMMAR TO AVOID GIVING CLUES TO THE RIGHT ANSWER.
The best way to increase the reliability of a test is to:
A. increase the test length B. removing poor quality items C. Tests should be readable for all test
takers.
What’s wrong with this item?
California:A). Contains the tallest mountain in the United StatesB). Has an eagle on its state flag.C). Is the second largest state in terms of area.*D). Was the location of the Gold Rush of 1849.
What is the main reason so many people moved to California in 1849?A). California land was fertile, plentiful, and inexpensive.*B). Gold was discovered in central CaliforniaC). The east was preparing for a civil war.D). They wanted to establish religious settlements.
Stem should state the problem
“FOCUS ON A SINGLE PROBLEM”; ITEMS WITH MULTIPLE CLAUSES MAY HAVE MULTIPLE CORRECT ANSWERS DEPENDING ON WHICH ASPECT THE STUDENT FOCUSES.
Bleeding of the gums is associated with gingivitis, which can be cured by the sufferer himself by brushing his teeth daily.
A. true B. false
The United States should adopt a foreign policy based on:A). A strong army and control of the North American continent.B). Achieving the best interest of all nations.C). Isolation from international affairs.*D). Naval supremacy and undisputed control of the world’s sea lanes.
According to Alfred T. Mahan, the United States should adopt a foreign policy based on:A). A strong army and control of the North American continent.B). Achieving the best interest of all nations.C). Isolation from international affairs.*D). Naval supremacy and undisputed control of the world’s sea lanes.
What’s wrong with this item?More than One Possible Answer
Piloting and revising
Following these rules does not guarantee that items will perform
well empirically. Testing companies, using paid item-writers and detailed writing
guidelines, ultimately only use 1/3 (or fewer) of the items
operationally.
It is normal and expected to have to revise items after pilot-testing .
Piloting and revisingITEM ANALYSIS STEPS:
• Item difficulty - proportion of people who answered the item correctly- an item should not be too easy or too difficult- problems arise from poor wording, trick questions, or
speediness
• Item Discrimination - correlation of item and total test, should be higher than 0.2
- item as an indicator of the overall test score- the higher the better- can be (but shouldn’t be) negative
• Distractor Analysis - frequency of response selections- item is problematic if people with a high overall score are
selecting incorrect responses frequently
THE BIG IDEA Ultimately, you want to know
whether students are achieving your objectives
Tests are used to indirectly measure these knowledge, skills, attitudes, etc.
Items you write are a sample of all possible items to measure that objectivethe more you write (and the better your
items are!) the more reliably you can measure the objective
You want to be sure that the items you create are measuring achievement of the objective, and NOT test-wiseness, reading ability, or other factors
REFERENCES Haladyna, T. M. (1999). Developing and validating
multiple-choice test items. Mahwah, NJ: Lawrence Erlbaum Associates.
Rodriguez, M. C. (2005). Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research. Educational Measurement: Issues and Practice, 24, 3-13.
Downing & Haladyna (2006). Handbook of test development. Mahwah, NJ: Lawrence Erlbaum Associates.
Burton S.J. et al (1991). How to prepare Multiple-Choice Test Items: Guidelines for University Faculty. Brigham Young University Testing Services.