Date post: | 04-Aug-2015 |
Category: |
Health & Medicine |
Upload: | govt-medical-college-surat |
View: | 19 times |
Download: | 0 times |
• Advantage and limitations• Enumerate the methods of objective evaluation• Types of multiple choice question• Advantage and disadvantage of MCQ• The component of MCQ item• Guideline for construction of an MCQ• Item analysis
Objective
Advantage • Objective in nature• In depth evaluation of student !• Easy to evaluate• To test the candidate at the desired level of
difficulty • Good for formative evaluation and for
screening a large number of examinees• Standard of scoring can be kept constant for
many years
Disadvantage
• Poorly framed MCQ may be misleading• Tendency among teacher to test trivial and
less important data• Questions are difficult and time consuming to
construct• Not very useful for summative evaluation
Types • The single or one best response • Multiple completion type• Relationship analysis type• Multiple true false completion type• Matching type• Case history type• Pictorial/Graphical /Tabulated type• Comparison type
Testing Item Content Levels• Recall – testing knowledge of
isolated facts• Interpret* – asking to review
and reach some conclusion• Problem solve* – asking to
take some action after reading a situation
*Higher order thinking skills.
Writing for Different Cognitive Levels
• Recall– Which of the following is the process of mitosis?
• Interpret– Which of the following can be classified as a
meiosis error?• Problem-solving– A maternal blood sample revealed decreased
levels of AFP and estriol and elevated hCG. What would be the next logical test to perform?
Purposes…
• Motivate students• Identify areas of deficiency or further learning• Determine final grades or make promotion
decisions• Identify areas where the course/curriculum is
weak
What Should Be Tested?
• Exam content should match course/clerkship objectives
• Important topics should be weighted more heavily
• The testing time devoted to each topic should reflect the relative importance of the topic
Direction for framing MCQs
• Design each item to measure an important learning outcome
• Present single, compact, clearly formulated stem
• Use simple and clear language• Stem should be a statement , not a word• State the stem in positive form as far as possible• The word ‘NOT’ and ‘EXCEPT’ should be
underlined
Direction …cont
• Grammar ,Style, length, plausible distractors • Avoid clue that suggest answers, similar length
and form• ‘All of the above’, ‘none of the above’• Item with numerical alternatives , arrange
them in rank order• Testwiseness• Irrelevant Difficulty
Issues Related to Testwiseness
• Grammatical cues• Logical cues• Absolute terms• Long correct answer• Word repeats• Convergence strategy• B or C position overused
Keep the alternatives simple• When your body adapts to your exercise load,
a. you should decrease the load slightly.b. you should increase the load slightly.*c. you should change the kind of exercise you are doing.d. you should stop exercising
• When your body adapts to your exercise load, you should
• a. decrease the load slightly.b. increase the load slightly.*c. change the kind of exercise.d. stop exercising
Implausible distractors:
• In a study of the effect of diet on risk of diabetes, the researcher can manipulate a number of variables including the amount of food, carbohydrates, proteins or fats consumed. During the experiment the amount of food, protein and fat subjects consumed remained the same. Only the amount of carbohydrates consumed changed. What was the independent variable in this study?
a. amount of food consumedb. amount of carbohydrates consumedc. amount of protein consumedd. amount of fat consumed
Plausible distractors:
• In a study of the effect of diet on risk of diabetes, the researcher measured how likely the subjects were to get diabetes and how severe their symptoms were if they developed the disease. To prevent amount of exercise from influencing the results, the researcher held it constant in the two groups he was studying. What was the independent variable in this study?
a. likelihood of developing diabetesb. severity of symptoms c. dietd. amount of exercise
dietary pattern
Possibility of a “Random Pass”
Depends on the number of answer options per questionand the number of questions!
Number of Questions
Percent Pass (≥50%) by Chance2 choices 3 choices 4 choices 5 choices
1 50 33 25 20
2 75 56 44 36
4 69 41 26 18
6 66 32 17 10
10 62 21 8 3
20 59 9.2 1.4 .3
50 56 1 .01 .0004from Brown, 2001
Adjustment for Guessing
Negative Marking…– Elimination strategy reduces odds of wrong
answer penalty
– Subtracting a percentage of the number of wrong answers obtained from the final grade
– Give a grade of 4 for a correct answer and a score of -1 for a wrong answer on a 4 choice question
Item analysis
• Difficulty index
• Discrimination index
• Distractor effectiveness
• To create a question bank• Used in classroom teaching to modify teaching content
or methodology
Purpose of Item Analysis
–Evaluates the quality of each item
–Rationale: the quality of items determines the quality of test (i.e., reliability & validity)
– May suggest ways of improving the measurement of a test
• Difficulty index– The item difficulty for item i, pi , is defined as the
proportion of examinees who get that item correct.
• Method for Dichotomously Scored Item
• Method for Polytomously Scored Item
• Grouping Method
N
RP
maxX
XP
2LU PP
P
Item Difficulty (cont.)
Interpreting the p-value...
example: 100 people take a test 15 got question 1 right
What is the p-value?Is this an easy or hard item?
Item Difficulty (cont.)
Interpreting the p-value...
example: 100 people take a test 70 got question 1 right
What is the p-value?Is this an easy or hard item?
0.5 + 10
Too EasyToo Difficult
Level of DifficultyIndex Range Difficulty Level
0.00-0.20 Very Difficult
0.21-0.40 Difficult
0.41-0.60 Average/Moderately Difficult
0.61-0.80 Easy
0.81-1.00 Very Easy
ITEM DISCRIMINATION
• The extent to which an item differentiates people on the behavior that the test is designed to assess.
• the computed difference between the percentage of high achievers and the percentage of low achievers who got the item right.
Index of Discrimination
• (used for dichotomously scored items)
D = PH - PL • We need to set one or two cutting scores to divide the
examinees into upper scoring group and lower scoring group.
• PH is the proportion in the upper group who answer the item correctly and PL is the proportion in the lower group who answer the item correctly.
• Values of D may range from -1.00 to 1.00.
Example
There are 140 students attending a world history test. If 18 examinees in upper group answer item 5 correctly, and 6 examinees in lower group answer it correctly, then calculate the discrimination index for item 5.
Example 2 50 Examinees’ Test Data on 8-Item Scale About Job
Stress.
Item 1 2 3 4 5 6 7 8
PHPL
.54 .81 .47 .32 .51 .18 .63 .56 .32 .56 .11 .05 .10 . 23 .25 .19
D .18 .25 .36 .27 .41 -.05 .38 .37
Guidelines for Interpretation of D Value
• D≥.40, the item is functioning quite satisfactorily• .30≤ D≤.39, little or no revision is required• .20 ≤ D≤.29, the item is marginal and needs revision• D≤.19, the item should be eliminated or completely
revised
0 + 1- 1
Good Discrimination
Reverse Discrimination
What’s a “Good” Distracter?A good distracter is a plausible answer:• CM = common student mistakes, a recurrent
misconception about topic• TG = too general option, that is too broad to
answer the question• TS = too specific option, option focuses on one
detail in stem• OT = off topic, option misses the answer
completely
Guidelines for constructing a MCQ paper
• Include a sufficient number of items to have validity
• Time allotted • Group according to format• Instructions for negative marking• Place easy questions first, easy moderate and
difficult questions in sufficient numbers• All part of an item should on same page
design
Pilot study
ReliabilityInternal consistency – reliability coefficients (coefficient alpha or KR 20)Equivalence - alternate form correlation
Content validity - expert panelFace validity - expert panel
Construct validity - 'key check' / item discrimination / item difficulty
Further reading…
• http://www.teambasedlearning.org/• Constructing Written Test Questions For the
Basic and Clinical Sciences Third Edition (Revised)•Multiple Choice Questions Not Considered
Harmful by
Karyn Woodford, Peter Bancroft accessed from http://crpit.com/confpapers/CRPITV42Woodford.pdf on 09 Aug14