Date post: | 16-Dec-2015 |
Category: |
Documents |
Upload: | byron-reynolds |
View: | 245 times |
Download: | 2 times |
Multiple Choice ExamsMultiple Choice Exams
“ “ Written properly, multiple choice Written properly, multiple choice exams correlate strongly with exams correlate strongly with
assessments by descriptive tests ”assessments by descriptive tests ”
Brown, Robert, “Multiple Choice Versus Descriptive Tests”, Brown, Robert, “Multiple Choice Versus Descriptive Tests”, Frontiers in Education Conference, Reno NC 2001.Frontiers in Education Conference, Reno NC 2001.
Use of Multiple Choice ExamsUse of Multiple Choice Exams
Not limited to Factual KnowledgeNot limited to Factual Knowledge
Making inferencesMaking inferences
Application of modelsApplication of models
EvaluationEvaluation
Multiple Choice vs. Descriptive Multiple Choice vs. Descriptive ExaminationsExaminations
DescriptiveDescriptive
QuestionsQuestions
Multiple Multiple ChoiceChoice
Setting the examSetting the exam EasyEasy DifficultDifficult
Grading TaskGrading Task DifficultDifficult EasyEasy
Grading TimeGrading Time LongLong ShortShort
Grade ConsistencyGrade Consistency VariesVaries HighHigh
Structure of a Structure of a Multiple Choice QuestionMultiple Choice Question
StemStem
AnswersAnswers
Formulating GoodFormulating GoodMultiple Choice QuestionsMultiple Choice Questions
Wording the stem so that there is a single Wording the stem so that there is a single correct answercorrect answer
Finding plausible wrong answersFinding plausible wrong answers
Sensitivity to ESL studentsSensitivity to ESL students- Negative voice in the stemNegative voice in the stem- Conditional “if… would” structuresConditional “if… would” structures- Cultural contexts for questions Cultural contexts for questions
StemsStems
StatementsStatements
““A river that flows through a desert is called..”A river that flows through a desert is called..”
QuestionsQuestions
“ “ Which of the following descriptions best fitsWhich of the following descriptions best fits
the character of Alsatian wines?”the character of Alsatian wines?”
NeNeggative voiceative voice
“ “ Which of the following grapes is Which of the following grapes is notnot used to used to
make Amarone?”make Amarone?”
StemsStems
Conditional VoiceConditional Voice
“ “ If you want a VQA Niagara red wine thatIf you want a VQA Niagara red wine that
was very fruity and required only a shortwas very fruity and required only a short
ageing period you would select: ”ageing period you would select: ”
Answer OptionsAnswer Options
Single correct answer optionsSingle correct answer optionsCollective answer options:Collective answer options:
“ “ (a) and (c) are both correct ”(a) and (c) are both correct ”
“ “ all of the above are correct ”all of the above are correct ”
“ “ none of the above are correct none of the above are correct ””
Plausible Answer StrategiesPlausible Answer Strategies
Good questions contain several plausible Good questions contain several plausible answer optionsanswer options
Grey area between designing “tricky” Grey area between designing “tricky” questions and providing answer options questions and providing answer options that test knowledge thoroughly.that test knowledge thoroughly.
Inadvertently giving away the answers Inadvertently giving away the answers
Giving away the answer…Giving away the answer…
A verbal question:A verbal question:
The forbidden fruit that Eve offered The forbidden fruit that Eve offered Adam was an:Adam was an:
(a) apple(a) apple (b) pear(b) pear
(c) banana(c) banana (d) melon(d) melon
(e) carrot(e) carrot
Giving away the answerGiving away the answer
i.e. a numerical question:i.e. a numerical question:
It costs a vendor $1.00 each time s/he It costs a vendor $1.00 each time s/he sells a hot dog to a customer. If the price sells a hot dog to a customer. If the price of the hot dog is $3.00, what is the of the hot dog is $3.00, what is the contribution margin percentage?contribution margin percentage?
(a) 17.5%(a) 17.5% (b) 22.5%(b) 22.5%(c) 47.5%(c) 47.5% (d) 66.7%(d) 66.7%(e) 82.0%(e) 82.0%
““Plausible” Answer StrategyPlausible” Answer Strategy
It costs a vendor $1.00 each time s/he sells It costs a vendor $1.00 each time s/he sells a hot dog to a customer. If the price of a hot dog to a customer. If the price of the hot dog is $3.00, what is the the hot dog is $3.00, what is the contribution margin percentage?contribution margin percentage?
(a) 300%(a) 300% (b) 200%(b) 200%
(c) 67.7%(c) 67.7% (d) 50%(d) 50%
(e) 33.3%(e) 33.3%
Plausible Answer StrategiesPlausible Answer Strategies
Use Nonsense Words Use Nonsense Words ::
Colio Wines in Ontario produce an interesting Colio Wines in Ontario produce an interesting product classified as Vin de Curé in much the product classified as Vin de Curé in much the same way as Amarone is made in Veneto. same way as Amarone is made in Veneto. The Italian word describing the drying process The Italian word describing the drying process used to desiccate the grapes is:used to desiccate the grapes is:
a) Ripassoa) Ripasso b) Appasimentob) Appasimentoc) Raisinatoc) Raisinato d) Pommaced) Pommacee) Prunelloe) Prunello
Plausible Answer StrategiesPlausible Answer Strategies
Use related, but different terms….Use related, but different terms….
The word that refers to the addition of The word that refers to the addition of sugar to a finished wine is:sugar to a finished wine is:
(a) Amelioration(a) Amelioration
(b) Chaptalization(b) Chaptalization
(c) Edulcoration(c) Edulcoration
(d) Elaboration(d) Elaboration
(e) Oxidation(e) Oxidation
Adding Uncertainty..Adding Uncertainty..
The term "fortified wine" means:The term "fortified wine" means:
a) the bottle had been reinforceda) the bottle had been reinforced
b) the wine has had tannins and acid b) the wine has had tannins and acid
addedadded
c) the wine has had sugar addedc) the wine has had sugar added
d) the wine has had SOd) the wine has had SO22 added added
e) none of the foregoinge) none of the foregoing
Adding difficulty..Adding difficulty..Choose the Choose the bestbest answer. answer.
Rootham’s, a well-known local manufacturer of Rootham’s, a well-known local manufacturer of brandied fruit preserves sells their product brandied fruit preserves sells their product through gourmet food shops and at craft fairs through gourmet food shops and at craft fairs throughout Canada. Their recipes use genuine throughout Canada. Their recipes use genuine brandy and all natural ingredients. They are brandy and all natural ingredients. They are able to sell these legally because:able to sell these legally because:
(a) the alcohol used is produced by fermentation(a) the alcohol used is produced by fermentation(b) the product is made with ethyl alcohol(b) the product is made with ethyl alcohol(c) the alcohol used is purchased from the LCBO(c) the alcohol used is purchased from the LCBO(d) the alcohol concentration is less than 1.2%(d) the alcohol concentration is less than 1.2%(e) the label says “made with genuine brandy”(e) the label says “made with genuine brandy”
Making Multiple Choice Questions Making Multiple Choice Questions Interesting…Interesting…
A gracious, but parsimonious hostess wants to A gracious, but parsimonious hostess wants to receive guests in the private dining room of your receive guests in the private dining room of your restaurant with a glass of champagne just to restaurant with a glass of champagne just to tantalise the taste buds and set the mood for the tantalise the taste buds and set the mood for the evening. evening.
1.You suggest that if the wine is being poured at 1.You suggest that if the wine is being poured at the bar, she should consider serving a Canadian the bar, she should consider serving a Canadian sparkling wine that is very similar to Champagne sparkling wine that is very similar to Champagne in quality but considerably less expensive. Such in quality but considerably less expensive. Such a wine would be a: a wine would be a:
2. The cheapest sparkling wine would be 2. The cheapest sparkling wine would be obvious to sophisticated guests because obvious to sophisticated guests because they would know that in the best sparkling they would know that in the best sparkling wines…wines…
3. If the gracious hostess insists on having 3. If the gracious hostess insists on having “French” Champagne and you are left to “French” Champagne and you are left to rationalise costs, the most cost effective rationalise costs, the most cost effective solution is to serve Champagne from.. solution is to serve Champagne from..
4. Occasionally, a patron asks you to do 4. Occasionally, a patron asks you to do something that will distinguish their event something that will distinguish their event from others. One client wants to obtain a from others. One client wants to obtain a large bottle of Champagne to celebrate a large bottle of Champagne to celebrate a particular event. Which is the largest particular event. Which is the largest bottle of champagne on the list below? bottle of champagne on the list below?
5. If someone wanted to serve a "glass of 5. If someone wanted to serve a "glass of bubbly" during a reception, which of the bubbly" during a reception, which of the following would be the most following would be the most inainapppproroppriateriate type of wine to select as the pre-prandial type of wine to select as the pre-prandial (before dinner) libation?(before dinner) libation?
Evaluating Multiple Choice Evaluating Multiple Choice QuestionsQuestions
Difficulty Factor = R/NDifficulty Factor = R/N
R = # of times the correct answer is R = # of times the correct answer is
selectedselected
N = # of people taking the examN = # of people taking the exam
0.5 + 10
Too EasyToo Difficult
Difficulty FactorDifficulty Factor
Values greater than 0.8:Values greater than 0.8: The question is too easyThe question is too easy Stem wording gives away the answerStem wording gives away the answer Not enough plausible incorrect answer optionsNot enough plausible incorrect answer options
Values less than 0.2:Values less than 0.2: Guessing effect can be a factorGuessing effect can be a factor The question is defectiveThe question is defective Material wasn’t coveredMaterial wasn’t covered Stem wording is unclear, confusingStem wording is unclear, confusing Answer options need work Answer options need work
Evaluating Multiple Choice Evaluating Multiple Choice QuestionsQuestions
Discrimination Index = Discrimination Index = Proportion of capable students that select an Proportion of capable students that select an answer option – proportion of weak students answer option – proportion of weak students that select the answer optionthat select the answer option
i.e. D = Fi.e. D = Ftop quartertop quarter – F – Fbottom quarterbottom quarter
0 + 1- 1
Good Discrimination
Reverse Discrimination
Discrimination IndexDiscrimination Index
For the Correct Answer OptionFor the Correct Answer Option
Values ~ 0.5 give good discriminationValues ~ 0.5 give good discrimination
< 0.2 consider dropping< 0.2 consider dropping
negative… drop the questionnegative… drop the question
For Incorrect Answer OptionsFor Incorrect Answer Options
Values should be negative.Values should be negative.
Knowledge or Successful Knowledge or Successful Guessing?Guessing?
Multiple Choice Exam StrategiesMultiple Choice Exam Strategies
Improve odds by eliminating 1 or more Improve odds by eliminating 1 or more infeasible or unlikely answer optionsinfeasible or unlikely answer options
Descriptive Exam StrategiesDescriptive Exam Strategies
Brain dumpingBrain dumpingPart marksPart marksConsideration for perfect answers to Consideration for perfect answers to
questions that were not askedquestions that were not asked
Possibility of a “Random Pass”Possibility of a “Random Pass”
Depends on the number Depends on the number
of answer options per questionof answer options per question
and the number of questions!and the number of questions!
Number of Number of QuestionsQuestions
Percent Pass (Percent Pass (≥≥50%) by Chance50%) by Chance2 choices2 choices 3 choices3 choices 4 choices4 choices 5 choices5 choices
11 5050 3333 2525 2020
22 7575 5656 4444 3636
44 6969 4141 2626 1818
66 6666 3232 1717 1010
1010 6262 2121 88 33
2020 5959 9.29.2 1.41.4 .3.3
5050 5656 11 .01.01 .0004.0004from Brown, 2001
Adjustment for GuessingAdjustment for Guessing
Negative Marking…Negative Marking…
Elimination strategy reduces odds of wrong Elimination strategy reduces odds of wrong answer penaltyanswer penalty
Subtracting a percentage of the number of Subtracting a percentage of the number of wrong answers obtained from the final gradewrong answers obtained from the final grade
Give a grade of 4 for a correct answer and a Give a grade of 4 for a correct answer and a score of -1 for a wrong answer on a 4 choice score of -1 for a wrong answer on a 4 choice questionquestion
Negative Marking..Negative Marking..
A score of less than zero is possibleA score of less than zero is possible
Students hate negative markingStudents hate negative marking
Negative marking is not practised in Negative marking is not practised in descriptive examinationsdescriptive examinations
A poor substitute for a test that is A poor substitute for a test that is too short with too few answer too short with too few answer optionsoptions
Teaching WithTeaching WithMultiple Choice QuestionsMultiple Choice Questions
Prepare lectures incorporating all material Prepare lectures incorporating all material covered in the questions to be usedcovered in the questions to be used
After each lecture, check coverage by After each lecture, check coverage by reviewing questionsreviewing questions
Provide students with multiple choice Provide students with multiple choice questions for self-evaluation at the end of questions for self-evaluation at the end of readingsreadings
Teaching WithTeaching WithMultiple Choice ExaminationsMultiple Choice Examinations
Use stem questions to teach additional Use stem questions to teach additional or ancillary terms not covered in the or ancillary terms not covered in the course:course:
i.e. …. For post-prandial (after dinner) i.e. …. For post-prandial (after dinner) drinks you want to stock a supply of drinks you want to stock a supply of fortified wines and brandy. The term fortified wines and brandy. The term "fortified wine" means…."fortified wine" means….
Provide Students the Stem Provide Students the Stem Questions Prior to the Exam!Questions Prior to the Exam!
Exam questions should cover all the Exam questions should cover all the course material being testedcourse material being tested
Provides more detail than a course outlineProvides more detail than a course outlineStudents focus their study effort on the Students focus their study effort on the
topics they understand the leasttopics they understand the least Increases learningIncreases learningReduces stressReduces stress
Thanks for your interest in my topic!Thanks for your interest in my topic!
I look forward to your questions and ideas.I look forward to your questions and ideas.
J.E. (Joe) BarthJ.E. (Joe) Barth
School of Hospitality and Tourism ManagementSchool of Hospitality and Tourism Management
University of Guelph, Ontario CANADAUniversity of Guelph, Ontario CANADA