DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE
School of Medicine, Flinders University
Train the Trainer WorkshopOctober 28, 2003
The sit uati on of Moun t Lof ty was fo
f rom hence and from some ot her cr
beari ngs, to be 34¡ 59 ' south and 13
east. N o l and w as vi si bl e so f ar to th
nort h as where t h e trees appeare d a
t he horizon , w hi ch show ed the coas
be v er y low, and our soundings we
f ast dec reasing.
noon t o six o'cl ock we ran th irt y t o t he nor thward, sk irt ing a sandy at the d istance of fi ve, and t he nce t mi les; the dept h was then 5 s, and we dropped the anchor upon
om of sand , mi xed wit h p ieces of o ral.
CONSTRUCTED RESPONSE ITEMS:
MEQ & SAQ
Hong Kong International Consortium
CONSTRUCTED RESPONSE ITEMS
ASSUMPTIONS
• Higher cognitive processes
• Increased reliability through wider sampling
ARE THE ASSUMPTIONS JUSTIFIED?
MODIFIED ESSAY QUESTIONS: MEQs
• 1970s – 1980s
• RCGP (Hodgkin & Knox 1975)
• Newcastle Australia (Feletti 1980, Feletti and Smith 1986)
• Jefferson USA (Rabinowitz 1987, Rabinowitz and Hojat 1989)
• Learning objectives, concepts, deep structures
• Clinical scenario
• 7 – 10 questions
• Sequential
(information question information question ...)
• Integrated (content, media – Rabinowitz 1985)
MODIFIED ESSAY QUESTIONS: MEQs
HIGHER COGNITIVE ORDER(Bloom 1956, Buckwalter et al 1981)
1. Knowledge Level 1 Recognition Recall
2. Comprehension Level 2 ComprehensionInterpretation
3. Application
4. AnalysisLevel 3 Application
5. Synthesis
6. Evaluation
(Irwin and Bamber 1982)
MODIFIED ESSAY QUESTIONS: MEQs
LEVEL 1 RECOGNITION AND RECALL
Know
• What is …..?
• List …..
• Identify …..
MODIFIED ESSAY QUESTIONS: MEQs
MODIFIED ESSAY QUESTIONS: MEQs
LEVEL 2 COMPREHENSION, INTERPRETATION
Understand
• Give reasons ….
• Explain ….
• Interpret ….
LEVEL 3 APPLICATION
Apply, Utilise, Judge
• How does X assist in explaining Y?
• What are the components of X?How are they related?How do they affect Y?
• How does XYZ affect treatment and management?
• Assess, judge X in terms of ….
MODIFIED ESSAY QUESTIONS: MEQs
MODIFIED ESSAY QUESTIONS – VALIDITY
• CONTENT
(Blueprints, objectives, deep structures)
• CONSTRUCT
(Level 1, 2, 3 questions)
MODIFIED ESSAY QUESTIONS – VALIDITY
• CONSTRUCT
Norman et al 1987Effect of educational levelMCQ and MEQ not different skillsUEQ may be a different skillSmall effect of context
Feletti 1980More Level 1 and 2 in first year
undergraduate
Irwin and Bamber 1986More Level 3 in final year undergraduate
• PREDICTIVE
Irwin and Bamber 1986 Correlation with clinical examination 0.41
Rabinowitz 1987 MCQ better predictor NBME I & II
Rabinowitz and Hojat 1989 Higher correlation for MEQ than MCQ with overall clerkship ratings
MODIFIED ESSAY QUESTIONS – VALIDITY
• Inter-rater reliability (model answers)
Norman et al 1987Inter-rater reliability satisfactory
• Internal consistency (cumulative errors)
Feletti 1980Alpha(60) 0.57 – 0.91
Stratford and Pierce-Fenn 1985Alpha(60) 0.39
MODIFIED ESSAY QUESTIONS – RELIABILITY
• Administration
• Marking
• Higher order questions
MODIFIED ESSAY QUESTIONS – FEASIBILITY
• Model PBL process
• Determine objectives
• Construct scenario
MINI-CASES: THE FLINDERS MEQ
John Taylor (age 55) has been brought to the E.D. by his wife. John
woke up this morning (one hour ago) with nausea and vertigo. He
managed to walk to the bathroom but had to hold onto the furniture
and walls to steady himself. He vomited when he reached the
bathroom.
1. Give 3 hypotheses to account for John’s condition.
2. Explain the mechanisms which give rise to John’s condition.
3. What further information would you like to distinguish between hypotheses. Explain how this information will distinguish between hypotheses.
MINICASES: THE FLINDERS MEQ
• Provide further information
• Cumulative error cues
• Continue scenario
MINI-CASES: THE FLINDERS MEQ
John has had no significant medical problems in the last few years. He spent the previous evening at home watching TV and did not drink any alcohol or take any other substance.
….. The left pupil is smaller than the right …..
….. John’s uvula deviates to the right …..
….. Movements of the left hand are clumsy.
MINICASES: THE FLINDERS MEQ
• Further questions
• Continue unfolding scenario
• Page breaks, cumulative cues
• Conclude with SAQs (Level 2/3)
MINI-CASES: THE FLINDERS MEQ
• Face validity
• Construct validity
• Model answers
• Higher order questions
MINI-CASES: THE FLINDERS MEQ
• Reform of RCGP MEQ– Swanson– Mullholand & McAleer
• Increase number (10 – 12 / 2 hours)– Reflect reality
• Increase independence
• Single marked – schedules
• SAQ
(Lockie et al 1990)
SHORT ANSWER QUESTIONS - SAQ
Evaluation: “The ability to make a value-judgement in relation to
relevant criteria”
Bloom 1956
contextual information + Q responsecriteria
Opportunity to target highest cognitive skills
• application - analysis - synthesis - evaluation
Flinders SAQ strategies
1. Provide multiple, detailed, relevant information
• Patient
• Problem
• Clinical care setting
• Your role
contextual information + Q responsecriteria
Targeting “evaluation” response
• Ron Harkin is a 32 year-old father of two and self-employed
gardener. He falls heavily. He is unable to stand and is in
considerable pain. He is taken to hospital where you are the Intern
in the Emergency Department. You note swelling and that his foot
appears to be at an unusual angle to his leg.
• A 7 year-old Aboriginal girl, Alice Mundi, presents with a mild
fever and painful knee to a small rural hospital. Her mother has
brought her 50km to the hospital. You are the GP called to see
her.
Example scenario information
2. Pose question requiring value-judgement• focus the response
• assume criteria are understood
• relate to what you would do
3. Extend information / probe further (optional)
contextual information + Q responsecriteria
Targeting “evaluation” response
Example questions targeting “evaluation”
Rationalising the diagnosis
• What further information from the history do you require?
• On what aspects of the physical examination will you focus?
• What investigations would you undertake to establish the diagnosis?
• What are your differential diagnoses? Which do you consider most likely?
Formulating an appropriate management plan
• What would be your immediate management goals?
• Which treatments would you advise?
• For what potential complications would you monitor?
• What longer-term management strategy would you advise?
• What advice would you give in relation to this new development?
Example questions targeting “evaluation”
Mr Dawes, a 58 yo man, presents to the Emergency Dept with a 24h history
of vomiting and colicky abdominal pain. He has had no bowel movement for
24h. He had an appendicectomy at age 30 to remove a gangrenous
appendix. On examination you find he has a distended abdomen with no
focal tenderness. There is no palpable hernia. He has symptoms and signs
consistent with dehydration of about 10%. Fluid resuscitation is
commenced. A plain abdominal Xray shows distended loops of small bowel
with air fluid levels and normal calibre large bowel.
Q1. What is your most likely diagnosis for this patient? (2 marks)
Q2. What possible causes of this would you consider? (3 marks)
After 48h of conservative management, Mr Dawes’ condition is such that surgical intervention is warranted.
Q3. What clinical indications would suggest to you that Mr Dawes needs surgery to resolve his problem at this time?
(5 marks)
Fairly reward sound responses
• Model answer (checklist)
• Writers / markers who understand:
- curriculum content
- learner stage
• Ongoing item improvement
Flinders SAQ strategies
• Are the claims justified?
• Do they measure anything different?(Maquire, Skakun, Triska 1997)
• What role do they have in a balanced assessment program?
CONSTRUCTED RESPONSE ITEMS
REQUIREMENTS
• Write to a blueprint
• Sample widely
• Increase level 2 and 3 questions
• Model answers
CONSTRUCTED RESPONSE ITEMS
THE END
You see Rebecca and Robert who are residents at a local training
centre for 27 young mentally handicapped patients. They are
accompanied by their care helper. Rebecca and Robert have
scabies.
Discuss your management
• communication
• clinical management
• health in the community
• sensitivity to the mentally handicapped
• use of extended team
• ethical issues
Deep/Surface Structures
(Lockie et al 1990)
SHORT ANSWER QUESTIONS - SAQ
VALIDITY
Construct
Wright et al (1983) – Independent cognitive abilities
Predictive
Wass et al (2001) – correlate with clinical tests
RELIABILITY
Low to moderate
Des Marchais and Vu (1996)
Wakeford and Roberts (1984)
Good
Wright et al (1983) – marking schedules
SHORT ANSWER QUESTIONS - SAQ
FEASIBILITY
• Marking
• Higher order questions
SHORT ANSWER QUESTIONS - SAQ
• One clinical scenario – multiple questions
PAQ (Des Marchais and Vu 1996) – no sequence
SAQ (Flinders) sequence
• Level 1, 2 and 3 questions on one scenario
• Reliability
PAQ – low to moderate
• Feasibility
HYBRID APPROACHES
• Face validity
• Construct validity
• Model answers
• Higher order questions
THE FLINDERS HYBRID MODEL