Date post: | 26-Mar-2015 |
Category: |
Documents |
Upload: | john-mcleod |
View: | 216 times |
Download: | 0 times |
Multiple Mini Interviews in the UKJon Dowell - University of Dundee
Aileen O’Brien - St George's University of London
RationalePersonal qualities considered ‘important’ but not very reliably
assessed by interview and without much predictive validity. (Goho + Blackman 2006. cognitive r 0.06, clinical r 0.17)
Introduced for assessing medical school applicants at McMaster University, Canada. (Reiter, Eva et al)
Piloted 2002 – 12 x 8 min. Generalisability since +/- 0.7 Predicted OSCE performance ß 0.44 (interview nil)Clerkship ratings ß 0.57 (interview nil)MCC Part 1 (selected components) ß 0.3-0.4 (interview nil or negative)
Supporting evidence fromCanberra - Harris and Owen 2007
Calgary - Brownell et al 2007
Dundee - Ponnamperuma 2008 / 4 station trial 2007-8
Domains assessedDundee
• Interpersonal and communication skills
• Critical thinking, problem solving
• Team work• Robustness against stress• Ethical/moral reasoning• Integrity/honesty• Preparation and motivation
for medical school
Georges• Empathy
• Communication
• Organisation and problem solving
• Team work
• Initiative and resilience
• Insight and integrity
• Effective learning style
Academic ability & intellect
ability & intellect
(already tested)
Dundee Stations2010
Role play OCD discussion
1:1 Professional Integrity
Task Sort it out - cards
1:1 Issues in Medicine
Task Puzzle
Role play Fitness to Practice
1:1 Medicine as a career
1:1 About you - UCAS
Role play Medications Muddle
1:1 Moral Dilemmas
How it runs
MarkingTraining 30 min general 20 min station specific Optimal criteria for each domain defined
Name Label
Stations St George’s
• Standard interview questions• Prioritising• Scenario/ breaking bad news• Presentation• Observation• Communication/ giving instructions
Candidates name………………………………………………
Motivation to do medicine
What are you looking forward to about being a doctor?
Points for guidance:Personal satisfaction from helping peopleSatisfaction from achieving something worthwhileWorking with a diverse range of peopleCombining an interest in science with caring for peopleVariety of career opportunities once qualifiedJob status and securityGood incomeThe opportunity to work abroadIntellectual rigourLifelong learning and clear professional career structureCamaraderie from clinical teams Research collaboration
Please mark out of 5Poor motivation 1 2 3 4 5 good motivation
Please also assign an overall impression of the candidate’s suitability to be offered a place at medical school based on their performance at this station (please circle and comment as appropriate).EXCELLENT ACCEPTABLE CAUSE FOR CONCERNRed card Comments:
George’s Marking Scheme
George’s results
• Graduates and non graduates did equally well• No correlation with– Gender– Age– 4 year and 5 year course– UKCAT and GAMSAT scores
– Reliable (kappa 0.68)
George’s Conclusions
Successful pilot 2009Ran this year with 1000 candidatesSmooth (two breakdowns!)Graduates and school leavers same cut off score
for admission
Dundee results
• 2009 n = 452 Cronbach’s Alpha = .70• 2010 n = 477 Cronbach’s Alpha = .69• No correlation in 2009 with– Gender– Age– Overseas candidates– School type– Small age gender interaction
2010 MMIs: Gender
•Females performed significantly better than males.•t(475) = 3.19, p = < .01•Effect size: 0.14
2010 MMIs: Applicant Group
•Significant differences between groups.
•F(4) = 2.50, p < .05
•Graduates/Mature perform significantly better than all other groups
Domain Reliability
Number of Stations
Number of Scores
Cronbach’s alpha
Communication 9 12 .767
Critical Thinking 6 6 .453
Integrity 1 1 -
Moral Reasoning 4 4 .170
Motivation 3 4 .515
Teamwork 2 3 .395
Correlations between scales
No correlation between UKCAT and MMI scores
No correlation between our UCAS form and MMI scores
Dundee MMI - Conclusions•Good reliability in last two years.
oAll stations were reliable enough. No negative Item-total correlations; no reason to reject any station.
•Successfully ran for over 900 candidates with no significant problems (no breakdowns, tears etc!)•Putting more weight on MMI may increase female and graduate/mature candidates compared to the Pre-Interview Score
MMI/Pre-Interview 40/60 60/40 80/20
Male 129 122 121Female 201 208 209A-Level 81 79 81Scottish 189 182 179G/M 27 35 38EU 16 17 17OS 17 17 18
48 40
88SelectionDifference
So what do the punters think?
Assessors’ Survey Dundee• 116 /157 assessors (88% of students and 50%
of staff) completed online survey to find out how they viewed MMI.
• 91% thought the process fair.• 88% thought stress on applicants ‘moderate’
or less.
Candidates’ Survey Dundee• 324 /433 (75%) of Candidates completed
online survey.• 94% felt MMI was ‘fair’.• 90% felt it is a ‘valid way to assess candidates’.• 71% preferred MMI to traditional interview.
• ‘Student Counselor’ (role play) cited as the most enjoyable (153/172).
Assessors’ Survey George’s
• Equally fair• Equally accurate• Equally able to pick best candidate
• Interviews “coached”• MMIs can be boring• MMIs good distinguishing tool
Candidates’ survey George’s
• 5 years- MMI more fair• -MMI more accurate• -interview more difficult
• 4 years- MMI more difficult• -equally fair and accurate
Conclusions and next steps• MMIs becoming an interview tool at many
schools. Is it justifiable not to use them?• How to develop new stations? Collectively?• Reliability potentially increased (above 0.8)
using Rasch analysis – alters decision for 10%• Predictive validity in UK? – could UKCAT help?
Questions...
Options to improve?? Rasch Modeling
5 6 7 8 9 104321
0
4
5
7
4
ExaminerExaminer
A
B
C
D
E
Station
Variation in outcomes can be enhanced by using multiple tracksTrack
Track
Yardstick
Effect of using Rasch
•Reliability based on ‘fair scores’ 0.84•10% different candidates would be given offers. (+ 10% not)•Only the top 169 ‘stable’. •Could not be 100% confident in process this year, so not applied. (complex black box stuff)