Post on 24-Feb-2016
description
transcript
Changes in selection to the Foundation ProgrammeProfessor Paul O’Neill
Chair, ISFP Project GroupMember UKFPO Rules GroupLead for Research and Evaluation Selection
Plan for Talk• Background to change – robust & numbers• Evidence around selection• SJTs
• Evidence • Piloting
• EPM• Algorithm• Academic FP
Selection (appointment)
Best prediction of the right person to do the best job
Should be done fairly
PON experience:• House jobs (Foundation)• CVs. application forms and letters• References & ‘putting a word in’• Interview panels – numbers, questions• Presentations• Occupational pyschologists• Personality testing
Now:• Mostly national• Application - anonymous, standardised and
assessed against criteria• Use of standardised tests (e.g. SJT)• Selection centres – standardised, multimodal,
some competency testing• Interview panels more & more standardised
Evolution2009 DH commissioned a review of selection
methods. The Improving Selection to the Foundation Programme project was set up and overseen by the Medical Schools
Council
2010/2011 New selection methods were piloted successfully
FP 2012Full-scale Parallel Recruitment Exercise (PRE)
FP 2013New selection methods implemented
A collaborative venture between
Educational Supervisors
Scottish Foundation BoardFoundation School Directors
Selection in MedicineUndergraduate
• Single School• UKCAT
Postgraduate• Foundation• Postgraduate – Birmingham Review
Substantive – Consultant & GP
Literature Reviews3 commissioned
WarwickNewcastleDurham
OverlappingSpecify methods usedIdentify gaps and where judgement will be needed
Durham: McLachlan & Turnbull• 236 References
• Behaviour predictive of future behaviour• Conscientiousness significant component of
concerns
Warwick: Thistlethwaite et al• 197 papers 1990-2009, medical school and
residency. • Most from USA
• Mostly looked at face validity and reliability NOT predictive validity
• Lack of consensus
• Non-medical literature – focus on employability skills (job analysis)
Newcastle: Illing et al• 202 references
• Foundation programme – narrow range of applicants, very few cannot do the job
• Selection methods must satisfy their stakeholders (employers, students etc)
• Cognitive tests are moderately predictive of later cognitive tests
• Non-cognitive elements need to be considered to ensure that a doctor is able to perform well
• Assessment centres allow for a range of methods to be used
Selection MethodsCombination of:
• Cognitive ability (academic)• Non-cognitive
Academic range very narrow• High ‘justice’
Incorporate measures of big 5 personality applied to medicine• Not IQ or general aptitude tests
Selection MethodsBig 5 Model of Personality is predictive of job and
academic performance• Extraversion (outgoing, sociable, impulsive)• Emotional stability (calm, relaxed)• Agreeableness (trusting, co-operative, helpful)• Conscientiousness (hardworking, dutiful,
organised)• Openess to experience (artistic, cultured, creative)
Assessment CriteriaReliability
ValidityConsistencyGranularity
Longevity
Will the process pick the ‘best’ applicants
across the UK?
Will the process be good for many
years?
Assessment Criteria
Will the process support or undermine educational
objectives?Educational Impact
How onerous is the process for applicants ?
Applicant Burden
To what extent does the process distract from
service delivery?
Clinician Time
Assessment CriteriaWill the process pick
minimise cheating or malpractice?
Compliance
Can the applicants see exactly where the
goal posts are?Transparency
Is there a level playing field?
Fairness (Justice)
Public Opinion Person in the street thinks that it is fair
Summary of Results
Better score
Lower 5-year cost
Selection to FP 2013Situational Judgement Test (SJT)
• SJTs will replace the ‘white space’ application form questions • This is an invigilated, machine markable test in exam conditions • The SJT will consist of 70 questions in 2 hours 20 mins
Educational Performance Measure (EPM)• The EPM will replace the academic quartile scores. The EPM score is
comprised of three elements: 1. medical school performance in deciles2. additional degrees3. academic achievements
The EPM and SJT will each be worth 50 points from a 100 point application score
The case for change
• ‘White space’ questions not sustainable as a selection tool• Will become steadily less discriminatory between eligible applicants (limited range of new questions that can be generated)
• SJTs draw upon bank of items to be available for each application round •Situations experienced in the Foundation Programme varied and complex• New items built incrementally and continuously against Job Analysis
• ‘White space’ questions non-invigilated conditions , model answers concerns about risk of plagiarism and coaching
• SJTs in invigilated conditions in the UK (3 national dates)• Not possible to revise for the SJT (scenarios complex, answers relate to judgement rather than knowledge)
Concern SJT and EPM
The case for change• Long-term technical reliability and validity could be improved
• 30 year evidence for reliability of SJTs • SJT pilots demonstrate the technical reliability, internal reliability, and validity for use for FP selection
•Academic quartile system - difficult to compare fairly between applicants from different medical schools (not standardised or subject to quality assurance across medical schools)
• EPM - standardised framework for deciles• Medical schools and students decide ‘basket of assessments’ • Schools will be required to publish their locally agreed deciles framework, which will facilitate transparency and quality assurance from the wider community. • Deciles fairer to applicants at margins
Concern SJT and EPM
Situational Judgement Test
What is a Situational Judgement Test?SJTs are:
• a test of aptitude• designed to assess the professional attributes expected of a
Foundation doctor• based on a detailed job analysis of an FY1 doctor
SJT questions assess your judgement by presenting you withchallenging situations you are likely encounter at work during the first year of an integrated Foundation Programme
Example SJT QuestionsThere are two question formats:
• Rank the five responses in the most appropriate order• Choose the three most appropriate responses from eight
You should answer what you ‘should’ do in the scenariodescribed, not what you ‘would’ do
Example Question 1 – rankingMr Reese has end-stage respiratory failure and needs continuous oxygen therapy.While you are taking an arterial blood gas sample, he confides in you that he knows heis dying and he really wants to die at home. He has not told anyone else about this as he thinks it will upset his family, and the nursing staff who are looking after him so well.
Rank in order the following actions in response to this situation (1= most appropriate;5= least appropriate).
A. Tell Mr Reese that whilst he is on oxygen therapy he will need to stay in hospitalB. Reassure Mr Reese that the team will take account of his wishesC. Discuss his case with the multi-disciplinary team*D. Discuss with Mr Reese's family his wish to die at homeE. Discuss Mr Reese's home circumstances with his General Practitioner
Answer to Question 1B. Reassure Mr Reese that the team will take account of his wishesC. Discuss his case with the multi-disciplinary team*E. Discuss Mr Reese's home circumstances with his General PractitionerD. Discuss with Mr Reese's family his wish to die at homeA. Tell Mr Reese that whilst he is on oxygen therapy he will need to stay in hospital
This question is focusing on effective communication with patients. Ensuring that patients’ informed wishes are met in relation to their care is central to your approach to patient care and this needs to be communicated to the patient in a reassuring manner even in situations relating to end of life care (B). These wishes should have been sought when addressing the management plan for Mr Reese and once identified the multidisciplinary team needs to be made aware of them in order to ensure that as far as possible Mr Reese’s views in relation to his end of life care are implemented (C). The management of Mr Reese will require the active involvement of his GP and communication with the GP is therefore of importance (E). Any decision to discuss Mr Reese’s wishes in relation to his end of life care with his family can only be made with the full agreement of Mr Reese (D). It would not be appropriate to give the patient inaccurate information in order to engineer a different medical pathway (A).
SJTs (Literature Review 77 papers)• Management, university, police, engineers• Large scale selection – short-listing• Construct validity not clearly identified
• Single construct (e.g. ‘practical intelligence’)• Can be designed to measure differing constructs
• Predictive validity will depend on what criterion is targeted
• SJT designed to test interpersonal skills will more likely predict inter-personal orientated performance
SJTs
• Used nationally to select GP registrars and other ‘high stakes’ occupations
• significant validity in predicting job performance • incremental validity over methods such as ability tests
and personality questionnaires • typically relate to general experience and ability, rather
than job-specific knowledge or experience • tend to show smaller differences between candidate
groups (e.g. based on race) than cognitive ability tests
Job analysis of FY1 doctor• Commitment to professionalism• Coping with pressure• Effective communication• Learning and professional development• Organisation and planning• Patient focus• Problem solving and decision-making• Self-awareness and insight• Working effectively as part of a team
Job analysis of FY1 doctor
Item development
Review, concordance,
piloting
Previously piloted & refined
(89 items)
Item writing workshops(43 items)
CIT interviews (78 items)
You have been prescribed codeine for persistent back pain which has become worse in the last few weeks. You have noticed that during shifts you are becoming increasingly tired, finding it difficult to concentrate and your performance, as a result, has been less effective. Choose the THREE most appropriate actions to take in this situation
Example Question 2 – multiple choice
A. Ask a colleague to assist with your workload until you finish your codeine prescription
B. Make an effort to increase the number of breaks during your next shiftC. Stop taking the codeine immediatelyD. Make an appointment to see your General PractitionerE. Seek advice from a specialist consultant about your back pain F. Arrange to speak with your specialty trainee (registrar)* before your next shift and
make them aware of your situationG. Seek advice from your clinical supervisor* regarding further support H. Consider taking some annual leave
Answer to Question 2D. Make an appointment to see your General PractitionerF. Arrange to speak with your specialty trainee (registrar) before your next shift
and make them aware of the situationG. Seek advice from your clinical supervisor regarding further support
This question looks at how you demonstrate commitment to professionalism and self-awareness. The essential problem is that as an FY1 doctor the level of your clinical performance is dropping. This constitutes a risk to the patients you are caring for and will impose a greater workload on your colleagues. In this circumstance you should inform and seek the advice of the senior clinician responsible for your work (G) and alert your colleagues (F). This matter is most likely to be related to your prescribed medicine and you should therefore consult with your GP (D) rather than any other specialist (E). It is not your place to re-allocate workload (A). Increasing the number of breaks is unlikely to improve a situation that is likely to be due to an adverse effect of a drug (B). You should not make any unilateral decisions about your medical treatment (C) and should seek the advice of others (D). You should not be seeking to use your annual leave (H) to compensate for a medical problem.
Scoring of the SJT Scoring key, determined through:• Consensus at item review stage (item writers, SMEs)• Expert judgement in concordance panel review• Review and analysis of the pilot data
Scoring not “all or nothing”, but based on how close to scoring key
Scoring of the SJT – ranking
• Up to 20 marks available
• Up to 4 marks available for each response (points for “near
misses”)
• No negative marking
Scoring of the SJT – multiple choice
• Points for each correct answer
• No negative marking
• 4 points for each correct answer
• Up to 12 points per item
A B C D E F G H
0 4 0 0 0 0 4 4
Parallel Recruitment Exercise (PRE)• New selection methods trialled alongside the normal selection
methods during 2012 FP application round
• Aims: logistics, awareness, pilot new SJT content
• All 31 medical schools involved
• SJT – 1 hr, 30 questions
• EPM – each medical school consulted study body on ‘basket of assessments’ to be used
Parallel Recruitment Exercise (PRE)• 90+% of FP applicants participated in the SJT
• Valuable learning experience ahead of live implementation
• Feedback to inform live implementation:
• Applicants
• Medical schools
• Was the final step in ensuring the selection methods can be consistently and robustly applied for implementation for FP2013
PRE - SJT30 item, one hour SJT
6,842 medical students took part in the PRE Participants included:
• Final year medical students• Students who had been pre-allocated to the Defence Deanery• Students who had chosen to take a year out post-graduate• International students returning overseas after graduation
30 medical schools (plus two centres for Eligibility Office applicants) delivered the SJT in 72 venues
Psychometric analysis shows that a 60 item SJT is a reliable measurement
Sheffield SJT pilot
Descriptive StatisticsInternal Reliability:
• Adjusted for a 60 - item test that included only robust items (such as would be used in an operational paper), all papers had an estimated reliability of α = 0.80 or above (α = 0.80 to α = 0.87)
• Demonstrates that the SJT is a reliable test in this context with items testing different things
Histograms showing the distribution of scores for Paper One (left) and Paper Two (right)
Descriptive StatisticsMean:
• Overall Mean = 79.5%; Range from 78.0% to 80.6%.So not too easy – differentiating appropriately
Standard deviations:• Mean SD = 18.6; Range from 17.3 to 20.0.
Distributions:• 305 to 468 (out of a maximum of 512) – as expected
given length of paper• Appears to be slightly negatively skewed, (more people
towards top end) although results do show a close to normal distribution
Item Facility (difficulty)Ranked Items
Maximum score 20Score 18 = ‘very easy’; Score 11.6 = ‘very hard’
Mean facility similar across all papers (approx 16)Range of facility values differed across papers
SD range similar for all papers, except for Paper 1 where one item
had a very high SD
Multiple Choice ItemsMaximum score 12
Score 10.8 = ‘very easy’;Score 3.6 = ‘very hard’
Mean facility across papers ranged from
7.9-9.0Range of facility values differed across papers
SD range was similar for all papers
Student views
The content of the assessment seemed relevant to the Foundation Programme
The scenario content seemed appropriate for my training level
Educational Performance Measure
Why change to EPM? • A clear framework with agreed principles used to calculate the EPM,
ensuring that it is fair, transparent and consistent across the schools of the UK
• Splitting cohorts into deciles rather than quartiles provides a wider spread of scores, which makes it easier to differentiate between applicants, and will be more fair for applicants at the margins
• It makes more sense for all the academic components of the application to be one part of the application
How is the EPM calculated? Score produced by applicant’s medical school to reflect achievementand performance compared to rest of cohort
EPM = 3 parts (maximum 50 points):1. Medical school performance in deciles (34 – 43 points)
E.g. Top 10% = 43; Top 20% = 42; etc2. Additional degrees (max 5 points)3. Educational achievements (presentations, prizes and
publications (max 2 points)
Schools have consulted with students about how the decile points for the EPM will be calculated.
PRE - EPM• 27 of 30 medical school initiated a new consultation and review of
framework• (3 schools consult annually and framework aligns)• Other schools do consult annually – but undertook new consultation with
students for the PRE• Majority of schools pleased with student engagement, especially amongst
later years• Benefit of raising awareness with students & staff• Students felt sense of ownership
Meetin
g stud
ent re
ps
Committe
e
Email to
all stu
dents
Online s
urvey
Open m
eetin
g
Studen
t reps
cons
ulted
Conve
ned w
orking
grou
p0
2
4
6
8
10
12
14
16
18
20
Consultation on frameworks
PRE EPM – Decile Points Decile/ Quartile
4th 3rd 2nd 1st
1st 2 6042nd 1 3 18 6463rd 1 5 341 3344th 2 13 649 275th 10 52 606 86th 13 625 50 27th 37 629 19 38th 337 348 109th 680 30 810th 671 7 2
PRE EPM – successes• All medical schools have agreed a ‘basket of assessments’ in
consultation with students
• All medical schools aligned with EPM framework
• All medical schools calculated EPM deciles, with around 10% in each decile (some ties)
• All medical schools confident they can calculate EPM deciles in line with common principles
Academic Foundation Programme2012 Same timetable as for standard applicationApply to 2 UoAHave to sit SJT
Appointed – application, EPM, interview (+/-)
If not appointed, then revert to standard process
Algorithms• Was initially triggered by student preferences
• Unstable, unfairness• Worse if increasing number applicants/school
• Changed for FP2012• Now triggered by application score
• Has changed patterns of applications
Selection to the Foundation Programme – improving and evolving
FP 2005Foundation Programme introduced
FP 2006National timetable and application process
FP 2007 Online application – white space & quartiles
2009-2011 Improving Selection to Foundation Programme
FP 2012Full-scale Parallel Recruitment Exercise (PRE)
FP 2013New selection methods implemented
‘The greatest forward step in the baking industry since bread was wrapped –
Missouri, 1928
What’s Wrong with SJTsLawrence Clinical Pharmacology
Wonderful
Not fit for my dog
Useful in some situations
More informationUKFPO - www.foundationprogramme.nhs.uk
• FP 2013 Applicant Handbook• Introductory videos• SJT monograph• SJT practice paper• FAQs (FP 2013, SJT, EPM)
Archived ISFP website – www.isfp.org.uk
Any questions?