W EB - BASED S CREENING OF A PPLICANT P ERSONAL C HARACTERISTICS C AN N OW B E D ONE R ELIABLY AND V ALIDLY S HOULD Y OU U SE I T ? T HE D EVELOPMENT AND R EAL - WORLD I MPLEMENTATION OF C OMPUTER - BASED A SSESSMENT FOR S AMPLING P ERSONAL C HARACTERISTICS -Harold Reiter -Kelly Dore Stemmler Fund W EB - BASED S CREENING OF A PPLICANT P ERSONAL C HARACTERISTICS W HY I M W ORRIED Length of talk Its the economy, stupid COMBINING T WO C ONCEPTS If you nail two things together that have never been nailed together before some s****k will buy it T HING 1 & T HING 2 THING 1 situational judgment testing THING 2 online testing S ITUATIONAL J UDGMENT T ESTING Scores responses to hypothetical but realistic scenarios 1950s, 1960s used for managerial assessments 1990s for workforce, medical and dental schools P LAY FIRST CASPER VIDEO HERE Question 1 What is your decision and why? Question 2 Would your response be any different if this family has four younger children expected to attend the same school in future? Question 3 Would your response be any different if the school was in a district with a wide range of income some very privileged families, some underprivileged? P ERSONAL /P ROFESSIONAL C HARACTERISTICS FINANCIAL COST $50,000,000,000 S TANDARDIZED T ESTS IN 2011 ESSAYLIB.com High-quality custom papers starting from $10/page All papers written from scratch We hire professional writers only Original and creative work Custom Personal Statements S TANDARD MEASURES OF PERSONAL / PROFESSIONAL CHARACTERISTICS P ERSONALITY T ESTING B IG 5 F ACTOR, CPI GPA MMI USMLE Step Clinical Clerk MCCQE I cogn MCCQE I C2LEO MCCQE II pass COGNITIVE PERSONAL/PROFESSIONAL CHARACTERISTICS 3 rd yr GPA Predictive Validity C HARACTERISTICS OF THE DESIRED INSTRUMENT Absolute Requirements Multiple samples Withhold control from examinees Relative Requirements Highly context dependent Score hard with hard, soft with soft Societal Requirements Cheap Fair to the less privileged M EDICAL C OLLEGE A DMISSIONS T EST 94,000 H OW SMALL PROBLEMS LOOM LARGE WHEN CHANGING MEDIUM OPTION # 1 Moving from MMI to online video-based SJT B ELGIAN M EDICAL & D ENTAL S TUDIES A DMISSIONS E XAMINATION 1999 ,444 test-takers Video stems 30 MCQ responses 723 admitted Internship performance R = 0.22, p < 0.05 Stopped after financial and technical constraints CASPer Computer-based Assessment for Sampling Personal characteristics 2003 2008: Psychometric research 2008 2010: Technological development 2010 2012: Implementation 8 Video Vignettes & 4 Self-Descriptive Non-clinical 3 follow-up questions 5 min short answer Scored by two raters CASPer C omputer-based A ssessment for S ampling Per sonal characteristics P LAY SECOND CASPER VIDEO HERE S TUDENTS CHEAT OPTION # 2 How can a non-proctored online test be secure? CASPer as a pre-interview Screen 2006 MCC 110 applicants NBME 167 applicants ReliableInter-Rater Correlation with MMI GPA MMI USMLE Step Clinical Clerk MCCQE I cogn MCCQE I C2LEO MCCQE II pass COGNITIVE PERSONAL/PROFESSIONAL CHARACTERISTICS 3 rd yr GPA Predictive Validity 2006 MCC & 2007 NBME STUDIES : MCC 110 63 MCCQE Part I, 53 MCCQE Part II NBME 167 46 MCCQE Part I, 12 MCCQE Part II Stemmler Fund CASPer COGNITIVE PERSONAL/PROFESSIONAL CHARACTERISTICS MCCQE I C2LEO MCCQE I PHELO MCCQE II C2LEO MCCQE Part I med expert Predictive Validity p < 0.05 M ICHAEL G. D E G ROOTE S CHOOL OF M EDICINE TARGET < 5% Incomplete O NGOING RESEARCH OPTION # 3 What further research can be done/is being done? W HAT S IMPORTANT TO ME OPTION # 4 Under what circumstances, if any, does CASPer become useful to me? T HANKS FOR P ARTICIPATING ! Harold Reiter MD MEd FRCPC DABR Kelly Dore PhD H OW SMALL PROBLEMS LOOM LARGE WHEN CHANGING MEDIUM OPTION # 1 Moving from MMI to online video-based SJT S TUDENTS CHEAT OPTION # 2 How can a non-proctored online test be secure? O NGOING RESEARCH OPTION # 3 What further research can be done/is being done? W HAT S IMPORTANT TO ME OPTION # 4 Under what circumstances, if any, does CASPer become useful to me?