Admissions Screening for People Skills
Admissions needs defensible strategies
Widening Access
Higher number of top
applicants
Evidence-Based Tools+ =
Current screening tools…
…aren’t perfect.
InterviewsReference
Letters
Personal
Statements
or Essays
Current screening tools:
Are
burdensome on
admissions and
faculty
Contribute to
lack of access
and diversity
Do not predict
for what we
want according
to research
Give applicants
opportunity to
‘fake good’
Unfit students drain resources
XX hours in
remediation
time
$XX in costs
+1
Problematic
Student
=
The ideal screening test prioritizes:
Access and
Diversity
Acceptable, accessible, and equitable
to a wide range of candidates
regardless of geography,
backgrounds, or social barriers
Research-Based
Efficacy
Validation and continuous
research including QA/QI and
collaborations with our
partnered institutions
Optimize
Resources
Maximize reliability and
prediction of performance
while minimizing burden on
admissions and faculty
Efficiency
Accessibility
Validation
Complementary screening
tool for people skills
Introducing
THE
CASPer®
ASSESSMENT
Sources: Juster F, Reiter HI, Dore KL, Baum R. Are Medical Student Demographics Impacted by Implementation of a Situational Judgment Test?
presentation April 17, 2016, Montreal, Canadian Conference on Medical Education
Based on NYMC 2015, over 20% of US med school
applicants comparing GPA, MCAT and CASPer
Less impact on
UIM / URM
One-third the effect
of GPA or MCAT
Higher scores
for female
applicants
Opposite that of
GPA or MCAT
Higher scores
for applicants with
paid employment
before 18
Opposite that of
GPA or MCAT
Demographic Impact
8.4 (Very important)
7.8 (More likely to apply)
6.3 (Somewhat well)
6.3 (Somewhat satisfied)
7.8 (Very satisfied)
1 2 3 4 5 6 7 8 9 10
17.7%
On a scale of 1-10 (with 10 being very satisfied/well/likely/important):
AUS Teacher’s Education programapplicants
ACCEPTIBILITY
6,003 applicants wrote CASPer™ for the 2017
cycle. The applicants were asked a series of
questions about their experience and thoughts
of the test.
How satisfied are you with the ease
of signing up for the CASPer test?
How satisfied are you with the
smoothness of the CASPer test
delivery?
How satisfied are you with the
overall CASPer test experience?
How satisfied are you with the
customer support you received?
How well do you think the CASPer
test will differentiate your personal
characteristics compared to other
applicants?
How likely does having the CASPer
test as a requirement make you
more likely or less likely to apply to
a school?
How important is it to you that the
program you are applying to values
getting an objective measure of
your personal characteristics?
8.7
(Very
satisfied
)
8.2
(Very
satisfied
)
Applicant Physical Location While Writing
Applicants have taken
the CASPer test from
177 different countries
Demographics3Coaching Effect1
Lowers SJT scores CASPer ongoing
Practice Effect2
Occurs less in SJTCASPer ongoing
Sources: 1Cullen MJ Int J Selection & Assessment 2006;14:142-55; 2,3Patterson F Med Educ 2012;46:850-68
Why SJT?
Sources: 1Lievens F J Appl Psychol 2005;90:442-52, Lievens F J Appl Psychol 2006;91:1181-8; 2Ginsburg S Acad Med 2000;75:S6-
S11; 3Kylonnen P AERA Montreal 2005; 4CASPer-12, BMDSAE-30, UK Work Psych Group-70; 5Dore KL Acad Med 2009;84:S9-S12
Videos
High predictive validity1
Open Text Response
High predictive validity2
No ceiling effect3
Betterfeasibility4
Audio responses lose anonymity, increase rater
time, lose MMI correlation5
SJT Benefits
• ESL scored .13 SD lower than EFL
applicants
• Effect is 2/3 smaller than closed-
response SJT (UKCAT-SJT
Monash)
• CASPer r=.12 vs UKCAT SJT r=.36
Comparison between ESL (n = 284) and EFL (n = 1,425) applicants.
Difference is significant (t(1707) = 3.86, p < .001), but minimal.
Difference is
minimal.
ESL vs. EFL
• At Grade 7 reading level
= no impact on score
Reading Level
Score vs Flesch Grade Level
• n = 40,457
US MD applicants
• Correlation between
age and CASPer score
is minimal (r= -0.10)
Figure. Stem-and-leaf plot of CASPer scores across age. The midline represents
the mean, the top of the box represents the top 25th percentile and the bottom of
the box represents the bottom 25th percentile. The line represents the general
spread of CASPer scores, with individual dots representing outliers.
Age
Effect is
minimal.
Effect of Age on
CASPer Z Score (USMEDICINE)
1 2 3 4 5 6 7 8 9
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
Pe
rce
nta
ge
of M
issp
elle
d w
ord
s in
Re
sp
on
se
Score
Box Plot of Percentage of Misspelled Words in Response vs. CASPer Score
Correlation coefficient = -0.01
Do spelling mistakes matter? No.After analyzing 27,000+ CASPer™ responses,CASPer score does not change at all with increased spelling mistakes.
Our PartnersCanadian Partners
Dentistry
Laval University
McGill University
University of British Columbia
University of Montreal
Graduate Occupational
Therapy & Physical
Therapy (OT/PT)
McGill University (Masters PT)
Sherbrooke University (OT/PT)
University of Alberta (OT)
Graduate Nursing
Laval University
McGill University
University of Montreal
Nursing
Cape Breton University
College of the Rockies
Conestoga College
Dalhousie University
McGill University
McMaster University
Mohawk College
Nipissing University
North Island College
St. Francis University
Trent University
University of British Columbia
University of Ontario
Institute of Technology
Vancouver Island University
York University
Optometry
University of Montreal
University of Waterloo
Pharmacy
Laval University
University of Montreal
University of Waterloo
Postgraduate Medicine
Dalhousie University -
Family Medicine
University of Manitoba
University of Saskatchewan
Social Work
Nipissing University
Speech Therapy
McGill University
Undergraduate Medicine
Dalhousie University
McGill University
McMaster University
Sherbrooke University
University of Alberta
University of Ottawa
Undergraduate Occupational
Therapy & Physical Therapy
McGill University
University of Sherbrooke
Veterinary
University of Montreal
Australian Partners
Medicine
Macquarie University
Wollongong Graduate
Medical School
Postgraduate Teacher
Education
Australian Catholic University
Deakin School of Education
Federation University Australia
La Trobe University
Monash University
RMIT University
Swinburne Institute of
Technology
Victoria University
Undergraduate Teacher
Education
Australian Catholic University
Deakin School of Education
Federation University Australia
La Trobe University
Monash University
RMIT University
Swinburne Institute
of Technology
Victoria University
United States Partners
Anesthesiologist Assistant
University of Colorado Denver
Osteopathic Medicine
Des Moines University
Touro University Harlem
Physician Assistant
Campbell University
Christian Brothers University
Colorado Mesa University
Eastern Virginia Medical School
Hardin-Simmons University
Idaho State University
Keiser University
Marquette University
Oklahoma University
Pacific University Oregon
Presbyterian College
Rocky Mountain University
of Health Professions
University Evansville
University of Kentucky
West Liberty University
Undergraduate Medicine
Albany Medical College
Central Michigan University
Drexel University
Florida Atlantic University
Medical College of Wisconsin
Mercer University
New York Medical College
Quillen College of Medicine
Rosalind Franklin University
Ross University
Rutgers University
Temple University
Texas A&M University
Texas Tech University
Tulane University
University of Colorado
Medical School
University of Illinois at Chicago
University of Michigan
University of Vermont
West Virginia University
New Zealand Partners
Veterinary
Massey University
UK Partners
Medicine
University of Central
Lancashire
Rating and Scoring
SCORE
SCORE
Raters are blinded
Each rater focuses on 1 scenario, gets anonymized responses,likert 1-9 RPM
CASPer score is an average of 12 unique impressions
Diverse rater pool of more than 200 people
Human Raters
4 5 4 4 3 6 5 4 4 3 4 6
SCORE
• Monitored daily with
structured QA system
• Responses from single
section oversampled
• Raters cannot rate applicant
more than once
• Rater QA metrics monitored include:
• low inter-rater reliability (IRR)
• overly low or overly high mean ratings
(too lenient or too strict)
• low standard deviation (is not using the full
range of the scale to differentiate applicants)
• excessively fast reading speed
• misuse of flagged responses Sample printout of rater descriptions (raters are anonymized for this document)
Rater QAEIRR Responses 2017-07-09T14Responses for USMEDICINE test on 2017-07-09T14
The test ID: DMYzuLZKx8u3RHGRThe inter-rater reliability intraclass correlation (ICC) for this test was: 0.95
The scenario oversampled was: Public Shaming (Set A) which was a video scenario.
MrEqDKcRmGmrptxdQ
qSEe2kEEPziz7C5Lw
wcr3m8pbpHGwaayXp
BAfnsq5EnJeqZjtX6
336i7FHh9fJBzECmt
RAom2nirf3WiqmNXQ
eBlwNbMA3Pw44BJ6H
tXCHd5aa0qfvTDT2C
U{fJWf8teslW,NbAX
siZLcnHNuj9ri9Hjg
XPLsp7Mn2PmTS3bux
A2iw8Cb6q2DS2ZRcB
iZQrH8wJFHYqAdy3K
SoZGXSyupCiAYXNjE
nm28adLt5gBnxopX
g3EPpyrtzwoTi9ZrD
ygSWgJFQRmdCSCua8
5.9
5.6
5.7
5.45
6.25
5.05
6.05
5.8
5.15
5.1
5.6
3.9
5.85
6.6
5.2
4.7
5.05
1.21
1.67
1.26
1.28
1.33
0.76
0.51
1.82
1.84
0.85
1.6
1.55
1.39
1.76
1.2
1.53
0.94
SCORE
SCORE
Raw score is converted to
z-score
Scoring
4 5 4 4 3 6 5 4 4 3 4 6 4.33
SCORE
Z-ScoresRaw scores are converted into z-scores
Easier to interpret
Understand
where student is
in relation to
their peers
Normalize
scores between
test slotsAccounts for
parallel test
forms
Needed for
formulaic method
Use common scale
when combining
multiple measures
Calculating Z-Scores
Global Ranking
Specific to Test Timeslot
Normalizes across
different content
Student compared
to all test takers in
the cohort
Mean of the group is set to zero (0)Applicants above the mean have positive z-score
Standard Deviation
Cumulative Percentile
Applicant with z-score of
+1.0 sits in the 84th percentile
Z-Scores and Percentiles
USING Z-SCORES IN SELECTION
Threshold
Applicants must meet a certain
threshold to be considered for
admission to the program
Eg. Z-Score > -1.0 for CASPer
Formulaic Ranking
CASPer and cognitive z-scores
combined to create overall score for
applicant. Applicants can then be ranked
based on their overall score.
%+%
Combination
Step 1: ATAR Threshold
Step 2: CASPer Threshold
Step 3: Formulaic Ranking
FORMULAIC RANKING
• CASPer and cognitive z-scores combined to
create overall score for applicant
• Applicants can then be ranked
123 Applicant CASPer Z-
Score(50%)
ATAR Z-Score(50%)
Overall Z-
Score
Jane Doe 1.12 0.96 1.04
Overall = 50% of CASPer + 50% of GPA
= (1.12 x .5) + (0.96 x .5)
= (0.56) + (0.48)
= 1.04
Screen In
Screen Out
Screen Out / Screen In