The Interviewer Fallacy:Evidence from 10 years of MBA interviews
Uri Simonsohn Francesca GinoHBS
Photo not necessary
Motivation
• How is a journal editor like a venture capitalist?• Continuous flow of judgments
“random” “daily” subsets.• Research question: Impact of subsetting?
Narrow bracketing+Belief in law of small numbers interviewer fallacy
Definition. Reluctance to create subsets of judgments that differ too much from expected distribution.
Paper in one slide
• Data: 1-5 Rating of MBA interviewees – Handful per day.
• corr[avg(so far), this interview]<0• Ruled out alternatives:
– Contrast effects– Non-random sequence
Data Description
• A business school gave us data• 10 years: N=9,323, k=31
***INTERRUPT THIS TALK TO COMMENT ON ANOTHER PROJECT***
False-Positive (PsychScience2011): “list all your variables”Naysayers: “love to, have too many”Authors of False-Positive: “really?”Uri: “watch me.”
Note:The .pdf weighs 13Kb.The Wharton logo from slide 1:
11kb
A hardliner may say: Only reason to choose not to post is to hide information from readers.
Back to this talkData Description
• A business school gave us data• 10 years: N=9,323, k=31*
– Interviews per day M=4.5, SD=1.9– Cluster SE [repeated measures]
• Info on:– Applicant (e.g, GMAT scores, experience, race, gender)– Interviewer identity– Interview: time, date– Ratings (1-5 likert)
• 5 subscores: communication, leader, etc.• Overall score (M=2.9, SD=0.9)
• Would like to analyze like gambler fallacy– HHHHpr(T)↑
• Problem– Non-binary data– Covariates– Different interviewers
Instead:
Scorek,i = OLS(average score so fari , covariates)
k: Interviewee, 1 to N that day.i : Interviewer
Prediction:
<0
(1) (2) (3) (4)
Dependent variable:
Specification BaselineInterviewee
controlsInterviewcontrols
Score (1-5) of written
application
Average interview score -0.116*** -0.110*** -0.105*** -0.088**Given by same interviewer to previous interviewees that day (1-5) (0.038) (0.035) (0.036) (0.035)
GMAT score of applicant (/100) 0.244*** 0.250*** 0.079**(0.036) (0.035) (0.032)
Months of job experience of applicant (/100) 0.324*** 0.319*** 0.254***(0.057) (0.055) (0.055)
Number of interviews by same interviewer that dayTotal -0.000 0.001
(0.012) (0.012)
So far -0.018 -0.010(0.013) (0.014)
Score given by reader of application 0.340***(0.044)
Other controlsMonth*year dummies (k=12*9) Yes Yes Yes Yes
Interviewer dummies (k= 21) Yes Yes Yes Yes
Interviewee gender, race (k=9), age & age-squared No Yes Yes Yes
Interview's time (k=12) & location (k=4) No No Yes Yes
Interview Score(1-5)
Effect Size
• Average interview 1 point higher,• Equivalent to losing:
– 40 GMAT points, or – 30 months of experience.
Contrast vs. Interviewer Fallacy
Two divergent predictions:1) Same effect on the interview subscores? Explanation Prediction
Contrast: yes, and strongerInt.Fallacy: no, or at least weaker.
Data:- Every one of five subscores: n.s.- Average a-la Robyn Dawes: n.s.- Biggest point estimate, ¼ as big- one is >0
Contrast vs. Interviewer Fallacy
Two divergent predictions:2) Effect as end of day approaches. Explanation Prediction
Contrast: weaker (arguably)Int.Fallacy: stronger (absolutely)
Data:Estimate same regressions for:• last interview of day• 1 interview left• 2 interviews left
• If better candidates follow bad ones or vice-versa spurious finding.• Can we predict objective quality with average-interview-score-so-far?• Test:
GMAT=OLS(avg.score)Job Experience = OLS(avg.score)
Same table + 2 new columns(1) (2) (3) (4) (5) (6)
Dependent variable:GMAT
(250-800)Experience (in months)
Specification BaselineInterviewee
controlsInterviewcontrols
Score (1-5) of written
application
Same as (3)
Same as (3)
Average interview score -0.116*** -0.110*** -0.105*** -0.088** 0.085 0.251Given by same interviewer to previous interviewees that day (1-5) (0.038) (0.035) (0.036) (0.035) (2.063) (0.959)
GMAT score of applicant (/100) 0.244*** 0.250*** 0.079** -- 1.140**(0.036) (0.035) (0.032) -- (0.495)
Months of job experience of applicant (/100) 0.324*** 0.319*** 0.254*** 10.363** --(0.057) (0.055) (0.055) (4.541) --
Number of interviews by same interviewer that dayTotal -0.000 0.001 0.845 0.504*
(0.012) (0.012) (0.676) (0.290)
So far -0.018 -0.010 -0.461 0.008(0.013) (0.014) (1.275) (0.360)
Score given by reader of application 0.340*** 24.190***(0.044) (1.719)
Other controlsMonth*year dummies (k=12*9) Yes Yes Yes Yes Yes Yes
Interviewer dummies (k= 21) Yes Yes Yes Yes Yes Yes
Interviewee gender, race (k=9), age & age-squared No Yes Yes Yes Yes Yes
Interview's time (k=12) & location (k=4) No No Yes Yes Yes Yes
PLACEBOSInterview Score
(1-5)