46
on human and model-based decision making Chris Snijders
on human and model-baseddecision making
Chris Snijders
www.chrissnijders.com/eth2012
- Overview of course content on a lecture-by-lecture basis
- Inspirational material
- Assignments
www.chrissnijders.com/eth2012
Chris [email protected]
Eindhoven University of Technology
Background in mathematics (game theory / econometrics)
PhD in Sociology, now into Decision Making
www.chrissnijders.com/me
www.chrissnijders.com/eth2012
www.chrissnijders.com/eth2012
Passing the course …• Presence and participation
• Create a “CaseFile” based on the SuperCrunchers book (individually or in groups of 2) + evaluate others’ work
• Write assignment about your own “Super Cruncher” idea + evaluate others’ work
www.chrissnijders.com/eth2012
Overview of today
• Some famous examples
• The science behind it
• Computers as decision makers
www.chrissnijders.com/eth2012
Case: Cook county hospital
Emergency Department
- 250.000 patients per year- many persons without insurance- not enough rooms, overworked staff- 1996: Brendan Reilly director
(see Gladwell, 2005)
www.chrissnijders.com/eth2012
Problem 1: acute chest painDiagnose through:blood pressure, stethoscope: fluid in the lungs, how long have you been experiencing pain, how does it feel precisely, where does it hurt, does it always hurt or only when you exercise, have you had heart problems before, how about your cholesterol, do you have diabetes, let's look at your ECG, are there any heart problems in the family, do you use drugs, how old are you, are you in shape, do you smoke, do you drink, check appearance: stressed, overweight, ....
High risk : 8
Medium risk : 12
Go home30 p/day
www.chrissnijders.com/eth2012
Reilly finds Goldman: obv 10,000 cases
Only 4 things matter
ECGBlood pressureFluid in your lungs"unstable angina"
www.chrissnijders.com/eth2012
Great! So let's do that! Or not...
Implementation: … physicians protest …
A test: 20 cases were given to several physicians
Hardly any agreement between physicians!
www.chrissnijders.com/eth2012
Reilly tests Goldman’s idea
vs
82% 95%
physician Goldman’s scheme
www.chrissnijders.com/eth2012
A literature check …Clinical versus statisticalprediction
For instance (zie Grove et al., 2000)– Survival probabilities in medical procedures– Probability of recidivism– Probability of success of starting firms– Choice of job candidates– Diagnosing schizofrenia– Predicting school success– …
www.chrissnijders.com/eth2012
The results …
Over 160 studies
When given the same info,the number of cases in whichthe expert wins = ??
www.chrissnijders.com/eth2012
0
www.chrissnijders.com/eth2012
Models beat Humans (quite often) How can this be?
www.chrissnijders.com/eth2012
... we have some clues ...
We emphasize the improbable’ (Stickler)Confirmation bias (Edwards,
Wason)
Hindsight bias (Fischhoff)Cognitive dissonance (Festinger)
“Dealing with probabilities / Base rate neglect”(Bar-Hillel)
Mental sets (Redelmayer, Tversky)
Our memory fools us (Wagenaar)
www.chrissnijders.com/eth2012
And there are more of these
"Mental Floating Frankfurters"
www.chrissnijders.com/eth2012
Restriction 1: “Mental sets”Connect the 9 dots with at most 4 straight lines, without lifting your pen from the paper.
• • • • • • • • •
www.chrissnijders.com/eth2012
Restriction 2: Memory
“Where were you, when …”
Shuttle Columbia Crew Lost Feb. 1, 2003
www.chrissnijders.com/eth2012
Restriction 3: the “availability heuristic”
Which is more likely, a plane crash or a car crash?
www.chrissnijders.com/eth2012
Restriction 4: dealing with probabilities
Suppose: a manager has a good intuition in business:– when a problem will arise: he gets a gut-feeling that something is wrong
with probability 90%
– when no problem will arise: he gets a gut-feeling that something is wrong with probability 10%
On average, there is a problem in 5% of the transactions.
The manager starts a transaction, and he gets a gut-feeling that something might be wrong.
What is the probability that something is indeed wrong?
www.chrissnijders.com/eth2012
Restriction 4: dealing with probabilities
A murder has been committed. The only evidence available is DNA, found at the murder scene. DNA-research shows a match with your DNA.
The probability that two persons are diagnosed as having the same DNA is about 1 in 100.000.
How likely is it that you are the murderer?
www.chrissnijders.com/eth2012
Restriction 5: overconfidence
Trivial Pursuit: estimate how many questions you answer correctly
Estimates are generally too high ... and this gets worse with expertise!
www.chrissnijders.com/eth2012
Restriction 6:Finding non-existent patterns
www.chrissnijders.com/eth2012
Restriction 7: the noble art of finding a broken leg
www.chrissnijders.com/eth2012
Restriction 8: where is the feedback?
www.chrissnijders.com/eth2012
Restriction 9: Hindsight bias
http://www.hss.cmu.edu/departments/sds/media/pdfs/fischhoff/HindsightEarlyHistory.pdf
www.chrissnijders.com/eth2012
A list of biases …
www.chrissnijders.com/eth2012
Decision making =Store, retrieve, combine
www.chrissnijders.com/eth2012
Where were we?
End of a big set of reasons why humans (even expert humans) are often outperformed by computer models.
www.chrissnijders.com/eth2012
The competition:
• “Naturalistic decision making”
• Fast and frugal heuristics
(covered only to some extent)
www.chrissnijders.com/eth2012
Intuition at its finesthttp://dsc.discovery.com/tv-shows/dirty-
jobs/videos/chicken-sexer.htm
www.chrissnijders.com/eth2012
www.chrissnijders.com/eth2012
Reilly finds Goldman: obv 10,000 cases
Only 4 things matter
ECGBlood pressureFluid in your lungs"unstable angina"
www.chrissnijders.com/eth2012
AND?are we using this today?
www.chrissnijders.com/eth2012
NO!
www.chrissnijders.com/eth2012
Back to the course ...
... the science behind many more questions that you can ask in relation to such topics
• This is an innovation adoption process that needs to take standard hurdles.
• Can we find consistencies across topics?• Which kind(s) of crunchers are more likely to be
adopted?• etc...
idea implementation
www.chrissnijders.com/eth2012
www.chrissnijders.com/eth2012
Diagnosing Actinic Keratosis
The power of intuitive judgment
vs
The rigor of statistical prediction
www.chrissnijders.com/eth2012
When and why do the models win?
Can we use the experts’ knowledge somehow?
When are the models used, and when not (and why is that)?
What can/should you do when you want to have a model-based solution?
What prevents people from using models?
Typical questions in the area
www.chrissnijders.com/eth2012
About the Supercruncher book
www.chrissnijders.com/eth2012
The Supercrunchers book
Intro
Who's doing your thinking for you?
Creating your own data with the
flip of a coin
Government by chance
Evidence based medicine
Experts vs equations
Why now?
Are we having fun yet?
On the web site [CaseFile] [Example] [Issue] [Method]
www.chrissnijders.com/eth2012
“Super Crunching”, what is that?• Using (lots of) data to
predict something (think Twitter, Blogs, Airmiles, …) that we normally cannot predict
• Using data to predict something that humans normally tend to predict– Experts vs models– Experiment– “Natural experiments”
www.chrissnijders.com/eth2012
show website (if I had not done that before)
www.chrissnijders.com/eth2012
To do• Read the book – cover to cover, asap
• Think about the different cases you encounter, try to uncover general patterns
• Upcoming assignment will be to create a “casefile” for one of the topics in the book: check for topics that interest you …
• Next lecture at 17:00 (it’s a colloquium), and tomorrow at 13:00 again.
www.chrissnijders.com/eth2012
Top 6 of statistically speaking total bogus professions
1. Weather predictors
2. Sports predictors3. “Profilers”4. Art critics5. Wine experts6. Stock market experts
