"Let's leave that up to the computer" Chris Snijders.

transcript

"Let's leave that up to the computer"

Chris Snijders

www.chrissnijders.com/supercrunchers2012

Chris Snijders, Super Crunchers, 2012

Chris Snijdersc.c.p.snijders@gmail.com

Martijn Willemsenm.c.willemsen@tue.nl

Kees van Overveldk.van.overveld@wxs.nl

Passing the course …

• Presence and active participation, including presentation, evaluation of others, …

• Create “CaseFile” based on the SuperCrunchers book

• Write assignment about own “Super Cruncher” idea

Overview of today

• A famous example: Cook county hospital• The science behind it• Computers as decision makers

• Warning! Upcoming assignment

Case: Cook county hospital

Emergency Department

- 250.000 patients per year

- many persons without insurance

- not enough rooms, overworked staff

- 1996: Brendan Reilly director

(see Gladwell, 2005)

Problem 1: acute chest pain

Diagnose through:

blood pressure, stethoscope: fluid in the lungs, how long have you been experiencing pain, how does it feel precisely, where does it hurt, does it always hurt or only when you exercise, have you had heart problems before, how about your cholesterol, do you have diabetes, let's look at your ECG, are there any heart problems in the family, do you use drugs, how old are you, are you in shape, do you smoke, do you drink, check appearance: stressed, overweight, ....

High risk : 8

Medium risk : 12

Go home30 p/day

Reilly finds Goldman: obv 10,000 cases

Only 4 things matter

ECGBlood pressureFluid in your lungs"unstable angina"

Great! So let's do that! Or not...

Implementation: … physicians protest …

A test: 20 cases were given to several physicians

Hardly any agreement between physicians!

Reilly tests Goldman’s idea

82% 95%

physician Goldman’s scheme

A literature check …

Clinical versus statisticalprediction

For instance (zie Grove et al., 2000)

– Survival probabilities in medical procedures– Probability of recidivism– Probability of success of starting firms– Choice of job candidates– Diagnosing schizofrenia– Predicting school success– …

What would a typicaltopic be, where you

would expect the humanto win?

The results …

Over 160 studies

When given the same info,the number of cases in whichthe expert wins = ??

Models beat Humans (often)How can this be?

... we have some clues ...

We emphasize the improbable’ (Stickler)

Confirmation bias (Edwards, Wason)

Hindsight bias (Fischhoff)

Cognitive dissonance (Festinger)

“Dealing with probabilities / Base rate neglect”(Bar-Hillel)

Mental sets (Redelmayer, Tversky)

Our memory fools us (Wagenaar)

And there are more of these

"Mental Floating Frankfurters"

Restriction 1: “Mental sets”

Connect the 9 dots with at most 4 straight lines, without lifting your pen from the paper.

• • • • • • • • •

Restriction 2: Memory

“Where were you, when …”

Shuttle Columbia Crew Lost Feb. 1, 2003

Restriction 3: the “availability heuristic”

Which is more likely, a plane crash or a car crash?

Restriction 4a: dealing with probabilities

Suppose: a manager has a good intuition in business:

– when a problem will arise: he gets a gut-feeling that something is wrong with probability 90%

– when no problem will arise: he gets a gut-feeling that something is wrong with probability 10%

On average, there is a problem in 5% of the transactions.

The manager starts a transaction, and he gets a gut-feeling that something might be wrong.

What is the probability that something is indeed wrong?

Restriction 4a - solution

1,000 transactions

50 are problematic950 are not problematic

45xsignal

5xNo signal

855xNo signal

95xsignal

If a signal, then the probability of a problem = 45 / (45+95) = 0.32

Restriction 4b: dealing with probabilities

A murder has been committed. The only evidence available is DNA, found at the murder scene. DNA-research shows a match with your DNA.

The probability that two persons are diagnosed as having the same DNA is about 1 in 100.000.

How likely is it that you are the murderer?

Restriction 5: overconfidence

Trivial Pursuit: estimate how many questions you will know

Estimates are generally too high ... and this gets worse with expertise!

Restriction 6:Finding non-existent patterns

Restriction 7: the noble art of finding a broken leg

Restriction 8: where is the feedback?

Decision making =Store, retrieve, combine

Where were we?

End of 1 set of reasons "why" humans (even expert humans) are often outperformed by computer models.

The competition:

• “Naturalistic decision making” (cf. work by Shanteau)

• Fast and frugal heuristics (cf. Gigerenzer)

The timelines of ideas

... and the science behind many more questions that you can ask in relation to such topics

• This is an innovation process • Can we find consistencies across topics?• Which kind(s) of crunchers are more likely to

be adopted?• etc...

idea implementation

Reilly finds Goldman: obv 10,000 cases

Only 4 things matter

ECGBlood pressureFluid in your lungs"unstable angina"

AND?are we using this today?

When and why do the models win?

Can we use the experts’ knowledge somehow?

When are the models used, and when not (and why is that)?

What can/should you do when you want to have a model-based solution?

What prevents people from using models?

The Supercrunchers book

Who's doing your thinking for you?

Creating your own data with the

flip of a coin

Government by chance

Evidence based medicine

Experts vs equations

Why now?

Are we having fun yet?

On the web site

[CaseFile]

[Example]

[Issue]

[Method]

“Super Crunching” = using data

• Supercrunching = Using (lots of) data to predict something (think Twitter, Blogs, Airmiles, …) that we normally cannot predict

• Supercrunching = (also) Using data to predict something that humans normally tend to predict– Experts vs models– Experiment– “Natural experiments”

Passing the course …

• Presence and active participation, including presentation, evaluation of others, …

• Create “CaseFile” based on the SuperCrunchers book

• Write assignment about own “Super Cruncher” idea (or: we take a single topic and each do parts of it)

Up next …

• Next week: no class (read! And start preparing the first Assignments)

• The week after that: we get creative…

To do for next week

• Read the book – cover to cover

• Think about the Cook-county case, try to think about general patterns

• Upcoming assignment will be to create a “casefile” for one of the topics in the book: check for topics that interest you, if possible choose one already, and let me know

"Let's leave that up to the computer" Chris Snijders.

Documents