GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
GeoMET 2013 Carrasco Lecture, 30Sept2013
Uncertainty, Decisions, Models & People
Steve BeggAustralian School of
PetroleumUniversity of Adelaide
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Outline
• Uncertainty, Decisions & Modeling– Uncertainty & Business Perfomance– Uncertainty & Decision-making– Need for integrated models
• Uncertainty, Judgments & People– The Nature of Uncertainty – Uncertainty & Value Maximization
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Industry Performance: Comments & Observations
• Super-major:– “Every one of our 10 most important projects failed to
generate the desired return.”
• Large independent:– “The actual performance of our key assets wasn’t even within
the P1 to P99 range.”
• CEO to manager:– “I want your guarantee that we will not spend more than the
P50 on this project!”
• IPA:– “The bigger and more important a project gets, the more likely
it ends up in the “disaster” category (1 in 8 major projects)”
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Uncertainty
A fundamental problem
Business outcomes not living up to expectations or possibilities!
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
A fundamental problem
Business outcomes not living up to expectations or possibilities!
• People tend to grossly under-estimate uncertainty– number of uncertain factors and the magnitude of uncertainty and is
its consequences (good or bad)– complexity of the relationships between them and therefore un-
anticipated non-intuitive outcomes)
• Naive understanding of NPV “rule” – Uncertainty (and/or delay) = Value Loss. Biased to risk mitigation.
• Better decision-making requires accurate (= unbiased & appropriate range) uncertainty assessment & response– Reduce uncertainty only IF it can change a decision AND expected
benefit of reduction is less than its cost– Mitigate downside risk AND capture upside opportunities– Exploit interactions – dependencies, correlations
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Uncertainties matter: they are everywhere in the (Oil & Gas) evaluation “system” ……
EconomicsAsset A
. . .
OOIP Model
Production Data
ReservoirSimulation
Petro-physics
Portfolio
EconomicsAsset 1
EconomicsAsset n
Prices
Taxes
PredictedProduction
Royalties/PSC
Decline
Seismic
Geology
Drilling
OpEx
CapEx
ProductionAllocation
Export
Processing
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Uncertainties matter: they are everywhere in the evaluation “system” ……
EconomicsAsset A
. . .
Resource Model
Production Data
ResourceSimulation
Petro-physics
Portfolio
EconomicsAsset 1
EconomicsAsset n
Prices
Taxes
PredictedProduction
Royalties/PSC
Depletion
Geophysics
Geology
Mining
OpEx
CapEx
Export
Processing
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
….. and occur at many levels …….
• Each domain must address nested layers/types of uncertainty, e.g. sub-surface description, …...
Data
Spatial/Temporal variability
Parameter / TI
Interpretation/Model
Monte Carlo
Geostatistics (or mp)
Experimental Design
Scenario Modeling, Discrete Probabilities
…... with appropriate modeling techniques
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
…. and we spend a lot of money without really knowing which ones matter
Average Porosity
Rig Cost
Recovery Factor
Gross Rock Volume
Fiscal Terms
Net:Gross
Saturation
Continuity
Oil Price
Facilities
UncertaintyParameter Impact on NPV
Take more core
Renegotiate contract
Build simulator model
Buy more seismic
Fire lawyers
More gamma logs
Different rock model
Survey Analogues
Hedge with futures
Cheaper Steel Supply
Action
"There is nothing so inefficient as very efficiently doing the wrong things".
Peter Drucker
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
…. and we spend a lot of money without really knowing which ones matter
Average Porosity
Mining Cost
Recovery Factor
Gross Ore-body Volume
Fiscal Terms
Net:Gross
Grade
Continuity
Copper Price
Processing Facilities Cost
UncertaintyParameter Impact on NPV
Take more core
Renegotiate contract
Build simulator model
Drill more holes
Fire lawyers
More gamma logs
Different rock model
Survey Analogues
Hedge with futures
Cheaper Steel Supply
Action
"There is nothing so inefficient as very efficiently doing the wrong things".
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
A fundamental problem
Better uncertaintymanagement
BetterDecisions
Betterperformance
Flawed Decision-Making
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Decision-making is about ranking:Case 1: we don’t even need a decision criterion
A
Unc
erta
inty
in d
ecis
ion
Crit
erio
n, e
gN
PV
B DCAlternatives (choices)
We only need to predict values precisely enough to determine which is the best alternative
Here, B is clearly the best choice
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Decision-making is about ranking:Case 2: need a decision criterion
A B DCAlternatives (choices)
Unc
erta
inty
in d
ecis
ion
Crit
erio
n, e
gN
PV
We only need to predict values precisely enough to determine which is the best alternative
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Decision Criteria
• OK, so we’ve got our PDFs of some decision variable (egNPV, tonnage) for each decision alternative
• What number should we use as a decision criterion?– the P10, P50, P90?– the mode?– the mean? NPV
0
• Answer: Expected Value (for a “risk-neutral” decision-maker)- BUT don’t “expect” the Expected Value!
• By “decision criterion”, we mean a number that can be used– to compare the different decision alternatives, so that we can
choose the best (if alternatives are mutually exclusive)– or compare against a minimum acceptable hurdle (if there is only
one alternative, or multiple non-mutually exclusive alternatives)
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Decision Criterion under Uncertainty: Choose the alternative with maximum Expected Value
• Intuitively– Defines what would happen “on average” if we repeat the situation
numerous times
• Mathematically– The probability weighted average of the possible values (discrete PDF)
n
Expected Value = pi xi1
• BUT, don’t “expect” the Expected Value!• No other metric (Mode, P10/50/90, etc) will give a higher
total value over multiple (different) decisions• For a “risk-neutral” decision-maker, it is value they would
attribute to the alternative if all uncertainty was removed.
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Avoiding Confusion: Decision Criteria v Predictions
• The PDF is the prediction of all possible outcomes (and their associated probabilities)
• The mode is the Most Likely outcome - So, perhaps, it is the best prediction of outcome
• The Expected Value is a Decision Criterion – ie used to make the best decision – it is not a prediction of the outcome!- at least, it is no more of a prediction than any of the other possible
outcomes in the PDF are- it might not even be a possible outcome!
• For decision making we need unbiased estimates of the EVs of our decision criteria (eg NPV, Reserves)- which requires propagating (unbiased) assessments of uncertainty
in input variables through to uncertainty of the decision criteria
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Technical Work in the context of Decisions and Uncertainty: Knowing when enough is enough.
• Technical work is fundamentally about uncertainty assessment for the purpose of making decisions
– First priority: Accurate (=unbiased) uncertainty assessment – Second priority: Uncertainty reduction.
• But if you have a “make the best possible prediction” focus, there is no stopping rule
– you can always reduce uncertainty a bit more (more data, more time, more detail, more accurate physics/geology/tax, more analysis, ...)
• A decision-driven (ranking) focus gives a trivially simple stopping rule
– Stop when further analysis doesn’t change the decision!!
The main role of a geoscientist, engineer or economist is to support decision-making
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Propagating Uncertainty:Flaw of Using Averages (after Savage)
Log, Price,NCF...
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Averages don’t always work
• For non-linear processes,– reservoir simulation– volumetrics with cut-offs– development alternatives
even if only a single, “best” estimate is required, we still need to use complete range of inputs - cannot use an average input
• Also, P10 (P90) results are NOT given by taking P10 (P90) inputs and running them through the model
ResultY = 4
ModelY = X2 Z2/
3 21 1 8
Mean = 10 Mean = 5
x z
Y
Simulation Result
0 30
True Mean~ 7.8
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
The Flaw of Averages: implications for decision metrics
• If f is a decision metric (eg NPV) computed from a model with uncertain input variables x,y,z,…
unless f is linear
, ,[ ( ]....)x y zf
( [ ], [ ], [ ], )...x y zf
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
On Models ……
“…. when you can measure what you are speaking about and express it in numbers, you know something about it. But when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind. It may be the beginning of knowledge, but you have scarcely in your thoughts advanced to the state of science, whatever the matter may be.”
Lord KelvinTaken from Davis, 1986
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
On Models ……
What could be cuterThan to feed a computerWith wrong informationBut naïve expectationTo obtain with precisionA Napoleonic decision
Major Alexander P. de SeverskyTaken from Davis, 1986
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Remember:
Average Porosity
Rig Cost
Recovery Factor
Gross Rock Volume
Fiscal Terms
Net:Gross
Saturation
Continuity
Oil Price
Facilities
UncertaintyParameter Impact on NPV
Take more core
Renegotiate contract
Build simulator model
Buy more seismic
Fire lawyers
More gamma logs
Different rock model
Survey Analogues
Hedge with futures
Cheaper Steel Supply
Action
Expected Value
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
The industry has been over-focussed on: 1. Developing and using ever more efficient tools for trying to
come up with precise “single number” estimates2. Modelling approaches based on simply “wrapping” a
probabilistic framework around our (very efficient!) classical deterministic tools
3. Modelling information (uncertainty) in the absence of its impact on decisions
How?
"There is nothing so inefficient as very efficiently doing the wrong things".
Peter Drucker
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
On Models
• Once you accept decisions really are made under uncertainty, and can be optimally so, it impacts the whole way you view “technical” work (geological, engineering, economic, commercial, legal)
“All models are wrong, some models are useful”
Box
“I would rather be vaguely right than precisely wrong”
Keynes
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
How?
Model “Richness”HiLo
Lo
Hi
IntegratedUncertainty Evaluation
(No. of“runs” orScenarios)
Too expensiveMissing the
point (ranking)High P(wrong)
Two Components
1. Modelling “philosophy”
2. Type of ModelsClassicalmodels
(precisely wrong)
Vaguely Right (opportunity to create
value)
Trade-off some “domain” rigor for ability to perform integrated uncertainty evaluation
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Decision, or feed to portfolio
Monte Carlo Simulation
Sens
itivi
ty A
naly
sis
G&GModel
ProductionModel
DrillingModel
ProcessingFacility Model
ExportModel
Scheduling of Decisions & Implementation
Economics
Taxes &Royalties
Costs
Prices
Scenario and Decision Analysis
Classical Modeling
Uncertainty Estimates, Calibration, Surrogates
Prob
Stochastic Integrated Asset Modelling System (Oil&Gas). Insightful?
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
The role of “Classical” Modeling
• Uncertainty Estimates of Input Parameters– Simple sensitivity studies
• Calibration of simplified models • Generation of high-fidelity simple surrogates
– Experimental Design and Response Surface Modeling
Classical Modeling
Uncertainty Estimates, Calibration, Surrogates
Holistic Evaluation System
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
End
Middle
Start
ClassicalApproach
SIAM-basedApproach
Study
Model Scope
Mod
el D
etai
l
BroadNarrow
Sim
plifi
edD
etai
led
Application of Holistic Modelling System
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Outline
• Uncertainty, Decisions & Modeling– Uncertainty & Business
Performance– Uncertainty & Decision-
making– Need for integrated models
• Uncertainty, Judgments & People– The Nature of Uncertainty– Uncertainty & Value
Maximization
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Probability: The Language of Uncertainty
• Classical (Theoretical)Number of outcomes representing the occurrence of an event
Total number of possible outcomes– e.g. 30 red balls and 70 green balls in a bag. P(Red) = 30%
• Relative Frequency – Proportion of times an event occurs in the long run– Estimated from sample data ASSUMING identical events
e.g. 15 out of 20 wells drilled were dry holes. P(Dry) = 75%– More accurate with greater sample size. May not apply to future.
• Subjective– Personal degree of belief of the likelihood of a future event occurring
(or of the unknown outcome of a past event)– May be based on some past similar / analogous occurrences
“All business proceeds on beliefs, or judgments of probabilities, and not on certainties".
Charles W. Eliot
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
The nature of uncertainty
Throw a die and hide top face. What is the probability of a 3?
What is the probability of a 3 now?
Now you getinformation.
Uncertainty is a function of what you know. There is no “right” uncertainty (or PDF)!
1/6
Has the top face changed? NoHas the probability of a 3 changed? Yes!
1/3
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
0.333
1 2 3 5 64Outcome
Prob
.
1 2 3 5 64Outcome
Prob
.
0.333
Uncertainty is in OUR heads – it’s personal, a function of our state of knowledge
Different people can, legitimately, hold different views about the uncertainty of an unknown quantity
Person A Person B
So we should not talk about THE probability of some outcome/statement, but about MY (or OUR) probability
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Uncertainty is in OUR heads – it’s a function of our state of knowledge
Probability is not an inherent “parameter” of the “system”!
• The systems that we deal with are essentially deterministic, its just our knowledge of them that is probabilistic
• The Die:- when the die is tossed, there will be one face that comes up
- in theory, if we knew (precisely) its initial conditions and we could model (precisely) all of the processes involved with tossing it, we could predict how it would land – but that is, practically, impossible
• And so with Project reserves,cost, schedule etc:- its our knowledge that is probabilistic
- for events that have occurred, we could collect information to reduce uncertainty - complete information would resolve it
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Probability is subjective (personal) and depends upon your information
Person or
CompanyA
Info
SharedInfo
A’s PDF B’s PDF
Person or
CompanyB
Info
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Uncertainty vs Variability
Variability of all sand-body widths
Width
Uncertainty in individual sand-body width??
Width
Sand 1
?? Sand 2
A distribution that describes the variability
of a natural phenomenon is not usually appropriate to describe the uncertainty
of a single occurrence
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Uncertainty vs Risk
Uncertainty Risk
• A Risk (noun!) is one possible consequence of uncertainty. It has a negative connotation, which is “personal” to the D-M
– an event that, if it occurs, has a negative impact on DM objectives– it is specified by defining the event and assessing its probability,
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Risk isn’t the only possible aspect, or consequence, of uncertainty
Consequences of Uncertainty
Risk Possibility of loss or injury
A dangerous element or factor
The degree of probability of loss
Opportunity
Possibility of exceeding expectations
Upside potential
A wonderful element or factor
• Risk is one outcome of uncertainty - but so is Opportunity!– often over-looked - is a source of value creation
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Uncertainty, decisions & people
• Bias in central value (mean): – eg “rose-tinted glasses”
• Bias in width of distribution:– eg assessing the range of uncertainty to be much less
than it really is with respect to your true state of knowledge
With respect to uncertainty, the main enemy of good decision making is bias, not the uncertainty being too
great
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Common biases and traps in judgment and probabilistic assessment
• Illusion of control
• The confirmation trap
• Overconfidence
• Availability and Vividness
• Anchoring
• The Law of Small Numbers
• Intuition and Repetition
• Hindsight Foresight and the “curse of knowledge”
• Knowing when enough is enough
Human beings are not endowed with rational probabilistic thinking and optimal
behaviour under uncertainty.
Bias & error => poor decisions and judgements
=> undesirable outcomes
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Overconfidence Results: Large O&G Industry Sample
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0 1 2 3 4 5 6 7 8 9 10
Questions Correct /10
Prop
ortio
n of
Par
ticip
ants Expected
Observed
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Does it Matter? Economic impact of over-confidence in O&G project
Estimated PDF (Biased)
12500 15000 17500 20000 22500Area
• Field investment decision where the development plan depended on the reserve estimate.
• Investigate the impact on value (NPV) of overconfidence in reserve parameters, eg
P10 of biased (overconfident) PDF
“True”, unbiased PDF
= P20 of unbiased PDF if the estimatedone was 20% over confident (20OC)Welsh, Begg & Bratvold (2007) SPE 110765
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
-100
0
100
200
300
400
0% 5% 10% 15% 20% 25% 30%
E(N
PV),
$Mill
ion
Overconfidence
True NPV
EVusing EVs of inputs
Economic Impact of Overconfidence in O&G project
Assessed (Overconfident) Value
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Value of Flexibility to Exploit Upside Potential:Extra well slots to manage Volume uncertainty
NPV
$600$200$0EMV = $260
100 kbd PlatformHigh = 30%
Med = 40%Low = 30%
$300$250$100EMV = $220
High = 30%
Med = 40%Low = 30%
60 kbd Platform
Expand
No action
Expand
No action
Expand
No action
Flexible Platform
High = 30%
Med = 40%
Low = 30%
EMV = $303
$180
$270
$580$270
-$20
$70
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Value of Flexibility to Exploit Upside Potential:Extra space to manage resource uncertainty
NPV
$600$200$0EMV = $260
100 kbd MillHigh = 30%
Med = 40%Low = 30%
$300$250$100EMV = $220
High = 30%
Med = 40%Low = 30%
60 kbd Mill
Expand
No action
Expand
No action
Expand
No action
FlexibleDevelopment
High = 30%
Med = 40%
Low = 30%
EMV = $303
$180
$270
$580$270
-$20
$70
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Summary
• The main purpose of technical work is assessment of uncertainty to aid decision-making - the focus should be ranking, not “best prediction”
• Modeling approach should be decision-driven- holistic models that focus on uncertainty & the complexities
of the system – not the most accurate physics/geology/…• Uncertainty is a function of what we know about a
situation – its in our heads, not a “system” parameter- there is no single, “right” probability for an uncertain event- variability is not the same thing as uncertainty
• Accurate, unbiassed, uncertainty assessment is required to assess the value of decision alternatives
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Acknowledgements & Further Info
• People– Prof. Reidar Bratvold, University of Stavanger, Norway– Dr. Matthew Welsh, University of Adelaide– Prof. Michael Lee, University of California, Irvine
• Papers (downloadable from www.onepetro.org)– SPE 71414: “Improving Investment Decisions with a Stochastic
Integrated Asset Model”
– SPE 77509: “Would You Know a Good Decision if You Saw One?”
– SPE 77586: “The value of Flexibility in Managing Uncertainty in Oil & Gas Investments”
– SPE 96423 “Cognitive Biases in the Petroleum Industry: Impact and Remediation”
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
O&G Evidence of Bias (IPA data)
0 50 100 150 200 250 300
Basis for development sanction
All 1000+projectsin the study
No projects
0% 50% 100% 150% 200% 250% 300%
If the Forecastedproduction is the “Base Case”, we should have approximately as many projects producing more than expected as less than expected !!
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Gambling (probability = repeated outcomes) vs. “Real World” (probability = degree of belief)
Uncertainty Quantification
KnownDistribution
Type
UnknownDistribution
Type
KnownParameters
UnknownParameters
1. Identify Possible Outcomes 2. Assign Probabilities to Outcomes
Games of Chance,Geostatistics?
Oil & Gas:Subjective
AllIdentified
Some missed or unknowable
Some denied or ignored
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Understanding Risk attitudes
Play game 1
YesOutcome
Tails
Heads
50%
50%
No0
(1)
3
E=1
(10)
30
E=10
(100)
300
E=100
(1,000)
3,000
E=1,000
(10,000)
30,000
E=10,000
(100,000)
300,000
E=100,000
(1,000,000)
3,000,000
E=1,000,000
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Understanding Risk attitudes
Play game 2
Outcome
Tails
Heads
Yes
No
50%
50%
0
3
1
E=1.5
30
10
E=15
300
100
E=150
3,000
1,000
E=1,500
30,000
10,000
E=15,000
300,000
100,000
E=150,000
3,000,000
1,000,000
E=1,500,000
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Risk Attitudes – Formal Definitions
• Risk-Averse– Prefer a sure option (P=1) whose
value is less than or equal to the EV of an uncertain option
– we put extra “value” on certainty
• Risk-Neutral– No preference between a sure
option (P=1) and an uncertain option whose EV is the same (as the sure value)
EV
PDF Uncertain Option
Sure option
EV
Sure- prefer this
Uncertain
• Risk-Seeking– Prefer an uncertain option whose
EV is less than or equal to a sure option (P=1)
– we put extra “value” on uncertaintyEV
PDF Uncertain - prefer this
Sure
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Impact of Risk Attitudes
• In both cases (risk-seeking and risk-aversion) the loweractual value option is preferred (chosen)!
• Therefore both Risk-Aversion and Risk-Seeking attitudes lead to lower total long-run (multiple-decision) outcomes!– this is consistent with earlier statement that EV maximizes $ value –
any other criterion loses $ value
• E.g. assuming the EV lies between the P10 and P90 then using either of the following decision criteria– invest if P10>0 (very risk-averse), or
– invest if P90>0 (very risk-seeking)
are both value-losing compared to using EV– invest if EV>0 (risk-neutral)
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Knowing When Enough is Enough
• In our desire to reduce uncertainty, we often ask for too much information
• We believe –mistakenly – that more information will increase accuracy
• More information helps only to the extent we can use it intelligently
Confidence does increase
Accuracy does not increase
5 10 20 40
10%
20%
30%
40%
Items of information available
% correctpredictions
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Heuristics, Biases, Uncertainty & Decisions
• Heuristics– simple rules of thumb and mental shortcuts
• Biases– systematic errors that can result from the use of heuristics
• With respect to uncertainty, the main enemy of good decision making is bias, not the uncertainty being too great
– bias in central value (mean): eg “rose tinted glasses”
– bias in width of distribution: eg assessing the range of uncertainty to be much less than it really is with respect to your true state of knowledge
• Our “mental wiring” is just not good when it comes to uncertainty
– Intuition and “gut feel” often significantly wrong
Human beings are not endowed with rational probabilistic thinking and optimal
behaviour under uncertainty.
Bias & error => poor decisions and judgements
=> undesirable outcomes
From http://www.freegifthome.co.uk/blog/best-optical-illusions/- may not be original source
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Estimate the gray %
Using a scale of 0% (black) to 100% (white) estimate the % gray of squares A and B
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
*Shepard’s Table
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
*Shepard’s Table
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Perceptual Limitations – visual illusions are a metaphor for cognitive illusions
• Awareness of the illusion, by itself, does not produce a more accurate perception.
• Illusions, therefore, can be extremely difficult to overcome.
Human beings are not endowed with rational probabilistic thinking and optimal
behaviour under uncertainty.
Bias & error => poor decisions or judgements
=> More frequent undesirable outcomes
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
GeologyReservoir
DrillingFacilitiesGeophysics
Asset/Project Evaluation Portfolio Mgmt.
Psychology of Judgments & Decision-making
Decision-Making Under Uncertainty
Actual Performance
Cognitive Science
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Main Research Themes
1. Psychological (cognitive science) aspects of judgment and decision-making– Elicitation of un-biased (accurate) expert knowledge for
input to decisions – Matching appropriate decision tools and processes to
decision types– Overcoming barriers–to-adoption
2. Development of models and processes for improved asset/project/portfolio economic valuation and decision-making under uncertainty – Incorporating “real option” thinking – valuing learning (incl
information gathering) and flexibility. – Focus on accurate, rather than ‘precise’, holistic (systems)
models
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
CIBP Publications: Phase 3
1Welsh, M. & Navarro, D. (in press). Seeing is Believing: Priors, Trust and Base Rate Neglect. Organizational Behavior and Human Decision Processes. Accepted February 7th 2012.
2Willigers, B.J.A., Begg, S., & Bratvold, R.B. (2011). Valuation of Swing Contracts by Least-Squares Monte Carlo Simulation” SPE Economics & Management, Vol 3, No 4, pp 215-225.
1&2Welsh, M. & Begg, S. (in press). Personal ruin versus corporate profit: why individual risk attitudes lessen economic outcomes. APPEA Journal. Accepted 10th January 2012.
1Sykes, M., Welsh, M. & Begg, S. (2011). SPE 146230 - Don't Drop the Anchor: Recognizing and Mitigating Human Factors When Making Assessment Judgments Under Uncertainty. Proceedings of the 2010 Society of Petroleum Engineers Annual Technical Conference and Exhibition. Austin, TX: SPE.
1Bruza, B, Welsh, M, Navarro, D. & Begg, S. (2011). Does anchoring cause overconfidence only in experts? In L. Carlson, C. Hölscher, & T. Shipley (Eds.), Proceedings of the 33rd Annual Conference of the Cognitive Science Society (pp. 1947-1952). Austin, TX: Cognitive Science Society.
1Welsh, M., Delfabbro, P., Burns, N. & Begg, S. (2011). Individual differences in anchoring: numerical ability, education and experience. In L. Carlson, C. Hölscher, & T. Shipley (Eds.), Proceedings of the 33rd Annual Conference of the Cognitive Science Society (pp. 3193-3198). Austin, TX: Cognitive Science Society.
1Welsh, M, Navarro, D. & Begg, S. (2011). Number preference, precision and implicit confidence. In L. Carlson, C. Hölscher, & T. Shipley (Eds.), Proceedings of the 33rd Annual Conference of the Cognitive Science Society (pp. 1521-1526). Austin, TX: Cognitive Science Society.
1Welsh, M., Alhakim, A., Ball, F., Dunstan, J. & Begg, S. (2011). Do personality traits affect decision making ability: can MBTI type predict biases? APPEA Journal, 51(1), pp 359-368.
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
CIBP Publications: Phase 3
2Willigers, B.J.A., Begg, S., & Bratvold, R.B. (2010). Valuation of Swing Contracts by Least-Squares Monte Carlo Simulation” Paper #133044 in Proc SPE Asia Pacific Oil and Gas Conference, Brisbane.
1Bratvold, R.B. and Begg, S.H. (2010) Making Good Decisions. Richardson, Texas, USA: Society of Petroleum Engineers.
2Bratvold, R.B., Begg, S.H. & Rasheva, S. (2010) A New Approach to Uncertainty Quantification for Decision Making, Paper #130157, in Proc. SPE Hydrocarbon Economics and Evaluation Symposium, Dallas, TX, Mar 8-9
Welsh, M. (2010). Of parrots and parsimony: reconsidering Morgan's canon. Proceedings of the Cognitive Science Conference.
1Bruza, B., Welsh, M., Navarro, D. & Begg, S. (2010). Effect of presentation order and question format on subjecive probability judgments. Proceedings of the Cognitive Science Conference.
1Mackie, S., Begg, S., Smith, C. & Welsh, M. (2010). Human Decision-Making in the Oil and Gas Industry. Proceedings of the Society of Petroleum Engineers Asia Pacific Oil and Gas Conference.
1Welsh, M. & Begg, S. (2010). Don't let it weigh you down: how to benefit from anchoring. Proceedings of the 2010 Society of Petroleum Engineers Annual Technical Conference and Exhibition.
1Welsh, M., Rees, N., Ringwood, H. & Begg, S. (2010). The Planning Fallacy in Oil and Gas Decision Making. APPEA Journal, 50 (1), pp 389-401.
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
CIBP Publications: Phase 1
1Mackie, S., Welsh, M. & Lee, M. (2006). An oil and gas decision-making taxonomy. Proceedings of the 2006 Asia Pacific Oil and Gas Conference of the Society of Petroleum Engineers. Richardson, TX: SPE.
1Welsh, M., Begg, S. & Bratvold, R. (2006). Correcting common errors in probabilistic evaluations: efficacy of debiasing. Proceedings of the 82nd Annual Technical Conference and Exhibition of the Society of Petroleum Engineers. Richardson, TX: SPE.
2Al-Harthy, M., Khurana, A., Begg, S., Bratvold, R., (2006) Sequential and Systems Approaches for Evaluating Investment Decisions: Influence of Functional Dependencies and Interactions. APPEA Journal, Vol 46, part 1, pp511-523
2Chapman, T., Nettelbeck, T., Welsh, M. & Mills, V. (2006). Investigating the construct validity associated with microworld research: a comparison of performance under different management structures across expert and non-expert naturalistic decision-making groups. Australian Journal of Psychology, 58(1), pp. 40-47. Drop by to look at the
2Yao, Y., Begg, S.H., Bratvold, R.B., Behrenbruch, P., van der Hoek, J. (2006) "A Case Study for Comparison of Different Real Option Approaches", Paper # 101031 in Proc. SPE Asia Pacific Oil and Gas Conference and Exhibition Adelaide, Australia, September, 2006.
2Bratvold, R. and Begg, S.H. (2006) "Education for the Real World: Equipping Petroleum Engineers to Manage Uncertainty", Paper #103339, in Proc. SPE ATCE, San Antonio
1Welsh, M., Bratvold, R. & Begg, S. (2005). Cognitive biases in the petroleum industry: impact and remediation. Proceedings of the 81st Annual Technical Conference and Exhibition of the Society of Petroleum Engineers. Richardson, TX: SPE.
2Bratvold,R.B., Laughton,D., Enloe,T., Borison,A., Begg, S.H., (2005) “A Critical Comparison of Real Option Valuation Methods: Assumptions, Applicability, Mechanics, and Recommendations”, Paper #97011, in Proc. SPE ATCE, Dallas
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
CIBP Publications: Phase 2
1Welsh, M. & Begg, S. (2009). Repeated judgment elicitation: tapping the wisdom of crowds in individuals. Proceedings of the 85th Annual Technical Conference and Exhibition of the Society of Petroleum Engineers
1Welsh, M., Lee, M & Begg, S. (2009). Repeated judgments in elicitation tasks: efficacy of the MOLE method. In N.A. Taatgen & H. van Rijn (Eds), Proceedings of the 31st Annual Conference of the Cognitive Science Society (pp1529-1534). Austin,TX:CognitiveScience Society.
1Bruza, B., Welsh, M. & Navarro, D. (2008). Does Memory Mediate Susceptibility to Cognitive Biases? Implications of the Decision-by-Sampling Theory. In V. Sloutsky, B. Love, & K. McRae (Eds.) Proceedings of the 30th Annual Conference of the Cognitive Science Society.
1Heywood-Smith, A., Welsh, M. & Begg, S. (2008). Cognitive Errors in Estimation: Does Anchoring Cause Overconfidence? Proceedings of the 84th Annual Technical Conference and Exhibition of the Society of Petroleum Engineers.
1&2Welsh, M. & Begg, S. (2008). Modeling the Economic Impact of Individual and Corporate Risk Attitude. Proceedings of the 84th Annual Technical Conference and Exhibition of the Society of Petroleum Engineers.
1Welsh, M., Lee, M. & Begg, S. (2008). More-Or-Less Elicitation (MOLE): Testing a heuristic elicitation method. In V. Sloutsky, B. Love, & K. McRae (Eds.) Proceedings of the 30th Annual Conference of the Cognitive Science Society.
2Begg, S.H., and Bratvold, R.B. (2008) Systematic Prediction Errors in O&G Project and Portfolio Selection., Paper #116525, in Proc. SPE ATCE, Denver.
2Bratvold, R. and Begg, S.H. (2008) I would rather be Vaguely Right than Precisely Wrong , AAPG Bulletin, Vol 92, No.10 (October 2008), pp. 1373-1392
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
CIBP Publications: Phase 2
2Cunningham, P. and Begg, S.H., (2008) Using Value of Information to Determine Optimal Well Order in a Sequential Drilling Program , AAPG Bulletin, Vol 92, No.10 (October 2008), pp. 1393-1402
2Smalley, P.C., Begg, S.H., Naylor, M., Johnsen, S. & Godi, A. (2008) Handling Risk and Uncertainty in Petroleum Exploration and Asset Management: An Overview , AAPG Bulletin, Vol 92, No.10 (October 2008), pp. 1251-1261
1&2Mackie, S., Begg, S., Smith, C. & Welsh, M. (2008). "Real World" Decision-Making in the Upstream Oil and Gas Industry - Prescriptions for Improvement. APPEA Journal, 48(1), pp 329-343.
1&2Welsh, M., Begg, S. & Bratvold, R. (2007). Modeling the economic impact of cognitive biases on oil and gas decisions. Proceedings of the 83rd Annual Technical Conference and Exhibition of the Society of Petroleum Engineers. Richardson, TX: SPE.
1Welsh, M., Begg, S. & Bratvold, R. (2007). Efficacy of bias awareness in debiasing oil and gas judgments. Proceedings of the 29th Annual Conference of the Cognitive Science Society.
1Welsh, M., & Navarro, D. (2007). Seeing is believing: priors, trust and base rate neglect. Proceedings of the 29th Annual Conference of the Cognitive Science Society.
1&2Mackie, S., Begg, S., Smith, C. & Welsh, M. (2007). Decision type - a key to realizing the potential of decision-making under uncertainty. APPEA Journal, 22(1), pp. 307-317.
2Al-Harthy, M., Begg, S., Bratvold, R., (2007) “Copulas: A New Technique to Model Dependence in Petroleum Decision Making”. Journal of Petroleum Science and Engineering, 57, pp195-208
2Begg, S.H., and Smit, N. (2007) “Sensitivity of Project Economics to Uncertainty in Type and Parameters of Oil Price Models“, Paper #110812, in Proc. SPE ATCE, Anaheim
1Elliot, T., Welsh, M., Nettelbeck, T. & Mills, V. (2007). Investigating Naturalistic Decision Making in a simulated micro-world: What questions should we ask? Behavior Research Methods, 39(4), pp 901-910.
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
CIBP Publications: Phase 1
Lee, M., Pincombe, B. & Welsh, M. (2005). An empirical evaluation of models of text document similarity. Proceedings of the 27th Annual Conference of the Cognitive Science Society. Mahwah, NJ: Erlbaum
1Welsh, M., Begg, S., Bratvold, R. & Lee, M. (2004). Problems with the elicitation of uncertainty. Proceedings of the 80th Annual Technical Conference and Exhibition of the Society of Petroleum Engineers. Richardson, TX: SPE.
2Begg, S.H. and Bratvold, R., (2004) “The value of flexibility”, Southern China Oil & Gas Journal, vol17, no.4, p59.
2Begg, S.H., Bratvold, R., Campbell, J.C, (2004) “Abandonment Decisions and The Value of Flexibility “, Paper #91131, in Proc. SPE ATCE, Houston
2Laughton, D., Bratvold, R., Begg, S.H., Campbell, J.C. (2004) “Development as the continuation of appraisal by other means “, Paper #90155, in Proc. SPE ATCE, Houston
2Lee, M., O'Connor, T. & Welsh, M. (2004). Decision making on the full information secretary problem. Proceedings of the 26th Annual Conference of the Cognitive Science Society. Mahwah, NJ: Erlbaum.
1&2Begg, S.H., Bratvold, R., Campbell, J.C. (2003) “Shrinks or Quants: Who will improve decision-making? “, Paper #84238, in Proc. SPE ATCE, Denver
2Bratvold, R., Begg, S.H., Campbell, J.C. (2003) “Even Optimists should optimize“, Paper #84329, in Proc. SPE ATCE, Denver
2Begg, S.H., Bratvold, R., Campbell, J.C, (2003) “Decision-making Under Uncertainty”, In Proc. 7th
International Symposium on Reservoir Simulation, Baden-Baden 2Campbell, J.C, Begg, S.H., Bratvold, R.B. (2003) “Portfolio Optimization: Living up to Expectations?“,
Paper #82005, in Proc. SPE Hydrocarbon Economics and Evaluation Symposium, Dallas
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Outline
• Introductions, Questionnaire• Overview of Performance of O&G
Industry Capital Investment Decisions • Underlying Concepts• Major Psychological Factors• Limits of Intuition in Complex & Uncertain
Situations• What can we do… RISC Process
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Uncertainty Assessment Exercise
Question 1
Question 2
Question 3
etc
Lower Limit(P10)
UpperLimit(P90)
80%Chance
10% 10%
172 cm
E.g. What is my height in cm?
• Your goal is to set ranges such that they are- Narrow enough not to contain the actual value more than 8 out 10
times (80%) on average, or
- Wide enough to contain the actual value 8 out of 10 times
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Anchoring QuestionLarge Industry Sample• Question given to two groups. One group had a High
anchor, other group had Low anchor
• High“Were world proved oil reserves in 2003 greater or less than 1721Billion Barrels?” Yes [ ] No [ ]
• Low“Were world proved oil reserves in 2003 greater or less than 574Billion Barrels?” Yes [ ] No [ ]
• Both versions then asked“What is your best estimate of the world proved oil reserves in 2003?”
( )
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Anchoring ResultsLarge Industry Sample
19321722
574 682
0
500
1000
1500
2000
2500
3000
3500
Low High
Anchor Group
Mea
n Es
timat
ed W
orld
Pro
ved
Res
erve
s 20
03 +
/- 1s
d AnchorEstimate
Common approach in E&P project evaluation:
“Let’s start with a base case and then build some scenarios around it.”
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Major Psychological Issues Relevant to Project Cost and Time Estimates and Outcomes
• Overconfidence, Optimism and Superiority Biases
• Anchoring
• Unpacking & the Planning Fallacy
• Availability, Recency and Vividness
• Hindsight Bias
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
REMOVE THIS SLIDE• I am not convinced about giving these definitions (next 3
slides) up front – liable to lead to unfruitful discussion to distinguish them before they have really learnt what they are
• I also really struggled to make / modify the illustrative “quotes” to keep them pithy and descriptive of the bias/factor
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
RISC view of primary Psychological Factors affecting E&P projects
Overconfidence: “There is an 90% chance the cost will be less than $10MM”; placing higher probabilities on events (or tighter probability distributions) than is warranted by our true state of knowledge. Due to this bias, actual outcomes will lie outside our ranges more frequently than expected.
Optimism: “There is only a 5% chance of a delay of more than 3 months”; assigning lower chances to the attainment of undesirable outcomes (and/or higher chance to desirable outcomes) than objective criteria, experience or logical analysis warrants. Due to this bias, outcomes will be systematically worse than expected.
Positive Illusions (Superiority Bias): “We can do it better than anyone else” causes people to overestimate, relative to others, their positive qualities, skills and abilities and to underestimate their negative qualities - leading them to believe that THEY are less at risk of experiencing a negative event compared to others who are doing the same thing. Also leads to the Illusion of Control - under attributing the role of chance, in decision outcomes
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
RISC view of primary Psychological Factors affecting E&P projects
Anchoring: “I wish the time to build XXX was less than 4 months”; an initial piece of information, typically a number (perhaps irrelevant) causes people (often sub-consciously) to “centre” on that information and to not adjust sufficiently far away when considering other possibilities
Planning Fallacy: “If everything goes to plan we’ll be on-line in 2 years” a tendency for people and organizations to rely on best-case scenarios, leading to optimistic estimates of how long it will take to complete complex tasks, even when they have experience of similar tasks over-running (their own or others).
Packing/Unpacking Effect: “lets not make this too complex and go with a high-level breakdown of tasks”
When broad tasks, or causes, are broken down (unpacked) into explicitly-identified sub-components, the estimates of times, costs, %-contribution for each task, and thus the overall total for all tasks, is more accurate. Failure to unpack adequately is thought to be a contributor to the planning fallacy – “out of sight is out of mind”
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
RISC view of primary Psychological Factors affecting E&P projects
Availability, Recency & Vividness: “I have an excel spreadsheet handy with cost estimates from our last project .....”
causes individuals to over-weight the most easily accessed, remembered or recent information when assessing the likelihoods of the possible outcomes of future events
Hindsight bias: “I knew that was likely to happen, ...”;the inclination, in retrospect, to assign higher chances to outcomes that have already occurred than were assigned before the event took place –can apply to one’s own, or other peoples, estimates of the chance of the outcome occurring
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Major Psychological Issues Relevant to Project Cost and Time Estimates and Outcomes
• Overconfidence, Optimism and Superiority Biases
• Anchoring
• Unpacking & the Planning Fallacy
• Availability, Recency and Vividness
• Hindsight Bias
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Refresher: Interpreting PDFs & Percentiles
0.10
P10 P90
80% ofarea
10% ofarea
10% ofarea
The area under the PDF between any two points is theprobability of X lying between those two points
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Uncertainty Assessment Exercise
Question 1
Question 2
Question 3
Question 4
Question 5
Question 6
Question 7
Question 8
Question 9
Question 10
Lower Limit(P10)
UpperLimit(P90)
Actual80%Chance
10% 10%
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Probability Estimation Exercise
• Excel - Interactive
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
*Assessing Uncertainty
• Some conclusions from Capen study (3000+ sample):– People who are uncertain about answers to a question have
almost no idea of the degree of their uncertainty. They cannot differentiate between a 30- and a 98-percent probability interval
– The more people know about a subject, the more likely they are to construct a large probability interval (one that has a high chance of catching the truth), regardless of what kind of interval they have been asked to use. The converse seems to hold as well; the less known, the smaller the chance that the interval will surround the truth
– People tend to be a lot prouder of their answers than they should be
– Even when people have been told that probability ranges tend to be too small, they cannot bring themselves to get their ranges wide enough
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Time
Min
Max
Discovery
Conceptselect
Detailed design
Practical completion
Procurement
Concept Construction CommissionDesign
Perceived uncertainty
Actual uncertainty
* Experts & Over-confidence
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
We’re not alone: Experts & Over-confidence
• Heavier-than-air flying machines are impossible
– Lord Kelvin, British mathematician, physicist, and president of the British Royal Society, spoken in 1895
• A severe depression like that of 1920-21 is outside the range of probability
– Harvard Economic Society, Weekly Letter, November 16, 1929
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
We’re not alone: Experts & Over-confidence
• That idea is so damned nonsensical and impossible that I'm willing to stand on the bridge of a battleship while that nitwit tries to hit it from the air
– Newton Baker, U.S. secretary of war in 1921, reacting to the claim of Billy Mitchell (later Brigadier General Mitchell) that airplanes could sink battleships by dropping bombs on them
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
We’re not alone: Experts & Over-confidence
• They couldn't hit an elephant at this dist…
– General John B. Sedgwick, Union Army Civil War officer's last words, uttered during the Battle of Spotsylvania, 1864
• I think there is a world market for about five computers
– Thomas J. Watson, chairman of IBM, 1943
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
We’re not alone: US DoE Price Forecasts
Trends Predicted Beginning From the Actual Price of Year Listed
after U.S. Department of Energy, 1998
120
100
80
60
40
20
01975 1980 1985 1990 1995 2000 2005
Year
1982
1981
1984
1985
1986 1987
1991
1995Actual
Dol
lars
per
Bar
rel
1983
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
We’re not alone: Yr 2000 Price Forecasts
U.S. Department of Energy, 1998
35
30
25
20
15
102000 2005 2010 2015 2020
IEA
DOE High
Mobil
DRI
DOE Base
Nat. Res. Canada
Nat. West Sec.
DOE Low
Pet. Econ. Ltd.
$/B
BL
(199
6 D
olla
rs)
Year
Widely Divergent ForecastsMake Planning Difficult
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Does it matter? Overconfidence in the Stock Market
• Based on a study of monthlypositions and trading records of 88000 investors over a 10 year period and $2 million commonstock trades (Barber & Odean 2000).
• In contrast to a buy-and-hold strategy, the average investor traded 75% of their investment in any given year.
• The investors traded frequently because they thought they could beat the market
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Overconfidence in the Stock Market: You Are What You Trade
• The average investor earned a return of 16.4% during the booming market.
• The overall market return for this same period was 17.9%.
• The 20% of accounts (more than 12000) that had the highest turnover rates earned a return of just 11.4%.
• Overconfident investors under diversify.
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Overconfidence in the Stock Market Boys will be Boys
• The data (Barber & Odean 2001) show that women performed better than men
– Not because they were better at picking stocks but because …
• Men tend to be more overconfident than women.– Men trade 45 percent more actively than women.
– For single men and women, the difference is 67%.
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Calibration
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
Estimated Accuracy (Confidence)
Act
ualA
ccur
acy
OverConfidence
UnderConfidence
PerfectCalibration
Typical O&Gresult at 80% confidence
• Calibration: the degree to which estimated accuracy (Confidence) matches actual accuracy
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Expert Calibration: Physicians
Data: Physicians, after completing history and physical examination, estimated the probability that patients had pneumonia (Source: Christensen-Szalanski & Bushyhead, 1981)
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
Subjective Probability of Pneumonia
% R
adio
grap
hica
lly a
ssig
ned
pneu
mon
ia
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
What Can We Do?Expert Calibration: US Weather Forecasters
Source: Russo and Schoemaker
• Why are physicians lousy and weather forecasters great?- A significant reason is because they get frequent, immediate, and
accurate feedback
161
146
282575
589
257
172
203147
159 82
38 9
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
Forecast Probability of Rain
Act
ual F
ract
ion
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
In the resources industries pressures towards overconfidence are rife
• Experts may feel pressure (motivational or cognitive) to demonstrate their expertise relative to peers or competitors and therefore place overconfident bounds on uncertain quantities
– Narrower bounds are used to imply “I(we) know more than you”
• Managers may create a climate that discourages a true assessment of uncertainty:
– “you are paid to know, not to not know …”
– “you are just covering your …”
– “why can’t you just tell me the answer”
– “I need a single number to make a decision”
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Optimism
• A bias towards – assigning higher chances to the attainment of desirable outcomes– assigning lower chances to the attainment of undesirable outcomes
than objective criteria, experience or logical analysis warrants.
• Optimism can be the result of– Dispositional: a personality trait– Situational: e.g. a motivational bias, caused by an incentive to be
optimistic rather than realistic• This bias raises the odds that the projects chosen for
investment will be those with the most optimistic forecasts –and hence the highest probability of disappointment
– outcomes will be systematically worse than expected (worse being context dependent – ie higher costs, lower production)
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Overconfidence v Optimism: Impact on Assessed Probabilities, Pictorially
“True” Uncertainty
Over-Confidence- assigning too-narrow a PDF
compared to your true state of knowledge
- driven by mis-assessing our level of knowledge
Optimism- assigning higher chance to
more desirable outcomes (eg“lower” valued outcomes, such as costs, are better)
- driven by our personality or by preferences for some of the outcomes
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Overconfidence & Optimism!
“True” Uncertainty
- in this case “higher” outcomes are more desirable (egproduction rates)
Over Confident and Optimistic
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Optimism and Organizational Pressure: Motivational perspective• Every company has limited investment funds and time to
devote to new projects.– Competition for this time and money is intense, as individuals and units
present their own proposals as being the most attractive for investment.
• The selection process often– favours the most communicative and articulate – not necessarily the most
knowledgeable
– discourages “realism” (unbiasedness), encourages exaggeration
– causes bearers of bad news, or people who point out problems, to be labelled as “not a team player” - we do “shoot the messenger”!
Big incentives to accentuate the positive in project forecasts.
Ability of the organization to think critically is under-minded
Optimistic individual views are self-reinforcing and unrealistic views of the future are validated by the group
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Using outcomes of multiple estimates to check for unbiasedness
P20 P60 P100P0 P40 P80
% outcomes lying within estimated range
20%Unbiased
!! !!P20 P60 P100P0 P40 P80
20%Biased% outcomes
lying within estimated range
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Questions
how good a decision-maker are you?
• Compared to your peers, on a scale of 1 to 10, where- 1 is poor- 5 is average - 10 is very goodhow good a driver are you?
how good is your intuition?how good a probability estimator are you?
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Positive Illusions (Superiority Bias)
• People/organizations overestimate their positive qualities, skills and abilities (and underestimate their negative qualities) relative to other people/organizations
• This can lead them to believe that THEY are less at risk of experiencing a negative event (e.g. cost/time overrun) compared to OTHERS in the same situation.
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
*Lake Wobegon effect:“… and all the children are above average” – Garrison Keillor
• 82% of people say they are in the top 30% of safe drivers;
• 86% of MBA students say they are better looking than their classmates;
• 68% of lawyers in civil cases believe that their side will prevail;
• Doctors consistently overestimate their ability to detect certain diseases;
• 81% of new business owners think their business has at least 70% chance of success, but only 39% think that any business like theirs would be likely to succeed.
Source: Russo & Schoemaker
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Lake Wobegon effect:“… and all the children are above average” – Garrison Keillor
• 82% of people say they are in the top 30% of safe drivers;
• 86% of MBA students say they are better looking than their classmates;
• 68% of lawyers in civil cases believe that their side will prevail;
• Doctors consistently overestimate their ability to detect certain diseases;
• 81% of new business owners think their business has at least 70% chance of success, but only 39% think that any business like theirs would be likely to succeed.
Source: Russo & Schoemaker
• When assessing their position in a distribution of peers on almost any positive trait, 90% of people say they are in the top half.
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
*Illusion of Control
OutcomeDeciding
• the thinking and decision process
Doing• implementation
and other factorsunder your control
Chance• uncontrollable
factors, luck
Skill Presumed CauseSuccess
FailurePresumed Cause
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Illusion of Control
OutcomeDeciding
• the thinking and decision process
Doing• implementation
and other factorsunder your control
Chance• uncontrollable
factors, luck
Skill Presumed CauseSuccess
FailurePresumed Cause
• Illusion of control frequently causes people to repeat actions that in the past were followed by success.
• This is true even if there’s no reason to believe the actions did anything to cause the success.
• Only by realistically assessing the role of chance in successes can you learn which of your actions you should repeat and which could be improved.
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Illusion of Control:
• People often (knowingly and unknowingly) take credit for positive outcomes and attribute negative outcomes to external factors, no matter what their true cause.
• Study of letters to shareholders:– Executives tend to attribute favourable outcomes to factors
under their control, and
– Unfavourable outcomes were more likely to be attributed to uncontrollable external events such as weather or inflation.
“Victory has a thousand fathers; defeat is an orphan.”
-the Duke of Wellington
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Thinking about “Luck”
• By definition, luck (or unluck) is something over which you do not have control, therefore:
– People are not inherently lucky or unlucky. – Lucky or unlucky things happen to people.
• People cannot “create their own luck” – but they can plan to exploit good luck when it happens,
and minimize the impact of bad luck when that happens– “Plan”: by creating an environment, contingencies or
opportunities to respond to events, rather than living with fate.
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Summary of Overconfidence, Optimism and Positive Illusions (Superiority)
Overconfidence: “There is an 90% chance the cost will be less than $10MM”; placing higher probabilities on events (or tighter probability distributions) than is warranted by our true state of knowledge. Due to this bias, actual outcomes will lie outside our ranges more frequently than expected.
Optimism: “There is only a 5% chance of a delay of more than 3 months”; assigning lower chances to the attainment of undesirable outcomes (and/or higher chance to desirable outcomes) than objective criteria, experience or logical analysis warrants. Due to this bias, outcomes will be systematically worse than expected.
Positive Illusions (Superiority Bias): “We can do it better than anyone else” causes people to overestimate, relative to others, their positive qualities, skills and abilities and to underestimate their negative qualities - leading them to believe that THEY are less at risk of experiencing a negative event compared to others who are doing the same thing. Also leads to the Illusion of Control - under attributing the role of chance, in decision outcomes
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Effect of Overconfidence, Optimism and Positive Illusions (Superiority)
• Collectively these bias’s affect the estimates and uncertainty ranges that are generated and used as part of Project cost and schedule workflows
• Ultimately the effect is to provides estimates that are too low and uncertainty ranges that are too narrow
• No-one is immune to these psychological issues and unless people are made aware of them (and even then) they will always skew our work
• The culture and incentive systems in organisations (and Project teams) actively drives overconfidence and optimism in the people within
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Mitigating Overconfidence, Optimism and Positive Illusions (Superiority)• Be aware of them (not just now, continually remind yourself)• Ask yourself why you might be wrong (as opposed to right) –
actively think of all the reasons why the P10 might be smaller and the P90 larger.
• Get Feedback– Privately record lots of predictions (probability for discrete events; P10-
P90 ranges for continuous variables) of things you will soon get to know the answer to – and record the outcomes as well.
– Enhance your motivation to be well-calibrated by making small bets with friends, colleagues and family on the outcomes
• Don’t limit your assessment of uncertainty to finding data (of the event in question) and fitting a distribution to them
– Unless the data are very numerous and truly repeat outcomes of the same “event” of interest
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Mitigating Overconfidence, Optimism and Positive Illusions (Superiority)• Take an “outside view”
– look at the performance of other people/teams/companies, or how you think others would perform if doing your projects
• Managers/Supervisors– Create a climate where honesty about uncertainty is encouraged– Reward for quality of predictive process – not actual outcome– Ask people to justify why their ranges are so narrow (half the P10,
double the P90)
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Major Psychological Issues Relevant to Project Cost and Time Estimates and Outcomes
• Overconfidence, Optimism and Superiority Biases
• Anchoring
• Unpacking & the Planning Fallacy
• Availability, Recency and Vividness
• Hindsight Bias
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Anchoring - Subtle changes in wording of a question can significantly impact responses
Question OutcomeDo you get headaches frequently? How often do you get them?Do you get headaches occasionally? How often do you get them?
2.2 / week
0.7 / week
Question OutcomeHow long was the movie?
How short was the movie?130 min100 min
OutcomeQuestion OutcomeQuestionHow tall was the basketball player? 79 inches
69 inchesHow short was the basketball player?
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Anchoring - Subtle changes in wording of a question can significantly impact responses
Question Outcome
Question Outcome
How wide are the channels?
How narrow are the channels?
How big is the fault throw?
How small is the fault throw?
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
• Participants watched a one-minute film that include a four-second multiple car crash.
• The participants were then divided into 5 groups who were asked:
– “About how fast were the cars going when they smashed into each other?”
– “About how fast were the cars going when they collided with each other?”
– “About how fast were the cars going when they bumped into each other?”
– “About how fast were the cars going when they hit each other?”
– “About how fast were the cars going when they contacted each other?”
Anchoring and Adjustment
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Anchoring and Adjustment
Descriptor Mean Speed
Smashed 40.8
Collided 39.3
Bumped 38.1
Hit 34.0
Contacted 31.8
Ref.: Loftus and Palmer
“About how fast were the cars going when they ??? into each other?”
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Anchoring Question
• Group A?
• Group B?
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Anchoring & Adjustment
• Describes a heuristic commonly used by people when estimating values.
• Use any given number/statistic/fact as a starting point (anchor) that from then on dominates the thinking process.
– adjust away from there to reach estimate
– generally people adjust too little so their estimates cluster near the anchor
• Random anchors can have just as large effects as credible anchors.
– Quattrone et al (1984) asked whether the average temperature in San Francisco was greater or less than 558º and still found people anchoring on this value
• Subtle changes in wording of a question can have significant impact on how people respond.
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Anchoring - Implications
• Referendums and opinion polls.• Jurors:
– “Strike that comment”
• Independence of second opinions.• Skilled negotiators often start by setting a suitable
anchor.• Used car salesmen and real estate brokers.• Resource industry managers?• Common approach in project evaluation:
– “Let’s start with a base case and then build some scenarios around it.”
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Summary of Anchoring
Anchoring: “I wish the time to build XXX was less than 4 months”; an initial piece of information, typically a number (perhaps irrelevant) causes people (often sub-consciously) to “centre” on that information and to not adjust sufficiently far away when considering other possibilities
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Anchoring: Effect and Mitigation
• Effect- This bias can affect the initial estimates (and then uncertainty ranges)
that are generated and used as part of Project cost and schedule workflows
- A bias that is used extensively when negotiating with someone who is aware of the effect
- Difficult to overcome even when aware of the effect- Framing of questions is extremely important to reduce the effect
• Mitigation- Be aware of it – not just now, remind yourself- Deliberately use multiple Anchors- NEED MORE
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Major Psychological Issues Relevant to Project Cost and Time Estimates and Outcomes
• Overconfidence, Optimism and Superiority Biases
• Anchoring
• Unpacking & the Planning Fallacy
• Availability, Recency and Vividness
• Hindsight Bias
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
The Planning Fallacy
• The tendency to underestimate completion times of projects because planners construct a single mental scenario, comprised of broad stages, in which most things go according to plan, – DESPITE knowing that similar projects (whether our own! or others)
have gone over time & budget
• The “because” above is the strict/narrow definition of the Planning Fallacy and is thought to be due to– Failing to distinguish between “best guess” and “best case” scenarios– Failure to unpack tasks, especially “other problems”, or ignoring them
completely
The whole workshop could be considered to be about a broader definition of the “planning fallacy” (lower case) and thus a result of all the psychological factors we are considering
– Overconfidence; Optimism (as a personality trait or motivational bias); Illusion of Control and Superiority bias; ……
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
The Planning Fallacy & Unpacking Effect
• When asked how long a 1-hour lecture takes to prepare, lecturers estimate around 3 hours
• When asked to estimate how long each task involved in preparing the same lecture takes, the sum of these task times is significantly greater than 3 hours
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
• 1st Unpacking Question (Packed)– “What % of world proved oil reserves are in
the following areas: Saudi Arabia, Iraq, Omanand All Others?”
• 1st Unpacking Question (Unpacked)– “What % of world proved oil reserves are in
the following areas: Saudi Arabia, Iraq, OmanVenezuela, USA, China, Russia, Nigeria andAll Others?”
Unpacking Questions
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Unpacking Question 1 Results
0
10
20
30
40
50
60
70
80
SaudiArabia
Iraq Oman All Others
Region
Estim
ated
% o
f Wor
ld P
rove
d Res
erve
s (2
003)
+/-
1SD
True:PackedUnpacked
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Unpacking example: estimating time to drill a well
• Four groups were asked to estimate completion times, in hours, for a real-world drilling scenario– 3rd yr ASP Pet. Eng. undergraduates (no decision-making training)– 4th yr ASP Pet. Eng. undergraduates (some decision-making training)– “Conversion” Masters of Pet. Eng. (little Pet. Eng. Knowledge)– Industry petroleum engineers (with average 10 yrs experience)
• Approximately half were given a Packed version of the scenario which consisted of 4 components
– Drilling, Tripping, Rigging and All associated problems
and the rest given an version where “All associated problems” was Unpacked into 6 explicit possibilities
– Mud conditioning; Well-control operation; Fishing operations; Severe weather; Rig repair; Logistics delays
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Unpacking example: estimating time to drill a well
‘An oilfield is located offshore Louisiana in about 350 ft of water. The field contains four wells that are currently producing 1,000 bbl of oil and 18 MMSCF of gas per day. Production is declining and it has been decided to drill an infill well in order to improve production. The well is required to be drilled to 10,000 ft drilling overbalance with mud. For this example, we will assume the well can be drilled straight to 10,000 ft without the need for staged casing. The rig is already in place and ready to commence drilling. From other well data, the stratigraphy can be assumed to be reasonably homogeneous and consolidated. The well needs to be ready up to the point where casing could then be run if required.’
Bourgoyne et al (1986)
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Unpacking Results: Number of hours of drilling problems
Unpacked
Packed
Estim
ated p
roble
m ho
urs,
mean
and 9
5% C
I
Group
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Unpacking Effect
• Out of sight is out of mind– Explicitly stating an option makes it available and
therefore increases its estimated likelihood– Unpacking a general category into specific
subcategories therefore increases the total likelihood assigned to that category despite the two being logically equivalent
• The Planning Fallacy– A specific instance of the unpacking effect– Refers to the tendency of people to underestimate
completion times for complex tasks
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Summary of Planning Fallacy & Unpacking Effect
Planning Fallacy: “If everything goes to plan we’ll be on-line in 2 years” a tendency for people and organizations to rely on best-case scenarios, leading to optimistic estimates of how long it will take to complete complex tasks, even when they have experience of similar tasks over-running (their own or others).
Packing/Unpacking Effect: “lets not make this too complex and go with a high-level breakdown of tasks”
When broad tasks, or causes, are broken down (unpacked) into explicitly-identified sub-components, the estimates of times, costs, %-contribution for each task, and thus the overall total for all tasks, is more accurate.
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Mitigating Planning Fallacy & Unpacking Effect
• If you insist on using single-point estimates – make a “best guess” estimate rather than a “best case” estimate
• Unpack general categories (such as “all other problems”) by just listing their sub-components
– this will improve the estimate of the general category, even if you do not estimate each of the subcomponents
• Take an “outside view” – look at the performance of other people/teams/companies, or how
you think others would perform if doing your projects
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Major Psychological Issues Relevant to Project Cost and Time Estimates and Outcomes
• Overconfidence, Optimism and Superiority Biases
• Anchoring
• Unpacking & the Planning Fallacy
• Availability, Recency and Vividness
• Hindsight Bias
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
*Availability and Vividness
• Are there more words in English that have the letter ‘k’ as their first letter (e.g., kill) or as their third letter (e.g., ink)?
a) first letter
b) third letter
• Experimental results (Tversky and Kahneman):– 2/3 of people asked thought that words with the letter k
in the first position were more probable
• Reality:– There are approximately twice as many words with k in
the third position as there are words that begin with a k.
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
*Availability and Vividness
• Which of the following is more likely to kill someone in the USA?a) Shark Attack?b) Falling airplane parts?– Falling airplane parts 30 times more likely than shark
attack in the US (cited in Plous, 1993)
• Which of the following caused more deaths in Australia in 2003a) Renal Failure?b) Car and other transport Accidents?– Renal failure - 15000 compared to 2100 for transport
accidents (Australian Bureau Statistics, 2003)
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Availability and Vividness Bias – Seeing What We Believe
• The tendency people have to base estimates of frequencies (probabilities) on the most readily available, recent and vividinformation they can remember
– how many events of a particular type are availableto memory
– more available events are judged more likely
• Memory is limited to 7 “chunks”
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Bias from the Availability Heuristic & Saliency Heuristic
• This heuristic leads to bias where it is easier to recall events for reasons other than their frequency, e.g. news coverage
– Shark attacks provoke a media feeding frenzy making them easier to recall
– Car accidents often make the news over long weekends with multiple deaths whereas renal failure only ever affects one person at a time and gets no coverage
• Another contaminating factor in our ability to use the availability heuristic accurately is the salience of events, e.g. how relevant they are to us personally
– People whose houses have burnt down inflate the likelihood of house fires when asked
– Knowing someone who has died of a particular cancer tends to cause inflation of estimates of its occurrence
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Bias from the Availability Heuristic & Saliency
• Tversky and Kahneman (1974)– “decision makers assess the frequency of a class or
the probability of an event by the ease with which instances or occurrences can be brought to mind.”
• Managers conducting performance appraisals:– Working from memory, vivid instances of an
employee’s behaviour (either positive or negative) will be most easily recalled from memory and will appear more numerous than more commonplace instances.
– Managers give more weight to performance during the three months prior to the evaluation than to the previous nine months of the evaluation period.
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Bias from the Availability Heuristic & Saliency
• Engineers who are evaluating durations for potential project tasks.
• Commonly done by comparing the tasks with tasks that have been done previously and where, as a consequence, the durations are known.
• Potential biases induced with this strategy are– previous personal experience with particular risks can
be overweighted
– no comparison is made with other risks that have not been experienced but that will potentially effect the tasks
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Summary of Availability, Recency and Vividness
Availability, Recency & Vividness: “I have an excel spreadsheet handy with cost estimates from our last project .....”
causes individuals to over-weight the most easily accessed, remembered or recent information when assessing the likelihoods of the possible outcomes of future events
• Difficult to overcome due to the amount of information that can be contained in the memory
• Highlights that memory can not be relied on when making estimates
• When recalling from memory, write down a list and try to think of the most common occurrences (not the most vivid or recent)
• Search for the most pertinent data, not the most available
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Major Psychological Issues Relevant to Project Cost and Time Estimates and Outcomes
• Overconfidence, Optimism and Superiority Biases
• Anchoring
• Unpacking & the Planning Fallacy
• Availability, Recency and Vividness
• Hindsight Bias
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Hindsight Bias:Remembered Probabilities of Once-Future Things
• Prior to President Nixon’s trip to China and Russia in 1972, students were asked to consider 15 possible outcomes:– “the US will establish a permanent diplomatic mission in
Peking, but not grant diplomatic recognition,”
– “Nixon will meet Mao Tse-tung at least once,”
– “Nixon will meet Soviet demonstrators,”
– …
• The students assigned probabilities to each outcome.
Ref: Fischhoff and Beyth (1975)
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
*Hindsight Bias:The “We Knew it All Along” Phenomenon
• After the trip the students were asked in hindsight to assess the likelihood of these various outcomes and also asked to remember or reconstruct their original probabilities.
• Two weeks between pre- and post-trip– 67% thought their original estimates were closer to the truth
then they really were.
• Four to eight months between pre- and post– 84% thought they had predicted the outcome.
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Hindsight Bias:The “We Knew it All Along” Phenomenon
• After the trip the students were asked in hindsight to assess the likelihood of these various outcomes and also asked to remember or reconstruct their original probabilities.
• Two weeks between pre- and post-trip– 67% thought their original estimates were closer to the truth
then they really were.
• Four to eight months between pre- and post– 84% thought they had predicted the outcome.
“people even misremember their own predictions so as to exaggerate in
hindsight what they knew in foresight”Fischhoff and Beyth
How will this bias impact our ability to, in hindsight, judge the quality of our
predictons?
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Summary Hindsight Bias
• Makes learning from previous estimates much more difficult
• Highlights that memory can not be relied on when doing “lessons learnt” workshops
• Records of previous predictions/estimates, and reasons for them, should be made at the time of the prediction/estimate and checked against actual performance to help with calibrating the mind
Hindsight bias: “I knew that was likely to happen, ...”;the inclination, in retrospect, to assign higher chances to outcomes that have already occurred than were assigned before the event took place –can apply to one’s own, or other peoples, estimates of the chance of the outcome occurring
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Outline
• Introductions, Questionnaire• Overview of Performance of O&G Industry
Capital Investment Decisions • Underlying Concepts• Major Psychological Factors• Limits of Intuition in Complex & Uncertain
Situations– Uncertainty Propagation– Updating Estimates with New Information– Complexity
• What can we do… RISC Process
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Estimating Odds
• Gilbert Video (2005G) Pt 1 – Odds
00.00-8:19
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Judging likelihoods of events
• Linda is a 31 years old, single, outspoken and very bright. She majored in philosophy. As a student she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations.
Which is the more likely alternative?a) Linda is a bank tellerb) Linda is a bank teller and active in the feminist
movement.
Answer:______
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
*Probability Rules: Multiplication Rule for Independent Events
• Multiplication Rule for independent events
When event A and Event B can occur together
Not A or B
A B
Joint probability = P (A and B)
P(A and B) = P (A) * P(B)
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
P(Bank Teller) > P(Bank Teller AND Feminist)
Discussion of Linda Question
• Nearly 90% of respondents choose the second alternative (bank teller and active in the feminist movement), even though this is logically incorrect
bank tellers feminists
feminist bank tellers
Junctions (“ands”) are always less likely than stand-alone statements.
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Cognitive Illusions
• Kahneman & Tversky (1982)– “As the amount of detail in a scenario increases, its probability
can only decrease steadily, but its representativeness and hence its apparent likelihood may increase.”
– “The reliance on representativeness, we believe, is a primary reason for the unwarranted appeal of detailed scenarios and the illusory sense of insight that such constructions often provide.”
• Implications: consider a “rich” description of a reservoir depositional environment
• The description of Linda is more representative of a feminist bank teller so people, wrongly, conclude it is more likely that she is a feminist and a bank teller
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
High production days
• Daily production has been tracked for two oil fields for a year and the average production computed for each– The first oil field has 45 wells and the second 15.
• The number of days when production was above average for 60% or more of the wells in a field has been calculated.
• Which field do you think would have recorded more such days over the course of a year? a) The 45-well field.
b) The 15-well field.
c) Approximately the same (within 5%).
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Insensitivity to Sample Size Results – Industry Respondees
0
10
20
30
40
50
60
70
Larger Smaller Equal
No o
f Par
ticip
ants
Answer
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Insensitivity to Sample Size
• This question requires intuitive judgement as no data for calculation is available
• Central Limit Theorem:– The smaller the sample, the more likely it is to be a
deviant sample -> one with a mean and standard deviation significantly different from those of the population from which the sample was drawn
– Answer b, the 15-well field, is the most likely
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Insensitivity to Sample Size
• People tend to ignore sample size they are dealing with when drawing conclusions from it.
• This is not consistent with the Central Limit Theorem which describes the changing characteristics of samples according to their size.
• Example – which is more likely:– Getting 6 heads in 10 flips of a coin, or
– Getting 6000 heads in 10000 flips of a coin.
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Sequences - Representivity Heuristic:The “Law of Small Numbers”
• I have just tossed a fair coin 7 times. You have not seen the result
• You are invited to play a betting game to guess which of the three sequences below is the one I actually observed.
• Which sequence would you bet on
a) HHHHTTTb) THHTHTTc) TTTTTTT
• Using multiplicative rule for independent eventsP(A&B&C&D.) = P(A)*P(B)*P(C)*P(D) ….
P = (1/2)7 = 1/128P = (1/2)7 = 1/128P = (1/2)7 = 1/128
• ALL sequences have the SAME probability and are thus EQUALLY likely (or equally rare!)
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
The “Law of Small Numbers” (Tversky & Kahneman)
• We expect to see the same behaviour in small sequences that we would observe in large sequences – the mathematical “Law of Large Numbers” informs us of
behaviours that are approximately true for large sequences, and rigorously true for sequences near to infinity
• Some sequences are seen as more balanced or more “typical” and are thus thought to be more probable. Typicality is mistaken for probability.– with the result that we over-estimate probability
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Sequences of Events, 2
• You are playing a game with a fair coin:– Heads you win $10, Tails you loose $10 (EV=$0)
• You have played 10 times so far and have had a luckystreak of 8 wins and 2 losses, so you are up $60– What is your expected value if you play a total of 1000 times
• You have played 10 times so far and have had an unlucky streak of, 3 wins and 7 losses, so your cumulative position is down $40– What is your expected outcome if you play a total of 1000 times
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
*Sequences of Events are not self-correcting :The “Gamblers Fallacy”
• You are playing a game with a fair coin:– Heads you win $10, Tails you loose $10 (EV=$0)
• You have played 10 times so far and have had a luckystreak of 8 wins and 2 losses, so you are up $60– What is your expected value if you play a total of 1000 times
– $60
• You have played 10 times so far and have had an unlucky streak of, 3 wins and 7 losses, so your cumulative position is down $40– What is your expected outcome if you play a total of 1000 times
– $-40
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Sequences of Events are not self-correcting :The “Gamblers Fallacy”
• After holding bad cards on ten hands of poker, the poker player believes he’s “due” for a good hand.
• After winning $10,000 in the lottery, a woman changes her regular lottery number – after all, how likely is it that the same number will come up twice?
• The “hot hand”– If your favourite player has made his last four shots, is the
probability of making the next shot higher, lower, or the same as the probability of his making a shot without the preceding four hits?
• After 6 successive failures in an exploration play with a 30% chance of success “the next well is bound to be a discovery”
• Tversky and Kahneman:– “Chance is commonly viewed as a self-
correcting process in which a deviation in one direction induces a deviation in the opposite direction to restore the equilibrium. In fact, deviations are not corrected as a chance process unfolds, they are merely diluted.”
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
The Law of Small Numbers –We are not consistently inconsistent.
• The Law of Small Numbers is ubiquitous in the world of business decision-makers, where it lends unfounded credibility to the claims of those who have been successful for a few years in a row.
• An employee who does well several years in a row is surprised if performance is thought to be mean-reverting.
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Probability Concepts: Multiplication Rule• The case of two dependent events
can be generalized for many events to
• If events are independent
which can be generalized for many events to
P(A and B) = P (A) * P(B|A)
P(A and B and C..) = P (A) * P(B|A) * P(C|BA)…
P(A and B and C…) = P (A) * P(B) * P(C) …
P(A | B) = P (A) and P(B | A) = P (B)
P(A and B) = P (A) * P(B)
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Using P50s to assess completion times
• Suppose a project is made up of five, parallel, independent tasks, all of which must be completed for the project to be completed.
• You have estimated the P50 time for completion of each task to be 6 weeks.
• What is the chance that the project will be completed within 6 weeks?
• By definition, the probability of each task being completed on or before the P50 time (6 weeks) is 50%. Since the tasks are independent, then,
P(All 5 completed by 6 weeks) = 0.55 ~ 3%
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Reliability of Predictors
• Historical estimates suggest one in every 1000 blow-out preventers (BOPs) has serious cracks.
• Suppose x-ray analysis is a very good, but not perfect, detector of these cracks.– If a BOP has cracks, x-rays will correctly say it has them
99% of the time– If a BOP does not have cracks, x-rays will wrongly say
that it has them 2% of the time
• A BOP has been x-rayed at random and the result was positive!– What is your intuitive estimate of the chance that is has
cracks?
Estimated Answer %
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Use Probability Rules: in this case, Bayes’ Rule
• Given a collection of n mutually exclusive and collectively exhaustive events B1, B2, …. Bn and another event A
(likelihood) * (prior probability)posterior probability =
{(likelihood) * (prior probability)}
• The denominator (summation) is the total probability of A, that is, all the ways that A can occur and P(A|Bi) is called the likelihood function
P (A|Bi) P(Bi) P (A|Bi) P(Bi) P(Bi|A) = = nP (A) P (A|Bi) P(Bi)
i=1
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Use Probability Rules: in this case, Bayes’ Rule
• Given a collection of n mutually exclusive and collectively exhaustive events B1, B2, …. Bn and another event A
(likelihood) * (prior probability)posterior probability =
{(likelihood) * (prior probability)}
• The denominator (summation) is the total probability of A, that is, all the ways that A can occur and P(A|Bi) is called the likelihood function
P (A|Bi) P(Bi) P (A|Bi) P(Bi) P(Bi|A) = = nP (A) P (A|Bi) P(Bi)
i=1
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
What’s the relevance?
We make extensive use of conditional probabilities, and their formulation as Bayes’ Rule, to answer questions such as
How should prior probabilities be revised in the light of new information?
- e.g. how should we revise initial CoS estimate as a drillingprogram progresses
What is the value of taking 3D Seismic?
What is the value of adding extra slots to a platform in case OOIP is higher than thought?
What is the value of coring a well?
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
*Insensitivity to Base Rate
• P(BOPC) = 1/1000 = 1x10-3
• P(test positive | BOPC) = 99%• P(test positive | no BOPC) = 2% • A = test positive, B = BOPC
( | ) ( )( | )( | ) ( ) ( | ) ( )
P A B P BP B AP A B P B P A B P B
3
30.99 10( | )
0.99 10 0.02 0.9990.047 4.7%
P SHRD test pos
BOPC
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Insensitivity to Base Rate Results
0
10
20
30
40
50
60
70
80
90
0 20 40 60 80 100
Estimated Posterior Probability
No
of P
artic
ipan
ts
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Uncertainty Propagation
Input 1
Model Decision Objective
AssessedUncertainties
Input n
Decision Criterion
PropagatedUncertainty Single Value
For non-linear models with uncertain inputs: the correct value of the decision criterion must be calculated by propagating the input uncertainty through to the decision objective uncertainty
eg NPV E[NPV] Equations(not brain!)
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Uncertainty Propagation
Input 1
Model Decision Objective
AssessedUncertainties
Input n
Decision Criterion
PropagatedUncertainty Single Value
For non-linear models with uncertain inputs: the correct value of the decision criterion must be calculated by propagating the input uncertainty through to the decision objective uncertainty
eg NPV E[NPV] Equations(not brain!)
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Averages don’t always work - misplaced
• For non-linear processes,– reservoir simulation– volumetrics with cut-offs– development alternatives
even if only a single, “best” estimate is required, we still need to use complete range of inputs - cannot use an average input
• Also, P10 (P90) results are NOT given by taking P10 (P90) inputs and running them through the model
ResultY = 4
ModelY = X2 Z2/
3 21 1 8
Mean = 10 Mean = 5
x z
Y
Simulation Result
0 30
True Mean~ 7.8
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
The Flaw of Averages:
• Unless model (f) is linear (“model” is a calculation)
where x,y,z,… are uncertain quantities, and
f is a calculation using x, y, z ....
average of f( x,y,z,…. )
f( average(x), average(y), average(z),….)
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Flaw of Using Averages (after Savage)
NCF, Log, ...
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Insensitivity to Base Rate: Frequency view
Applying the following reliabilities (CONDITIONAL probabilities)
P(T|R) Has Has NotHas 99% 2%
Has Not 1% 98%Test says
Real world
100% 100%
Has Has NotHas 99 1998
Has Not 1 97902Test says
Real worldto a sample of 100,000 people gives the following frequencies
100 99,900
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Outline
• Introductions, Questionnaire• Overview of Performance of O&G Industry
Capital Investment Decisions • Underlying Concepts• Major Psychological Factors• Limits of Intuition in Complex & Uncertain
Situations– Uncertainty Propagation– Updating Estimates for New Information– Complexity
• What can we do… RISC Process
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
*
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
*
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
*
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Change Detection and Blindness
• People fail to notice large changes to visual arrays and scenes if they are briefly occluded
– e.g., during saccades, eye blinks, or presentations using simple masks
– the task is called ‘change detection’
– the inability to perceive change is called ‘change blindness’
• The previous simple example works for about a third to a half of people, and was created by Mondy and Coltheart (2002)
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Counting Passes:Basketball movie
Basketball Movie
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
The Monty Hall Problem
• US TV Game show in early 1970s: “Let’s Make a Deal!”
– Show host Monty Hall
• Later discussed in newspaper column by Marilyn vos Savant
– Created an enormous response
– More than 10000 letters (many from professional mathematicians) denouncing her answer.
– Discussed in “The American Statistician.”
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
The Monty Hall Problem
1/3 1/3 1/3There are three doors. One of these doors contains a prize. The other two do not. Therefore, the probability that any one of the doors contains the prize is 1/3.
You choose one door, say, door A.
A B C
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
The Monty Hall Problem
1/3 1/3 1/3There are three doors. One of these doors contains a prize. The other two do not. Therefore, the probability that any one of the doors contains the prize is 1/3.
You choose one door, say, door A.
A B C
Given the option, should you stick with your original choice of door A, or switch to door B? Or does it matter?
I open one of the two remaining doors, say, door C, and reveal to you that it does not contain a prize.
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
*Solution: The Monty Hall Problem
1/3 2/3
A B C
Not behind A= 2/3
• If your original guess was correct and you switch, you lose.
• If your original guess was wrong and you switch, you win.
• Since your original guess would be wrong two out of three times, if you switch you’ll win two out of three times.
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Use Probability Rules: in this case, Bayes’ Rule
• Conceptually, Bayes tells you how to update your prior probability of some event occurring (or model being true) after getting some information/data
based on the likelihood (probability) of observing the information/data, giventhat the event/model is true
(likelihood) * (prior probability)posterior probability =
{(likelihood) * (prior probability)}
• The denominator (summation) is the total probability of the data/information, that is, all the ways that it can occur
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
The Monty Hall Problem: Bayes’ Theorem
( | ) P(X and Y) P(X |Y) P(Y)P Y XP(X) P(X)
( and ) ( ) ( | )( | )P( ) P( )
P B open C P B P open C BP B open Copen C open C
( and ) ( ) ( | )( | )P( ) P( )
P A open C P A P open C AP A open Copen C open C
Bayes’ Theorem
We want to know the probability of the prize lying behind A or B, given the host opens C
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Remember: Decomposing Total (or Marginal) Probability
BAB1
B2
B3
P(A) = P(A&B1) + P(A&B2) + P(A&B3)= P(A|B1)P(B1) + P(A|B2)P(B2) + P(A|B3)P(B3)
nP(A) = P (A|Bi) P(Bi)i=1
In “Monty Hall”:
A = open CB1 = prize behind AB2 = prize behind BB3 = prize behind C
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
The Monty Hall Problem: Probability that host will open door C
The a priori probability that the prize is behind door A, B, or CP(A) = P(B) = P(C) = 1/3
The probability that host opens door C if the prize is behind A:
The probability that host opens door C if the prize is behind B:
This image cannot currently be displayed.
P(open C|A) = 1/2
P(open C|B) = 1
The probability that host opens door C if the prize is behind C:P(open C|C) = 0
The total probability for host opening door C is then
P(open C) = P(A)*P(oC|A) + P(B)*P(oC|B) + P(C)*P(oC|C)= (1/3)*(1/2) + (1/3)*(1) + (1/3)*(0) = 1/2
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
The Monty Hall Problem: Solution using Bayes’ Rule
1 1( ) ( |B) 23( | )1P( ) 3
2
P B P oCP B oCoC
1 1( ) ( |A) 13 2( | )1P( ) 3
2
P A P oCP A oCoC
Switch to door B!
So the probabilities we require are:
P(open C) = P(A)*P(oC|A) + P(B)*P(oC|B) + P(C)*P(oC|C)= (1/3)*(1/2) + (1/3)*(1) + (1/3)*(0) = 1/2
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
21 movie
21 Movie
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
You Choose
Prize Behind
Host opens
Outcome (no Switch) Switch to Outcome
A A B or C Win B or C LoseA B C Lose B WinA C B Lose C WinB A C Lose A WinB B A or C Win A or C LoseB C A Lose C WinC A B Lose A WinC B A Lose B WinC C A or B Win A or B Lose
*Alternative Solution:List all possibilities
When you switch, you win 2/3 of the time- if you stick you win 1/3 of the time
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Some of the things the letters said
• “I’m very concerned with the general public’s lack of mathematical skills. Please help by confessing your error.”
» Robert Sachs, PhD, George Mason University
• “There is enough mathematical illiteracy in this country, and we don’t need the world’s highest IQ propagating more. Shame!”
» Scott Smith, PhD, University of Florida
• “I’m in shock that after being corrected by at least three mathematicians, you still do not see your mistake.”
» Kent Ford, Dickinson State University
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Some of the things the letters to vos Savant said:
• “…Your solution is the correct one and any REAL mathematician can produce a proof of its correctness. REAL mathematicians consider this a trivial problem… WHAT discipline do these respondents have their PhDs in? Is it adolescent behavior? If it is in mathematics, my second question is what institution granted it?”
» Professor Stephen J. Turner, Babson College
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Some of the things the letters said
• “In a recent column, you called on math classes around the country to perform an experiment that would confirm your response to a game show problem. My eight-grade classes tried it, and I don’t really understand how to set up an equation for your theory, but it definitely does work!”
» Pat Gross, Ascension School, Missouri
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Some of the things the letters said
• “After considerable discussion and vacillation here at the Los Alamos National Laboratory, two of my colleagues independently programmed the problem, and in 1 million trials, switching paid off 66.7 percent of the time. The total running time on the computer was less than one second.”
» G.P. DeVault, PhD, Los Alamos National Laboratory
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Outline
• Introductions, Questionnaire• Overview of Performance of O&G Industry
Capital Investment Decisions • Underlying Concepts• Major Psychological Factors• Limits of Intuition in Complex & Uncertain
Situations– Uncertainty Propagation– Updating Estimates with New Information– Complexity
• What can we do… RISC Process
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Develop Strategyand SetFinancial Goals
Manage theBalance Sheet
Manage HumanResources
Use ProjectFinance for Middle East Build LNG
Capability inAlaska
Achieve 60/40Oil/Gas PortfolioSplit
Assume Operatorshipin CO2 Flood
Produce in theRockies
Develop VenezuelanHeavy Oil
Explore inColombia
Partner with XXin Deepwater
Trade GOM Shelffor Indonesia
Sell SouthLouisiana
Buy More Blocksin Viet Nam
A generic US Independent Executive Decision-Maker
• Complex problem-7 chunks of data
• “Optimum” is not intuitive-models can help
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
*Intuition: Steel Band Problem
• Assume the earth is perfectly smooth and a band of steel is places around the equator, which has a circumference of 50,000 km.
• Now we add in an extra 10m of steel, which slightly forces the band off the surface of the earth.
• What is your intuitive estimate of how much?
• Answer: 1.6 metres
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Imagine two one-mile long pieces of railroad track, put end to end, and attached to the ground at the extremes. When it gets hot, each track length expands by one inch, forcing it to rise above the ground.
How high is the track off the ground at peak?
Give a high and low estimate such that you are 90% sure the correct answer lies between them.
Geometry Problem
x?
Ref: Richard Thaler
1 mile + 1 inch
1 mile
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
• Give a high and low estimate such that you are 90% sure the correct answer lies between them.
• Typical Answers:– Median Low Guess: ½ Inch
– Median High Guess: 2 Inches
– Ranges containing True Value: 15.9%
*Intuition: Geometry Problem
Ref: Richard Thaler
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
*Normative Rule
• Pythagorean Theorem2 2 2 2 2z x y x z y
2 2
2 2
x (1 m ile 1 in ch ) (1 m ile )
(5, 2 8 0 * 1 2 1) (5, 2 8 0 * 1 2 )
• Descriptive reality:– Most people underestimate x
– Why: We anchor on 1 inch and adjust insufficiently
1 2 6, 7 2 1 3 5 5 .9 8 in ch es 2 9 .6 fee t!
Ref: Richard Thaler
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Intuition is Overrated
• Many decision-makers believe that intuition, repeated experience and their general intelligence will see them through
• Human beings are imperfect information processors
• We can’t always trust our intuition and perception. particularly in an uncertain environment!
• We need to use the appropriatetools and frameworks to address the uncertainties and decisions.
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Intuition
• Intuition can be useful when– It is educated by making the same kind of decision
multiple times and observing the outcome – that is, when you are well calibrated
– The possible outcomes of the decision are not important
• Intuition can lead you astray when– Even slight degrees of complexity are involved– Uncertainty is involved, particularly if there are multiple
uncertainties which you have to combine to reach a conclusion
– Or when it is trained in one situation but then applied to another
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
When to trust intuition
• Daniel Kahneman (Heuristics & biases, prescriptive DA) and Gary Klein (naturalistic decision-making) debated when you can trust intuition or gut-instinct. They agreed that to protect decisions against bias, four tests should be passed:
– The familiarity test: Have we frequently experienced identical or similar situations?
– The feedback test: Did we get quick & reliable feedback on the outcomes of past decisions/judgments?
– The measured-emotions test: Is our thinking clouded by emotions we have experienced in similar or related situations? (“no”= pass!)
– The independence test: Are we likely to be influenced by any inappropriate personal motivations or biased thinking (“no”= pass!)
• If a situation fails even one of these four tests, we need to strengthen the decision process to reduce the risk of a bad outcome.
After a McKinsey Quarterly article (May 2010) based on “Conditions for intuitive expertise: A failure to disagree”, American Psychologist, Sept 2009
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
*Probability Training
• In terms of practical applicability, probability theory is comparable with geometry;
– both are branches of applied mathematics that are directly linked with the problems of daily life.
• While most people have a natural feel for geometry (at least to some extent), many people clearly have trouble developing a good intuition for probability.
• In no other branch of mathematics is it so easy to make mistakes as in probability theory.
– Conditional probabilities, and Bayes theorem in particular, are especially difficult
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Probability Training
• In terms of practical applicability, probability theory is comparable with geometry;
– both are branches of applied mathematics that are directly linked with the problems of daily life.
• While most people have a natural feel for geometry (at least to some extent), many people clearly have trouble developing a good intuition for probability.
• In no other branch of mathematics is it so easy to make mistakes as in probability theory.
– Conditional probabilities, and Bayes theorem in particular, are especially difficult
“The theory of probabilities is at bottom nothing but common sense reduced to calculus; …
It teaches us to avoid the illusions which often mislead us;
… there is no science more worthy of our contemplations nor a more useful one for admission to our system of public education.”
Laplace – Theorie Analytique des Probabilites
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Developing subjective Probabilities
Have you made repetitive forecasts of such an event in the past
Did you receive timely feedback on the accuracy of your forecasts
Is there a reference class of events that is similar and on which relative frequency information exists
Go ahead and assess subjective probability
Use the relative frequency information as subjective probability
Beware of impact of potential errors due to biases and inappropriate heuristics
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Eliciting Subjective Probabilities
• The de Finetti game:– Bruno de Finetti
u An Italian statistician (1906 – 1985)
u Worked in the middle ground between mathematics and psychology
– A device to objectively measure subjective probability
• Most people ”lie” about probability without even being aware of it – they even lie to themselves.
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Eliciting Subjective Probabilities
• Suppose your friend just took an exam and feels good about how she did.
• She might tell you:– ”I aced it; I’m one hundred percent sure I’ll get a
perfect score.”
• The de Finetti game is a way to measure how sure she really is about having aced the exam.
• We need to ask your friend a series of questions to assess her true subjective probability of having aced the test.
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Eliciting Subjective Probabilities
• Tell you friend the following:– ”Let’s play a game. You have a choice.
1. you can either draw a ball from a bag that has 98 red balls and 2 black balls. If you happen to draw a red ball, I will give you one million dollars, or
2. you can decide to wait to see how you did on the exam, and if you receive a perfect score on that exam I will give you one million dollars.
– What’s you choice: draw or wait?”
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Eliciting Subjective Probabilities
• If your friend says:– ”Draw from the bag.”
– She then isn’t 100% sure that she has aced the exam – actually she must be less then 98% certain that she has aced the test if she chooses to draw from the bag.
• Now you ask the next question:– ”Now there are 80 red balls in the bag and
20 black ones.”
– ”Do you want to draw, and if you obtain a red ball get a million dollars, or wait to see how the exam went?”
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Eliciting Subjective Probabilities
• If your friend says:– ”Wait for the exam result.”
– Then you know that he is more than 80% sure that she really aced the test.
• Now choose a value in between, such as 90 red balls and ask:– ”Now there are 90 red balls in the bag and
10 black ones.”
– ”Do you want to draw, and if you obtain a red ball get a million dollars, or wait to see how the exam went?”
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Eliciting Subjective Probabilities
• At some point, say at 83 red balls (83%), your friend may say:– ”I’m indifferent between drawing a ball and waiting
for the exam result.”
• This (83%) is then her subjective probability of having aced the test.
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Perceptual Limitations – visual illusions are a metaphor for cognitive illusions
• Previous examples were Cognitive/Probability Illusions
• Cognitive biases or illusions are similar to optical illusions in that the error can remain compellingeven when one is fully aware of its nature.– Awareness of the bias, by itself, does not
necessarily produce a more accurate perception.
• Cognitive biases, therefore, are also exceedingly difficult to overcome.
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
What Does This Have to Do With You and Me?
• These biases, which are well known to psychologists, have been shown to influence real decision-making behaviour.
• Evidence confirms oil & gas professionals are as prone to these biases and illusions as anyone else.
• In any decision-making situation it pays to pause and ask a few key questions– What are the non-intuitive factors might be present, particularly with
respect to uncertainty assessment and probability
– Am I motivated to see things a certain way?
– What expectations did I bring into the situation?
– Would I see things differently without these expectations and motives?
– Have I consulted others who don’t share my motives and expectations?
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
De-biasing
• Create awareness of the problems & recognition of situations when they may be present
– Anchoring– Overconfidence– Availability, Vividness & Recency
• Don’t rely on memory alone – write lists
• Actively challenge ourselves.– Stop to consider reasons why your judgement might be wrong.
• Abandon false comfort of single-point predictions.– Use ranges instead of single-point estimates.– Use multiple anchors.
• Calibration.– Feedback and accountability.
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
Understanding the Limits of Our Knowledge
• To know that we know what we know, and that we do not know what we do not know, that is true knowledge
Confucius
“It’s not what we don’t know that gets us into trouble, it’s what we know that
ain’t so”Will Rogers
GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg
The Pioneers
• Kahneman and Tversky– 2002 Nobel Price in Economics
• Russo and Schoemaker– Decision Traps– Winning Decisions
• Thaler– The Winner’s Curse
• Bazerman– Judgment in Managerial Decision Making
• Plous– The Psychology of Judgment and Decision Making