The Two Faces of The Two Faces of
Decision Analysis Decision Analysis Practice (1)Practice (1)
James L. Corner
Department of Management Systems
University of Waikato,
Hamilton, New Zealand
Aalto University
Sept/Oct 2009
AgendaAgenda
1. Decision analysis is alive and well
(the first hour of the course)
2. Decision analysis is not so alive and well
(the last hour of the course)
The Nature of Decision AnalysisThe Nature of Decision Analysis
• For hard decision problems with many competing objectives, much uncertainty
• Process (called Value Focused Thinking)– Elicit an objective function (value vs utility)
– Identify alternatives (decision trees, IDs, etc)– Identify alternatives (decision trees, IDs, etc)
– Determine probability density function (pdf) over alternative outcomes
– Integrate utility function over pdf for each alternative
– Choose one with maximum utility, or
– Use sensitivity analysis to find out the value of more information
Perspective on Decision Analysis
Applications in the Operations
Research Literature
James L. Corner
Department of Management Systems
University of Waikato, NZ University of Waikato, NZ
withDonald L. Keefer
Craig W. Kirkwood
Department of Supply Chain Management
Arizona State University
Tempe, AZ, USA
Purpose
•Present highlights from and stimulate interest in:
“Perspective on Decision Analysis Applications, 1990-2001”
(Decision Analysis, 2004)
•Companion technical report, “Summary of Decision Analysis Applications in
the Operations Research Literature, 1990-2001”Available at www.public.asu.edu/~kirkwood
•Accompanying comments on above article, with our responses•Accompanying comments on above article, with our responses
•Paper identifies and classifies DA applications from 16 major English
Language OR and closely related journals from 1990-2001
•Period-to-period comparisons with Corner and Kirkwood (Operations
Research, 1991) covering 1970-1989
•Perspective and opinions on trends and developments
•Practice competitions.
•Decision Analysis Affinity Group.
•Needs and concerns.
Ground Rules for Selecting DA Applications
•Is an application a “decision analysis” application?
•Include if it explicitly analyzes alternatives using judgmental
probabilities and/or subjectively assessed utility/value
functions.
•Include if it, on balance, uses a DA approach.
•Is it an “application”? •Is it an “application”?
- Include if it represents a case history of the use of DA to
address a specific decision problem.
- Include if it describes an analysis performed to provide
background for policy making.
- Include for reference if material is clearly of direct
interest for applications, even if an explicit application is
not included.
No. of Articles % Change in Ave.
No. per Year1970-1989 1990-2001
Total Number of Articles 85 86 69%
Total Articles in Common Journals 85 63 24%
Decision Sciences 4 1 -58%
European Journal of OR 5 0 -100%
IEEE Trans. on Engr. Mgmt. Not Cov 4 —
IEEE Trans. on SMC 7 1 -76%
Interfaces 18 40 270%
Number of DA Applications Articles, by Journal, with Trends
Journal of MCDA Not Pub 6 —
Journal of the OR Society 18 1 -91%
Management Science 10 4 -33%
Military OR Not Pub 8 —
Omega 3 1 -44%
Operations Research 15 11 22%
OR Letters 0 0 0%
Reliability Engr. & Sys. Safety Not Cov 2 —
Research• Tech. Mgmt. Not Cov 2 —
Risk Analysis 6 4 11%
Theory and Decision Not Cov 1 —
•Positive overall rate of change (1970-1989 to 1990-2001) over 32 years.
•69% increase in average rate per year if all 16 journals included.
•24% increase if confined to 10 journals included in both surveys.
•Applications articles concentrated in fewer journals for 1990-2001.
- Interfaces: 47% of total, versus 21% for 1970-1989.
Observations
- Interfaces: 47% of total, versus 21% for 1970-1989.
- Operations Research: 13%.
- Military OR (new): 9%.
•Substantial decreases in European OR journals, except for new Journal of MCDA.
•Implications.
- For DA field
- For new journal Decision Analysis
Classification by Area of Application
Each article classified into exactly one of six application areas and, where
appropriate, a subarea — numerous judgment calls required.
Area
Subarea
Increase Decrease
Energy (26) Envt Risk Regulation
Product Selection Site Selection
Project Selection
Strategy
Mfg Services (23) R&D Proj Selection
Medical (5) Down in OR lit from 16
Military (13) Large increase, new area
Public Policy (13) Standard Setting
General (6)
Classification by Methodological and Implementation Issues
Article included in (multiple) classification area(s) only if it provides
significant details about issue in question.
•Strategy and/or Objectives Generation (42) — substantial increase.
•Problem Structuring/Formulation (34) — substantial increase.
•Probability Assessment (22).
•Utility/Value Assessment (28).
•Sensitivity Analysis (23) — substantial increase.
•Communication/Facilitation (29).
•Group Decision Making (12).
•Implementation (27) — substantial increase.
General Trends and Developments
•Powerful PC software for DA.
- Wider use of IDs, strategy tables, tornado diagrams, etc.
- Solution of large problems.
•Decision conferencing — computer-assisted group meeting focused on
decision problem, with facilitator(s).
- Synthesis of techniques from DA, DSS, and group management. - Synthesis of techniques from DA, DSS, and group management.
- Typically apply MAV models within military or public sector.
•Stochastic trees — combine continuous-time Markov chains with DTs.
- Medical applications with time as continuum.
•Value-focused thinking — values as primary driver for structuring and
analysis, including generation of alternatives.
•Numerous military, strategic, and public policy applications.
Interdisciplinary Trends and Developments
•Normative systems — DA and AI.
- AI systems based on influence diagrams or belief nets (Bayesian
principles).
- Applications in medical diagnosis, energy price and demand
forecasting, and machine vision.
•Organizational processes — DA and OB.
- Well-defined procedures for conducting and implementing DA in large-
scale strategic analyses.
- Managing interactions between analysts and other stakeholders
•Real options — DA and Finance.
- Synthesis of methods from DA and finance proposed; few such
applications in OR literature so far.
- Increased emphasis on sequential (downstream) decisions.
Practice Competitions — Edelman Award Finalists
•Burnett, W. M., D. J. Monetta, B. G. Silverman. 1993. How the Gas Research Institute
(GRI) helped transform the US natural gas industry. Interfaces 23(1) 44−58.
•Long-term use of project appraisal methodology within GRI’s annual five-year R&D planning
process.
•Estimated benefits in tens of billions of dollars.
•Paté-Cornell, M.-E., P. S. Fischbeck. 1994. Risk management for the tiles of the space
shuttle. Interfaces 24(1) 64-86.
•PRA of failure of exterior surface tiles on US space shuttle orbiter.
•Highlighted high-risk zones of surface and organizational factors contributing to tile failure
risks.
•Von Winterfeldt, D., E. Schweitzer. 1998. An assessment of tritium supply alternatives
in support of the US nuclear weapons stockpile. Interfaces 28(1) 92−112.
•Analysis to help US DOE choose tritium-supply alternatives to replenish tritium for US nuclear
weapons stockpile.
•Influential in final choice by US Secretary of Energy.
Practice Competitions — DAS Practice Award Winners
1999. Mazen A. Skaff and Donald W. Spillman, “A Portfolio Management Process
and System for an Upstream Oil and Gas Organization.”
-Process and system to help manage large portfolio of upstream oil and gas
assets in GOM and support variety of related decisions.
2000. David A. Mauney, “Best Practices in the Application of Decision/Financial
Analysis to Repair/Replacement Decisions of Plant Components.”
-Process to aid in planning timing of major maintenance investments for fossil--Process to aid in planning timing of major maintenance investments for fossil-
fuel power plants.
2001. Eric Johnson, “Life Cycle Strategy Analysis.”
- Helped client pharmaceutical firm reach consensus on development strategy
for cancer drug.
2002. Jeff Stonebraker, “Commercial Evaluation of a Blood Clot-Busting Drug.”
- First decision analysis project done by Bayer Biological Products.
Decision Analysis Affinity Group (DAAG)
•Founded in 1995 to promote use of DA in industry and to further
development and careers of industrial practitioners.
- Fills important need for DA practitioners in industry.
•Annual conferences focusing on use and implementation of DA within major
corporations.
- Emphasis on mutual support and information-sharing among industrial
practitioners.
- Attendance by consultants and academics generally discouraged until
2002 meeting. 2002 meeting.
•DAS-DAAG relationships.
- Some joint members of DAS and DAAG.
•Most DAAG members not interested in publishing, DAS/INFORMS
technical sessions, etc.
- A number of DAAG-organized sessions at INFORMS meetings.
- Future relationships?
Needs and Concerns
•Status of DA in companies and universities.
- DA and DA groups in industry continue to fall in and out of favor.
- Ambiguity of “DA” even within INFORMS community.
•Contents in functional area for Interfaces or in keyword searches in
indices.
•Coverage in OR/MS textbooks.
•Key role of small group of institutions and individuals in DA education and
applications-oriented innovations.
- Effects of retirements, acquisitions, and reorientations?
- “Who’s gonna fill those shoes?”
•Better Tools.
- Modeling and assessing probabilistic dependence.
- Time-dynamics (sequential nature) of decision problems.
Concluding Remarks
1. DA alive and well as evidenced by applications published in the OR
literature over the last dozen years.
•Significant numbers of DA applications published outside the OR
literature.
2. Small fraction of real-world applications of DA written up and
published. published.
•Beneficial to DA field if more practitioners and consultants
published or presented applications.
3. How representative are publications (OR literature or otherwise) of
DA applications and practice in the real world?
Critique of the AboveCritique of the Above
Scott Cantor – Dept of Biostats, University of Texas
Raimo Hämäläinen – Aalto University
Some valid comments…Some valid comments…
• Don’t forget the specialized journals, eg in medicine:
– JAMA, NEJM, AIM, Medical DMg• Clinical DA, individual patient is DMer, vs
• Cost-effectiveness analysis – what’s best for society?
• Does theory drive practice, or the reverse?
– Medical field has advanced theory in:– Medical field has advanced theory in:• value-of-life,
• managing uncertainty
• value of information
• Just what is decision analysis anyway?
– AHP? Multicriteria Optimization?• If it helps, shouldn’t we use it and consider it DA?
• Problem first, solution technique second
Some valid comments (cont’d)…Some valid comments (cont’d)…
• Be more critical
– What worked, what didn’t, why?
– How do we measure decision success?• Monetary?
• Happiness?
• Transparency?
– How did the application deal with behavioral issues• Understanding of the technique
• Cognitive biases
• Effect of computer tool usage
• Group issues
• Use multiple methods (esp AHP) and heuristics
• Use online help
ConclusionConclusion
• Decision analysis is alive and well
• No. of publication outlets is increasing
• Some concerns but not fatal
• Challenges remain• Challenges remain
The Two Faces of The Two Faces of
Decision Analysis Decision Analysis Practice (2)Practice (2)
James L. Corner
Department of Management Systems
University of Waikato,
Hamilton, New Zealand
Aalko University
Sept/Oct 2009
AgendaAgenda
1. Decision analysis is alive and well
(the first hour of the course)
2. Decision analysis is not so alive and well
(the last hour of the course)
We already know so much about...We already know so much about...
• Descriptive decision processes –
what we do, unaided
• Prescriptive decision processes –
how we can do betterhow we can do better
• Normative decision processes –
what we should do ideally
6 Lessons From Considering
Descriptive Decision Processes
Decision Making
Alternatives
Values
Politics
Constraints & ResourcesTradeoffs
Information
Preferences
Culture
Lesson 1. The Basic Descriptive Decision Process
Decision Making
StructureObjectives
Recognition
Uncertainty & Risk
Implementation
Outcomes
Ethics
Choice
Decision Making
2. Alternatives
7. Values
Politics
5. Constraints & Resources10. Tradeoffs
4. Information
8. Preferences
Culture
Alternative Focused Thinking
Decision Making
6. Structure9. Objectives
1. Recognition
3. Uncertainty & Risk
12. Implementation
13. Outcomes
Ethics
11. Choice
Lessons LearnedLessons Learned
1. People are alternative focused, not value focused.
Lesson 2. A Question of OntologyLesson 2. A Question of Ontology
• Realism vs nominalism in decision making
• Need for a dialectical approach
Intuition (Nominalism) Rationality (Realism)
- automatic - intentional
- holistic - analytic
- non-verbal - verbal
- associated with emotion & - emotion-free
Two Parallel Systems of Knowing
- associated with emotion &
feeling
- emotion-free
- encodes information as
examples, images, stories
- encodes information as
abstract symbols & concepts
- operates in the sub-conscious - operates in the conscious
Lessons LearnedLessons Learned
1. People are alternative focused, not value focused.
2. People need to be both intuitive and rational.
Lesson 3. Two Basic Human DifferencesLesson 3. Two Basic Human Differences
• Holistic vs. Non-Holistic evaluation
• Compensatory behaviour
Schoemaker’s FrameworkSchoemaker’s FrameworkSubjective Probabilities
Subjective Weights
Decision
Holistic
EU Models with Probability Transformations
Expected Utility Theory
Mean-Risk Models
Additive Models
Satisficing
Garbage Can
Image Theory
Recognition Primed Decisions
Disjunctive Models
Conjunctive Models
Image Theory
Decision
Models
Non-Holistic
(Sequential
Elimination)
Recognition Primed Decisions
Comparisons against some Standard
Additive Difference Models
Dominance ModelsComparisons across Attributes
Lexicographic Model
Elimination By Aspects Model
Experience the Situation in a changing Context
Reassess
Situation
Seek More
Information
Are experiences
Is the situation
familiar?
Recognition has four aspects
Goals Cues
Are experiences
Violated? Expectancies Actions 1…n
Mental Simulation of Action (n)
Will it work?
Implement
Modify
Mapping of Descriptive ModelsMapping of Descriptive Models
Compensatory Non-CompensatoryHolistic
Recognition Printed Decisions
Image Theory
Garbage Can
SatisficingAdditive Difference
Additive
Non-Holistic E.B.A
Lexicographic
Conjunctive/Disjunctive
Additive Difference
Additive
Dillon, Buchanan, Corner 2005
Mapping of Empirical Results Mapping of Empirical Results
onto Descriptive Theoriesonto Descriptive Theories
Compensatory Non-Compensatory
Holistic
Recognition Printed Decisions
Image Theory
Garbage Can
XXXX XXXXXXXX XXXXXXXX XXXX
XXXXXXXX XXXX XXXX XXXX
Non-Holistic
Holistic
Garbage Can
Satisficing
E.B.A
Lexicographic
Conjunctive/Disjunctive
Additive Difference
Additive
XXXXXXXX
XXXX
XXXX
XXXX
XXXXXXXX XXXX XXXX XXXX
XXXXXXXX XXXX XXXX
XXXX
XXXXXXXX
XXXXXXXX
XXXX
Dillon, Buchanan, Corner 2005
Lessons LearnedLessons Learned
1. People are alternative focused, not value focused.
2. People need to be both intuitive and rational.
3. People are not normally compensatory and non-holistic.
Lesson 4. A Question of Framing Lesson 4. A Question of Framing ((LipshitzLipshitz’ ’ Framework 1993)Framework 1993)
Consequential Choice Matching Reassessment
Additive
Additive difference
Satisficing
Garbage CanGarbage Can
Conjunctive/Disjunctive
Elimination By Aspects
Lexicographic
Image Theory (2) Image Theory (1) Image Theory (3)
Recognition Primed Decisions (1) Recognition Prime Decisions (2)
Lessons LearnedLessons Learned
1. People are alternative focused, not value focused.
2. People need to be both intuitive and rational.
3. People are not normally compensatory and non-holistic.
4. Framing
1. People wish not to always look toward the future1. People wish not to always look toward the future
Four Types of Formulation Process Four Types of Formulation Process Nutt 2003Nutt 2003
Issue Processes (26% of cases, least successful)An issue is analysed for the decision process to consider. Concerns and difficulties imply problems, which in turn suggest solutions (DMs become problem solvers).
Idea Processes (33%, third most successful)Direction is provided with an initial solution imposed by the DM. Development is concerned with certifying the idea’s virtues and suggesting refinements.suggesting refinements.
Objective-Directed Processes (29%, second)Direction setting with objectives, goals, or aims. Freedom to search for alternatives.
Reframing Processes (12%, most successful)Focusing on solutions, problems, or new practices to justify the need to act. (Avoids Raiffa’s 1968 “errors of the third kind.”)
How to Reframe: How to Reframe:
Dynamic Problem Structuring Dynamic Problem Structuring (Corner (Corner et al. et al. 2001)2001)
VFT AFT
Alternatives Criteria
Lessons LearnedLessons Learned
1. People are alternative focused, not value focused.
2. People need to be both intuitive and rational.
3. People are not normally compensatory and non-holistic.
4. Framing
1. People wish not to always look toward the future1. People wish not to always look toward the future
2. Re-framing should be the goal
Lesson 5. Consider Individual PersonalityLesson 5. Consider Individual Personality
Extrovert
Sensor
Thinker
Introvert
Intuitor
FeelerThinker
Judgement
Feeler
Perception
Decision Making Aspects of Decision Making Aspects of
PersonalityPersonalityThinker (50% of population)
- likes many alternatives
- needs analysis time
- does not rush to choice
Feeler (50%)- more concerned with impact on others
- does not rush to choice- does not rush to choice
- tends to be holistic
- tends toward reassessment
Intuitor (25%)- tends to be consequential choice
- likes unique alternatives
- does not like detail
Sensor (75%)- likes few alternatives
- likes graphs and diagrams
- tends toward matching
Lessons LearnedLessons Learned
1. People are alternative focused, not value focused.
2. People need to be both intuitive and rational.
3. People are not normally compensatory and non-holistic.
4. Framing is very important
1. People wish not to always look toward the future1. People wish not to always look toward the future
2. Re-framing should be the goal
5. Decision behavior differs by personality
Lesson 6. Consider Decision Style
Instructions
Lessons LearnedLessons Learned
1. People are alternative focused, not value focused.
2. People need to be both intuitive and rational.
3. People are not normally compensatory and non-holistic.
4. Framing is very important
1. People wish not to always look toward the future1. People wish not to always look toward the future
2. Re-framing should be the goal
5. Decision behavior differs by personality
6. Decision behavior differs by individual style
Concluding Remarks
1. Our natural inclination does NOT follow the decision
analytic approach
2. People do not like to make and are not good at making
tradeoffs
3. People Match; they do not consider consequences
appropriately
4. Decision making is not a linear process4. Decision making is not a linear process
5. People do not know what they want
6. There is more than one way to structure a decision
problem
7. Framing is absolutely key in structuring and is often
overlooked
8. There is a need for model management systems
The question is…..
… How do we incorporate
this descriptive behavior
into our prescriptive
techniques?
Or, even better…
… What new prescriptive
techniques can we
envision that improve on
descriptive behavior?
Questions?
Comments?
Final thought – which way is the sun going?