Managing uncertainties by IPCC: battle of disciplines

by Rob Swart

Which changes took place since the 1990 1st assessment and the 2007 AR4?

What are similarities and differences between Working Groups?

Uncertainty developments in IPCC

• FAR (1990): careful formulation (WG1)

•AR4 (2007): all WGs use (improved) Guidance, but select different options, and uncertainty focus mainly in SPM

• TAR (2001): Guidance for authors, adapted by WG1, applied by WG2 and rejected by WG3

• SAR (1995): WG1 includes special section, WG2 introduced “confidence levels”; WG3: silence

Working Group I (climate scientists)

SAR: special section about uncertainties, “project”, “the balance of evidence suggests”

• AR1: “certain of”, “confident about”, ”predict”

TAR: guidance note; not accepted but adapted by WG1: likelihood scale, “new and stronger evidence” (repeated in AR4)

subjective (bayesian – “degree of belief”) perspective, but hidden in definition of

‘likelihoods’ (“judgmental”)

Working Group II (impact researchers)

AR1: little attention, cautious but inconsistent formulation of findings

SAR: introduction confidence levels (H, M. L)TAR: guidance note, selectively

applied but generally as intended: confidence levels and 2D qualitative scale (also in AR4)

Explicit bayesian perspective

FAR: hardly any attention

Working Group III (economists, engineers)

SAR: description of definitions, assumptions, methodsTAR: guidance hardly used, but •stresses importance choices made

•elaborates background methodologies (top-down versus bottom-up)

•sometimes uses undefined/subjective terminology

•probabilities for reduction potential options

AR4: 2D qualitative approach

IPCC Guidelines TAR (2001): stepwise approach

1. Identification of key factors2. Document ranges and distributions3. Determine precision4. Characterize value distribution: use confidence

levels (1D) or evaluate the amount of evidence athe level of agreement (2D)

5. Value and describe the underlying information6. Describe the origin (“traceable account”)7. (Use a probabilistic framework if applicable)

This is were the emphasis has been

Issues such as “framing”, unquantifiable uncertainties, selection of indicators

missing!

Types of uncertainty (e.g., Dessai) Epistemic uncertainty: incomplete knowledge (c.f. “myopia”) e.g., climate sensitivity, terrestrial carbon uptake, structural

uncertainty in models Stochastic uncertainty: system variability (c.f. “chance”)

e.g., non-linear behaviour of climate system, randomness, initial conditions

Human reflexive uncertainty: volition (c.f. “intentionality”) e.g., climate policies, behavioural responses to scientific

knowledge IPCC acknowledges different sources of uncertainty, but

does not translate this into diversity of uncertainty management and communication guidance

Method 1

qualitative description of level of evidence and level of agreement

Established but incomplete

Speculative

Well-established

Competing explanations

Choice WG3 in AR4

TARAR4

Method 2quantitatively calibrated levels of

agreement of scientists

Choice WG2 in AR4

Method 3

probabilities that the statements are true (“judgmental”)

Choice WG1 in AR4

But remember: words have different meanings for different people

Probability that subjects associated with the qualitative description

0.00.20.40.60.81.0

Almost certain

Probable

Likely

Good chance

Possible

Tossup

Unlikely

Improbable

Doubtful

Almost impossible

range of individual upper bound estimates

range of individual lower bound estimates

range from upper to lower median estimate

Qua

litat

ive

desc

riptio

n of

unc

erta

inty

use

dIPCC “likely”

IPCC “unlikely”

Profiles according to Weiss 2003:

1. Environmental absolutist

2. Cautious environmentalist

3. Environmental centrist

4. Technological optimist

5. Scientific absolutist

Even if there is agreement on the “level of evidence”, there will be disagreement about how to respond

Level of

Evidence

Text example from the Intergovernmental Panel on Climate Change WG I (2001)

“In the light of new evidence and taking into account the remaining uncertainties, most of the observed warming over the last 50 years is likely7

to have been due to the increase in greenhouse gas concentrations.”(SPM)

Example IPCC WG I (continued)7 In this Summary for Policymakers and in the Technical Summary, the following words have been used where appropriate to indicate judgmental estimates of confidence: virtually certain (greater than 99% chance that a result is true); very likely (90–99% chance); likely (66–90% chance); medium likelihood (33–66% chance); unlikely (10–33% chance); very unlikely (1–10% chance); exceptionally unlikely (less than 1% chance). The reader is referred to individual chapters for more details.

IPCC CO2 emission scenarios in 1990

Uncertainties in the level of

policy

Not only the text, also the graphics became more complex (better?)’…

BaU

Accelerated Policies

IPCC CO2 emission scenarios in 1992

Uncertainties in the driving forces

IPCC CO2 emissions scenarios in 2001

Figure 2-12: Global CO2 emissions from energy and industry, historical development from 1900 to 1990 and in 40 SRES scenarios from 1990 to 2100, shown as an index (1990 = 1). The range is large in the base year 1990, as indicated by an “error” bar, but is excluded from the indexed future emissions paths. The dashed time-paths depict individual SRES scenarios and the blue shaded area the range of scenarios from the literature (as documented in the SRES database). The median (50th), 5th, and 95th percentiles of the frequency distribution are shown. The statistics associated with the distribution of scenarios do not imply probability of occurrence (e.g., the frequency distribution of the scenarios in the literature may be influenced by the use of IS92a as a reference for many subsequent studies). The 40 SRES scenarios are classified into six groups. Jointly the scenarios span most of the range of the scenarios in the literature. The emissions profiles are dynamic, ranging from continuous increases to those that curve through a maximum and then decline. The coloured vertical bars indicate the range of the four SRES scenario families in 2100. Also shown as vertical bars on the right are the ranges of emissions in 2100 of IS92 scenarios, and of scenarios from the literature that apparently include additional climate initiatives (designated as “intervention” scenarios emissions range), those that do not (“non-intervention”), and those that cannot be assigned to either of these two categories (“non-classified”).

Uncertainties driving forces, different models

Detailed explanation

intervention/

no intervention

statistical information

Uncertainty range

Level of knowledge

Uncertainties: targeted by sceptics

Limitation to one source (where there were several available) made this graph

particularly vulnerable

Always check the axes!

Apparent correlations do not automatically imply causal links

Improvement in AR4: hierarchy of approaches

LEVEL OF INFORMATION/AGREEMENT

LEVEL OF INFORMATION/AGREEMENT

Ambiguous or unpredictable Describe governing factors

Trend/direction can be described

Explain basis/extent to which opposite changes would not be expected; use the language options

Ranges/orders of magnitude can be given

Use (also) confidence scale

Probabilities can be given Describe assumptions, sgtructural undertainties, use likelihood scale

Probability distribution functions can be provided

Provide PDF, methods used, and structural uncertainties

NB: simplified from original source!!

Three important uncertainty dimensions

frequentist: “truth” based on observations, repeatedexperiments

bayesian: level of “belief”

precise: quantified risk, large datasets unprecise: few and/or inconsistent data

observations-theories/models human choice/intentionality/volition

-> “agree to disagree”

The three dimensions visualized

findngs about:• observed GHG concentrations• observed radiative forcing• past GHG emissions• observed temperature

mostly WG1?

findings about:findings about:-- GHG emissions reduction potential GHG emissions reduction potential -- Costs of technologiesCosts of technologies-- Future climatic changesFuture climatic changes-- Future extreme eventsFuture extreme events-- Projected impacts Projected impacts -- Emission scenariosEmission scenarios•• Costs and benefits of stabilizing Costs and benefits of stabilizing

Mostly WG3?Mostly WG3?

findings about:• observed climate impacts• attribution climate change

Mostly WG2?

attribution(models/theories)

frequentist/objective bayesian/subjective

Likelihood

Confidence scale

Level of evidence and agreement

Explanatory factors

FysicalObservations/measurements

precise

unprecise

scenarios, human choices

Human systems

Natural

systems

Personal conclusions

+ Increasing appreciation by an increasing number of authors from AR1 to AR4

+ IPCC terminology adopted by other assessments (Millennium Ecosystem Assessment, Arctic Assessment), with mixed results

+ Major differences between physicists/climate scientists (WG1), biologists/ecologists (WG2) and economists/social scientists (WG3) not consistent with one-size-fits-all approach

+ Diificult balance between scientifically comprehensive and balanced account of uncertainties and effectiveness of communation with readers/policymakers.

Ideas about managing and communicating uncertainties evolve rather fundamentally over time as we learn

Differences between Working Groups (disciplines) not only acceptable, but useful to enhance understanding about the nature of uncertainties

Distinguish between findings based on observations, on models and theories, or on scenarios including human choices

Give much and early attention to choice of indicators, graphical representation and explanation of outer ends of ranges

Provide an account of the background and history of key findings (pedigree/traceable account)

Some lessons learned

Thanks

3 paradigms risks and uncertainties

uncertainty as failure Uncertainty is temporary Reduce uncertainty, develop ever complex models Tools: quantify, Monte Carlo, Bayesian belief networks

uncertainty as lack of agreement Independent comparative evaluation of research results (evidence evaluation) Tools: scientific consensus; multi disciplinary expert panels Focus on “robust “findings

uncertainty as unavoidable and useful information Uncertainty as intrinsic in complex systems Uncertainty as a result of knowledge production Acceptance that not all uncertainty is quantifiable Deal with deeper dimensions of uncertainty openly (problem frames, indeterminacy, ignorance,

assumptions, value loadings) Tools: Knowledge Quality Assessment “Working deliberatively within imperfections”

Adapted from Jeroen van der

Sluijs

How to act upon such uncertainty? Bayesian approach: 5 priors. Average and update likelihood of

each grid-cell being red with data (but oooops, there is no data & we need decisions NOW)

IPCC approach: Lock the 5 consultants up in a room and don’t release them before they have consensus

Nihilist approach: Dump the science and decide on an other basis Precautionary robustness approach: protect all grid-cells Academic bureaucrat approach: Weigh by citation index (or H-

index) of consultant. Select the consultant that you trust most Real life approach: Select the consultant that best fits your policy

agenda Post normal: explore the relevance of our ignorance: working

deliberatively within imperfections

Adapted from Jeroen van der

Sluijs

De zekerheidstrog(McKenzie, 1990)

Some issues to take into account

More research often leads to more uncertainties: Unforseen complexities Complex systems exhibit irreducible uncertainties (intrinsic or practical)

Scientific consensus model may lead to ignoring of weak early warning signals

Neglect of uncertainty management can lead to scandals and loss of trust in science and institutions

Non-quantifiable uncertainties dominate in many complex risks High quality risk assessment low uncertainty Uncertainty information is valuable input into policy debate Shift remains required from attention to uncertainty rewduction to making

uncertainties explicit and systematically deal with it

Adapted from Jeroen van der

Sluijs

Dimensions of uncertainty

Technical: (im)precision Methodological: (un)reliability Epistemological: ignorance Societal: societal (un)robustness

Example RIVM/MNP typology of uncertaintiesDimensions of uncertainty ->

Location of uncertainty

Level of uncertainty(from determinism, through probability and possibility, to

ignorance)

Nature of uncertainty

Qualification of knowledge

base

Value-ladenness of choices

Statistical uncertainty

Scenariouncertainty

Recognisedignorance

Epis-temic

Varia-bility

– 0 + – 0 +

ContextExpert judgmentMODEL

StructureImplemen-

tationParameters

Inputs

Data Outputs

Different issues can have different dominant locations and dimensions

Elements of Tool Catalogue

Sensitivity analysis (screening, local, global) Error propagation equations (“Tier 1”) Monte Carlo Analysis (“Tier 2”) Expert elicitation NUSAP (Numeral Unit Spread Assessment

Pedrigree) Scenario analysis PRIMA (Pluralistic fRamework of Integrated

uncertainty Management and risk Analysis) Checklist for model quality assistance Critical review of assumptions

Experience with RIVM/MNP uncertainty guidance, personal observations

Enhanced awareness of importance and hence Very broad support for development and application

from both management and researchers Successful application in special dedicated projects Limited but important applications in MNP’s top

products (annual National Environmental Balance, periodic National Environmental Outlooks)

Very limited application in other projects, mainly because of time and resource constraints