A Toolbox for Comprehensive, Efficient, and Robust Sensitivity and Uncertainty Analysis

Saman Razavi, First Annual General MeetingSaman Razavi, July 18-19, 2018

Uncertainty in Output Space

Model/Process Response Surface

(Uncertain)

Co

rrelatio

n

Effects

Inte

raction

s Effe

cts

Uncertainty in Input Space

θ1θ2

𝑍

uses information in the mismatch betweenmodel predictions and data to identify “good”values for the model “parameters”, and tocharacterize their posterior uncertainty.

(2) Inverse-Problem Approach

propagates assumptions on uncertainties ininputs and other system properties throughthe model to obtain some understanding onuncertainties in model predictions.

(1) Forward-Problem Approach

Joint Probability Distribution

Uncertainty in Input Space

Uncertainty in Output Space

Co

rrelatio

n

Effects

Joint Probability Distribution

Inte

raction

s Effe

cts

attributes the uncertainty in a modelprediction to the uncertainties in inputs, andseeks to answer the critical question:

when does uncertainty matter?

illuminates the controls on model behavior,thereby characterizing the dominant controlson predictive uncertainty.

guides research towards reducing theuncertainties that matter, as it points to themost important aspects of the problem.

θ1θ2

𝑍

(3) Sensitivity Analysis Approach

Model/Process Response Surface

(Uncertain)

A comprehensive, multi-approach, multi-algorithm software toolbox for sensitivity analysis of anycomputer simulation model, including Earth and environmental systems models.

Razavi, S., Sheikholeslami, R., Gupta, H., Haghnegahdar, A., VARS-TOOL: A Toolbox for Comprehensive, Efficient,and Robust Sensitivity and Uncertainty Analysis, submitted to Environmental Modelling & Software.

Important Features:

• Multi-Method Approach to Sensitivity Analysis

• Sensitivity Analysis of Dynamical Systems Models (NEW)

• Various Sampling Strategies, e.g., Progressive Latin Hypercube Sampling (NEW)

• Handling High-Dimensional Problems: A Grouping Solution to Curse of Dimensionality (NEW)

• Characterizing Confidence, Convergence, and Robustness

• Reporting and Visualization: Monitoring Stability and Convergence (NEW)

• Handling Model Crashes via Model Emulation (NEW)

• Interface with Any Computer Model and Linkage to OSTRICH toolkit (NEW)

• A Comprehensive Test Bed for Training and Research (NEW)

Most approaches to SA of Earth systems models ignore or, at best, do not adequately account for the

dynamical nature of such models. These approaches handle problems with only a single response.

0

5

10

15

20

20

05

-10

-01

20

05

-10

-08

20

05

-10

-15

20

05

-10

-22

20

05

-10

-29

20

05

-11

-05

20

05

-11

-12

20

05

-11

-19

20

05

-11

-26

20

05

-12

-03

20

05

-12

-10

20

05

-12

-17

20

05

-12

-24

20

05

-12

-31

20

06

-01

-07

20

06

-01

-14

20

06

-01

-21

20

06

-01

-28

20

06

-02

-04

20

06

-02

-11

20

06

-02

-18

20

06

-02

-25

20

06

-03

-04

20

06

-03

-11

20

06

-03

-18

20

06

-03

-25

20

06

-04

-01

20

06

-04

-08

20

06

-04

-15

20

06

-04

-22

20

06

-04

-29

20

06

-05

-06

20

06

-05

-13

20

06

-05

-20

20

06

-05

-27

20

06

-06

-03

20

06

-06

-10

20

06

-06

-17

20

06

-06

-24

20

06

-07

-01

20

06

-07

-08

20

06

-07

-15

20

06

-07

-22

20

06

-07

-29

20

06

-08

-05

20

06

-08

-12

20

06

-08

-19

20

06

-08

-26

20

06

-09

-02

20

06

-09

-09

20

06

-09

-16

20

06

-09

-23

20

06

-09

-30

-30-20-10

0102030

20

05

-10

-01

20

05

-10

-08

20

05

-10

-15

20

05

-10

-22

20

05

-10

-29

20

05

-11

-05

20

05

-11

-12

20

05

-11

-19

20

05

-11

-26

20

05

-12

-03

20

05

-12

-10

20

05

-12

-17

20

05

-12

-24

20

05

-12

-31

20

06

-01

-07

20

06

-01

-14

20

06

-01

-21

20

06

-01

-28

20

06

-02

-04

20

06

-02

-11

20

06

-02

-18

20

06

-02

-25

20

06

-03

-04

20

06

-03

-11

20

06

-03

-18

20

06

-03

-25

20

06

-04

-01

20

06

-04

-08

20

06

-04

-15

20

06

-04

-22

20

06

-04

-29

20

06

-05

-06

20

06

-05

-13

20

06

-05

-20

20

06

-05

-27

20

06

-06

-03

20

06

-06

-10

20

06

-06

-17

20

06

-06

-24

20

06

-07

-01

20

06

-07

-08

20

06

-07

-15

20

06

-07

-22

20

06

-07

-29

20

06

-08

-05

20

06

-08

-12

20

06

-08

-19

20

06

-08

-26

20

06

-09

-02

20

06

-09

-09

20

06

-09

-16

20

06

-09

-23

20

06

-09

-30

0

20

40

60

80

100

20

05

-10

-01

20

05

-10

-08

20

05

-10

-15

20

05

-10

-22

20

05

-10

-29

20

05

-11

-05

20

05

-11

-12

20

05

-11

-19

20

05

-11

-26

20

05

-12

-03

20

05

-12

-10

20

05

-12

-17

20

05

-12

-24

20

05

-12

-31

20

06

-01

-07

20

06

-01

-14

20

06

-01

-21

20

06

-01

-28

20

06

-02

-04

20

06

-02

-11

20

06

-02

-18

20

06

-02

-25

20

06

-03

-04

20

06

-03

-11

20

06

-03

-18

20

06

-03

-25

20

06

-04

-01

20

06

-04

-08

20

06

-04

-15

20

06

-04

-22

20

06

-04

-29

20

06

-05

-06

20

06

-05

-13

20

06

-05

-20

20

06

-05

-27

20

06

-06

-03

20

06

-06

-10

20

06

-06

-17

20

06

-06

-24

20

06

-07

-01

20

06

-07

-08

20

06

-07

-15

20

06

-07

-22

20

06

-07

-29

20

06

-08

-05

20

06

-08

-12

20

06

-08

-19

20

06

-08

-26

20

06

-09

-02

20

06

-09

-09

20

06

-09

-16

20

06

-09

-23

20

06

-09

-30

𝑄𝑡

incm

s𝑇𝑡

in°𝐶

𝑃𝑡

inm

m

References:Razavi, S., Gupta, H., A General Approach to Multi-Method Sensitivity Analysis of Dynamical Systems Models, submitted to Environmental Modelling & Software.

Gupta, H.V., and Razavi, S., Rethinking the Fundamental Basis of Sensitivity Analysis for Dynamical Earth Systems Models, submitted to Water Resources Research.

“Time-varying” sensitivity indices:

time series that reveals time-dependent

sensitivities of model responses to factors.

“Time-aggregate” sensitivity indices:

summary statistics that aggregate the

dynamical sensitivity information.

VARS-TOOL includes “Generalized Global Sensitivity Matrix” approach to account for models’ thedynamical nature and generate:

0.01

0.1

1

10

100

1000

10000

2005

-10

-01

2005

-10

-08

2005

-10

-15

2005

-10

-22

2005

-10

-29

2005

-11

-05

2005

-11

-12

2005

-11

-19

2005

-11

-26

2005

-12

-03

2005

-12

-10

2005

-12

-17

2005

-12

-24

2005

-12

-31

2006

-01

-07

2006

-01

-14

2006

-01

-21

2006

-01

-28

2006

-02

-04

2006

-02

-11

2006

-02

-18

2006

-02

-25

2006

-03

-04

2006

-03

-11

2006

-03

-18

2006

-03

-25

2006

-04

-01

2006

-04

-08

2006

-04

-15

2006

-04

-22

2006

-04

-29

2006

-05

-06

2006

-05

-13

2006

-05

-20

2006

-05

-27

2006

-06

-03

2006

-06

-10

2006

-06

-17

2006

-06

-24

2006

-07

-01

2006

-07

-08

2006

-07

-15

2006

-07

-22

2006

-07

-29

2006

-08

-05

2006

-08

-12

2006

-08

-19

2006

-08

-26

2006

-09

-02

2006

-09

-09

2006

-09

-16

2006

-09

-23

2006

-09

-30

Time (daily)

0.01

0.1

1

10

100

1000

10000

2005

-10

-01

2005

-10

-15

2005

-10

-29

2005

-11

-12

2005

-11

-26

2005

-12

-10

2005

-12

-24

2006

-01

-07

2006

-01

-21

2006

-02

-04

2006

-02

-18

2006

-03

-04

2006

-03

-18

2006

-04

-01

2006

-04

-15

2006

-04

-29

2006

-05

-13

2006

-05

-27

2006

-06

-10

2006

-06

-24

2006

-07

-08

2006

-07

-22

2006

-08

-05

2006

-08

-19

2006

-09

-02

2006

-09

-16

2006

-09

-30

Time (daily)

TTETFPMK2

IVARS 5

0𝑄

𝑡Revisiting the Fundamental Basis of Global Sensitivity Analysis

for Dynamical Environmental Models

𝑼𝒕 = 𝑼𝒕𝟏, … , 𝑼𝒕

𝑫𝒖Sequences of Inputs:

𝑿𝟎 = 𝑿𝟎𝟏, … , 𝑿𝟎

𝑫𝒙Initial State: 𝒀𝒕(𝚯) = 𝒀𝒕𝟏(𝚯),… , 𝒀𝒕

𝑫𝒚(𝚯)

Sequences of Fluxes:

Time Step:

𝑿𝒕(𝚯) = 𝑿𝒕𝟏(𝚯),… , 𝑿𝒕

𝑫𝒙(𝚯)Sequences of States:

𝚯 = 𝜽𝟏, … , 𝜽𝑵𝚯Parameter Set:

𝑫𝒖 , 𝑵𝚯 , 𝑫𝒙 and 𝑫𝒚 are the

dimensions of the input, parameter,state, and flux vectors, respectively.

𝒕 = 𝟏 to 𝑻

𝑭𝒋 = 𝑭 𝒀|𝒁, 𝚯𝒋 =𝟏

𝑻

𝒕=𝟏

𝑻

𝒇𝒁𝒕 − 𝒇𝒀𝒕 𝚯𝒋

𝟐

TransformationFunction

Performance Metric: 𝒀𝒌 𝚯𝒋 = 𝒀𝟏𝒌 𝚯𝒋 , … , 𝒀𝑻

𝒌 𝚯𝒋

Simulated Time Series

𝒁 = 𝒁1, … , 𝒁𝑻Observed Time Series

flux k for point j in parameter space

Τ𝒅𝑭𝒋 𝒅𝜽𝒊 =−2

𝑻

𝒕=1

𝑻

𝒇𝒁𝒕 − 𝒇𝒀𝒕 𝚯𝒋 ∙ ቤ

𝒅𝒇

𝒅𝒀𝒕 𝚯𝒋∙ ቤ𝒅𝒀𝒕𝒅𝜽𝒊 𝚯𝒋

𝑭𝒋 =𝟏

𝑻

𝒕=𝟏

𝑻

𝒇𝒁𝒕 − 𝒇𝒀𝒕 𝚯𝒋

𝟐

The critical issue is that the result is obscured by mix effects of the residual term (“goodness ofmodel fit” at that time step), the nature of transformation function, and sensitivity coefficient.

Unjustified insensitivity of the time steps and parameter locations at which the model fits data well

(i.e., where 𝐫𝐭 𝚯𝐣 ~ zero). Counter-intuitively, the result is biased to represent time steps and

parameter locations where the model performance is not good (where 𝐫𝐭 𝚯𝐣 is far from zero).

Such approaches depend on availability of system state/output observational data, and therefore,the analysis they provide is necessarily incomplete.

Residual(Error)

SensitivityCoefficient

TransformationEffect

𝛻𝑭𝒋 = Τ𝒅𝑭𝒋 𝒅𝜽1 , … , ൗ𝒅𝑭𝒋 𝒅𝜽𝑵𝜽

Magnitude and Sign of ‘Local Sensitivity’

Gradient Vector Representing

Τ𝒅𝑭𝒋 𝒅𝜽𝒊 =−2

𝑻

𝒕=1

𝑻

𝒓𝒕 𝚯𝒋 ∙ 𝜷𝒕 𝚯

𝒋 ∙ ቤ𝒅𝒀𝒕𝒅𝜽𝒊 𝚯𝒋

(1) ‘Sensitivity’ Analysis versus ‘Identifiability’ Analysis: The Need for a Clear Distinction

o The former is a specific attribute of the ‘forward’ problem to establish which parameters exertstronger (or weaker) controls on the models’ dynamical input-state-output behavior.

o The latter is an attribute of the ‘inverse’ problem to establish which parameters are morereadily identifiable when observational data regarding the system behavior is available.

(2) Methodological Focus on Single-Response Problems: Weakly Informative on Dynamics

o Most sensitivity analysis approaches are primarily designed for applications where thesensitivity of only a single model output to factor perturbations is of interest.

o Sensitivity analysis of models with time series outputs is mainly handled by computing some‘performance metric’ that measures the goodness-of-fit to observed data.

o The performance metric-based approach is typically extended (e.g., by a moving windowapproach) to account for the time-evolving nature dynamical models.

VARS-TOOL is home to the novel “Variogram Analysis of Response Surfaces” or VARS framework, which

can be seen as a “unifying theory” for SA and encompasses the pre-existing, widely used derivative-

based and variance-based methods as special/limiting cases.

𝐶ℎ𝑖

𝛾ℎ𝑖

or

Derivative-Based Approach

ℎ𝑖 → ∞

ℎ𝑖

Variance-based Approach

𝛾 ℎ𝑖 =1

2𝑉 𝑍 𝜽 + ℎ𝑖 − 𝑍 𝜽

𝐶 ℎ𝑖 = 𝐶𝑂𝑉 𝑍 𝜽 + ℎ𝑖 , 𝑍 𝜽

ℎ𝑖 → 0

Variogram

Covariogram

Summary Derivations:

𝑆𝑖𝑇𝑂 =

𝛾 ℎ𝑖 + 𝐸 𝐶𝜽~𝑖(ℎ𝑖)

𝑉(𝑍)

If ℎ𝑖 → ∞֜𝛾 ℎ𝑖 = 𝑉(𝑍)

If ℎ𝑖 → 0֜

𝛾 ℎ𝑖 ∝ 𝑉𝑑𝑍

𝑑𝜃𝑖∝ 𝐸

𝑑𝑍

𝑑𝜃𝑖

2

“Elementary Effects” based Metrics of Morris

Variance of Response Surface

“Total-Order Effects” of Sobol’

References:Razavi, S., and H. V. Gupta, (2015), What do we mean by sensitivity analysis? The need for comprehensive characterization of ‘‘global’’ sensitivity in Earth and Environmental systems models, Water Resources Research.

Razavi, S., and Gupta, H. V., (2016), A new framework for comprehensive, robust, and efficient global sensitivity analysis: 1. Theory, Water Resources Research.

Razavi, S., and Gupta, H. V., (2016), A new framework for comprehensive, robust, and efficient global sensitivity analysis: 2. Application, Water Resources Research.

VARS-TOOL includes other derivative-based (Morris), variance-based (Sobol’), and Monte-Carlo Filtering

methods.

Sampling strategies are necessary fundamental components of any algorithm forsensitivity and uncertainty analysis of computer simulation models.

VARS-TOOL includes a variety of sampling strategies, including Latin Hypercube Sampling(LHS), Symmetric LHS, Progressive LHS (PLHS), Halton and Sobol Sequences, STAR, etc.

PLHS sequentially generates sample points while progressively preserving importantdistributional properties (Latin hypercube, space-filling, etc.), as the sample size grows.

Progressive Sample Size = 4, 8, 12, …

#1 #1 + #2 #1 + #2 + #3

References:Sheikholeslami, R., and Razavi, S., (2017), Progressive Latin Hypercube Sampling: An efficient approach for robust sampling-based analysis of environmental models, Environmental Modelling & Software.

Approximately, 70 percent of GSA applications in the environmental modelling literaturefocused on models with less than 20 parameters, suggesting GSA is paradoxically under-utilized where it should prove most useful.

References:Sheikholeslami, R., Razavi, S., Gupta, H., Becker, W., Haghnegahdar, A., Global Sensitivity Analysis of High-Dimensional Problems: How to Objectively Group Factors and Measure Robustness and Convergence of the Results?, subm. to Environmental Modelling & Software.

g7

g1

g2

g3

g4

g5

g6

VARS-TOOL includes an innovative bootstrap-based “factor grouping”strategy that employs a clustering mechanism to handle high-dimensional problems, involving tens to hundreds of factors. It:

o Estimates Optimal Numberof Groups

o Measures and Maximizes“Robustness”

VARS-TOOL is a comprehensive, multi-approach, multi-algorithm toolbox equipped with a set of tools to enable GSA for any application.