Post on 18-Dec-2021
transcript
A Toolbox for Comprehensive, Efficient, and Robust Sensitivity and Uncertainty Analysis
Saman Razavi, First Annual General MeetingSaman Razavi, July 18-19, 2018
Uncertainty in Output Space
Model/Process Response Surface
(Uncertain)
Co
rrelatio
n
Effects
Inte
raction
s Effe
cts
Uncertainty in Input Space
ฮธ1ฮธ2
๐
uses information in the mismatch betweenmodel predictions and data to identify โgoodโvalues for the model โparametersโ, and tocharacterize their posterior uncertainty.
(2) Inverse-Problem Approach
propagates assumptions on uncertainties ininputs and other system properties throughthe model to obtain some understanding onuncertainties in model predictions.
(1) Forward-Problem Approach
Joint Probability Distribution
Uncertainty in Input Space
Uncertainty in Output Space
Co
rrelatio
n
Effects
Joint Probability Distribution
Inte
raction
s Effe
cts
attributes the uncertainty in a modelprediction to the uncertainties in inputs, andseeks to answer the critical question:
when does uncertainty matter?
illuminates the controls on model behavior,thereby characterizing the dominant controlson predictive uncertainty.
guides research towards reducing theuncertainties that matter, as it points to themost important aspects of the problem.
ฮธ1ฮธ2
๐
(3) Sensitivity Analysis Approach
Model/Process Response Surface
(Uncertain)
A comprehensive, multi-approach, multi-algorithm software toolbox for sensitivity analysis of anycomputer simulation model, including Earth and environmental systems models.
Razavi, S., Sheikholeslami, R., Gupta, H., Haghnegahdar, A., VARS-TOOL: A Toolbox for Comprehensive, Efficient,and Robust Sensitivity and Uncertainty Analysis, submitted to Environmental Modelling & Software.
Important Features:
โข Multi-Method Approach to Sensitivity Analysis
โข Sensitivity Analysis of Dynamical Systems Models (NEW)
โข Various Sampling Strategies, e.g., Progressive Latin Hypercube Sampling (NEW)
โข Handling High-Dimensional Problems: A Grouping Solution to Curse of Dimensionality (NEW)
โข Characterizing Confidence, Convergence, and Robustness
โข Reporting and Visualization: Monitoring Stability and Convergence (NEW)
โข Handling Model Crashes via Model Emulation (NEW)
โข Interface with Any Computer Model and Linkage to OSTRICH toolkit (NEW)
โข A Comprehensive Test Bed for Training and Research (NEW)
Most approaches to SA of Earth systems models ignore or, at best, do not adequately account for the
dynamical nature of such models. These approaches handle problems with only a single response.
0
5
10
15
20
20
05
-10
-01
20
05
-10
-08
20
05
-10
-15
20
05
-10
-22
20
05
-10
-29
20
05
-11
-05
20
05
-11
-12
20
05
-11
-19
20
05
-11
-26
20
05
-12
-03
20
05
-12
-10
20
05
-12
-17
20
05
-12
-24
20
05
-12
-31
20
06
-01
-07
20
06
-01
-14
20
06
-01
-21
20
06
-01
-28
20
06
-02
-04
20
06
-02
-11
20
06
-02
-18
20
06
-02
-25
20
06
-03
-04
20
06
-03
-11
20
06
-03
-18
20
06
-03
-25
20
06
-04
-01
20
06
-04
-08
20
06
-04
-15
20
06
-04
-22
20
06
-04
-29
20
06
-05
-06
20
06
-05
-13
20
06
-05
-20
20
06
-05
-27
20
06
-06
-03
20
06
-06
-10
20
06
-06
-17
20
06
-06
-24
20
06
-07
-01
20
06
-07
-08
20
06
-07
-15
20
06
-07
-22
20
06
-07
-29
20
06
-08
-05
20
06
-08
-12
20
06
-08
-19
20
06
-08
-26
20
06
-09
-02
20
06
-09
-09
20
06
-09
-16
20
06
-09
-23
20
06
-09
-30
-30-20-10
0102030
20
05
-10
-01
20
05
-10
-08
20
05
-10
-15
20
05
-10
-22
20
05
-10
-29
20
05
-11
-05
20
05
-11
-12
20
05
-11
-19
20
05
-11
-26
20
05
-12
-03
20
05
-12
-10
20
05
-12
-17
20
05
-12
-24
20
05
-12
-31
20
06
-01
-07
20
06
-01
-14
20
06
-01
-21
20
06
-01
-28
20
06
-02
-04
20
06
-02
-11
20
06
-02
-18
20
06
-02
-25
20
06
-03
-04
20
06
-03
-11
20
06
-03
-18
20
06
-03
-25
20
06
-04
-01
20
06
-04
-08
20
06
-04
-15
20
06
-04
-22
20
06
-04
-29
20
06
-05
-06
20
06
-05
-13
20
06
-05
-20
20
06
-05
-27
20
06
-06
-03
20
06
-06
-10
20
06
-06
-17
20
06
-06
-24
20
06
-07
-01
20
06
-07
-08
20
06
-07
-15
20
06
-07
-22
20
06
-07
-29
20
06
-08
-05
20
06
-08
-12
20
06
-08
-19
20
06
-08
-26
20
06
-09
-02
20
06
-09
-09
20
06
-09
-16
20
06
-09
-23
20
06
-09
-30
0
20
40
60
80
100
20
05
-10
-01
20
05
-10
-08
20
05
-10
-15
20
05
-10
-22
20
05
-10
-29
20
05
-11
-05
20
05
-11
-12
20
05
-11
-19
20
05
-11
-26
20
05
-12
-03
20
05
-12
-10
20
05
-12
-17
20
05
-12
-24
20
05
-12
-31
20
06
-01
-07
20
06
-01
-14
20
06
-01
-21
20
06
-01
-28
20
06
-02
-04
20
06
-02
-11
20
06
-02
-18
20
06
-02
-25
20
06
-03
-04
20
06
-03
-11
20
06
-03
-18
20
06
-03
-25
20
06
-04
-01
20
06
-04
-08
20
06
-04
-15
20
06
-04
-22
20
06
-04
-29
20
06
-05
-06
20
06
-05
-13
20
06
-05
-20
20
06
-05
-27
20
06
-06
-03
20
06
-06
-10
20
06
-06
-17
20
06
-06
-24
20
06
-07
-01
20
06
-07
-08
20
06
-07
-15
20
06
-07
-22
20
06
-07
-29
20
06
-08
-05
20
06
-08
-12
20
06
-08
-19
20
06
-08
-26
20
06
-09
-02
20
06
-09
-09
20
06
-09
-16
20
06
-09
-23
20
06
-09
-30
๐๐ก
incm
s๐๐ก
inยฐ๐ถ
๐๐ก
inm
m
References:Razavi, S., Gupta, H., A General Approach to Multi-Method Sensitivity Analysis of Dynamical Systems Models, submitted to Environmental Modelling & Software.
Gupta, H.V., and Razavi, S., Rethinking the Fundamental Basis of Sensitivity Analysis for Dynamical Earth Systems Models, submitted to Water Resources Research.
โTime-varyingโ sensitivity indices:
time series that reveals time-dependent
sensitivities of model responses to factors.
โTime-aggregateโ sensitivity indices:
summary statistics that aggregate the
dynamical sensitivity information.
VARS-TOOL includes โGeneralized Global Sensitivity Matrixโ approach to account for modelsโ thedynamical nature and generate:
0.01
0.1
1
10
100
1000
10000
2005
-10
-01
2005
-10
-08
2005
-10
-15
2005
-10
-22
2005
-10
-29
2005
-11
-05
2005
-11
-12
2005
-11
-19
2005
-11
-26
2005
-12
-03
2005
-12
-10
2005
-12
-17
2005
-12
-24
2005
-12
-31
2006
-01
-07
2006
-01
-14
2006
-01
-21
2006
-01
-28
2006
-02
-04
2006
-02
-11
2006
-02
-18
2006
-02
-25
2006
-03
-04
2006
-03
-11
2006
-03
-18
2006
-03
-25
2006
-04
-01
2006
-04
-08
2006
-04
-15
2006
-04
-22
2006
-04
-29
2006
-05
-06
2006
-05
-13
2006
-05
-20
2006
-05
-27
2006
-06
-03
2006
-06
-10
2006
-06
-17
2006
-06
-24
2006
-07
-01
2006
-07
-08
2006
-07
-15
2006
-07
-22
2006
-07
-29
2006
-08
-05
2006
-08
-12
2006
-08
-19
2006
-08
-26
2006
-09
-02
2006
-09
-09
2006
-09
-16
2006
-09
-23
2006
-09
-30
Time (daily)
0.01
0.1
1
10
100
1000
10000
2005
-10
-01
2005
-10
-15
2005
-10
-29
2005
-11
-12
2005
-11
-26
2005
-12
-10
2005
-12
-24
2006
-01
-07
2006
-01
-21
2006
-02
-04
2006
-02
-18
2006
-03
-04
2006
-03
-18
2006
-04
-01
2006
-04
-15
2006
-04
-29
2006
-05
-13
2006
-05
-27
2006
-06
-10
2006
-06
-24
2006
-07
-08
2006
-07
-22
2006
-08
-05
2006
-08
-19
2006
-09
-02
2006
-09
-16
2006
-09
-30
Time (daily)
TTETFPMK2
IVARS 5
0๐
๐กRevisiting the Fundamental Basis of Global Sensitivity Analysis
for Dynamical Environmental Models
๐ผ๐ = ๐ผ๐๐, โฆ , ๐ผ๐
๐ซ๐Sequences of Inputs:
๐ฟ๐ = ๐ฟ๐๐, โฆ , ๐ฟ๐
๐ซ๐Initial State: ๐๐(๐ฏ) = ๐๐๐(๐ฏ),โฆ , ๐๐
๐ซ๐(๐ฏ)
Sequences of Fluxes:
Time Step:
๐ฟ๐(๐ฏ) = ๐ฟ๐๐(๐ฏ),โฆ , ๐ฟ๐
๐ซ๐(๐ฏ)Sequences of States:
๐ฏ = ๐ฝ๐, โฆ , ๐ฝ๐ต๐ฏParameter Set:
๐ซ๐ , ๐ต๐ฏ , ๐ซ๐ and ๐ซ๐ are the
dimensions of the input, parameter,state, and flux vectors, respectively.
๐ = ๐ to ๐ป
๐ญ๐ = ๐ญ ๐|๐, ๐ฏ๐ =๐
๐ป
๐=๐
๐ป
๐๐๐ โ ๐๐๐ ๐ฏ๐
๐
TransformationFunction
Performance Metric: ๐๐ ๐ฏ๐ = ๐๐๐ ๐ฏ๐ , โฆ , ๐๐ป
๐ ๐ฏ๐
Simulated Time Series
๐ = ๐1, โฆ , ๐๐ปObserved Time Series
flux k for point j in parameter space
ฮค๐ ๐ญ๐ ๐ ๐ฝ๐ =โ2
๐ป
๐=1
๐ป
๐๐๐ โ ๐๐๐ ๐ฏ๐ โ แค
๐ ๐
๐ ๐๐ ๐ฏ๐โ แค๐ ๐๐๐ ๐ฝ๐ ๐ฏ๐
๐ญ๐ =๐
๐ป
๐=๐
๐ป
๐๐๐ โ ๐๐๐ ๐ฏ๐
๐
The critical issue is that the result is obscured by mix effects of the residual term (โgoodness ofmodel fitโ at that time step), the nature of transformation function, and sensitivity coefficient.
Unjustified insensitivity of the time steps and parameter locations at which the model fits data well
(i.e., where ๐ซ๐ญ ๐ฏ๐ฃ ~ zero). Counter-intuitively, the result is biased to represent time steps and
parameter locations where the model performance is not good (where ๐ซ๐ญ ๐ฏ๐ฃ is far from zero).
Such approaches depend on availability of system state/output observational data, and therefore,the analysis they provide is necessarily incomplete.
Residual(Error)
SensitivityCoefficient
TransformationEffect
๐ป๐ญ๐ = ฮค๐ ๐ญ๐ ๐ ๐ฝ1 , โฆ , เต๐ ๐ญ๐ ๐ ๐ฝ๐ต๐ฝ
Magnitude and Sign of โLocal Sensitivityโ
Gradient Vector Representing
ฮค๐ ๐ญ๐ ๐ ๐ฝ๐ =โ2
๐ป
๐=1
๐ป
๐๐ ๐ฏ๐ โ ๐ท๐ ๐ฏ
๐ โ แค๐ ๐๐๐ ๐ฝ๐ ๐ฏ๐
(1) โSensitivityโ Analysis versus โIdentifiabilityโ Analysis: The Need for a Clear Distinction
o The former is a specific attribute of the โforwardโ problem to establish which parameters exertstronger (or weaker) controls on the modelsโ dynamical input-state-output behavior.
o The latter is an attribute of the โinverseโ problem to establish which parameters are morereadily identifiable when observational data regarding the system behavior is available.
(2) Methodological Focus on Single-Response Problems: Weakly Informative on Dynamics
o Most sensitivity analysis approaches are primarily designed for applications where thesensitivity of only a single model output to factor perturbations is of interest.
o Sensitivity analysis of models with time series outputs is mainly handled by computing someโperformance metricโ that measures the goodness-of-fit to observed data.
o The performance metric-based approach is typically extended (e.g., by a moving windowapproach) to account for the time-evolving nature dynamical models.
VARS-TOOL is home to the novel โVariogram Analysis of Response Surfacesโ or VARS framework, which
can be seen as a โunifying theoryโ for SA and encompasses the pre-existing, widely used derivative-
based and variance-based methods as special/limiting cases.
๐ถโ๐
๐พโ๐
or
Derivative-Based Approach
โ๐ โ โ
โ๐
Variance-based Approach
๐พ โ๐ =1
2๐ ๐ ๐ฝ + โ๐ โ ๐ ๐ฝ
๐ถ โ๐ = ๐ถ๐๐ ๐ ๐ฝ + โ๐ , ๐ ๐ฝ
โ๐ โ 0
Variogram
Covariogram
Summary Derivations:
๐๐๐๐ =
๐พ โ๐ + ๐ธ ๐ถ๐ฝ~๐(โ๐)
๐(๐)
If โ๐ โ โึ๐พ โ๐ = ๐(๐)
If โ๐ โ 0ึ
๐พ โ๐ โ ๐๐๐
๐๐๐โ ๐ธ
๐๐
๐๐๐
2
โElementary Effectsโ based Metrics of Morris
Variance of Response Surface
โTotal-Order Effectsโ of Sobolโ
References:Razavi, S., and H. V. Gupta, (2015), What do we mean by sensitivity analysis? The need for comprehensive characterization of โโglobalโโ sensitivity in Earth and Environmental systems models, Water Resources Research.
Razavi, S., and Gupta, H. V., (2016), A new framework for comprehensive, robust, and efficient global sensitivity analysis: 1. Theory, Water Resources Research.
Razavi, S., and Gupta, H. V., (2016), A new framework for comprehensive, robust, and efficient global sensitivity analysis: 2. Application, Water Resources Research.
VARS-TOOL includes other derivative-based (Morris), variance-based (Sobolโ), and Monte-Carlo Filtering
methods.
Sampling strategies are necessary fundamental components of any algorithm forsensitivity and uncertainty analysis of computer simulation models.
VARS-TOOL includes a variety of sampling strategies, including Latin Hypercube Sampling(LHS), Symmetric LHS, Progressive LHS (PLHS), Halton and Sobol Sequences, STAR, etc.
PLHS sequentially generates sample points while progressively preserving importantdistributional properties (Latin hypercube, space-filling, etc.), as the sample size grows.
Progressive Sample Size = 4, 8, 12, โฆ
#1 #1 + #2 #1 + #2 + #3
References:Sheikholeslami, R., and Razavi, S., (2017), Progressive Latin Hypercube Sampling: An efficient approach for robust sampling-based analysis of environmental models, Environmental Modelling & Software.
Approximately, 70 percent of GSA applications in the environmental modelling literaturefocused on models with less than 20 parameters, suggesting GSA is paradoxically under-utilized where it should prove most useful.
References:Sheikholeslami, R., Razavi, S., Gupta, H., Becker, W., Haghnegahdar, A., Global Sensitivity Analysis of High-Dimensional Problems: How to Objectively Group Factors and Measure Robustness and Convergence of the Results?, subm. to Environmental Modelling & Software.
g7
g1
g2
g3
g4
g5
g6
VARS-TOOL includes an innovative bootstrap-based โfactor groupingโstrategy that employs a clustering mechanism to handle high-dimensional problems, involving tens to hundreds of factors. It:
o Estimates Optimal Numberof Groups
o Measures and MaximizesโRobustnessโ
VARS-TOOL is a comprehensive, multi-approach, multi-algorithm toolbox equipped with a set of tools to enable GSA for any application.