Sensitivity analysis:An introduction
Stefano TarantolaEuropean Commission, Joint Research Centre,
Ispra (I)
Fifth CourseOn Impact Assessment
BrusselsJanuary 20-21, 2015
Simulation (or computer) models are used in manydisciplinesto understand complex phenomena (natural or social) and consequently as tools to support decisionsand policy.
x y
Knowledge base is often flawed by uncertainties (partly irreducible, largely unquantifiable), imperfect understanding, subjective values.
A few examples … x y
Uncertainties in model parameters that govern surface and ground water transport, …
Courtesy of
Models in hydrology
Ex: biological model© 2008 Zi et al; licensee BioMed Central Ltd.
… Uncertainties of kinetic parameters in a chemical process…
Models in bio-chemistry
A B
C
D
E
F
• Parameters of the supply model are mostly uncertain (but kept fixed in the usual practice)
Models in traffic simul.
Uncertainty analysis the analyst can scrutinize uncertainties in model parameters, input data, subjective assumptions and alternative model structures how they propagate through the model effect on predictions identification of the best policy alternative.
Uncertainty analysis ‘forward process’
Sensitivity analysis ‘backward process’. “The study of how the uncertainty in the output of a model can be apportioned to different sources of uncertainty in the model input”. identify which inputs are most influential for the prediction
On those important inputs one should focus to see whether their uncertainty can be reduced
improve prediction accuracy.
Various types of uncertain inputs:
Data
Parameters
Assumptions
Scenarios
Alternative model specifications
x y
Sensitivity analysis: what for
1. Prioritising acquisition of information
If model prediction is too uncertain SA identify important factors reduce uncertainty of important factors increase robustness of results
Sensitivity analysis: what for
2. Model understanding
Is the model doing what we expect from it?Discover inputs interactions.
Sensitivity analysis: what for
3. Model simplification
Identify inputs with no effect on the prediction
Sensitivity analysis: what for
4. Model simplification
Identify critical regions in the space of inputsExample …
x y
y
P(y)
Sensitivity analysis: what for
5. Are policy options distinguishable given the uncertainties?
Example …
Traffic modelling:
Average Travel timeA BPolicy A: traffic lightsPolicy B: roundabouts
Deterministic assessment
Traffic modelling:
A B
Average Travel time
Probabilistic assessment
A B
Average Travel time
A better than B (given the otheruncertainties)
The other uncertainties obfuscatethe effect of the policies
- Identify the factors responsible for the overlap
-More knowledge on those factors could allow the decisionto be taken
Other uncertainties
A vs B A B
Average Travel Time
A B
Average Travel time
A B
Average Travel Time
Local, One at a Time
and Global Sensitivity Analysis
Local SA
xr
xr = nominal value
0 1x
2x
),( 21 xx = space of input
- evaluation of partial derivatives - works in the neighborhood of nominal point- use of Taylor-like formulas
0xxiXY
rr=
∂∂
x y
One at a time SA
xr = nominal value
0 1x
2x
),( 21 xx = space of input
xr
- SA performed by changing one input variable by one while keeping others at their baseline nominal values
- the other inputs are kept fixed
Global SA
xr = nominal value0 1x
2x
),( 21 xx = space of input
- full exploration of uncertainty- Monte Carlo methods to generate samples
Regression / correlationScreening techniques
Variance decompositionMoment- independent
Statistical testsGraphical tools
At large dimension of input space OAT exploresa negligible volume with respect to GSA
Limitations of OAT
Area circle / area square = 0.78Volume sphere / volume cube = 0.5
In 10 dimensions:Vol hyper-sphere / vol. hyper-cube = 0.0025
… the modeller is afraid his model willcrash in that region as global SA explores the boundary of the input space
OAT is still widely used because …
Model OutputInput
x1
x2
x3
…
x4
xk
y
xr )(xfy r= y
Monte Carlo approach to uncertainty analysis
Space of uncertainty
...X1
Positive Impactof policy
Policy 1 Policy 2NO policy
10
20
30
40
50
60
Monte Carlo approach to uncertainty analysis
X2 X3
XjXk
Specification of the model inputs
)( ii xp
x1
x2
x3
x4
Characterise the uncertainty of each input.
Assign a pdf using all available information
eg experiments, estimations, physical bounds
considerations, scientific knowledge and
expert opinion.
A very delicate step: it may require significant
resources.
Extended peer-review should be considered
to ensure quality in the treatment of
uncertainty
A major issue in global sensitivity analysis is the number of model runs required to conduct the analysis.
Our preferred methods for SA: variance-based
concise and easy to communicate
Variance-based method’s best formalization is based on the work of Ilya M. Sobol’(1990) who extended the work of R. I. Cukier (1973).
First-order sensitivity indices
x y
)()]|([
yVarxyEVarS i
i =
)]|([)]|([)( ii xyVarExyEVaryVar +=
Easy to prove using V(•)=E(•)2-E
2(•)
-60
-40
-20
0
20
40
60
-4 -3 -2 -1 0 1 2 3 4
-60
-40
-20
0
20
40
60
-4 -3 -2 -1 0 1 2 3 4
The ordinate axis is always Y
The abscissa are the various factors Xi in turn.
The points are always the same!
-60
-40
-20
0
20
40
60
-4 -3 -2 -1 0 1 2 3 4
-60
-40
-20
0
20
40
60
-4 -3 -2 -1 0 1 2 3 4
Which variable is the most important?
-60
-40
-20
0
20
40
60
-4 -3 -2 -1 0 1 2 3 4
-60
-40
-20
0
20
40
60
-4 -3 -2 -1 0 1 2 3 4
These are ~1,000 points
Divide them in 20 bins of ~ 50 points
-60
-40
-20
0
20
40
60
-4 -3 -2 -1 0 1 2 3 4
-60
-40
-20
0
20
40
60
-4 -3 -2 -1 0 1 2 3 4
Compute the bin’s average (pink dots)
( )iXYEi~XEach pink point is ~
-60
-40
-20
0
20
40
60
-4 -3 -2 -1 0 1 2 3 4
( )( )iX XYEVii ~X
Take the variance of the pinkies
-60
-40
-20
0
20
40
60
-4 -3 -2 -1 0 1 2 3 4
( )( )iX XYEVii ~X
First order effect =
= the expected reduction in variance that would be achieved if factor Xi could be fixed.
Why?
( )( )( )( ) )(
~
~
YVXYVE
XYEV
iX
iX
ii
ii
=+
+
X
X
Because:
( )( )( )( ) )(
~
~
YVXYVE
XYEV
iX
iX
ii
ii
=+
+
X
X
Because:
The variance that would be left (on average) if Xi could be fixed.
Variance decomposition (ANOVA)
( )
kiji
iji
i VVV
YV
...123,
...+++
=
∑∑>
)(YVVi
i ≈∑
For additive systems one can decompose the total variance as a sum of first order effects
)],|([)],|([)( jiji xxyVarExxyEVaryVar +=
Joint effects
)()],|([
yVarxxyEVar
S jiij
tjoin =
45
The expected amount of variance that would remain unexplained (residual variance)
if xi, and only xi, were left free to vary over its uncertainty range.
Use: for model simplification, to identify unessential inputs in the model, which are not important neither singularly nor in combination with others.
An input with a small value of its total effect sensitivity index can be frozento any value within its range.
)(/)]|([ YVarxYVarES iTi −=
Total effects
)]|([)]|([)( ii xYVarExYEVarYVar −− +=
We cannot use Si to fix a factor; Si =0 is a necessary but not sufficient condition for Xi to be non‐influential.
Xi could be influential at the second order.
Example …
Si ?
-60
-40
-20
0
20
40
60
-4 -3 -2 -1 0 1 2 3 4
Other (non variance-based) techniques
Screening techniques (Morris, 1991)
Graphical methods
Derivative-based techniques
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
∫ ⎟⎟⎠
⎞⎜⎜⎝
⎛∂∂
= dxxy
ii
2
ν
Global sensitivity analysis.
The Primer
A textbook of methods to evidencehow model-based inference depends
upon modelspecifications and assumptions,
John Wiley, 2008
Saltelli, A., Ratto M., Andres, T., Campolongo, F., Cariboni J., Gatelli
D., Ratto, M., Saisana, M., Tarantola, S.
List of References
Sobol’ and Kucherenko (2009) Derivative based global sensitivity measures and their link with global sensitivity indices, Mathematics and Computers in Simulation 79, 3009–3017
Bolado, Castaings and Tarantola (2009) Contribution to the sample mean plot for graphical and numerical sensitivity analysis, Reliability Engineering and System Safety 94, 1041–1049
Tarantola, S., V. Kopustinskas, R. Bolado-Lavin, A. Kaliatka, E. Uspuras, M. Vaisnoras (2012) Sensitivity analysis using contribution to sample variance plot: Application to a water hammer model, Reliability Engineering and System Safety 99, 62–73
Morris, M.D. (1991) Factorial Sampling Plans for Preliminary Computational Experiments, Technometrics 33: 161–174
Saltelli A., P. Annoni I. Azzini, F. Campolongo, M. Ratto and S. Tarantola (2010) Variance based sensitivity analysis of model output. Design and estimator for the total sensitivity index, Computer Physics Communications 181, 259–270
Kucherenko S., S. Tarantola, P. Annoni (2012) Estimation of global sensitivity indices for models with dependent variables, Computer Physics Communications 183, 937–946
Thank you for your attention!
Questions?
