1

Citation: Gupta, H. and Razavi, S. (2017), Chapter 20 - Challenges and Future Outlook of Sensitivity Analysis, In Sensitivity Analysis in Earth Observation Modelling, edited by George P. Petropoulos and Prashant K. Srivastava, Elsevier, Pages 397-415, ISBN 9780128030110, http://dx.doi.org/10.1016/B978-0-12-803011-0.00020-3.

ChallengesandFutureOutlookofSensitivityAnalysis

HoshinGupta1andSamanRazavi21DepartmentofHydrologyandAtmosphericSciences,TheUniversityofArizona,Tucson,Arizona,USA

2 Global Institute for Water Security & School of Environment and Sustainability, University ofSaskatchewan,Saskatoon,Saskatchewan,Canada

2 Department of Civil and Geological Engineering, University of Saskatchewan, Saskatoon,Saskatchewan,Canada

Abstract

AsEarthandEnvironmentalSystemmodelshaverapidlybecomemorecomplexandcomputationally intensive, growing in parameter dimensionality as they reflect ourgrowingunderstandingabout thenatureand functioningof theworld, theneed forrobust, informativeandcomputationallyefficientsensitivityanalysistechniquesandtools has become evermore pressing. To date, a variety of different approaches tosensitivity analysis have been proposed in the literature, each of which has itsstrengths and weaknesses. Further, their application is typically inhibited bycomputationalexpenseandtheirusefulnesslimitedbytheinabilitytoextractusefuldiagnosticinformationfromthemodelresponse.Thischapterreviewsandcontrastssome of themajor strategies for sensitivity analysis that have been proposed, anddiscussesseveralchallengesthatneedcriticalattention.Perhapsthemostimportantof these is to establish a clear definition of how to “compactly” characterize thesensitivityofamodelresponsetoperturbationsinitscausalfactorsinsuchamannerthatthe“diagnostic”informationprovidedbytheanalysisismaximized.

1. Introduction

Sensitivity Analysis (SA) is an important tool in the development and application ofdynamicalmodelsofEarthandEnvironmentalSystem(EES)observations,andisbecomingeven more important as such models become progressively more realistic. While theobjectives of a SA can be varied (seeRazaviandGupta2015), by far themost commonpurposeistoidentify(andprioritizeandscreen)whichoftheparametersinthemodelarethemost important/influential, in that perturbations to these parametersmost stronglyaffect the magnitude, variability and dynamics of model response. This is commonlyreferred to as a “parameter sensitivity analysis” and (although not quite grammaticallycorrect) a parameter that causes a “stronger” change in the model response whenperturbedistypicallysaidtobea“moresensitiveparameter”,whileaparameterthatcausesa “weaker” change in themodel responsewhen perturbed is typically said to be a “lesssensitiveparameter”(notethatsensitivityisthereforearelativeconcept).

Themainimportanceofaparametersensitivityanalysisisthatitenablesustoassesshowmuch caremust be taken in the specification of the values of variousmodel parameterswhen using the model to help make decisions in a specific situation. If parameter is

2

relatively “moresensitive” thanparameter (meaning that themodel‐baseddecision ismoresensitivetounitchangesinparameter thantounitchangesinparameter ),thengreatercarewillneedtobetakeninspecifyingthevalueofparameter .

Asacorollarytothis,ifsubstantialperturbationstosomeparameter (suchasvaryingitacross its entire feasible rangeof values) result in little orno effect on themodel‐baseddecision ,thensuchaparameteristypicallyreferredtoas“insensitive”.Oncedetected,itiscommonpracticetofixsuchaparameteratsomereasonablenominalvalueandnotgiveit further attention; this knowledge can sometimes also be used to guide structuralsimplificationsofthemodel.

TheabilitytounderstandtheimportanceandroleofthevariousparametersinadynamicalmodeloftheobservationsrelatedtoanEarthandEnvironmentalSystem(EES)cannotbeunderstated.Suchmodelscancoverawiderangeofspatio‐temporalscales(fromground‐to satellite‐based), involvemany different sources of data, and dealwithmany differenttypesofprocesses.Parametersensitivityanalysisthereforeprovidesanessentialtoolthatcanhelptheearthobservationmodelernavigatethemyriadquestions/problemsthatcanarisewhile seeking answers thatwill inform a bettermodel‐based decision. This abilitybecomes even more relevant when dealing with climate and human‐induced non‐stationaritiesinthestructureandfunctioningofEarthandEnvironmentalSystems(Razavietal.,2015).

2. BriefReviewofSomeCommonlyUsedSensitivityAnalysisMethods

Consider that the model of interest has a set of parameters , … , , where representsthedimensionoftheparameterspace(thenumberofparametersinthemodel).The value of each parameter can vary across the range , , , , such that defines the feasiblespaceofvalues that themodelparameterscan jointly takeon; in thecase that the parameters can vary independently over their full range, defines ahypercube in the parameter space. Hence, , … , represents a single point in whereeachoftheparameters takesonaspecificvalue.

By running the model using the parameter values indicated by we generate a modeloutput . Note that can be as simple as a single value, or be considerably morecomplex, such as a time series of values computed for some specific location in space, aspatiallydistributedfieldofvaluescomputedforsomespecifictime,or(mostgenerally)aspatio‐temporalvaryingfieldofvalues.

Thebehaviorofthismodeloutputisthentypicallysummarizedintermsofasmallsetofscalar quantities that are assumed to represent the “response” of the model. Forexample, is often taken to be a model performance metric that measures the (scalar)distancebetweenthemodeloutput andsomedatarepresentingobservationsmadeon the real system. Alternatively, can represent a model‐based decision (or set ofdecisions) . For purposes of further discussionwewill assume that is a single scalarvalue;vectorvalueswillrequireasimilarbutmorecomplexanalysis.

Tosummarize,themodelparameters canvaryoverafeasiblespace ,andateachpointin this feasible spacewe can compute a value (or set of values) that represents somemodel‐basedquantity(orquantities)ofinterest.Ifperturbationofoneoftheparameters

3

causes a relatively large change in , then that parameter will be considered to be(relatively)sensitive,andifperturbationofanotherparameter causesarelativelysmallchangein ,thenthatparameterwillbeconsideredtobe(relatively)insensitive.

2.1LocalAssessmentofParameterSensitivity

Atanyspecificpoint ,theideaofparametersensitivityismathematicallywelldefinedintermsofthevectoroflocalpartialderivatives ⁄ | ⁄ , … , ⁄ whichindicateshowsensitivethemodelresponse istoinfinitesimallysmallindividualchangesin each of the parameters , … , at the point (Figure1). In practice the value of

⁄ | is often computed via a finite‐difference approximation using small parameterperturbationsaroundthenominalvalue.

Now,intheunlikelycaseofaperfectlylinearrelationshipbetween and (thisdoesnotgenerally happen in EES models) the magnitude of each term in ⁄ | remainsconstant throughout the feasible space and hence the vector ⁄ (evaluated at anypointin )canbeconsideredtobeacompletecharacterizationofthesensitivityof to .However,inEESmodelstherelationshipistypicallynon‐linear(andcanbequitecomplex).In general, therefore, ⁄ changes from point to point and hence a proper (i.e.,informative) characterization of sensitivity requires a more “global” investigation, asdiscussedinthenextsection.

Figure1:A2‐parametersurface ‘R1’ showinghowtheresponse varieswith thevaluesof theparameters and . Thedirectional gradients (partialderivatives)

⁄ ⁄ , ⁄ areshownforaselectednominalpoint .

4

The exception to needing a global characterization would be when we have alreadyprecisely determined the “best” values ∗ for , and are only interested in thecharacterization of relative sensitivity of to at that specific location. But, in the vastmajority of cases these “best” values are not known before conducting the parametercalibrationprocess,andwewouldliketo liketoknowhowsensitive(inarelativesense)theoverallresponseistoeachoftheparameterssothatwecandecidewhichparameterstooptimizeandwhichtokeepfixed.Alternatively, theremaysimplyexistmorethanonepointofinterestinthefeasiblespace .

2.2GlobalAssessmentofParameterSensitivity

So, for all but the trivial case of a perfectly linear relationship between and , ourassessmentofparametersensitivitymusttakeintoconsiderationthefactthatthenatureofthe relationship varies throughout the feasible space . Over the past severaldecades, several attempts have been made to represent the “global” nature of thesensitivityofmodelresponse.Someoftheearlyoneswerebasedinconceptssuchasthestatisticaldesignofexperiments(factorialdesign)andregressionandcorrelationanalysis(seediscussioninRazaviandGupta2015);theseapproachesweretypicallybasedinstrongassumptions regarding the mathematical form (e.g., linear or polynomial) of the relationship. ForEESmodels, such assumptions about the formof response surfaces aregenerallyunjustifiableandwewillthereforenotdiscussthosemethodshere.

Instead, we consider the following five different (but related) strategies that have beenproposedintheliterature,andthataregenerallyapplicabletoEESmodels.Eachofthesestrategies is based in a somewhat different philosophical approach to the definition ofsensitivity.

a) The‘One‐DimensionalCrossSection’Strategyb) The‘DistributionofDerivatives’Strategyc) The‘AnalysisofVariance’Strategyd) The‘AnalysisofCumulativeDistributions’Strategye) The‘VariogramAnalysisofResponseSurface’Strategy

Webrieflydiscusseachofthesestrategiesbelow,withaviewtocomparingandcontrastingtheirsimilaritiesanddifferences.

2.2.1 The‘One‐DimensionalCrossSection’Strategy

Thesimplestandcomputationallycheapestwaytoconducta ‘quickanddirty’assessmentofrelativeparametersensitivityacross the feasiblespace is toexamine thenatureof thedifferentone‐dimensional cross‐sectionsof the response aseachparameter is variedbetween its minimum and maximum values , , , , while other parameters arefixed at nominal values (Figure 2). If the cross‐section for parameter tends to havesteeperslopesacrosstheparameterrangethanthecross‐sectionforparameter ,and/orthecross‐section forparameter coversa largerrangeof theresponse thanthecross‐sectionforparameter ,thenwecanconcludethattheparametersensitivityishigherfortheformerthanforthelatter.Suchplotsare,therefore,intuitivelyeasytounderstand.

5

Figure2: Themain plot illustrates one‐dimensional cross‐sections in each of theparameter directions and for the 2‐parameter response surface ‘R1’. Thesectionshavebeenconstructedtopass throughthenominalpoint (indicatedbytheverticaldashedline).Thetwosubplotsshowthetwo versusZcrosssectionalplotssoobtained.

However, although relatively simple and computationally inexpensive to perform, thisstrategyhassomeseriousweaknesses.First,theanalysisisreallyonlyvalidforaparticularpoint in theparameterspace,as thispointmustbespecifiedbefore thecross‐sectionscan be conducted; in this sense it is really just a slight extension of the local parametersensitivityapproach.Second,theassessmentreliesuponasubjectivevisualexaminationofthenatureofthecross‐sections.Toestablishanobjectiveanalysisonewouldhavetodefineamathematicalmetrictocharacterizethenatureofeachcrosssectionintermsofasinglenumerical “sensitivity” value. In doing so, one must necessarily accept the fact that,whatever the strategy used for such characterization, many different cross‐sectionalshapeswill inevitablyhavethesamenumerical “sensitivity”value(e.g., seecross‐sectionsC1andC2inFigure3).

6

Figure3:A cartoon illustrationofa2‐parameter responsesurface ‘R2’, forwhichthe one‐dimensional versus Z cross‐sections C1 and C2, corresponding toparameters and respectively, have been constructed. These cross‐sectionshavedifferentshapesbut identicalaverage slope(andalsoaverageabsoluteslopeand average squared slope) over the parameter ranges. If a metric based onaveragingsome functionof theslope isused tocharacterizeparametersensitivityassociatedwitheachcross‐section,thentheresponsewillbedeemedtobeequallysensitivetobothparameters.

Third, to be truly globally representative, the analysiswould need to be conducted at arepresentativesampleoflocations intheparameterspace,inwhichcasetheassessmentis no longer simple, nor computationally inexpensive. Such a “sampled cross section”strategyisactuallyembeddedintheSTAR‐VARSimplementation(RazaviandGupta2016b)oftheVARSframeworkintroducedinsection2.2.5below,butweleavethedetailsforthereadertoexplore.

7

2.2.2 The‘DistributionofDerivatives’Strategy

Giventhatlocalparametersensitivityatapoint canbemathematicallycharacterizedbythe vector of partial derivatives ⁄ | , a logical extension is to define “global”parametersensitivityintermsofthedistributionofdifferentvaluesof ⁄ obtainedasis varied throughout the feasible space (Figure 4). Since each directional partial

derivative ⁄ is now characterized by a probability distribution ⁄ , we cancompactly characterize global parameter sensitivity in terms of the statistics of thisdistribution.

Figure4:Themainplot shows the2‐parameter surface ‘R1’ onwhichdirectionalgradients(partialderivatives) ⁄ areindicatedatasampleofdifferentnominalpoints in the feasible parameter space . The two subplots show the probabilitydistributions ⁄ and ⁄ ofthedirectionalgradients,evaluatedoveralloftheparameterlocationsin .Note,thatthedistributionforparameter hasalargermean( ⁄ ⁄ )indicatingalargeraveragelocalsensitivity,and larger standard deviation ( ⁄ ⁄ ) indicating a largervariabilityofthelocalsensitivity.

8

Morris (1991) proposed simply using the mean ⁄ and standard deviation⁄ ofthedistributionofderivativestocharacterizeglobalsensitivity,with

higher parameter sensitivity corresponding to larger values of both and .Subsequently,Campolongoetal(2007)pointed out that since the derivatives ⁄ cantakeonbothpositiveandnegativevalues, itwouldbebetter touse instead themeanofabsolutederivatives ⁄ ,andSobolandKucherenko(2009)suggestedthemeanof

thesquaredderivatives ⁄ .Note,however,thattheseapproachesarebasedon

selecting summary statistics that are globally aggregatedmeasures of local sensitivities;recently Rakovec et al (2014) reiterated the importance of considering the entiredistributionofderivatives,insteadofonlythefirsttwomoments,tobettercharacterizetheavailableinformation.

In practice, the distribution of derivatives strategy is applied by obtaining a sample of⁄ valuescomputedata representativesetof locationsdistributeduniformlyacross

the feasible parameter space, and an experimental design strategy can be employed tominimize the computational cost associated with having to perform a large number ofmodelruns.

2.2.3 The‘AnalysisofVariance’Strategy

Analternativeglobalsensitivityanalysisstrategy,thathastheadvantageofnotrequiringpartialderivativestobecomputed,wasproposedbySobol’(1990)basedontheconceptofananalysisofvariance.Thebasisforthisapproachistherecognition(Figure5)thatasthemodelparameters varythroughoutthefeasiblespace ,themodelresponse takesonadistributionof values that canbe characterized in termsof its “total” unconditionalvariance .

Now, ifwefixoneof theparameters atsomenominalvalue whileallowingallof theother parameters to vary over their feasible ranges, we will obtain a conditionaldistribution | that has a different variance | ; in general thevarianceof this conditionaldistributionmaybe smalleror larger than theunconditionalvariance .

Repeating this computation for different nominal values of the “fixed” parameter sampledacrossitsfeasiblerangewillresultinaprobabilitydistributionofvalues forthe conditional variance. Sobol’ proposed that the expected value (average) of theconditionalvariancewillalwaysbesmallerthan(orequalto)thetotalvariance,andthattheir difference – the average reduction in variance∆ due to fixingparameter –canserveasameaningfulcharacterizationof thesensitivityof themodelresponse to that parameter. In otherwords, if fixing a parameter does not result in asizeableexpectedreduction∆ intotalvariance,thenclearlyvariationsofthatparameter(overitsfeasiblerange)didnotcontributesubstantiallytothetotalvariance .

9

Figure5: Themain plot shows the 2‐parameter surface ‘R1’ and illustrates one‐dimensional cross‐sections where parameter is varied across its range whileparameter isfixedatfourdifferentnominalvalues.Thefirstsubplot(below‐left)showstheprobabilitydistribution oftheresponse overtheentireparameterdomain ,forwhichthevariance 0.075.Thesubsequentfoursubplotsshowthe conditional probability distributions | evaluated for each of the cross‐sectional response surfaces indicated in the main plot; note that the high peakscorrespond to regions where the response surfaces are relatively flat. Theconditional variances | for these four cross‐sections are0.006, 0.048, 0.042, 0.082 ,forwhichthe(sampled)averagechangeinvarianceduetoconditioningon atonlythesefourdifferentnominalvaluesis0.0305.

This ‘expected’ reduction in variance∆ is called the “main effect” associated withparameter .Sobol’furtherpointsoutthatthismaineffectmay,however,onlybeapartofthe contribution by that parameter to the total unconditional variance (thereforecalledtheindividualor independentcontribution),andthatthefurthercontributionscancome through its interdependence (called “interactions”) with other parameters. Forexample, if we fix two of the parameters at nominal values so that and whileallowingtheremainingparameterstovaryovertheirfeasibleranges,wewillobtainaconditionaldistribution | , withvariance | , .

10

Repeating this at a representative sample of nominal points , will result in aprobability distribution of values from which the expected reduction in variance∆ due to simultaneously fixing the two parameters , can becomputed. Now the second‐order interaction effect, ∆ ∆ ∆ , betweenparameters and becomesclear.

More generally, one can simultaneously fix three ormoreparameters at a time to studyhigher‐order interaction effects, andSobol’(1990) has shown that the overall variance canbedecomposedintoitsconstituentelementsas:

∆

wherethe∆ termsrepresentthemaineffectsassociatedwithindividualparameters,thetermsrepresentsecond‐order interactioneffects, the termsrepresent third‐

orderinteractioneffects,andsoon.

To summarize, the sensitivity of response Z to parameter is characterized by the so‐called “total‐order” effect, consisting of the summation of all contributions due toparameter , including the main effect∆ and all of the second‐ and higher‐orderinteraction effects ( , , …) associated with parameter . Computationally, asimpleway to estimate the total‐order effect is by fixing all of the parameters except (several times at a sufficiently large representative sample of locations) and computing∆ ~ ~ ;herethenotation ~ meansthatparameter isnotfixed.

As with the Morris distribution of derivatives strategy, sampling strategies have beenproposed that enable efficient computation of the various Sobol sensitivitymetricswithminimalcomputationalcost(Saltellietal2008).

2.2.4 The‘AnalysisofCumulativeDistributions’Strategy

Amajor critique that can be leveled against the analysisofvariance strategy is that the‘variance’ can often be a poor summary statistic of variability (or more accurately the‘change’invariability)associatedwithadistributionofresponsevalues,particularlywhenthedistributioniseitherhighlyskewedormulti‐modal(i.e.,whenitdeviatessignificantlyfromasymmetricalandunimodaldistributionform).Forsuchcases,ithasbeenproposedthatalternativemeasuresofthevariabilityexpressedbythedistributionbeapplied,suchasitsentropy(e.g.,ParkandAhn1994).However,accuratenumericalestimatesofentropyaredifficulttoobtainwhenworkingwithsmallsamplesizes,andapromisingalternativestrategy has been proposed based on the use of cumulative probability distributionfunctions (cdf’s) that aremuch easier to construct and evaluate numerically than actualprobabilitydistributionfunctions(pdf’s).

Aswiththeanalysisofvarianceapproach,the‘AnalysisofCumulativeDistributions’strategyis based on the probability distribution obtained by representative sampling of themodel parameters in the feasible space , and on the corresponding conditionaldistributions |θ θ obtainedbyfixingeachoftheparametersθ atnominalvaluesθ while allowing all of the other parameters to vary over their feasible ranges. However,rather than characterizing the difference between the unconditional to conditional

11

distributionsbythereductioninvariance,the‘analysisofcumulativedistributions’strategyusessomemeasureofthe“distance”betweentheassociatedcumulativedistributions(zerodistanceindicatesnosensitivitywhilealargerdistanceindicatesastrongsensitivity).SincethestrategyissimilartotheSobol’approachwithtwoimportantdifferences(workingwithcumulativedistributionsandusingsomealternativestatisticthanthevariance),allofthedifferentorderandinteractiontermscan,inprinciple,beestimatedbyfixingoneormorecombinationsofparameters.

AspecificimplementationofthisstrategyisthePAWNapproachproposedbyPianosiandWagener(2015),inwhichtheKolmogorov‐Smirnovstatisticisusedtomeasurethedistancebetweenthetwo(unconditionalandconditional)one‐dimensionalcumulativeprobabilitydistributions.Theauthorspointoutthattheadvantagesofworkingwithcdf’s(ratherthanpdf’s) include the ability to test for statistical significance, the ease with whichbootstrapping can be performed to evaluate robustness, and the ability to focus theevaluation on segments of the distribution such as the extremes. Note, however, thatPianosiandWagener(2015) limit their analysis to only first‐order effects (second‐orderandhigher‐orderinteractioneffectsarenotaddressed).

2.2.5 The‘VariogramAnalysisofResponseSurface’Strategy

A major weakness of the ‘DistributionofDerivatives’ (e.g., Morris), ‘AnalysisofVariance’(e.g.,Sobol’)and‘AnalysisofCumulativeDistributions’strategies(e.g.,PAWN)isthatnoneofthem considers or accounts for the spatially ordered structure of the response in theparameter space (RazaviandGupta2015). In other words, they ignore the fact that theresponsevaluesarenotrandomlydistributedthroughoutthefeasibleparameterspace,butinstead there is a spatiallycontinuous correlation structure to the values of , andhencealso to the values of ⁄ . As a result, such approaches can actually assign identicalsensitivity estimates to response surfaces having totally different spatial correlationstructures (shapes), even though such an assignment of relative sensitivity may runcountertointuition(Figure3;seeexamplesanddiscussioninRazaviandGupta2015).

To address this problem, Razavi and Gupta (2016a,b) recently proposed a sensitivityanalysisapproachbasedonthepropertiesofthedirectionalvariogramsoftheresponse intheparameterspace(Figures6&7);theapproachiscalledVARS(VariogramAnalysisof Response Surfaces). Note that a variogram is constructed by computing thevarianceofthedifferences∆ betweenvaluesoftheresponsecomputedat(alargenumberof) pairs of points at different locations spaced distance apart throughout theparameter space. So, a directional variogram represents the variance computedatpoints spaced apart in thedirectionofparameter .Accordingly, ahigher value of for distance , corresponds to a higher ‘rate of variability’ (andtherefore ‘sensitivity’) of at that distance in the direction of parameter . Notably, thisrate of variability at a particular distance in the problem domain represents the ‘scale‐dependentsensitivity’oftheresponsesurfacetothecorrespondingparameter.Toaccountfor thisvariation inrelativesensitivityacrossscales(distances), theVARSapproachusesthe integrated variogram to represent the average sensitivity over all distances up to agivenscalelength.

12

Figure 6: The left plot shows the directional variograms and corresponding to parameters and for the 2‐parameter response surface ‘R2’(seeFigure3). Intuitivelyweconsidertheresponsesensitivitytoparameters tobe lower than that for parameter . This is reflected in the variogram forparameter which tends tobebelow thevariogram forparameter . Theright plot shows the corresponding integrated directional variograms and

;forthiscaseofresponsesurface‘R2’,theoverallsensitivitytoparameter is higher than to parameter regardless of the scale distance . For a morecomplexcaseseeFigure7.

Thevariogram‐basedsensitivityanalysisapproachhasseveralinterestingproperties:

1) Multi‐scale dependency: Because variograms are defined over the full range ofdistances/scales,theyprovideaspectrumofinformationaboutthesensitivityofthemodelresponsetothefactorofinterest;i.e.,theyhighlightthefactthattherelativesensitivityofparametersinEESmodelsisa‘scale‐dependent’concept.

2) Mathematical foundation: If we take the square root of , which gives us, anddivideby thedistance ,weget aquantity that corresponds to

the average finite difference approximation of∆ ∆ ∆ , which approaches⁄ as ∆ → 0 , and is therefore recognizable as being related to the

mathematicaldefinitionoflocalsensitivity.

3) Theoreticalbasis: Itcanbeformallyshown(seeRazaviandGupta2016a)thatthesensitivity ranking provided by VARS converges to that provided by the Morris(distribution of derivatives) strategy at small distances in the parameter space,andtothatprovidedbySobol’(analysisofvariance)strategyatlarge distancesinthe parameter space. In otherwords, theMorris and Sobol’ strategies are specialcases of the more general variogram‐based approach. Further, the STAR‐VARSimplementation (RazaviandGupta2016b) of VARS is based on the use of cross‐sections sampled at numerous locations in the parameter space, and so istheoreticallyalsorelatedtothe‘One‐DimensionalCrossSection’strategy.

4) EfficiencyandRobustness:Becauseconstructionofthevariogramisbasedonpairsofpoints (rather than individual points) sampled across theparameter space, andbecausethenumberofpairsgrowsrapidlyas~n2(wherenistherateofincreaseof

γ(h)

h H

Γ(H

)

0

0.05

0.1

0.15

0.2

0.25

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

0

0.02

0.04

0.06

0.08

0.1

0.12

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

13

number of points sampled in the feasible space), the VARS approach iscomputationally efficient and statistically robust, even for high‐dimensionalresponse surfaces. Accordingly, it is able to provide relatively stable estimates ofparameter sensitivity with much fewer samples than are required by the Morrisand/orSobol’ strategies (RazaviandGupta2016a,b found it tobe asmuchas twoordersofmagnitudemoreefficientontwoapplicationcases).

5) Confidence Intervals: The practical implementation ofVARSpresented byRazaviand Gupta (2016b) includes a bootstrap procedure that provides estimates ofconfidence intervals and reliabilities of the VARS‐based estimates of relativeparametersensitivity.

3. ChallengesandFutureOutlook

Over the past two decades, EES models have rapidly become more complex andcomputationally intensive, growing in parameter dimensionality as they reflect ourgrowing understanding about the nature and functioning of the world. Accordingly, theneed for robust, informativeandcomputationallyefficient sensitivityanalysis techniquesandtoolshasbecomeevenmorepressing.Thereare,however,severalchallengesthatneedcriticalattention;weidentifyanddiscusssomeofthesechallengesbelow.

3.1ComputationalEfficiency

Perhapsthemostobviouschallengeisthatthegrowingcomputationalexpenseassociatedwith increasinglymore complexmodels tends to inhibit thewidespread applicability ofrigorousapproachestoSA,andsopractitionersareoftenforcedtorelyuponsimplerad‐hoc strategies (such asOne‐DimensionalCrossSections) that are relatively inexpensive toimplement. This is understandable, since SAmethods such asMorris and Sobol’ requirevery large numbers of model runs to provide relatively stable parameter sensitivityrankings, andpractitionersmay insteadprefer to reserve their computationalbudget forthe parameter optimization stage of their analysis. In general, we can expect that thecomputationalefficiencyassociatedwithEESmodelscanbeimprovedby:

1) Moreefficient/intelligentextractionofinformationfromEESmodelsruns

2) Useofmorepowerfulandefficientcomputationalresources

In regard to the former, there have already been significant advances. For example, theVARSapproachcanbeusedtoobtainmoreefficientestimatesofbothMorris(shortscale)and Sobol’ (large scale)parameter sensitivity rankings,while alsoproviding informationaboutthestructureoftheresponsesurfaceatarangeofintermediatescales.Therearealsootherapproachessuchasthe“FAST”(Cukieretal1973;SchaiblyandKShuler1973;Cukieretal1975)and“extendedFAST”(Saltellietal1999)methodologiesbasedonuseofFourierseries expansion that are designed to bemore computationally efficient than the Sobol’typeapproachesbasedinMonteCarlosampling.Finally,surrogate‐modelingstrategiescanbeusedtodevelopcheaper‐to‐runsurrogatesofcomputationallyintensivemodels(Razavietal.,2012a&b).

Inregardtothelatter,high‐performancecomputing(HPC)resourcesarerapidlybecomingwidespread, making it increasingly feasible to run large numbers of EES modelconfigurations (e.g., different parameter values) simultaneously in a parallel manner.

14

BecauseSAtechniquesareessentiallybasedinsampling(withnofeedbackfromthemodelresponse surface required), theyareperfectlyparallelizable to takeadvantageof the fullcapacityofHPCresources.AsHPCresourcesbecomemorereadilyavailable,theiruseforperforming rigorous SA of large‐scale computationally expensive EES models will growaccordingly.

3.2Reliability(AccuracyandRobustness)

Closelyrelatedtotheissueofcomputationalefficiencyisthatofaccuracyandrobustness.Practical implementations of the various strategies to compute parameter sensitivityrequire a “sufficient” number of representative samples of model response behaviordistributed throughout the feasible parameter space . Asmodel complexity (andhenceparameter dimensionality) increase, the number of samples required for statisticallyaccurate and robust sensitivity estimates grows geometrically. And, whereas SA isgenerally most useful when the dimensionality of the problem is large (e.g., for factorscreeningorprioritization),itispreciselyinsuchproblemsthattheabilitytogeneratethatrequirednumberofsamplesbecomesrestrictive.

So,withlimitedcomputationalbudgets,theconfidencethatcanbeplacedintheaccuracyofestimated parameter sensitivity rankings can decline very rapidly, to the point of beingvirtually useless. For example, Figure 8 illustrates the reliability of factor rankingsobtainedwhen different sensitivity analysis approacheswere applied to a 45‐parameterEES model using two different computational budgets (sample sizes). Although thereliabilityofparameterrankings tends to increasewithsamplesize, theresultsobtainedusinganyofthesemethodscanstillbequitelow(below50%).Ingeneral,however,duetoits efficient use of information from pairs of points (rather than individual points), theVARS‐basedapproachtendstoprovidemorereliableresultsthanthoseprovidedbyMorrisand/orSobol’.]

The challenge for future SA methods is, therefore, to enable the reliable analysis ofparameter sensitivity forhigh‐dimensionalproblems.Complexmodelswithhundreds,oreven thousands, of uncertain parameters are likely to becomemorewidespread, and assuch,theroleofSAwillbecomemoreimportantthanever.Itisimportant,therefore,thatthe development and application of practical sensitivity analysis tools be designed tomaximizetheuseofinformationprovidedbytheΘZresponsesurface(whilebeingbasedin robust statistical approaches). For example, the VARS approach is able to exploit theinformationaboutdifferentialresponseprovidedbypairsofpointsatvariousdistancesinthe feasiblespace.Strategies(suchasVARS)thatcanexploit informationaboutresponsesurfacestructureneedtobeexploredinfurtherdepth.

In this regard, note that the statistical approachesmentioned in this chapter are basedprimarilyonthefirsttwomoments(meanandvariance/covariance)ofthedistributionofmodelresponse.Becausethesetwostatisticscanbeinsufficienttostatisticallycharacterizethe information contained in the underlying pdf (i.e., skewness and other higher ordermomentsareignored),itseemsdesirabletoexplorehowtheinformationcontainedintheentire distribution of model response can be better exploited (e.g., via more powerfulimplementationsofthe‘AnalysisofCumulativeDistributions’strategy).

15

Figure 8: Both plots show percent reliability of parameter sensitivity rankingobtained by different methods for a 45‐parameter computationally intensivephysicallybased landsurfacehydrologymodel; theresults in theupperplotwereobtained using a sample size of ~20,000 function evaluations (samples) and theresults in the lowerplotwere obtainedusing a sample size of~100,000 functionevaluations(a5‐foldincrease).Althoughreliabilityincreaseswithsamplesize,itcanbe still be quite low (around 10‐50%). The plots also show that the VARS‐basedsensitivity rankings (IVARS50 & VARS‐TO) tend to be more reliable than thoseprovided byMorris and Sobol’. [Source: Based on Figures 12& 13 in Razavi andGupta2016b].

3.3AmbiguityoftheDefinitionof‘Sensitivity’

Asexplainedearlier,theMorris(DistributionofDerivatives)strategyisbasedonextendingtheconceptoflocalsensitivitytotheentirefeasiblespace,whereastheSobol’(analysisofvariance)strategyisbasedontheconceptofattributingportionsofthevarianceinoverallmodelresponseZtoindividualparameters.Whileeachstrategyseems(onfacevalue)tobeintuitively meaningful, they can result in quite different interpretations of relativeparametersensitivity(andthereforedifferentconclusionsaboutwhichparametersaretobedeemed‘sensitive’and‘insensitive’).

RazaviandGupta(2016a)provideatheoreticalanalysisdemonstratingthatthereasonforthe different interpretations of sensitivity is primarily the ‘scale’ at which sensitivity isevaluated;i.e.,Morriscorrespondstoa‘smallscale’viewpointwhileSobol’correspondstoa‘largescale’viewpoint.Importantly,neitherapproachtakesintoconsiderationthefactthat

0

10

20

30

40

50

60

70

80

90

100

LAMIN4

MANN(I,M

)

MAXMASS4

ALVC4

LAMAX4

ALVC2

ROOT4

LAMAX2

LANZ01

MAXMASS2

ROOT2

ALVC1

ALI4

LANZ02

ZPLG

ZPLS

LANZ04

LAMAX1

ALI2

MAXMASS1

ROOT1

CLAY23

ALI1

CLAY22

RSM

N2

RSM

N4

CLAY21

LAMIN2

RSM

N1

CLAY12

SAND23

KS(I,M

)

LAMIN1

ZSNL

ZPLG

ZPLS

SAND21

SAND12

CLAY13

SAND13

SAND22

CLAY11

ZSNL

SAND11

WF_R2

Rel

iabi

lity

(%

)

Parameter Name

IVARS50 VARS‐TO Morris Sobol‐TO

0

10

20

30

40

50

60

70

80

90

100

LAMIN4

MANN(I,M

)

MAXMASS4

ALVC4

LAMAX4

ALVC2

ROOT4

LAMAX2

LANZ01

MAXMASS2

ROOT2

ALVC1

ALI4

LANZ02

ZPLG

ZPLS

LANZ04

LAMAX1

ALI2

MAXMASS1

ROOT1

CLAY23

ALI1

CLAY22

RSM

N2

RSM

N4

CLAY21

LAMIN2

RSM

N1

CLAY12

SAND23

KS(I,M

)

LAMIN1

ZSNL

ZPLG

ZPLS

SAND21

SAND12

CLAY13

SAND13

SAND22

CLAY11

ZSNL

SAND11

WF_R2

Rel

iabi

lity

(%

)

Parameter Name

IVARS50 VARS‐TO Morris Sobol‐TO

16

the continuous structural form of response surface must be taken into account for anevaluationofparametersensitivitytobemeaningful.

WhiletheVARSapproachhelpstoaddressthisproblem,itopensupthenewchallengethatsensitivityrankingmustbetreatedasascaledependentconcept,whichmakesthetaskofinterpretingsensitivityresultsmorechallenging;i.e.,itbecomesnecessarytoacknowledgethataparametermightshowhighsensitivityatsmallscalesbut lowsensitivityat longerscales(andviceversa).Theapplicationofthisnewinformationtothemodeldevelopmentprocess becomes model/problem dependent and (given the relative newness of thisfinding) the manner in which to interpret this additional information remains to beproperlyunderstood.

Theneedforacomprehensivecharacterizationofsensitivitybecomesevenmorepressinggiven the utility of SA for applications that go beyond the need for parameterscreening/prioritization (RazaviandGupta,2015) as a pre‐requisite to optimization formodelcalibration.Forexample,SAislikelytoseeextensiveapplicationasadiagnostictoolfortheevaluationofsimilaritiesbetweenthefunctioningofthemodelandtheunderlyingsystem,soastoassessfidelityofthemodelstructureandconceptualization.

Related to the above, development of improved visualization techniques and tools thatmore effectively communicate the sensitivity information to the modeler/user is muchneeded. Accordingly, SA techniques that can comprehensively characterize the complexnature of the high‐dimensional EES model response surfaces in terms of a lower‐dimensionalnumberofmetricsthatcanbereadilyvisualizedwillberequired.Forexample,the anisotropic variogram and covariogram functions of high‐dimensional responsesurfaces presented by RazaviandGupta (2016a,b) provide a way to represent the fullspectrumofsensitivityinformationviadirectional(uni‐dimensional)variograms.

3.4SpecificationoftheCriterionRepresentingModel‘Response’

In general, all of the approaches to assessingparameter sensitivity described above relyupontheuseofasinglepre‐selectedmodelperformancecriterionas ‘themeaningfulandinformative’representativeofmodelresponse.However,anyaggregateaveragemeasureofmodelperformance(suchastheMeanSquaredErrorcriterionetc.)unavoidablyactsasafiltertoemphasizesomeaspectsofmodelbehaviorwhilede‐emphasizingothers.

Forexample,useoftheMeanSquaredErrorcriterioninwatershedmodelingemphasizesthereproductionoffloodpeaks,andsotheaquifertransmissivityparameterswilltypicallybe“found”toberelativelyinsensitive.Incontrast,useoftheMeanSquaredErrorcriterionappliedtolog‐transformedflowsemphasizethereproductionofrecessionperiods,andsothe aquifer transmissivity parameters will appear to be more sensitive than those thatcontroltheformationoffloodpeaks.

AsEESmodelsgrowincomplexityandcapability,thereareactuallyagreatmanyaspectsofmodelresponsethatcharacterizeitsbehavior(e.g.,waterbalance,latentandsensibleheat‐fluxes, carbon and nitrogen fluxes, spatial patterns of the aforementioned, etc.). In suchmodels a single aggregate measure of model performance will fail to provide usefulinformation about parameter sensitivity. To address this challenge,Rosolemetal(2012)present a multiple‐criteria approach that helps to preserve and highlight the important

17

distinctions regarding sensitivity of different aspects of model response to variousparameters.

Webelievethatfurtherworkinthedirectionofmulti‐variatesensitivityanalysisshouldbeencouragedand,inparticular,theuseof“signatureproperties”(Guptaetal2008)asrobustand informative measures of model response should be pursued, given their ability toprovidediagnosticinformationaboutthestructureandfunctioningofthemodel(seee.g.,Yilmazetal2008;deVosetal2010;MartinezandGupta2010,2011;Pokhreletal2012;Heetal2015;Guseetal2016).

4. Conclusions

Sensitivity analysis is a critical tool in the development and application of EES models.However, its application can be inhibited by computational expense, and its usefulnesslimitedbytheinabilitytoextractusefuldiagnosticinformationfromthemodelresponse.Itis of paramount importance that ongoing research seeks to develop strategies forsensitivity analysis that are both effective and efficient. As illustrated by the variouschapters in this book, a variety of different sensitivity analysis approaches (includingMorris’andSobol’)havebeenproposedintheliterature.Asdiscussedinthischapter,eachapproachhasitsstrengthsandweaknesses.

The ‘Variogram Analysis of Response Surface’ strategy proposed by Razavi and Gupta(2016a,b), ‘AnalysisofCumulativeDistributions’ strategyproposedbyPianosiandWagener(2015), and the ‘Multiple‐Criteria’ approach presented byRosolemetal(2012) representdifferentaspectsofprogresstowardsresolvingsomeofthoseweaknesses,butmoreworkneedstobedone.Perhapsmostimportantistheneedtoestablishacleardefinitionofhowto“compactly”characterize thesensitivityofEESmodelresponses toperturbations in theircausalfactors,andtodosoinamannerthatmaximizesthe“diagnostic”informationprovidedbytheanalysis.

Acknowledgments

The firstauthorreceivedpartial support from theAustralianResearchCouncil through theCentre of Excellence for Climate System Science (grant CE110001028), and from the EU‐fundedproject‘‘SustainableWaterAction(SWAN):BuildingResearchLinksBetweenEUandUS’’ (INCO‐20011‐7.6 grant 294947). The second author is thankful to the University ofSaskatchewan’s Global Institute for Water Security and Howard Wheater, the CanadaExcellenceResearchChairinWaterSecurity,forencouragementandsupport.References

CampolongoF, JCariboniandASaltelli (2007),Aneffectivescreeningdesignforsensitivityanalysisoflargemodels,EnvironmentalModellingandSoftware,22(10),1509–1518

Cukier RI, CM Fortuin, KE Shuler, AG Petschek and JH Schaibly (1973), Study of thesensitivity of coupled reaction systems to uncertainties in rate coefficients. I Theory. TheJournalofChemicalPhysics,59(8),3873–3878

18

Cukier RI, JH Schaibly and KE Shuler (1975), Studyof the sensitivityof coupled reactionsystemstouncertaintiesinratecoefficients.IIIAnalysisoftheapproximations.TheJournalofChemicalPhysics,63(3),1140–1149

de Vos NJ, THM Rientjes and HV Gupta (2010), Diagnostic Evaluation of ConceptualRainfall–RunoffModelsUsingTemporalClustering,HydrologicalProcesses,PublishedonlineinWileyInterScience(www.interscience.wiley.com),DOI:10.1002/hyp.7698

GuptaHV,TWagenerandYQLiu(2008),ReconcilingTheorywithObservations:TowardsaDiagnosticApproachtoModelEvaluation,HydrologicalProcesses,Vol.22(18),pp.3802‐3813,DOI:10.1002/hyp.6989.

GuseB,MPfannerstill,MStrauch,D Reusser, S Lüdtke,M Volk,HGupta, andNFohrer(2016), On characterizing the temporal dominance patterns of model parameters andprocesses,HydrologicalProcesses,DOI:10.1002/hyp.10764

HeZ,FTian,HVGupta,HCHu,HPHu(2015),Diagnosticcalibrationofahydrologicalmodelinanalpineareabyhydrographpartitioning,Hydrology and Earth Systems Science, 19,1807‐1826, www.hydrol‐earth‐syst‐sci.net/19/1807/2015/, doi: 10.5194/hess‐19‐1807‐2015

MartinezGFandHVGupta(2010),TowardImprovedIdentificationofHydrologicalmodels:ADiagnosticEvaluationof the “abcd”MonthlyWaterBalanceModel for theConterminousUnitedStates,WaterResourcesResearch,46,W08507,doi:10.1029/2009WR008294

Martinez GF and HV Gupta (2011), Hydrologic Consistency as a Basis for AssessingComplexity ofWater BalanceModels for the Continental United States, Water ResourcesResearch,doi:10.1029/2011WR011229

Morris MD (1991), Factorial sampling plans for preliminary computational experiments,Technometrics,33(2),161–174.

Park C and K Ahn (1994), A new approach formeasuring uncertainty importance anddistributional sensitivity in probabilistic safety assessment, Reliability in EngineeringSystems,Saf,46,253e261

Pianosi F and T Wagener (2015), A simple and efficient method for global sensitivityanalysisbasedoncumulativedistributionfunctions,EnvironmentalModelling&Software,67,1e11,http://dx.doi.org/10.1016/j.envsoft.2015.01.004

Pokhrel P, K Yilmaz and HV Gupta (2012),Multiple‐CriteriaCalibrationofaDistributedWatershed Model using Spatial Regularization and Response Signatures, Journal ofHydrology 418‐419, pp 49‐60, Special Issue on DMIP‐2, doi: 10.1016/j.jhydrol.2008.12.004

RakovecO,MCHill,MPClark,AHWeerts,AJTeulingandRUijlenhoet(2014),DistributedEvaluation of Local Sensitivity Analysis (DELSA), with application to hydrologic models,WaterResourcesResearch,50,409–426,doi:10.1002/2013WR014063

Razavi S, A Elshorbagy, H Wheater and D Sauchyn (2015), Toward understandingnonstationarityinclimateandhydrologythroughtreeringproxyrecords,WaterResourcesResearch,51,1813–1830,doi:10.1002/2014WR015696

19

Razavi S and HV Gupta (2015),WhatDoWeMeanbySensitivityAnalysis?TheNeed foraComprehensiveCharacterizationof ‘Global’Sensitivity inEarthandEnvironmentalSystemsModels,WaterResourcesResearch,doi:10.1002/2014WR016527

RazaviSandHVGupta(2016a),ANewFrameworkforComprehensive,Robust,andEfficientGlobal Sensitivity Analysis: Part I ‐ Theory, Water Resources Research, 52, 423–439,doi:10.1002/2015WR017558

RazaviSandHVGupta(2016b),ANewFrameworkforComprehensive,Robust,andEfficientGlobalSensitivityAnalysis:PartII ‐Applications,Water Resources Research, 52, 440–455,doi:10.1002/2015WR017559

RazaviS,BATolsonandDHBurn(2012),NumericalAssessmentofMetamodellingStrategiesInComputationally IntensiveOptimization, Environmental Modelling and Software,34(0),67‐86

Razavi S, BA Tolson and DH Burn (2012), Review Of Surrogate Modelling In WaterResources,WaterResourcesResearch,48,W07401,doi:10.1029/2011WR011527

Rosolem R, HV Gupta, WJ Shuttleworth, X Zeng and LGG de Goncalves (2012), A FullyMultiple‐Criteria Implementationof theSobolMethod forParameterSensitivityAnalysis,JournalofGeophysicalResearch(117),D07103,doi:10.1029/2011JD016355

Saltelli A, S Tarantola and K Chan (1999), Aquantitativemodel‐independentmethod forglobalsensitivityanalysisofmodeloutput,Technometrics,41(1),39–56

SaltelliA,MRatto,TAndres,FCampolongo,JCariboni,DGatelli,MSaisanaandSTarantola(2008),GlobalSensitivityAnalysis:ThePrimer,JohnWiley,Hoboken,N.J.

Schaibly JH and K Shuler (1973), Study of the sensitivity of coupled reaction systems touncertainties in rate coefficients. IIApplications. The Journal of Chemical Physics, 59(8),3879–3888

Sobol’ IM (1990), On sensitivity estimation for nonlinear mathematical models,MatematicheskoeModelirovanie,2(1),pp.112–118[inRussian]

Sobol’ IMandSKucherenko (2009),Derivativebasedglobalsensitivitymeasuresandtheirlink with global sensitivity indices, Mathematics and Computers in Simulation, 79(10),3009–3017.

YilmazKK,HVGuptaandTWagener(2008),Aprocess‐baseddiagnosticapproachtomodelevaluation:ApplicationtotheNWSdistributedhydrologicmodel,WaterResourcesResearch,44,W09417,doi:10.1029/2007WR006716.