Available online at www.sciencedirect.com

Journal

www.elsevier.com/locate/jterra

Journal of Terramechanics 49 (2012) 315–324

ofTerramechanics

Sensitivity analysis, calibration and validation of a snowindentation model

Jonah H. Lee ⇑, Daisy Huang

Department of Mechanical Engineering, University of Alaska Fairbanks, Fairbanks, AK 99775-5905, United States

Received 12 September 2012; received in revised form 23 November 2012; accepted 4 December 2012

Abstract

Quantification of the mechanical behavior of snow in response to loading is of importance in vehicle-terrain interaction studies. Snow,like other engineering materials, may be studied using indentation tests. However, unlike engineered materials with targeted and repeat-able material properties, snow is a naturally-occurring, heterogeneous material whose mechanical properties display a statistical distri-bution. This study accounts for the statistical nature of snow behavior that is calculated from the pressure-sinkage curves fromindentation tests. Recent developments in the field of statistics were used in conjunction with experimental results to calibrate, validate,and study the sensitivity of the plasticity-based snow indentation model. It was found that for material properties, in the semi-infinitezone of indentation, the cohesion has the largest influence on indentation pressure, followed by one of the the hardening coefficients.In the finite depth zone, the friction angle has the largest influence on the indentation pressure. A Bayesian metamodel was developed,and model parameters were calibrated by maximizing a Gaussian likelihood function. The calibrated model was validated using threelocal and global confidence-interval based metrics with good results.� 2012 ISTVS. Published by Elsevier Ltd. All rights reserved.

Keywords: Indentation; Snow; Drucker-Prager; Validation; Calibration; Sensitivity; Bayesian; Metrics; Bias; Surrogate

1. Introduction

A key component in the modeling of vehicle-terraininteractions for soils and snow [1–3] is the pressure-sinkagerelationship of terrain material obtained using indentationtesting. Naturally occurring terrain materials have beencategorized as random heterogeneous materials [4] suchthat their properties should be treated statistically. Recentefforts in statistical modeling of vehicle-snow interactioninclude the interval analysis approach in [5], a metamodel-ing approach in [6], and a polynomial chaos approach in[7]. The pressure-sinkage relationship used in these effortswere, however, empirically-based.

Recently, a snow indentation model in [8] was developedbased on plasticity theory such that pressure-sinkage

0022-4898/$36.00 � 2012 ISTVS. Published by Elsevier Ltd. All rights reserve

http://dx.doi.org/10.1016/j.jterra.2012.12.001

⇑ Corresponding author. Tel.: +1 907 474 7136.E-mail address: jonah.lee@alaska.edu (J.H. Lee).

curves have a physical basis, which gives an improvementover the empirical nature of previous research work. Themodel uses only a few fundamental material propertiessuch as the cohesion, friction angle and hardening param-eters. However, due to limited test data, the material prop-erties of the model were assumed in an ad hoc fashionwithout consideration of the statistical variations of thepressure-sinkage curves of natural snow. Estimatingparameters of a physical model against test data is a statis-tical process that is usually difficult since it belongs to theclass of inverse problems. Indeed, estimating mechanicalproperties from indentation tests for engineering materialsin general poses as an inverse problem.

Parameter estimation is also called calibration and isintimately related to the validation of the model. Valida-tion of engineering and scientific models has drawn muchattention from academia and industry in recent yearsresulting in common terminology (e.g., [9]) for validationand related statistical frameworks [10,11]. One accepted

d.

316 J.H. Lee, D. Huang / Journal of Terramechanics 49 (2012) 315–324

definition of validation is ‘the process of determining thedegree to which a model is an accurate representation ofthe real world from the perspective of the intended usesof the model’ [9]. Characterization of uncertainties ofmodel and data is an integral part of the validation process.Toward this end, advancement of flexible and reasonablyrigorous statistical frameworks such as [10,11] has beenmade to address the issues of sensitivity, calibration ofparameters and validation of models including many appli-cations to road-load interaction in automotive engineering[11–13]. In addition, quantitative validation metrics havebeen an active research area that provide a more rigorousstatistical assessment of the agreement between test resultsand model predictions [14]. Although headway has beenmade in the statistical modeling of vehicle-snow interac-tion, no statistically rigorous efforts have been made to val-idate the various models developed recently.

This paper applies recent results in the field of statistics,to study the sensitivity of the snow indentation model, tocalibrate fundamental material properties of the modelusing newly obtained experimental results, and to validatethe model using calibrated material properties and severalvalidation metrics.

The paper is organized as follows. Section 2 discussesbackground in the snow indentation model, statisticalmethodology in global sensitivity analysis, Bayesian meta-model, calibration, and confidence-interval based valida-tion metrics. Section 3 presents new snow indentationtests. Section 4 discusses results, and Section 5 follows withdiscussion and conclusions.

2. Background and methodology

In the following, we first summarize the snow indenta-tion model where material constants are to be calibratedagainst test data toward the validation of the model. Wethen discuss the statistical methods and models used. Itshould be noted that there are two types of model discussedin this paper, one is the physical snow indentation model,and the other is the statistical model. Unless expressedexplicitly, ‘model’ by itself means the snow indentationmodel.

2.1. Snow indentation model

Three deformation zones have been approximately iden-tified in [8]. Zone I is a small, initially linearly elastic region.It is followed by zone II, a strain-hardening region wherethe pressure bulb developed underneath the indenter hasnot yet reached the bottom of the snow cover, i.e., it is azone of semi-infinite depth. Zone III, a finite-depth zone,is a region where the pressure bulb has reached the bottomof the snow cover.

The material properties for the plasticity indentationmodel to be calibrated in this paper are summarized below.Full details of the model can be found in [8] and referencestherein.

A simple Drucker-Prager yield function was used todevelop the plasticity solutions for the indentation modelusing two material parameters: the cohesion (pd) and thefriction angle (b) which can be related to the absolute ten-sile and compressive strengths (T and C) as

C� 1

3C tan b� pd ¼ 0 ð1Þ

Tþ 1

3T tan b� pd ¼ 0 ð2Þ

The hardening of the snow can be expressed by the loca-tion of the cap of the Drucker-Prager model, pa, which isparameterized as

log10pa ¼ c1 � c2 exp ��pv � c3 �

pv

� �3� �

ð3Þ

where �pv ¼

�pkk3

is the volumetric plastic strain, c1, c2 and c3

are constants.

2.2. Statistical methods

In this section, we present background of the essen-tial ingredients of the methodology used in this paper:global sensitivity analysis, Bayesian metamodel, calibra-tion, and validation. These include the Gaussian maxi-mum likelihood used in calibration, Gaussian processesused for metamodel and sensitivity analysis, and valida-tion metrics. The methods used are also summarized asflow charts in Figs. 2–4 which will be discussed in detailin Section 4.

2.2.1. Gaussian likelihood

Maximum likelihood estimation (MLE) is a commonmethod that estimates the parameters of a statistical modelsuch that there is a maximum probability of the observeddata based on the estimated parameters [15]. When the sta-tistical model uses a Gaussian distribution for the randomvariable x with unknown mean l and standard deviation r,one has

f ðx; l; r2Þ ¼ 1

rffiffiffiffiffiffi2pp exp �ðx� lÞ2

2r2

!ð4Þ

To estimate the mean and standard deviation of a sam-ple size of N (xi, i = 1, . . . , N) using MLE, the probabil-ity of the data set is first expressed as the product ofthe probabilities of each point, i.e., the likelihood func-tion L(l,rjx)

Lðl; rjxÞ ¼YNi¼1

1

rffiffiffiffiffiffi2pp exp �ðxi � lÞ2

2r2

!ð5Þ

The unknown parameters (l and r) are then obtained bymaximizing L which is equivalent in maximizing log(L)such that the normal log-likelihood function is

logðLÞ ¼ �N2ðlog r2 þ logð2pÞÞ �

XN

i¼1

ðxi � lÞ2

2r2

!ð6Þ

P

W

H

d

(a)

(b)

Fig. 1. Indentation test: (a) schematic of the test: diameter of the indenterd = 12.7 mm, diameter of the container W = 46 mm, depth of the snowH = 20 mm, P = force, and (b) experimental set up.

Fig. 2. Flow chart of global sensitivity analysis.

J.H. Lee, D. Huang / Journal of Terramechanics 49 (2012) 315–324 317

2.2.2. Global sensitivities

Analysis of the global sensitivities of a model is suchthat it does not rely on local derivatives of the functionagainst parameters of interest, but rather uses design ofexperiments to explore the entire space to efficiently figureout sensitivity [16]. This is especially useful for nonlinearfunctions such as the snow indentation model of interestin this paper.

A statistical model for sensitivity studies can beexpressed as

y ¼ f ðxÞ þ � ð7Þwhere y is the scalar output of the physical model, x = [x1,x2, . . . , xr] is a vector of r parameters, and � is an errorterm due to numerical computation with finite precision.� discussed here should not be confused with strain of snowindentation which uses the same symbol. � will be assumedto be small and will not be subsequently considered. Thesensitivity of the parameters x of the model refers to the rel-ative contribution of each of the parameters to the output

y, as well as the contribution of the interactions of theparameters to y. In this paper, we adopt the approach ofglobal sensitivities based on variance decomposition as op-posed to a local, derivative based approach; see Sec-tion 1.2.1 of [16] for more details. The sensitivity measureused in this paper is Tj for the jth parameter(j = 1, . . . , r) defined in [17] as

T j ¼EðVar½f ðxÞjxð�jÞ�Þ

Varðf ðxÞÞ ð8Þ

where E(�) and Var(�) are the expectation and variance of afunction, respectively; x(�j) = [x1, . . . , xj�1, xj+1, . . . , xr]represents all the parameters except xj. Tj defined here ac-counts for the contributions of parameter xj and its interac-tion with other parameters to y. In addition to obtainingthe sensitivity measure, it is also desirable to obtain its con-fidence interval, e.g., via a bootstrapping (resampling) tech-nique [17].

To obtain the sensitivity measure and its confidenceinterval, many model runs are needed, which can be daunt-ing for even relatively simple models. Consequently, meta-models are oftentimes developed to speed up thecalculations. This will be discussed in the next section. Inthe literature, metamodel is also called surrogate, emulatoror simulator. When metamodels are used for sensitivityanalysis, the sensitivity measure Tj will be designated as bT j.

2.2.3. Bayesian metamodel

There are two major views of statistics: the frequentistview and the Bayesian view, with the former based onhypothesis testing and the latter based on inference of

Fig. 3. Flow chart of the development of the Time Input (TI) metamodel.

Fig. 4. Flow chart of the calibration process.

318 J.H. Lee, D. Huang / Journal of Terramechanics 49 (2012) 315–324

(posterior) probabilities with assumed or known prior

knowledge using Bayes’ theorem [18]. See [19] for detaileddiscussions on these two views where the Bayesianapproach was considered to be more flexible and practical.

A common approach for a metamodel is to use aBayesian statistical method where the Gaussian processesare used as the prior probability distribution [10,20].Gaussian process [21], in contrast to Gaussian distributionwhich is a distribution over random variables, is a stochas-tic process and is defined as the probability distributionover functions f(x) such that the set of values of f(x) evalu-ated at arbitrary points x1, . . . , xn will have a joint Gauss-ian distribution, i.e., it is a distribution over functions. TheGaussian process has been used widely in many fields ofscience and engineering including application in the formof Gaussian random field used effectively in characterizingthe 3D microstructure of snow [22].

Given data from the simulations of the physical model,used as ‘training’ data, the posterior probability distribu-tions of the model output can be obtained, which is modeldependent. In particular, the output from a physical model

is considered as an unknown function modeled statisticallyas a Gaussian process by its mean function m(�) wherem(x) = E[f(x)], and its covariance function c(�) wherec(x,x0) = cov(x,x0). Following [20], for a random functionf(�) and a set of points [x1, . . . , xn] in its domain where eachxi, i = 1, . . . , n, has r parameters [x1, . . . , xr], the randomvector [f(x1), . . . , f(xn)] is assumed to be a multivariate nor-mal distribution with mean

E½f ðxÞjb� ¼ hðxÞT b; ð9Þwhere b is a vector of unknown coefficients, and h(�) the q

unknown regressor functions of the r parameters x = [x1,. . . , xr]; superscript T denotes the transpose of a vectoror matrix. The covariance is expressed as

cov½f ðxÞ; f ðx0Þjr2� ¼ r2cðx; x0Þ ð10Þwhere c(x,x0), the correlation function between the inputs x

and x0, is assumed to be

cðx; x0Þ ¼ exp½�ðx� x0ÞT Bðx� x0Þ� ð11Þwhere B is a diagonal matrix of positive ‘roughness’ param-eters li, i = 1, . . . , r; in statistical literature, the unknown r2

and roughness parameters li are called hyperparameters.Once the mean function and the covariance function are

chosen, one obtains a set of design points [x1, . . . , xn] inparameter space as input to the physical model f(�); thesepoints are called the experimental design or the design

matrix (with dimensions n � r). In this paper, the designpoints are generated using the popular Latin HypercubeSampling (LHS) approach [23] which obtains distributionsof parameters from a given multidimensional distribution.The outputs from the design matrix, d = [d1, . . . , dn], areobtained by running the physical model n times. In prac-tice, d are typically appended to the design matrix; for

J.H. Lee, D. Huang / Journal of Terramechanics 49 (2012) 315–324 319

convenience, the design matrix augmented by d will also bereferred to collectively as the design matrix.

Using Bayes’ theorem, the posterior mean and covari-ance functions can then be derived and given in Eqs. (3)and (4) in [20] and Eqs. (2) and (3) in [24]. The quality ofthe metamodel is assessed via cross-validation methodssuch as the leave-one-out analysis where data from n � 1model runs is used to predict the model data of the remain-ing run. The metamodel thus obtained can be used to pre-dict new outputs as an emulator of the original physicalmodel f(�) given a new set of inputs. The Bayesian meta-model approach effectively converts a deterministic physi-cal model into a stochastic model via Bayes’ theoremconditioned with model data.

For pressure–displacement relationship, given parame-ters x, the output of the model is a curve rather than a sin-gle pressure. Displacement, or strain, can be considered asa time-like parameter (t). The metamodel discussed previ-ously can then be extended to include an extra input t tak-ing values in 1, . . . , T

yðtÞ ¼ f ðx; tÞ þ � ð12Þto emulate a dynamic simulator resulting in a Time Input(TI) simulator discussed in [25]. The design matrix nowhas dimensions of nT � r.

2.2.4. Calibration

For calibration of model parameters, we used theGaussian likelihood method. Since the snow indentationmodel is an approximation of the real phenomenon, thereis a bias (or model inadequacy [10]) between the modeland test data that needs to be taken into consideration.Let the test data be represented by Yt,i, t = 1, . . . , T, andi = 1, . . . , N where N is the number of replications of testdata. The bias between the model f(x, t) and test data isapproximated as [13]

Bt ¼ Y t � �f ðx; tÞ ð13Þwhere at time t, Y t is the average of the test data over N

replications, and �f ðx; tÞ is the average of the surrogatemodel output over n sets of input parameters x.

We then assume the following statistical model for cali-bration [13]

Y t;i ¼ Bt þ f ðx; tÞ þ Eðx; tÞ þ �t;i ð14Þwhere f is the mean function of the surrogate model, E isthe surrogate error term having a multivariate Gaussiandistribution with a zero mean vector and covariance ex-pressed in Eq. (10), � is an observational random error termwith a Gaussian distribution. The likelihood function inEq. (5) then can be generalized as

LðxjYÞ¼YNi¼1

1ffiffiffiffiffiffiffiffiffiffiffiffiffiffidetðXÞ

p� exp �1

2ðY t;i�Bt� fðx; tÞÞT X�1ðY t;i�Bt� fðx; tÞÞ

� �ð15Þ

where X is the covariance function of the metamodel in Eq.(12); constant terms are ignored. The calibration parame-ters x are then obtained by maximizing Eq. (15) or itslogarithm.

2.2.5. ValidationOnce the calibration parameters have been obtained via

the Gaussian maximum likelihood approach, they aresubstituted into the metamodel of Eq. (12) to predict themean, variance, and confidence interval (called credible

interval using Bayesian terminology) of the output of inter-est. An evaluation of the prediction constitutes the valida-tion of the model.

Validation of models can be conducted qualitativelyand/or quantitatively. The former often involves expertopinion of, say, the comparison of prediction against testdata graphically. To validate models quantitatively, valida-tion metrics are needed which are more stringent thanqualitative validation. In this paper, we use three confi-dence-interval (CI) based validation metrics where the firsttwo were proposed in [14].

The first validation metric is based on the difference inthe means between model prediction and test data, andthe standard deviation of either test data or model predic-tion as follows.

We define the estimated error eE at a given time (strain)as the difference between the predicted mean of the cali-brated model f(xcalibrated, t), and the mean of test data overall replications (Y t in Eq. (13)):eEðtÞ ¼ f ðxcalibrated ; tÞ � Y t ð16ÞAt a given time t, the confidence interval (CI) of test data isconstructed based on the standard deviation of the testdata s(t) following standard statistical procedure. A plotof eEðtÞ against the test CI gives a comparison of the esti-mated error with the error of the test data as a functionof t; if the former is smaller than the latter, then one mayconsider the model to be adequate and one may want toimprove the test; otherwise, one may want to improve themodel if needed depending upon the intended purpose.

A variation of this metric is to construct the CI of modelprediction as opposed to that of the test data. Since we areusing a Bayesian surrogate model, the CI is constructedfrom the surrogate model. A plot of eEðtÞ against the modelCI gives a comparison of the estimated error with the errorof the model prediction as a function of t; if the former issmaller than the latter, then one may consider the modelto be adequate for predictive purpose; otherwise, onemay want to improve the model.

Since the above metric is a function of t, it is possiblethat the model performs adequately for part of t, but notfor others. Consequently, this metric gives a fine-grained(local) evaluation of the quality of the model and can pro-vide guidance on the improvement of the model.

The second and third validation metrics are global, i.e., ameasure for the entire time interval. They are normalized toprovide a high-level summary of the errors in percentage.

320 J.H. Lee, D. Huang / Journal of Terramechanics 49 (2012) 315–324

For the second metric, the average relative error metric isfirst defined as [14]:eEðtÞ

Y t

avg

¼ 1

tu � tl

Z tu

tl

f ðxcalibrated ; tÞ � Y ðtÞY ðtÞ

dt ð17Þ

where tu and tl are the upper and lower limits of t. Thisquantity can be considered as the absolute mean error be-tween test and model normalized by the mean test error.

The associated average relative confidence indicator isdefined next as:

CI

Y t

avg

¼ tp;m

ðtu � tlÞffiffiffiffiNp

Z xu

xl

sðtÞY ðtÞ

dt ð18Þ

where tp,m is the t-statistic of probability p, and degree offreedom m, N is the total number of test data replications.This indicator can be considered as an approximate nor-malized confidence interval of test data.

For this metric, similar to the first metric, if the averagerelative error is smaller than the average relative confidenceindicator, then one may consider the model to be adequate;otherwise, the model may need improvement.

The third validation metric is the coverage defined as thepercentage of the mean test data points that fall within,say, the 95% CI of the model prediction; this is a measureof the predictability of the model. A larger coverage meansa better model.

3. Experiments

To validate the model, indentation tests were performedon virgin, natural snows. Snow was collected in the Fair-banks, Alaska area where air temperature, snow surfacetemperature, and snow pit temperature under the surfacewere recorded at each snow collection. The experimentalprocedures used in this paper follow those in [26] wheremuch more details can be found.

Snow was characterized in situ using a high-resolutionsnow micropenetrometer, which gave information aboutpenetration resistance and average grain size. Then sampleswere removed and measured, weighed, and sieved on-siteto obtain density and grain size distribution. Specimenswere also collected and brought to the laboratory andstored in freezers which were maintained at �30 �C. There,they were additionally characterized via inspection with anoptical microscope, and scanning using 3D X-ray Micro-Tomography. From these measurements, the snow densi-ties were determined to range from about 170 to 240 kg/m3, and the average grain sizes were estimated to be 1 mm.

Indentation tests were performed using a CETR UMTtribometer shown in Fig. 1. The testing region was enclosed

Table 1Range of parameters for global sensitivity analysis.

pd (kPa) b (deg) c1 c2

(1,50) (10,50) (0.4,0.8) (3,5)

in a clear, insulative enclosure maintained at a temperatureof �20 �C using a low-temperature forced air stream. Dur-ing an indentation test, snow was placed into a small sam-ple holder underneath the indenter pin, and the indenterpin was manually moved down until it was close to, butnot touching, the snow surface. Then the pin was moveddown at the selected rate while resisting force was mea-sured by the load cell, which was coupled to the top ofthe pin. Resisting force versus indentation displacementwas output by the tribometer.

Tests were carried out with an indenter of 12.7 mmdiameter and an indentation speed of 5 mm/s for a snow20 mm deep. A 20-kg load cell with 0.01 N resolutionwas used. Two fine-grained snows were used in this paper,one was collected on December 29, 2009 and tested onSeptember 14, 2010; the other was collected on November16, 2010 and tested on March 18, 2011.

4. Results

The implementation of the methodology discussed inSection 2.2 uses computer programs coded in the R lan-guage [27]. Global sensitivity analysis uses the R packagein [28] which also employs metamodels to represent thephysical model; the Bayesian metamodel uses the R pack-age discussed in [24].

4.1. Global sensitivity analysis

For global sensitivity analysis, the ranges of parameters,selected by expanding the ranges used in [8], are shown inTable 1. The design points were generated using LatinHypercube Sampling with a uniform distribution wherethe parameters are normalized in [0,1]. Note that the strain� refers to the logarithmic strain. Out of the many meta-models available in [28], we used the Gaussian processmetamodel. To gain physical insight of the indentationprocess, we examine the sensitivities for zone II (hardeningor semi-infinite zone) and zone III (finite-depth zone) sep-arately. The dimensions of the design matrix (n � r) forzone II are 149 � 7, and those for zone III are 199 � 7.Fig. 2 presents the procedure of sensitivity analysis as aflow chart.

The sensitivity measure bT j and its 95% confidence inter-val for the parameters are shown in Tables 2 and 3 for zoneII and zone III, respectively. In interpreting bT j, it should benoted that bT j accounts for the contribution of the j-thparameter and its interaction with all other parameters;consequently, the sum of bT j is larger than 1.

For zone II, Table 2 shows that pd has the largest influ-ence on indentation pressure and explains between 58.5%

c3 Depth (m) �

(0.1,0.5) (0.2,0.6) (0,0.7)

Table 2Sensitivity measure and associated 95% confidence interval using theGaussian process metamodel in [28] for zone II.

Parameter bT 95% CI of bTpd 0.676 (0.585,0.757)b 0.068 (0.049,0.110)c1 0.010 (0.000,0.043)c2 0.106 (0.055,0.130)c3 0.000 (0.000,0.028)Depth 0.028 (0.019,0.056)� 0.220 (0.164,0.286)

Table 3Sensitivity measure and associated 95% confidence interval using theGaussian process metamodel in [28] for zone III.

Parameter bT 95% CI of bTpd 0.000 (0.000,0.039)b 0.217 (0.156,0.261)c1 0.045 (0.013,0.090)c2 0.012 (0.000,0.050)c3 0.032 (0.000,0.069)Depth 0.234 (0.214,0.300)� 0.732 (0.670,0.774)

Table 4Ranges of parameters to generate the design matrix; c1 = 0.66, c3 = 0.18;depth = 20 mm.

Snow sample pd (kPa) b (deg) c2

December 2009 (0.1,6.3) (10,20) (4,30)November 2010 (0.1,7.3) (10,20) (4,30)

0.0 0.1 0.2 0.3 0.4

020

0040

0060

0080

0010

000

1200

0Strain

Pres

sure

(Pa)

Fig. 5. Indentation pressure from model data (dashed lines) and test data(solid lines) for November 2010 snow.

J.H. Lee, D. Huang / Journal of Terramechanics 49 (2012) 315–324 321

and 75.7% of the indentation pressure with a 95% confi-dence; other parameters can be interpreted similarly. Thestrain � has the second largest influence on indentationpressure which is not surprising since indentation pressureincreases with strain. Next in importance is c2 and b withthe rest of the parameters of little importance. That thedepth has little influence over the indentation pressure forzone II is also expected since it is a semi-infinite zone. Insummary, the significance of material properties for zoneII, in decreasing order of significance is pd, c2, b.

For zone III, Table 3 shows that the significance ofparameters is ranked by � > depth > b. For a finite-depthzone, indentation pressure can increase dramatically withdeformation which explains the significance of �; similarlyfor depth. It is interesting to note that b becomes the mostimportant material property for zone III with pd playingalmost no role in indentation pressure, and with c2 playingan insignificant role–almost the opposite of the results forzone II.

4.2. Bayesian metamodel

The Bayesian metamodel is used for the calibration, pre-diction and validation of the physical model; the steps inbuilding the metamodel are given in Fig. 3. Based on theresults from Section 4.1, only a reduced set of the parame-ters, pd, b, and c2 will be allowed to vary with the rest fixed.A prerequisite of the values of parameters for building themetamodel is that the output from the physical model dueto the parameters need to cover the domain of interest, i.e.,the test data in our case. Using the Time Input model from[25] in Eq. (12), the design matrix is again created usingLatin Hypercube Sampling with initial dimensions ofn � r where n = 50, and r = 3. For each model run, one

uses T values of the model output such that the final designmatrix has dimensions of nT � r where T = 5, i.e., 250 � 3.The parameters are adjusted until the test data are coveredby the model output.

During the step in creating the design matrix, it wasfound that a slight modification of the original snow inden-tation model for stage I was needed to improve the cover-age of test data by model data. In particular, the strength atthe end of zone I is quite small and was set to a smallthreshold value of 100 Pa in all subsequent steps.

The ranges of parameters thus arrived at are given inTable 4 for both snow samples. Model output and the testdata are shown in Fig. 5 for the November 2010 snow withN = 6 data replications, as an example.

After the design matrix is established, the hyperparame-ters of the metamodel need to be estimated. Initial valuesare assumed and an MLE procedure was used in [24] tofind optimized values. A leave-one-out cross-validationwas then conducted to evaluate the quality of the meta-model shown in Fig. 6. The closer the data points to themain diagonal, the better and the performance of the meta-model seems adequate.

4.3. Calibration, prediction and validation

4.3.1. Calibration

After the Bayesian metamodel was developed, calibra-tion of the parameters was conducted by maximizing theGaussian likelihood function in Eq. (15). The steps forcalibration are summarized in Fig. 4. The calibrated con-

Table 5Calibrated material parameters.

Snow sample pd (kPa) b (deg) c2 T (kPa) C (kPa)

December 2009 3.4 11.9 13.6 3.1 3.6November 2010 3.9 11.7 25.6 3.6 4.7

0.0 0.1 0.2 0.3 0.4

020

0040

0060

0080

00

Strain

Pres

sure

(Pa)

Fig. 7. Predicted indentation pressure (solid line) with 95% confidenceinterval (dashed lines) using calibrated material constants compared withexperimental data for December 2009 snow.

0.0 0.1 0.2 0.3 0.4

020

0040

0060

0080

00

Strain

Pres

sure

(Pa)

Fig. 8. Predicted indentation pressure (solid line) with 95% confidenceinterval (dashed lines) using calibrated material constants compared withexperimental data for November 2010 snow.

0 2000 4000 6000 8000 10000 12000

020

0040

0060

0080

0010

000

1200

0

Actual model output (Pa)

Pred

icte

d pr

essu

re (P

a)

Fig. 6. Cross-validation results of the surrogate model for December 2009snow.

322 J.H. Lee, D. Huang / Journal of Terramechanics 49 (2012) 315–324

stants for two snow samples are given in Table 5 where thecompressive (C) and tensile strength (T) have been con-verted from pd and b using Eq. (1) for reference. The cali-brated material properties of the two snows are similareven though they were collected about a year apart andtested at different times.

4.3.2. Prediction

Predictions of the mean and variance of the metamodelare obtained by substituting the calibrated parameters tothe metamodel with results shown in Figs. 7 and 8 wherethe solid red line indicates the predicted mean and thedashed blue lines indicate the 95% confidence interval.The predictions appear to be good qualitatively especiallyin view of the large variations of the test data which isthe rule rather than the exception. A more rigorous evalu-ation of the predictions is given below using validationmetrics introduced previously.

4.3.3. Validation

Validation results using the first (time-dependent) metricare shown in Fig. 9a and b for the December 2009 snow.Fig. 9a shows that the estimated error lies within the CIof the test data indicating that the calibrated model hasgood performance over the entire range of strain. The esti-mated error increases with strain, following the trend of theCI of the test data, as a consequence of the calibrationprocedure.

Fig. 9b shows that the CI of the model prediction, con-stant over the entire range of strain, is almost half of that ofthe CI of the test data. Except at large strains, the esti-mated error still lies within the CI of the model. However,improvement of the model at large strains could be consid-ered if one is concerned about the prediction of the modelat that level of strain.

Fig. 10a and b show the results in applying the first val-idation metric for the November 2010 snow. For both fig-ures, the estimated errors lie within the CI of the test dataand model prediction. Consequently, the results are slightlybetter than those for the December 2009 snow.

The results in applying the two global metrics to the twosnows are shown in Table 6.

Strain

Estim

ated

erro

r and

exp

erim

ent c

onfid

ence

0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40

Strain0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40

−200

0−1

000

010

0020

00−1

000

−500

050

010

00

Estim

ated

erro

r and

mod

el c

onfid

ence

inte

rval

(Pa)

inte

rval

(Pa)

(a)

(b)

Fig. 9. (a) Estimated error and confidence interval for test, and (b)estimated error and confidence interval for model; December 2009 snow;mean error (solid lines), confidence interval (dashed lines).

Strain0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40

Strain0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40

−150

0−1

000

−500

050

010

0015

00−4

00−2

000

200

400

Estim

ated

erro

r and

exp

erim

ent c

onfid

ence

inte

rval

(Pa)

Estim

ated

erro

r and

mod

el c

onfid

ence

inte

rval

(Pa)

(a)

(b)

Fig. 10. (a) Estimated error and confidence interval for test, and (b)estimated error and confidence interval for model; November 2010 snow;mean error (solid lines), confidence interval (dashed lines).

Table 6Results of global validation metrics.

Snow type Coverage(%)

Relativeerror

Relative confidenceindicator

December 2009 94 0.12 0.73November 2010 100 0.12 0.44

J.H. Lee, D. Huang / Journal of Terramechanics 49 (2012) 315–324 323

Since the relative error is much smaller than the relativeconfidence indicator for both snows, the performance ofthe calibrated model is good. These summary metrics showclearly that the December 2009 snow has a larger confi-dence indicator than the November 2010 snow. This isdue to the top curve in Fig. 7 which is farther apart fromother curves. The coverage for the December 2009 snowis slightly less than that for the November 2010 snow indi-cating the model for the latter is slightly better than the for-mer. In addition, the 94–100% coverage of both snowsshows that the calibrated model performs quite well forthe indentation tests.

To the best of our knowledge, this is the first time thatfundamental mechanical properties of snow have been esti-mated from indentation tests with rigorous statistical pre-diction and validation of the indentation model thatperforms well.

5. Discussion and conclusions

In this paper, we used a few interrelated statistical meth-ods to calibrate parameters of the snow indentation model,and infer statistically the pressure-sinkage relationshipbased on new test data with replications. The Bayesianmetamodel is a key component of the methodology servingthe following roles: (1) an efficient emulator of the physical

324 J.H. Lee, D. Huang / Journal of Terramechanics 49 (2012) 315–324

model for the calibration, prediction and validation of themodel, and (2) as a non-intrusive stochastic model of thephysical model such that statistical inference can be made.

Global sensitivity analysis was conducted rigorously andefficiently for several purposes: (1) to gain physical insightof the influence of the multidimensional parameters tomodel output, (2) to select only influential parameters forcalibration which utilizes the Bayesian metamodel, (3) toavoid the possibility of getting physically unreasonableresults for the calibration process [13].

The results presented here are from the slightly-modifiedindentation model, as a consequence of applying the statis-tical methodology during the design of experiment stage,and before the validation metrics were applied, demon-strating another aspect of the usefulness of the methodol-ogy employed.

In summary, our results indicated that, in terms of mate-rial properties, pd, c2 influence significantly the pressure-sinkage relationship for zone II (semi-infinite hardening),whereas b has the most important contribution for zoneIII (finite-depth). The three interval-based validation met-rics used indicated that the calibrated model is quite ade-quate for the two snows used in the paper.

Acknowledgments

The authors gratefully acknowledge the support of thiswork by the US Army TACOM Life Cycle Command un-der Contract No. W56HZV-08-C-0236, through a subcon-tract with Mississippi State University. This work wasperformed in part for the Simulation Based Reliabilityand Safety (SimBRS) research program. Any opinions,findings and conclusions or recommendations expressedin this material are those of the authors and do not neces-sarily reflect the views of the US Army TACOM Life CycleCommand.

References

[1] Wong JY. Theory of ground vehicles. 3rd ed. Wiley Interscience;2001.

[2] Muro T, O’Brien J. Terramechanics. Land locomotion mechanics.A.A. Balkema Publishers; 2004.

[3] Lee JH. An improved slip-based model for tire-snow interaction. SAEInt J Mater Manuf 2011;4:278–88. http://dx.doi.org/10.4271/2011-01-0188.

[4] Torquato S. Random heterogeneous materials: microstructure andmacroscopic properties. 1st ed. Springer; 2001.

[5] Lee JH, Liu Q, Mourelatos ZP. Simulation of tire-snow interfacialforces for a range of snow densities with uncertainty. SAE Trans JMater Manuf 2006:408–18.

[6] Li J, Mourelatos ZP, Lee JH, Liu Q. Prediction of tire-snowinteraction forces using metamodeling. SAE congress. Paper 2007-01-1511; 2007.

[7] Li L, Sandu C, Lee J, Liu B. Stochastic modeling of tire-snowinteraction using a polynomial chaos approach. J Terramech2009;46:165–88.

[8] Lee JH. A new indentation model for snow. J Terramech2009;46:1–13.

[9] ANSI/ASME V& V 10-2006: guide for verification and validation incomputational solid mechanics. American Society of MechanicalEngineers; 2006. ISBN: 079183042X.

[10] Kennedy MC, O’Hagan A. Bayesian calibration of computer models.J R Stat Soc: Ser B (Stat Methodol) 2001;63(3):425–64.

[11] Bayarri MJ, Paulo R, Cafeo JA, Cavendish J, Lin C, Tu J. Aframework for validation of computer models. Technometrics2007;49(2):138–54.

[12] Drignei D. Functional ANOVA in computer models with time seriesoutput. Technometrics 2010;52:430–7.

[13] Drignei D, Mourelatos ZP. Parameter screening in dynamic computermodel calibration using global sensitivities. ASME J Mech Des134:081001. doi:10.1115/1.4006874.

[14] Oberkampf D, Barone MF. Measures of agreement between compu-tation and experiment: validation metrics. J Comput Phys2006;217:5–36.

[15] Press WH, Teukolsky SA, Vetterling WT, Flannery BP. Numericalrecipes: the art of scientific computing. 3rd ed. Cambridge UniversityPress; 2007.

[16] Saltelli A, Ratto M, Andres T, Campolongo F, Cariboni J, Gatelli D,et al. Global sensitivity analysis: the primer. Wiley-Interscience; 2008.ISBN: 0470059974.

[17] Storlie CB, Swiler LP, Helton JC, Sallaberry CJ. Implementation andevaluation of nonparametric regression procedures for sensitivityanalysis of computationally demanding models. Reliab Eng SystSafety 2009;94(11):1735–63.

[18] Bayes T, Price R. An essay towards solving a problem in the doctrineof chance. By the late Rev. Mr. Bayes, communicated by Mr. Price, ina letter to John Canton, M.A. and F.R.S.. Philos Trans R Soc Lond1763;1:370–418.

[19] Bayarri MJ, Berger JO. The interplay of bayesian and frequentistanalysis. Stat Sci 2004;19(1):58–80.

[20] Oakley J, O’Hagan A. Bayesian inference for the uncertaintydistribution of computer model outputs. Biometrika2002;89(4):769–84.

[21] Rasmussen CE, Williams CKI. Gaussian processes for machinelearning. MIT Press; 2006. ISBN: 026218253X.

[22] Yuan H, Lee JH, Guilkey J. Stochastic reconstruction of themicrostructure of equilibrium form snow and computation of effectiveelastic properties. J Glaciol 2010;56:405–14. http://dx.doi.org/10.3189/002214310792447770.

[23] McKay MD, Beckman RJ, Conover WJ. A comparison of threemethods for selecting values of input variables in the analysis ofoutput from a computer code. Technometrics 1979;21:239–45.

[24] Hankin RKS. Introducing BACCO, an R bundle for bayesiananalysis of computer code output. J Stat Softw 2005;14(16):1–21.

[25] Conti S, O’Hagan A. Bayesian emulation of complex multi-outputand dynamic computer models. J Stat Plann Infer 2010;140:640–51.

[26] Huang D, Lee JH. Mechanical properties of snow using indentationtests: size effects. J Glaciol 59. doi:10.3189/2013JoG12J064.

[27] R Development Core Team. R: A language and environment forstatistical computing. R foundation for statistical computing, Vienna,Austria; 2005. <http://www.R-project.org/>. ISBN: 3-900051-07-0.

[28] Storlie C. CompModSA: an R package for the sensitivity analysis forcomplex computer models. R package version 1.3.1; 2011. <http://cran.r-project.org/web/packages/CompModSA/index. html>.