8/7/2019 p557-mckay[1]

1/8

Proceedings of the 1992 Winter Simulation Conferenceed. J. J. Swain, D. Goldsman, R. C. Crain, and J. R. Wilson

LATIN HYPERCUBE SAMPLING AS A TOOL IN UNCERTAINTY ANALYSIS OF COMPUTER MODELS

Michael D. McKayStatistics Group, Los Ahunos National Laboratory

Los Alamos, New Mexieo 87545, U.S.A.

ABSTRACTTlis paper addresses several aspects of the analysis ofuncertainty in the output of computer models arising fromuncertainty in inputs (parameters). Uncertainty of thistype, which is separate and distinct from the randomnessof a stochastic model, most often arises when input valuesare guesstimates, or when they are estimated from data, orwhen the input parameters do not actually correspond toobservable quantit ies, e.g., in lumped-parameter models.Uncertainty in the output is quantified in its probabilitydistribution, which results from treating the inputs asrandom variables. The assessment of which inputs areimportant with respect to uncertainty is done relative tothe probability distribution of the output.

1 INTRODUCTIONUncertainty in a model output-how big it is and whatit is attributable to-is not a new issue. The aecuraeyof models has always been a concern, and things are nodifferent today with computationally intensive models oncomputers. Three things make the analysis of computermodels usually more difficult than that of other models.Fret, the nature of the relationship between the outputand the inputs is often very complex. Seeond, there canbe very many inputs for which the cost of data collectionis high. Finally, when the output is something like a fore-cast, comparison between predicted and calculated valuesis essentially impossible. Despite these difficulties, ques-tions like What is the uncertainty in the calculation?continue to be heard both in scientific circles and in po-litical ones, where the cost of decisions resting on modelcalculations can be high.

Although there are several useful questions onemight ask about computer models, there seems to be atendency to use the same kind of method to answer manyof them. It is unlikely, however, that any single methodof analysis exists that answers all questions irn modelevaluation. Movsover, an examination of typical ques-tions would likely suggest that different kinds of methods

are not only desirable but necessary. To find appropri-ate methods, one needs precise statements of objectives.Unfortunately for many investigators, what begins as anintuitive concept, like uncertainty or sensitivity, can eas-ily end up as an impuxisely stated and misleading ques-tion that suggests an inappropriate method of answeringit. This paper tries to address that problem by providinga framework within which questions of uncertainty andimportance are posed.

This discussion will be limited to particular consid-erations from the diversity of issues comprising modelanalysis. First of all, this paper is not an empirical com-parison or evaluation of methods currently used in theanalysis of computer models. Examples of such stud-ies are Saltelli and Homma (1992), Saltelli and Marivoet(1990), Iman and Helton (1988) and Downing. Gardnerand Hoftinan (1985). Secondly, we will consider ques-tions dealing with the values of the output and the input,and not, specifically, with the form or structure of theirrelationship. In particular, we will focus on the issue ofuncertainty in the calculated value due to uncertainty ininput values. To provide a setting, we will introduce cur-rent methods from two different perspectives related tomodel analysis. With that background, we will developa new and useful paradigm for analysis and methods de-velopment for issues related to uncertainty in the outputvalue.

2 TWO PERSPECTIVESModel analysis can be thought of as a collection ofquestions asked about output and input values, Althougha simplification, it is useful to dMin@h the questionsas arising from one of two perspectives. This idea,previously discussed by McKay (1978, 1988), allowsthe introduction of a new way of viewing importanceof inputs with regards to uncertainty in the output.

The first perspective is from the space of inputvalues, and tends to focus at one point, like a nominalinput value. Because of this, quantities of interest, likea derivative, can seem to be treated as constant over theinput space, so that the focal point really does not matter.

557

8/7/2019 p557-mckay[1]

2/8

558 McKay

This perspective is termed a local perspective relative tothe input space.

The second perspective is from the space of outputvalues. As such, its focus is not constrained a priori inthe input space, so that it is termed a global perspectiverelative to the input space.

The reason for making the distinction between localand global perspectives has to do with the problem ofidentifying important inputs. Although importance hasnot yet been defined, it seems reasonable to suppose thatqualities that make an input important locally are notnecessarily those that make it important globally, and viceversa. Therefore, it is necessary to realize the perspectiveof interest.

2.1 A Local PerspectiveLet us suppose that there is some value of the inputs,~., worth attention and that we are interested in changesin the output Y for small perturbations in inputs Xabout x O. A common question in this situation concernspropagation of error, characterized by the derivatives ofY with respect to the components of X. Similarly,one might be interested in the direction, not necessarilyparallel to a coordinate axis, in which Y changes mostrapidly, or in the change in Y in an arbitrary duection.Issues like these lead one to the concept of critical orimportant variable (or direction) as being one(s) thatmost accounts for change in Y. For propagation of error,it seems to make sense to talk about each component of Xas being important or not important. When the directionbecomes arbitrary, it seems natural to talk about subsetsof the components, rather than about individual ones.

2.2 A Global PerspectiveSuppose that interest lies in the event that Y, the output orprediction, exceeds some specified value, Questions thatcould arise in this case might be concerned with associ-ating particular inputs (components of X) or segments ofranges with that event. Objectives of study for this ques-tion might be related to controlling the event, or withreducing its probability of occurrence in the real worldby adjusting the values of some of the inputs. If costsare associated with the inputs, minimum cost solutionsmight be sought.

Clearly, both perspectives have a place in modelanalysis. In the local perspective, interest in X is re-stricted to a (small) neighborhood of a single point, andthe derivative seems to come into play. In the global per-spective, interest is in values of Y, which might translateinto a subset of, or possibly just a boundary in, the in-put space. In this case, the role of the derivative is lessclear. What tends to blur the distinction between the two

perspectives is the use of the derivative to answer ques-tions of a global nature. The practice is defensible if themodel is essentially linear, meaning that the derivativedoes not change substantially with ZO; or that, to firstorder approximation, an average derivative is sufficientto characterize the model, again meaning that the modelis essentially linear. In what follows, a global approachis taken and the role of the derivative is not paramount.

3 UNCERTAINTY ANALYSISWe are interested in the type of uncertainty in the outputof a model that can be characterized as being due tothe values used for the inputs. A related uncertainty,due to the structure or form of the model itself, will notbe addressed explicitly. Neither will we be concernedwith uncertainty due to errors in implementation of themodel on a computer. On the other hand, it is certainlyacceptable that the output might have the randomness ofa stochastic process. In that case, we will think of theoutput of the model as being the cumulative distributionfunction of the observable output value. With this inmind, the purpose of uncertainty analysis is to quantifythe variability in the output of a computer model due tovariability in the values of the inputs.

We proceed by fist describing the variability in theinputs with probability functions. Commonly, input val-ues are uncertain because they are guesstimates, or whenthey are estimated from data, or when the input parame-ters do not actually correspond to observable quantities,e.g., in lumped-parameter models. Treating the inputs asrandom variables introduces another layer of complica-tion, namely, that of assigning to them probability distri-butions. Everything that follows will depend on the dis-tributions used for the inputs, which means uncertainty inthe input probability distributions leads to correspondinguncertainty in the analysis. As an alternative to quantify-ing that uncertainty, some kind of variational study usedto measure the effect of the distributional assumptions isa possibility.

I When the inputs are treated as random variables, theoutput becomes a random variable because it is a trans-formation of the inputs. Uncertainty in the output, then,is characterized by its probability distribution. Therefore,when we consi&r questions related to uncertainty in theoutput, Y, we will Icok to the probability distribution ofY for answers.

We assume that interest in uncertainty in Y can besummed up by in these two questions: How big is it?and Can it be attributed to particular inputs? An obviousmotivation for these questions is a desire to minimizeuncertainty in the model output, which might be achievedby reducing the variance of some of the inputs. Thus, an

8/7/2019 p557-mckay[1]

3/8

Latin Hypercube Sampling 559

important problem might be how to find a minimum-costreduction in the variance of Y by reducing the variancesof components of X. This problem presupposes, ofcourse, that reduction in the variance of the inputs makessense.

4 MEASURING UNCERTAINTYWe look to the probability dkdribution function of theoutput Y for information about uncertainty. Questionslike What is the uncertainty in Y? might be answeredusing a probability interval constructed with quantiles ofthe distribution of Y. For example, the 0.05 and the0.95 quantiles define an interval covering 90% of theprobability content of the distribution of Y. Alternatively,the difference in the two quantiles provides a range of90% coverage. The use of probability internals as ameasure of uncertainty has au advantage over the useof the variance of the distribution in that the variancemay not directly relate to coverage. This is not thecase, though, in the familiar normal distribution wherequantiles depend in a straightforward manner on the meanand variance of the distribution.

Ideally, the probability distribution of Y would beknown once the distribution of X is specitied. Realis-tically, the distribution will have to be estimated, mostlikely, with a sample of runs using the model which re-lates Y to X. Simple random sampling (SRS) could beused for this purpose, as well as could other samplingschemes. Latin hypercube sampling (LHS), introducedby McKay, Conover and Beckman (1979), is a preferredalternative to SRS when the output is amonotone functionof the inputs. Additionally, Stein (1987) shows that LHSyields an asymptotic variance smaller than that for SRS.Besides being used to estimate the probability distribu-tion, sample values cmdd be used to construct a toleranceinterval, which covers at least a specitied portion of theprobability distribution of Y with a specified confidencelevel. (For a short discussion of probability intemd andtolerance interval, see Tietjen (1986, p, 36).) Generally,a tolerance interval, which corresponds to a probabilityinterval when the probability distribution is estimated, isbased on a random sample. If usual methods for con-structing tolerance intervals based on nonparametric tech-niques or on the normal distribution-e.g,, in Bowker andLiebennan (1972, pp. 309-316) where they are calledtolerance limits-are applied to the LHS, however, theresults are only approximate.

In the remainder of this paper, we will be mnca-nedonly with probabMy distributions and their moments.Furthermore, we will assume that sample sizes are suffi-ciently large to rule out concern about sampling error inall regions of interest in estimated distribution functions.

Although it has been suggested that a probability in-terval is a more appropriate measure of uncertainty thanis variance, the use of variance to partition or allocateuncertainty to components of X cannot be overlooked.In fact, three categories of techniques for the determi-nation of important inputs, relative to uncertainty inthe output, look primarily at the variance of Y. We willreview these techniques using variance as the measureof uncertainty before introducing the new paradigm foruncertainty analysis.

5 PARTITIONING UNCERTAINTYStatements like 20% of the uncertainty in Y is due toXl have an ice sound, but maybe very misleading with-out explanation. If we suppose that uncertainty in Y ismeasured by its variance, then a reasonable interpreta-tion of the statement is that the variance of Y can bewritten, approximately, as the sum of two functions, onedepending on the distribution of X1 alone and the otherindependent of the distribution of Xl. This picture cap-tures a motivation for, but dces not limit, the classes oftechniques to be discussed.5.1 Linear Propagation of ErrorWhen we are using variance to measure uncertainty,the problem of partitioning uncertainty reduces to thatof tiding suitable decompositions for the variance ofY. The simplest of these is the usual propagation oferror method in which Y is expressed as a Taylor seriesin the inputs X about some point XO. To iirst orderapproximation, the variance of Y is expressed as a linearcombination of the variances of the components of X bychoosing zo to be p, the mean value of X.

w(m)) X ~oi) + ~. .Y(x) = Y(xo) + ~ =( ~ a

V[Y] s! ~-) v[xili

When the derivatives of Y are not determined numeri-cally, but estimated by the coefficients from a linear re-gression of Y on X, one seems to be making a strongerassumption about the linear dependence of Y on X.However, it is generally unknown whether the value ofthe actual derivative of Y or the value of au average slopeis preferred in the variance approximation. In a techniquethat could be related to linear propagation of error, Wongand Rabitz (1991) look at the principal components ofthe partial derivative matr ix.

Although not precisely a variance decompmition,correl:iion coefficients have been used to indicate rel-ative importance of the inputs. They are mentioned here

8/7/2019 p557-mckay[1]

4/8

560 McKay

because they are closely related to linear regression coef-ficients. In a similar way, rank transformed values of Yand X have been used for rank correlation and rank re-gression by McKay, Conover and Whiteman (1976), mdIman, Helton and Campbell (1981a, 1981b ). Also,Oblow (1978) and Oblow, Pin and Wright (1986) use atechnique whereby the capability of calculating deriva-tives into the model is added using a precompiled calledGRESS.

5.2 General Analytical ApproximationThe natural extension of linear propagation of error, toadd more terms in the Taylor series, makes it ticult tointerpret variance decomposition component-wise for X.That is, the introduction of cross-prwluct terms bringscross-moments into the variance approximation, whichmakes the approximation no longer separable with respectto the inputs. Nevertheless, one may feel it necessary touse higher order terms in variance approximation. Theadequacy of the approximation to Y might be used asa guide to the adequacy of the variance approximation.However, there is no particular reason to think that oneimplies the other.

Similarly, the linear approximation of Y used in theregression can be generalized to an arbitrary analyticalapproximation from which, in theory, the variance of Ycan be derived either mathematically or through simula-tion. Alternatively, one can use a method proposed bySacks, Welch, Mitchell and Wynn (1989), which looks atthe model as a realization of a stochastic process. Thedifficulties in interpretation and assessing adequacy justmentioned for the higher order Taylor series expansionapply here, tco.

5.3 Sampling MethodsThis tlnal category of partitioning techniques relies on asample (usually, some type of random sample) of valuesof Y whose variability can be partitioned according to theinputs without an apparent assumed functional relationbetween Y and X. In this category is a Fourier methodof Cukier, Levine and Shuler (1978). The proceduresays that values of each component of X are to besampled in a periodic fashion, with different periods foreach component. The variability (sum of squares) ofthe resulting values of Y can be written as a sum ofterms corresponding to the different periods, and thusassociated with the different components. It is unclearhow this relates to linear propagation of error, but itmay be just another way to estimate the same quantities.The original Fourier method applies to continuous inputs.It is extended to binary variables by Pierce and Cukier(1981). Again, the relation to linear propagation oferror is unclear. Another procedure suggested by Morris

(1991) examines a probability distribution of the partialderivatives of the output arising from particular samplingdesigns.

Finally, I mention a partition of variance describedby Cox (1982). Though not actually a sampling method,the elements of the decomposition are likely to estimatedfrom sampled data, in practice, The identity used in-volves the variances of conditional expectations of theoutput given subsets of the inputs. As with general an-alytical approximation, it is not possible to isolate termsfor all the individual components of X.

6 MATHEMATICAL FRAMEWORKThe uncertainty in the output that we focus on is thatattributable to the inputs. Specifically, we are ignoringthe uncertainty in calculations due to the possibility thatthe structure of the model might be deficient. We letY denote the calculated output, which depends on theinput vector, X, of length p through the computer model,h(*). Because proper values of the components of X maybe unknown or imprecisely known, or because they canonly be described stochastically, it seems reasonable totreat X as a random variable and to describe uncertaintyabout X with a probability function. Uncertainty in thecalculation Y is captured by its own probability function,which is what we will study. In summary, then,

Y = h(x)X * f.(z) , XCRP

y w fY(Y) .For now, we will think of fz as known, although inpractice, knowledge about it is at best incomplete.

We look to the probability distribution, f~, for an-swers to the question What is the uncertainty in Y?That is to say, we can use the quantiles of the distribu-tion of Y to construct probability intervals. Alternatively,one might use the variance of Y to quantify uncertainty.In either case, under the assumption that fy can be ad-equately estimated, questions answerable with quantilesor moments are covered. However, as has already beenmentioned, the issue of how well f. is known will surelyhave to be addressed in practice.

We relate questions of importance of inputs to theprobability distribution of Y. That is, we will considerquestions like Which variables really contribute to (orailed) the probability distribution of the output? Whatit means to be important is defied in somewhat of abackwards way as beii the complement of unimportant.We say that a subset of inputs is unimportant if theconditional distribution of the output given the subset isessentially independent of the values of the inputs in thesubset. We now examine these ideas in more detail.

8/7/2019 p557-mckay[1]

5/8

Latin Hypercube Sampling 561

Suppose that the vector X of inputs is partitionedinto Xl and X2. Corresponding to the partition, wewrite Y = h(x)

= h(xl, xz) .Furthermore, we assume that Xl and Az are stochasti-cally independent, meaning that

xi wf~(z~) , i= 1, 2j-x(z) = f,(z~).f2,(z,)

We address the question of the unimportance of X2 bylooking at

.fvl~, = distribution Y given X2 = X2 .as compamd to ~Y. We say that X2 is unimportant if fYrmd fy 1~, are not substantially different for all values ofX2 of interest. Similarly, we say that Xl contains allthe important inputs if XZ is unimportant. Of course, theactual way to compare fv and ~y l., must be determined.

We use the term screening to mean au initialprocess of separating inputs into Xl, potentially impor-tant ones, and X2, potentially unimportant ones, In thenext section, a simple method of partitioning the inputs,following McKay, Beckman, Moore and Pickard (1992),will be discussed.

7 A SIMPLE SCREENING HEURISTICWe now describe a simple, twestep screening process.The first step is to partition X into a set of importantcomponents, Xl, and a set of unimportant components,X2. The second step is a partial validation to estimatehow the components in X2 actually change fY ICz, to beused to decide if X2 is really unimportant.7.1 Partitioning the Input SetWe say that X2, a subset of X, is (completely) unim-portant when the marginal distribution of Y, equals theconditional distribution of Y given X2.

fy = fyi., for ~1 ValW of X2 (1)A way to get an idea of how closely the equality in(1) holds is through the variance expression (2) whichexpresses the marginal variance of fy in terms of theconditional mean and variance of fu 1ZZ. The variance ofY can be written as

V[Y] = E[V[Y I X2]]+ V[E[Y \ X2]] . (2)Equality of the maqjnal and conditional distributions in(1) implies that the conditional mean and variance are

equal to their margimll counterparts for all values of XZ.Specifically, the variance (over XJ of the conditionalexpectation in (2) is zero. It is unlikely, of course, thatany (realistic set) of the inputs is completely unimportant.llhe~fore, the equality between marginal and conditionalquantities will be true only in approximation, with thedegree of approximation linked to the level of acceptanceof the difference between the marginal and conditionaldistributions of the output, Y.

By inference, if Xl, the complement to X2, is (com-pletely, singly) important, the conditional variance of Ygiven Xl is zero, and the variance of the conditional ex-pectation of Y given XI is the marginal variance. Asbefore, these relations usually hold only in approxima-tion. Nevertheless, a comparison of terms in (2) willoffer a way to look at the degree of importance.

The variance decomposition in (2) suggests a relatedidentity from a one-way analysis of variance, in whichthe total sum of squares is written as the sum of twocomponents, a between level component and a withinlevel component. It will be the analysis of varianceapproach we will use to suggest which components of Xbelong in X1 and which in X2. What we will do is touse r replicate Latin hypercube samples of size k. Thesame k values of each component of X will appear ineach replicate but the matching within each one will bedone independently. The k values will correspond to thek levels in the sum of squares decomposition.

In an LHS as introduced by McKay, Conover andBeckman (1979), when the inputs are continuous andstochaatically independent, the range of each componentof X is divided into k intervals of equal probabilitycontent. Simple modifications can be made to handlediscrete inputs (McKay 1988) and dependence (McKay1988, Stein 1987, Iman and Conover 1982). For a trueLHS, a value is selected from each interval accordingto the conditional distribution of the component on theinterval. For this application, it will be sufficient to usethe probability midpoiut of the interval as the value. Thek values for each input are matched (paired) at randomto form k input vectors. For the replicates needed in thisscreening heuristic, r independent matchings of the samevalues are used to produce the n = k x r input vectorsin total.

A design matrix, M, for an LHS is given in (3).Each column contains a random permutation of the kvalues for an input. Each row of the matrix correspondsto a random matcha computer run.

M=

; of values for the p inputs used inVll V12 . . . Vlp 17)21 l&z . . . v2p. . . . (3). . . .. . . .

?)k1 v~z . . . vkp ]

8/7/2019 p557-mckay[1]

6/8

562 McKay

A design matrix for any of the r replicates in this ap-plication is obtained by randomly and independently per-muting the values in every column of M.

After making the necessary n computer runs usingreplicated LHS, we begin by looking at the components ofX one at a time. Let U denote the component of interestiuX, anddenote thekvaluesof Ubyul, UZ, . . . . uk.We label n values of the output as y~j to correspond tothe ith value w, in the jth replicate (sample). The sumof squares partition corresponding to the input U takesthe formf-xYo - ?32=& (R-7)2+;5 (Yo -%)2i=l=l i=l i=l j=l

SST = SSB + SSW

where-r kiji = :~ ~ ~~i ~ijandgdJ=l 2=1

The statistic we have chosen to use to assess theimportance of U is R2 = SSB/SST. Although R2 isbounded between O and 1, the attainment of the boundsis not necessarily a symmetric process. The upper boundis reached if Y depends only on U. In that case, for anyfixed value of U, say u~, the value of Y will also befixed, making SSW equal to O. As a result, R2 will be 1.On the other hand, if Y is completely independent of U,we do not expect SSB (and, therefo~, R) to be O. Wenow examine this last point in more detail.

In general, the probability distribution of R2 willbe unknown. To gain a little insight, however, supposethat we arbitrarily partition a random sample of size nfrom a normal distribution to form R2. (An arbitrarypartition would correspond to Y independent of U.) Theexpected value of R2 is (k I)/(n 1), which goes tozero with k/n as n increases. Thus, one might consider(k I)/(n 1) as a working lower bound associatedwith a completely unimportant input.

Issues that still need to be addressed include theapportionment of n between r and k, the extension of thedesign and decomposition to more that one component ata time, and the interpretation of values of R2.

Whether or not one uses R2 or additional methodsto develop the sets Xl and X2, there remains the issueof evaluating the partition to see how effective it is insatisfying (1). In fact, iterating between a partition andvalidation is what one would do in practice. The nextsection discussion validation.

7.2 Validation of the PartitionVery simply stated, in the validation step we look at Xland X2 and try to assess how well the partition meetsthe objective of isolating the important inputs to Xl. Wepropose using a very elementary sequence of steps thatbegins with a sample design resembling Taguchis (1986)inner array/outer.

1. Select a sample, S2, of the X25 and asample, S1, of the Xls.

2. For each sample element Z2 E S2, ob-tain the sample of Y corresponding to{Z2 @ Sl}.

3. Calculate appropriate statistics for eachsample in Step 2, e.g., T(zz), s~(xz) andFy[z,.

4. Compare the statistics and decide if thedifference X2 makes is acceptable.

The differences seen in the statistics in Step 4 are due onlyto the different values of X2 because the sample valuesfor Xl are the same in each. Hence, the comparisonsare reasonable.

The reliability of any validation procedure needs tobe evaluated. In this case, S2 may not adequately coverthe domain of XZ, particularly as the dimension of X2increases. Merely increasing the size of S2 may not bean acceptable solution if the increase in the number ofruns to generate the sample of Ys becomes impossibleto accommodate. Inadequate coverage can be due to tworeasons. First, regions where the conditional distribu-tion of Y really changes with XZ alone may be missed.Second, there may be regions where the interaction be-tween X2 aud Xl in the model has a significant impact onthe conditional distribution of Y. Although it has obvi-ous deficiencies, LHS is an appropriate sampling methodfor generating S2 because it provides marginal stratifica-tion for each input in X2, meaning that the individualranges within the components likely have been sampledadequately. Whether or not interaction between Xl andX2 will be detected is unknown. As m alternative toLHS, one might use an orthogonal array as described byOwen (1991), which provides marginal stratification forall pairs of input variables.

8 APPLICATIONFor an application of these methods, the reader is referredto McKay, Beckman, Moore and Plckard (1992).

8/7/2019 p557-mckay[1]

7/8

Latin Hypercube Sampling 563

ACKNOWLEDGMENTSThis work was supported by the United States NuclearRegulatory Commission, Office of Nuclear RegulatoryResearch. The author expresses his thanks folr manyvaluable conversations with and help from Richard Beck-man, Richard Pickard, Lisa Moore, Thomas Bement, PaulKvam and other members of the Statistics Group at theLos Alamos National Laboratory.

REFERENCESBowker, A. H. and Lieberman, G. K. (1972). Engineer-

ing Statistics. Prentice-Hall, Englewood Cliffs, NewJersey.

Cox, D. C. (1982). An analytical method for uncertaintyanalysis of nonlinear output functions, with applicationto fault-tree analysis. IEEE Transactions on Reliabi li ty,R-31(5):265-68.

Cukier, R. I., Levine, H. B., and Shuler, K. E. (1978).Nonlinear sensitivity analysis of mukiparameter modelsystems. Journal of Computational Physics, 26:1-42.

Downing, D. J., Gardner, R. H., and Hoffman, F. O.(1985). An examination of response-surface method-ologies for uncertainty analysis in assessment of mod-els. Technometrics, 27(2):151-163.

Iman, R. L. and Conover, W. J. (1982). A distributionfree approach to inducing rank correlation among inputvariables. Communications in StatisticsSirr~ulationand Computation, B 11:311-334.

Iman, R. L. and Helton, J. C. (1988). An investigationof uncertainty and sensitivity analysis techniques forcomputer models. Risk Analysis, 8(1):71-90.

Iman, R. L., Helton, J. C., and Campbell, J. E. (1981a).An approach to sensitivity analysis of computm mod-els: Part Iintroduction, input variable selection andpreliminary variable assessment. Journal of Quali~Technology, 13(3):174-183.

Iman, R. L., Helton, J. C., and Campbell, J. E. (1981b).An approach to sensitivity analysis of computer mod-els: Part II-ranking of input variables, response sur-face validation, distribution effect and technique syn-opsis. Journal of QualiQ Technology, 13(4):232-240.

McKay, M. D. (1978). A comparison of some sensitivityanalysis techniques. Presented at the ORSALUMSannual meeting, New York.

McKay, M. D. (1988). Sensitivity and uncertainty analy-sis using a statistical sample of input values. In Ronen,Y., editor, Uncertainty Analysis, chapter 4, pages 145-186. CRC Press, Bcca Raton, Florida.

McKay, M. D., Beckman, R. J., Moore, L. M., and Picard,R. R. (1992). An alternative view of sensitivity inthe analysis of computer codes. In Proceedings of theAmerican Statistical Association Section on Physicaland Engineering Sciences, Boston, Massachusetts.

McKay, M. D., Conover, W. J., and Beckman, R. J.(1979). A comparison of three methods for selectionvalues of input variables in the analysis of output froma computer code. Technometrics, 22(2):239-245.

McKay, M. D,, Conover, W, J., and Whiteman, D. E.(1976). Report on the application of statistical tech-niques to the analysis of computer codes. TechnicalReport LA-NUREG-6526-MS, Los Alamos NationalLaboratory, Los Alamos, NM.

Morris, M. D. (1991). Factorial sampling plans for pre-liminary computational experiments. Technometrics,33(2):161-174.

Oblow, E. M. (1978). Sensitivity theory for reactorthermal-hydraulics problems. Nuclear Science andEngineering, 68:322-337.

Oblow, E. M., Pm, F. G., and Wright, R. Q. (1986). Sen-sitivity analysis using computer calculus: A nuclearwaste isolation application. Nuclear Science and En-gineering, 94:46-65.

Owen, A. B. (1992). Orthogonal arrays for computerintegration and visualization. Statistic Sinica, 2(2).

Pierce, T. H. and Cukier, R. I. (1981). Global nonlinearsensitivity analysis using Walsh functions. Journal ofComputational Physics, 41:427-43.

Sacks, J., Welch, W. J., Mitchell, T. J,, and Wynn, H. P.(1989). Design and analysis of computer experiments.Statistical Science, 4(4):409-435,

Saltelli, A. and Homma, T. (1992). Sensitivity analy-sis for a model outpu~ Performance of black boxtechniques on three international benchmark exercises.Computational Statistics & Data Analysis, 13:73-94.

Saltelli, A. and Marivoet, J. (1990). Non-parametric sta-tistics in sensitivity analysis for model outputi A com-parison of selected techniques. Reliabi li ty Engineeringand System Safety, 28:229-53.

Stein, M. (1987). Large sample properties of simula-tions using Latin hypercube sampling. Techrwmetrics,29(2):143-151.Taguchi, G. (1986). Introduction to Quality Engineering.Kraus International Publications, White Plains, NewYork.

Tietjen, G. L. (1986). A Topical Dictionary of Statistics.Chapman and Hall, New York.

Wong, C. F. and Rabitz, H. (1991). Sensitivity analysisand principal component analysis in free energy cal-culations. Journal of Physics and Chemistiy, 95:9628-9630.

8/7/2019 p557-mckay[1]

8/8

564 McKay

AUTHOR BIOGRAPHY Laboratory for 19 years. His current research interestsinclude uncertainty analysis of computer models and the

MICHAEL D. MCKAY has been a technical staff mem- tmatment of categorical variables as random effects inber in the Statistics Group at the Los Alamos National ordinary regression models.