This article was downloaded by: [North Carolina State University] On: 28 May 2014, At: 15:19 Publisher: Taylor & Francis Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK Technometrics Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/utch20 Integrated Analysis of Computer and Physical Experiments C. Shane Reese a , Alyson G Wilson b , Michael Hamada b , Harry F Martz b & Kenneth J Ryan c a Department of Statistics, Brigham Young University, Provo, UT 84602 b Statistical Sciences Group, Los Alamos National Laboratory, Los Alamos, NM 87545 c Department of Statistics, University of Illinois, Chicago, IL 60208 Published online: 01 Jan 2012. To cite this article: C. Shane Reese, Alyson G Wilson, Michael Hamada, Harry F Martz & Kenneth J Ryan (2004) Integrated Analysis of Computer and Physical Experiments, Technometrics, 46:2, 153-164, DOI: 10.1198/004017004000000211 To link to this article: http://dx.doi.org/10.1198/004017004000000211 PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content. This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions
Transcript
This article was downloaded by: [North Carolina State University]On: 28 May 2014, At: 15:19Publisher: Taylor & FrancisInforma Ltd Registered in England and Wales Registered Number: 1072954 Registered office:Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK
TechnometricsPublication details, including instructions for authors and subscriptioninformation:http://www.tandfonline.com/loi/utch20
Integrated Analysis of Computer and PhysicalExperimentsC. Shane Reesea, Alyson G Wilsonb, Michael Hamadab, Harry F Martzb & Kenneth JRyanc
a Department of Statistics, Brigham Young University, Provo, UT 84602b Statistical Sciences Group, Los Alamos National Laboratory, Los Alamos, NM87545c Department of Statistics, University of Illinois, Chicago, IL 60208Published online: 01 Jan 2012.
To cite this article: C. Shane Reese, Alyson G Wilson, Michael Hamada, Harry F Martz & Kenneth J Ryan(2004) Integrated Analysis of Computer and Physical Experiments, Technometrics, 46:2, 153-164, DOI:10.1198/004017004000000211
To link to this article: http://dx.doi.org/10.1198/004017004000000211
PLEASE SCROLL DOWN FOR ARTICLE
Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”)contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensorsmake no representations or warranties whatsoever as to the accuracy, completeness, or suitabilityfor any purpose of the Content. Any opinions and views expressed in this publication are the opinionsand views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy ofthe Content should not be relied upon and should be independently verified with primary sourcesof information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings,demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arisingdirectly or indirectly in connection with, in relation to or arising out of the use of the Content.
This article may be used for research, teaching, and private study purposes. Any substantialor systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, ordistribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can befound at http://www.tandfonline.com/page/terms-and-conditions
Alyson G. WILSON, Michael HAMADA,and Harry F. MARTZ
Statistical Sciences GroupLos Alamos National Laboratory
Los Alamos, NM 87545
Kenneth J. RYAN
Department of StatisticsUniversity of IllinoisChicago, IL 60208
Scienti� c investigations frequently involve data from computer experiment(s) as well as related physicalexperimental data on the same factors and related response variable(s). There may also be one or moreexpert opinions regarding the response of interest. Traditional statistical approaches consider each of thesedatasets separately with corresponding separate analyses and � tted statistical models. A compelling argu-ment can be made that better, more precise statistical models can be obtained if the combined data areanalyzed simultaneously using a hierarchical Bayesian integrated modeling approach. However, such anintegrated approach must recognize important differences, such as possible biases, in these experimentsand expert opinions. We illustrate our proposed integrated methodology by using it to model the thermo-dynamic operation point of a top-spray � uidized bed microencapsulation processing unit. Such units areused in the food industry to tune the effect of functional ingredients and additives. An important thermo-dynamic response variable of interest, Y, is the steady-state outlet air temperature. In addition to a setof physical experimental observations involving six factors used to predict Y, similar results from threedifferent computer models are also available. The integrated data from the physical experiment and thethree computer models are used to � t an appropriate response surface (regression) model for predicting Y.
Computer models are often used to perform experiments be-fore expensive physical experiments are undertaken. The com-puter models attempt to reproduce the physical properties ofa process by mathematically representing the individual physi-cal subprocesses. For example, in the food industry, � uidized-bed (or air-suspension) processes are increasingly used to coatfood particles with preservatives and � avor enhancers. Someof the physical principles that govern the operation of � u-idized beds are fairly well understood (e.g., heat transfer and� uid � ow), but others are less well characterized. As a result,computer models based on these thermodynamic principles ofphysics are constructed that resemble and simulate the actualphysical process. In this article we analyze data collected fromthree such computer models (with each model accounting fordifferent effects), as well as data collected from a correspond-ing physical experiment. We consider this example further inSection 3.
It is statistically ef� cient and desirable to � t a single com-mon response surface model that combines the physical exper-imental data and the computer model output data to express therelationship between the factors and the response variable. Al-though the response variables of interest in the computer andphysical experimentsmay not be the same, we assume that theycan be related by a known transfer function. Thus we effec-tively consider the same response variable in both types of ex-periments. However, the computed (or measured) value of theresponse variable need not be considered at the same factor val-ues in both experiments. We require only that there exist some
common set of factors (either all or at least some) for both ex-periments (see Sec. 2.3). For example, a broad (screening) com-puter experimentmay be performed � rst, followedby a physicalexperiment in a smaller region of particular interest (perhaps acorner) of the overall computer experiment design space.
In addition, one or more expert opinions may be availableregarding the response variable of interest. Traditional statisti-cal approaches consider each of these datasets separately withcorresponding separate designs, analyses, and results. A com-pelling argument can be made that better, more powerful statis-tical results can be obtained if we simultaneously analyze thecombined data using a recursive Bayesian hierarchical model(RBHM) that we propose in Section 2. As we illustrate, thesimultaneous analysis of such combined data permits the un-known coef� cients in an assumed overall regression (or re-sponse surface) model to be estimated more precisely, therebyproducing a better-� tting response surface.
In Section 2 we present the methodology, including our im-plementation of the RBHM. In Section 3 we describe the me-chanics and process variables involved in the � uidized bedexample and the experiment from which the data arise. Weapply the RBHM methodology to the � uidized bed study andpresent the resulting response surface in Section 3.3. We dis-cuss sensitivity to prior speci� cation in Section 4 and the resultsand methodology in Section 5.
TECHNOMETRICS, MAY 2004, VOL. 46, NO. 2DOI 10.1198/004017004000000211
153
Dow
nloa
ded
by [
Nor
th C
arol
ina
Stat
e U
nive
rsity
] at
15:
19 2
8 M
ay 2
014
154 C. SHANE REESE ET AL.
2. DATA INTEGRATION MODEL AND ANALYSIS
Fundamental to Bayesian estimation is the notion and use ofprior and posterior distributions. A good elementary discussionof prior and posterior probabilities and distributions was givenby Berry (1996). An RBHM provides a convenient way to se-quentially combine the data as follows. Initial informative, butdiffuse, prior distributions are de� ned, one for each unknownparameter. Any available expert opinion data that exist are thenused to update these priors to form correspondingposterior dis-tributions. This represents stage 1 of the combined analysis.These posteriors then become the prior distributionsfor the sec-ond stage, in which the computer experimental data are used toupdate these priors to form stage 2 posterior distributions. Atstage 2, the posteriors thus represent the combined use of onlythe expert opinion and computer data. Finally, these posteriorsbecome the priors for stage 3, in which the physical experimen-tal data are used to construct the � nal desired posteriors. In thisway, all available data are used recursively within the contextof the model to successively (and more precisely) estimate allof the desired parameters of interest.
The design and analysisof computer experimentshas evolvedas the power of computers has grown (although it has cer-tainly not kept pace). Sacks, Welch, Mitchell, and Wynn (1989)provided a review of techniques used in the analysis of out-put from complex computer codes, as well as issues for de-sign. Latin hypercube sampling had its genesis in the design ofcomputer experiments (McKay, Beckman, and Conover 1979).A Bayesian treatment of the design and analysis of computerexperiments was presented by Currin, Mitchell, Morris, andYlvisaker (1991). These authors were concerned primarily withissues when the only source of information is the output froma complex computer model.
Combining multiple sources of information had its genesisin the meta-analytic literature. Zeckhauser (1971) provided anearly treatment of meta-analysis, and Hedges and Olkin (1987)provided a nice review of meta-analytic techniques. Meta-analysis has not been viewed without strong criticism (Shapiro1994). M Ruller, Parmigiani, Schildkrout, and Tardella (1999)presented a Bayesian hierarchical modeling approach for com-bining case-control and prospective studies, where effects dueto different studies as well as different centers are allowed.
Craig, Goldstein, Rougier, and Scheult (2001) presented anapproach to forecasting from computer models that explicitlyincorporates two of the data sources that we consider, expertopinion and computer experiments. They considered the possi-bility of multivariate responses on the computer model (whichthey called computer simulators). Physical data in the form ofhistorical measurements are included by using this informationin prior (expert opinion speci� cation). Their approach is basedon a Bayesian treatment with no hierarchical modeling and in-ventive ways of including several types of expert opinion. Theprimary concern is improving prediction of the computer code.
Kennedy and O’Hagan (2001) considered the three sourcesof data that we consider in this article. Their approach uses ageneral Gaussian process model for the computer model as afunctionof inputs.They used physical data to calibrate the com-puter experimentaldata and to estimate unknown parameters ofthat model. They also found Bayesian hierarchical models to be
a useful tool in implementing their models. Their framework is� exible and, in the context of trying to improve computer mod-els, the appropriate approach. The essential difference betweentheir work and our proposed approach is that we are trying touse computer model outputs and expert opinion to improve es-timation and prediction of the physical process, and Kennedyand O’Hagan were trying to use physical experimentaldata andexpert opinion to improve the computer model.
The statistical notion of pooling data (sometimes also knownas “borrowing strength”) underliesour discussion of the RBHMand analysis. A commonly used and extremely powerfulmethod for borrowing strength is hierarchical Bayesian model-ing; a nice introduction to both hierarchical Bayesian modelingand borrowing strength was given by Draper et al. (1992). Thebasic idea involves the notion that when information concern-ing some response of interest arises from several independentbut not identical data sources, a hierarchical model is often use-ful to describe relationships involving the observed data andunobserved parameters of interest. For example, unobservedparameters might be the coef� cients and error variance in anassumed response surface model, as well as unknown biases.Each source of data provides perhaps biased information aboutthese parameters, in which case methods that borrow strengthwill be useful. We illustrate the practical advantages of bor-rowing strength for estimating the unknown parameters in Sec-tion 3.2.
We propose � tting models using information from threedistinct sources: expert opinion, computer experiments, andphysical experiments. The problem is dif� cult, because theinformation sources are not necessarily all available at eachof the design points. For example, physical experiments maybe performed according to a statistically designed experiment,whereas computer experiments may be collected at (possibly)different design points. In addition, expert opinions may beavailable at only a very limited set of design points, such asthe center or corners of the statistical design region. Our goal isto combine these sources of information using an appropriately� exible integration methodology that considers (and automati-cally adjusts for) the uncertainties and possible biases in eachof these three data sources.
Thus we begin by considering regression models of the form
Y D f .X; ¯/ C ";
where X is a design matrix, ¯ is a vector of unknown coef-� cients, and " is a vector of unobserved errors. Note that al-though this formulation can accommodate a general class ofmodels, f .¢/, that includes both linear and nonlinear regressionmodels, here we consider only linear models [i.e., f .X; ¯/ DX¯]. Although the strategy that we use is quite general, themodel and mathematics that we develop is applied to a normallinear model. In addition, we consider only quantitative vari-ables, although qualitative variables coded with indicator vari-ables � t naturally into this framework.
2.1 Physical Experimental Data
We assume that we are interested in estimating the parame-ters of a model that describes a physical experiment. For thisexample, assume that the physical experimental data can be de-scribed using the familiar model
Yp » N.Xp¯; ¾ 2I/;
TECHNOMETRICS, MAY 2004, VOL. 46, NO. 2
Dow
nloa
ded
by [
Nor
th C
arol
ina
Stat
e U
nive
rsity
] at
15:
19 2
8 M
ay 2
014
INTEGRATED ANALYSIS OF COMPUTER AND PHYSICAL EXPERIMENTS 155
where the subscript p denotes the “physical experiment.” Thusthe physical experimental data are assumed to be normally dis-tributed with mean Xp¯ , where Xp is a model matrix and ¯ isa vector of parameters that need to be estimated. We see thateach physical observation is independent of the others and hascommon (homoscedastic) variance ¾ 2, which also must be es-timated.
If physical experimental data were the only informationsource considered, then this model would typically be � t us-ing either standard least squares regression methods (Draperand Smith 1998) or standard Bayesian linear model methods(Gelman, Carlin, Stern, and Rubin 1995). However, we want toincorporate information both from experts and computer exper-imental data to “improve” our estimates of ¯ and ¾ 2.
2.2 Expert Opinion
Suppose that there are e expert opinions. These opinionsdo not have to be from distinct experts. The ith expert opin-ion (i D 1; : : : ; e) is elicited at design point xi . Some points inthe design space will have exactly one elicited expert opinion,whereas others will have many or none. Each expert observa-tion contains the following information:
² The expected response, yoi .² A subjectivecoverage probabilityon the physical response
yi, »i, and the quantile associated with that probability,q»i ;that is, Pr. yi · q»i/ D »i .
Typically, the analyst elicits a quantile of interest; that is, »i isspeci� ed. However, the expert may indicate which quantile heor she is most interested in specifying. The methods developedhere do not depend on which approach is taken. In addition,weconsider the elicited “worth” of the opinion in units of equiv-alent physical experimental data observations, m.e/
oi . In otherwords, suppose that a physical experiment could be conductedat xi that would yield one observation; if the expert’s opinionshould be weighted half of that observation, then m.e/
oi D :5. Attimes, the elicited values ( yoi , »i, q»i , m.e/
oi ) may be obtainedsimply by requesting them from the expert. However, it may bedif� cult for the expert to provide information directly on thesevalues (especially q»i and m.e/
oi ), and other elicitation techniquesmay be useful (Meyer and Booker 1990).
To use these data, we need to transform these individualpieces of information into probabilitydistributions that provideinformation about ¯ and ¾ 2. Assume for the moment that thethree aforementioned quantities can be used to create “data”with the following model:
Yo » N.Xo¯ C ±o; ¾ 26o/:
Like the physical experimental data, the expert data areassumed to be normally distributed. However, the mean isXo¯ C ±o, where ±o is a vector of location biases that areexpert-speci� c. The variances are also biased, and the matrix6o contains the scale biases for each expert. Besides locationbiases, in which an expert’s average value is high or low rela-tive to the true mean, scale biases often occur due to informationovervaluation and are well documented in the elicitation litera-ture. For example, an expert may be asked to provide what he
or she thinks is a .90 quantile, but responds with what is ac-tually only a .60 quantile (Meyer and Booker 1990). Althoughresponses from experts can be correlated by having nondiago-nal elements in 6o, we consider uncorrelated responses; thus
6o D
2
664
1=ko1 0 ¢ ¢ ¢ 00 1=ko2 0 ¢ ¢ ¢::: 0
: : : ¢ ¢ ¢0 ¢ ¢ ¢ ¢ ¢ ¢ 1=koe
3
775 :
In addition, we assume the following prior distributions forthe unknown parameters ¯ and ¾ 2:
¯j¾ 2 » N.¹o; ¾ 2Co/
and¾ 2 » IG.®o; °o/;
where IG.a;b/ is the inverse gamma distribution with densityfunction
f .zja;b/ / z¡.aC1/ exp
»¡
b
z
¼; z > 0:
Assume for the moment that we know ±o and mo, wheremo is a vector denoting the “worth” of the expert opinions.Con-tinue to assume that we have created “data” yo from the expertopinions, and write out the likelihood for the data model,³
1
j¾ 26oj:5
´
£ exp
»¡ 1
2¾ 2
£¡yo ¡ .Xo¯ C ±o/
¢06¡1
o
¡yo ¡ .Xo¯ C ±o/
¢¤¼:
Using Bayes’s theorem, we can use the data provided by theexpert opinions to update the prior distributions for ¯ and ¾ 2.The resulting stage 1 posterior/updated prior distribution for.¯; ¾ 2/, conditional on ´ D .±o;6o;mo;Co;¹o;®o; °o/, is
¼.¯j¾ 2;´; yo/
» N¡.X0
o6¡1o Xo C C¡1
o /¡1z; ¾ 2.X0o6¡1
o Xo C C¡1o /¡1
¢
and
¼.¾ 2j´;yo/
» IG
³®o C
PeiD1 moi
2;
°o C :5£.yo ¡ ±o/06¡1
o .yo ¡ ±o/
C ¹0oC¡1
o ¹o ¡ z0.X0o6¡1
o Xo C C¡1o /¡1z
¤´;
where z D X0o6¡1
o .yo ¡ ±o/ C C¡1o ¹o .
Given that the full vector of observations yo was not elicited(only suf� cient statistics were), we cannot immediately eval-uate any term in these expressions. We instead reexpress thecomponents in these posterior distributions in terms of theelicited values, so they can be evaluated. Suppose that moi ob-servations were elicited as yoi from the ith expert opinion.Then¡X0
o6¡1o .yo ¡ ±o/
¢j
D ko1 x1j
Á mo1X
nD1
¡yojn ¡ ±o1
¢!
C ¢ ¢ ¢ C koexej
Á moeX
nD1
¡yojn ¡ ±oe
¢!
D ko1 mo1 x1j¡yo1 ¡ ±o1
¢C ¢ ¢ ¢ C koemoexej
¡yoe ¡ ±oe
¢;
TECHNOMETRICS, MAY 2004, VOL. 46, NO. 2
Dow
nloa
ded
by [
Nor
th C
arol
ina
Stat
e U
nive
rsity
] at
15:
19 2
8 M
ay 2
014
156 C. SHANE REESE ET AL.
because yoi is the expected or average response for the designpoint.
Using a similar argument, we can show that
.yo ¡ ±o/06¡1o .yo ¡ ±o/
D ko1
Á mo1X
nD1
¡yojn ¡ ±o1
¢2
!
C ¢ ¢ ¢ C koe
Á moeX
nD1
¡yojn ¡ ±oe
¢2
!
DeX
iD1
koimoi
¡s2i C
¡yoi ¡ ±oi
¢2¢; (1)
where s2i D . yoi ¡ q»i/
2=Z2» , which is the variance approxima-
tion implicitly elicited from expert i. Equation (1) follows fromthe identity var.Y/ D E[Y2] ¡ E[Y]2.
By a similar argument,
.X0o6¡1
o Xo/ij DeX
nD1
knmon xnixnj:
These representations allow calculation of the quantities in theposterior distributions based on the elicited values rather thanon the actual observations.
For the unknown parameters ´ D .±o;6o; mo;Co;¹o;
®o; °o/, we propose the following prior distributions:
¹o D a¹o;
Co D aCoI;
®o D a®o;
°o D a°o ;
moi » uniform¡:5m.e/
oi; 2:0m.e/
oi
¢;
±oi
iid» N.µo; »2o /;
µo » N¡mµo ; s2
µo
¢;
»2o » IG
¡a» 2
o;b» 2
o
¢;
koi
iid» G.Áo;!o/;
Áo » G¡aÁo; bÁo
¢;
and!o » G
¡a!o ;b!o
¢;
where a and b subscripted indicate constants, and G.a;b/ in-dicates a gamma distribution with mean ab and variance ab2.These highly parametric speci� cations suggest that sensitivitymay result from choices of distributional form as well as hyper-parameter choices. As with any analysis, increasing the degreeof assumption increases the potential for sensitivity to those as-sumptions. For example, inadequate sample sizes will certainlyexacerbate these sensitivities. In Section 4 we consider a sensi-tivity study to examine the degree to which our results dependon the foregoing hyperparameter choices.
There are similarities between this approach to the quanti� -cation of expert opinion and Zellner’s approach using g-priordistributions (Zellner 1986; Agliari and Parisetti 1988). Bothapproaches rely on the natural conjugate prior for .¯; ¾ 2/.However, Zellner (1986) elicited posterior means for ¯ and ¾ 2 ,
whereas we elicit predicted observations yo. Agliari andParisetti (1988) extended Zellner’s methods to include a dif-ferent design matrix, XA; similarly, we do not require that thefactor levels where the expert elicitation occurs correspond tothe levels where the physical or computer experimentaldata arecollected.
2.3 Computer Experimental Data
We have used the expert opinion data to develop stage 1 pos-terior distributions for ¯ and ¾ 2. We continue to update ourknowledge about these parameters using data from computerexperiments. Let the computer data and associated model para-meters be indexed by c, where the jth element of the responsevector Yc is ycj . Consider the following model:
Yc » N.Xc¯ C ±c; ¾ 26c/;
¯j¾ 2 » N.¹c; ¾ 2Cc/;
and¾ 2 » IG.®c; °c/:
For this development, assume that 6c and Cc have the samediagonal form as 6o and Co. The “prior” distributions for¯j¾ 2 and ¾ 2 are the stage 1 posterior distributions given theexpert opinion data. The only other unspeci� ed prior distribu-tions are
±cj
iid» N.µc; »2c /;
µc » N¡mµc ; s2
µc
¢;
»2c » IG
¡a» 2
c;b» 2
c
¢;
kcj
iid» G.Ác;!c/;
Ác » G¡aÁc ;bÁc
¢;
and!c » G
¡a!c; b!c
¢:
Although assuming a diagonal structure for 6c yields amodel for the computer experiment where the observationsare conditionally independent given ¯ , ±c , ¾ 2 , and kc , theobservations are not unconditionally independent once the un-certainty in the unknown parameters is integrated out. For ex-ample, Broemeling (1985) derived the distribution for Yc forthe conjugate Bayesian linear model. Our model for the cor-relation structure differs from those proposed by Currin et al.(1991) and Welch et al. (1992), who assumed a distance-basedparametric form for 6c with the parameters selected usingcross-validation or maximum likelihood estimation. Althoughthese forms of prior distribution could be incorporated into ouranalysis, we have chosen to induce correlation through the hi-erarchical structure of the prior.
Computer models, especially when the physical processesare not well known, often produce estimates that are biasedwith respect to the physical data. These biases may be in themean structure (location bias) or in the variance (scale bias).Computer experimental data are especially likely to have scalebiases, because these data usually tend to be less variable thanphysical experimental data; in fact, there is often no stochastic
TECHNOMETRICS, MAY 2004, VOL. 46, NO. 2
Dow
nloa
ded
by [
Nor
th C
arol
ina
Stat
e U
nive
rsity
] at
15:
19 2
8 M
ay 2
014
INTEGRATED ANALYSIS OF COMPUTER AND PHYSICAL EXPERIMENTS 157
variability for given values of the factors, because a computercode is often deterministic. The variability occurs relative tothe assumed model. Another reason for the reduced variabil-ity relative to physical experimental data is that we know thatnot all factors generating the physical experimental data are in-corporated into the computer code—perhaps all of the factorscausing variability are unknown. Because the location bias ad-dresses only differences in the intercept term (¯0) between thecomputer and physical data, more general bias structures forthe parameters also can be modeled. In Section 3 we motivatethese ideas by introducing the operation of � uidized beds andthe computer models for that process.
Because the location biases are additive (instead of multi-plicative), the model only requires that data exist for a subset ofthe full set of factors. That is, if only one data source includesinformation on a factor, then only that source is used in esti-mating that effect. The precision with which those effects areestimated will be affected by the differing amounts of data usedin estimation. However, distributions can be calculated. If themodel is to be chosen based on the physical data only (as inour example), then all of the factors would need to be presentin the physical experimental data. Thus the framework is quitegeneral and does not require that all factors be present in eachdata source.
Other approaches that might be considered for modeling thecomputer experimental data often use a Gaussian process (GP)model (Santner, Williams, and Notz 2003). Although the GPapproach is commonly (and appropriately)used for many prob-lems, the RBHM provides an alternative that is useful and eas-ily interpreted for certain classes of problems. The bene� ts ofa linear models approach as outlined in the RBHM are that it iscomputationallytractable, easily interpretable,and easy to visu-alize. Disadvantages of using a linear models approach includethat they cannot act as an interpolator (whereas GP models havethis feature, which explicitly acknowledges the deterministicnature of computer experiments), they are not as � exible asGP models, and they require more observations when higher-order terms are needed in the model. Considering these bene� tsand limitations, researchers must ascertain the suitability of thisformulation (or any other modeling approach) when combiningdata sources that are diverse, such as computer experiments andphysical experiments.
In Section 3 we illustrate a problem that is well suited for ourproposed modeling approach (RBHM).
2.4 Incorporating Physical Experimental Data
Recall from Section 2.1 that the model for the physical ex-perimental data is
Yp » N.Xp¯; ¾ 2I/:
After incorporating the computer experimental data into theanalysis, we have a stage 2 posterior that is used as the priorfor .¯; ¾ 2/ in the stage 3 analysis.
The stage 3 analysis calculates the � nal distributions for theparameters of interest. These calculations cannot be done inclosed form, but are carried out using Markov chain MonteCarlo (MCMC). The Appendix provides general informationon MCMC and the Metropolis–Hastings algorithm.
3. APPLICATION OF RECURSIVE BAYESIANHIERARCHICAL MODELS TO FLUIDIZED
BED PROCESSES
Fluidized-bed microencapsulation processes are used in thefood industry to coat certain food products with additives.Dewettinck, Visscher, Deroo, and Huyghebaert (1999) de-scribed a physical experiment and several corresponding ther-modynamic computer models developed for predicting thesteady-state thermodynamicoperation point of a Glatt GPCG-1� uidized-bed unit in the top-spray con� guration. Figure 1 il-lustrates the simple geometry of this unit, which is essentiallyan upside-down truncated cone. The base of the unit containsa screen, below which is an air pump. Also, there are coatingsprayers at the side of the unit.
To use the unit, a batch of uncoated food product is placedinside the “cone,” and the air pump and coating sprayers areturned on. This “� uidizes” the product in the unit and coats theproduct as it passes by the sprayer. This is continued until thedesired coating thickness is achieved.
When room conditions and process conditions are constant,a � uidized-bed process will attain its steady-state thermody-namic operation point. This state can be described in terms ofthe temperature and humidity inside the unit. The importanceofthe steady-state operation point is that product characteristics,such as coating evenness and ef� ciency, are directly related toit.
Several variables potentially affect the steady-state thermo-dynamic operating point:
² Vf , � uid velocity of the � uidization air² Ta, temperature of the air from the pump² Rf , � ow rate of the coating solution² Ts, temperature of the coating solution² Md , coating solution dry matter content² Pa , pressure of atomization air.
The ambient room conditions inside the plant, such as temper-ature .Tr/ and humidity .Hr/, may also have an effect on thesteady-state process conditions.
3.1 The Data
Dewettinck et al. (1999) considered 28 process conditionsof particular interest (settings) for a GPCG-1 � uidized-bedprocess. In the experiment, distilled water was used as the coat-ing solution. Thus Md was 0 (no dry matter content) for all28 runs. Also, Ts was at room temperature (about 20±C) for
Figure 1. A Glatt GPCG-1 Fluidized Bed Unit.
TECHNOMETRICS, MAY 2004, VOL. 46, NO. 2
Dow
nloa
ded
by [
Nor
th C
arol
ina
Stat
e U
nive
rsity
] at
15:
19 2
8 M
ay 2
014
158 C. SHANE REESE ET AL.
Table 1. Process Variables
Hr (%) T r (±C) T a( ±C) Rf (g=min) Pa(bar) Vf (m=s)
all 28 runs. Table 1 gives the room conditions (i.e., Tr and Hr)and settings for the remaining four process variables (i.e., Ta ,Rf , Pa, and Vf ). Thus the six factors actually studied areTr; Hr;Ta; Rf ;Pa, and Vf .
For each factor combination,glass beads were put in the unit,and the process was run for 15 minutes to attain steady state.Then temperature inside the unit was measured at 20, 25, and
Table 2. Experimental and Computer ModelSteady-State Temperatures
30 minutes and their average was recorded. The average outletair temperature (the steady-state response of interest), T2;exp , isreported in Table 2. Dewettinck et al. (1999) also consideredthree unique computer models to predict the steady-state outletair temperature for each run. These computationalresponses arealso given in Table 2, denoted by T2;1, T2;2, and T2;3.
There are important differences among the three computa-tional models described in detail by Dewettinck et al. (1999).In summary, the � rst computer model does not include ad-justments for heat losses in the process. The second computermodel takes those heat losses into account. A further adjust-ment for the inlet air� ow represents the fundamental differencebetween the second and third computer models.
3.2 Modeling T 2,exp in Terms of Room andProcess Conditions
Table 3 shows the correlation matrix for the room condi-tions, process conditions, and observed steady-state tempera-ture T2;exp . Figure 2 is a matrix plot of these seven variables.Note that Ta has the highest correlation with T2;exp (r D :73).
Model choice is complicated by the fact that the underlyingdesign is not at all clear. The covariance matrix reveals thatsome of the covariates are highly correlated (as high as .82) in-dicating possible collinearity.We also note that the full second-order model is fully saturated.
Chipman, Hamada, and Wu (1997) described a Bayesianvariable selection procedure that places hierarchical prior distri-butions on second-order effects. In their approach, higher priorprobability is given to interactions if one of the main effects isin the model, and an even higher probability is placed on in-teractions when both main effects are in the model. Using theirapproach on the physical data, we obtain the variable selectionresults displayed in Table 4, which provide the most likely mod-els and their respective posterior probabilities.
To illustrate the RBHM approach, we use the most likelymodel from Table 4 to form X¯ , where X is composed of a col-umn of 1’s (for the intercept) and columns corresponding to Ta,Rf , Vf , and Rf £ Vf , whose respective regression parameters
Table 4. Bayesian Variable Selection Results
Model Pr(model | data)
¯1Ta C ¯2Rf C ¯3Vf C ¯4Rf £ Vf .1169¯1Ta C ¯2Rf C ¯3Vf C ¯4Hr £ Tr C ¯5Rf £ Vf .0349¯1Ta C ¯2Rf C ¯3Vf C ¯4H2
r .0155¯1Ta C ¯2Rf C ¯3Vf C ¯4Tr £ Ta C ¯5Rf £ Vf .0141¯1Ta C ¯2Rf C ¯3Vf C ¯4Rf £ Vf C ¯5V2
f .0136¯0 C ¯1Ta C ¯2Rf C ¯3Vf C ¯4H2
r .0132¯1Tr C ¯2Ta C ¯3Rf C ¯4Vf C ¯5Rf £ Vf .0130
TECHNOMETRICS, MAY 2004, VOL. 46, NO. 2
Dow
nloa
ded
by [
Nor
th C
arol
ina
Stat
e U
nive
rsity
] at
15:
19 2
8 M
ay 2
014
INTEGRATED ANALYSIS OF COMPUTER AND PHYSICAL EXPERIMENTS 159
Figure 2. Scatterplot Matrix of the Experimental Response With Each of the Six Covariates.
are ¯ D .¯0; : : : ; ¯4/0. Table 5 contains the OLS � t of the mostlikely model given in Table 4.
The hyperparameter values used in our example are given inTable 6. Note that we used the same hyperparameters for allthree computer experiments. Because we have no prior knowl-edge as to the sign of the location bias, we center the dis-tribution of ±ci at 0 (i.e., unbiased in location) and allow themean of that distribution to have a standard deviation of 10.Although we believe that the computer models are all reason-ably good approximations of the physical model, we do nothave a good idea about the degree of separation, and thus al-low a generous variability for the location biases [a» 2
cD 2,000
and b»2c
D 3.0 suggest a mean for the variance distribution of2,000=.3 ¡ 1/ D1,000 and a standard deviation of 2,000=..3 ¡1/ ¢ .3 ¡ 2// D 1,000]. The distribution of scale biases is alsosomewhat unknown. With little or no prior knowledge, wewould allow the mean of the scale biases to be unity (unbiasedin scale). Further, we believe that the standard deviation of thescale biases should be no greater than 15, and thus we let the
Table 5. OLS Fit for T2,exp D ¯0 C ¯1(Ta ¡ NTa) C ¯2(Rf ¡ NRf ) C¯3(Vf ¡ NVf ) C ¯3 ((Rf ¡ NRf ) £ (Vf ¡ NVf )) C ²
Parameter Standard T for H0 :Variable DF estimate error Parameter D 0 Pr > |T|
Figures 3(a)–(e) show the posterior for ¯ with only the phys-ical experimental data, the physical data with the computerexperimental data taken separately, and the � nal posteriordistribution for ¯ after incorporating all sources of informa-tion. Figure 3(f ) shows the corresponding posteriors for ¾ 2.The � gures indicate two important and appealing aspects of ourRBHM approach. First, the additional sources of informationreduce uncertainty in the distribution of the parameters, thusmaking our estimates more precise. Second, the additionaldata
Table 6. Hyperparameter Values for Parametersin Computer Experiments
Hyperparameter Value
Cc 1.0 £ 10¡4
®C 3.0¯C 3.0mµc 0s2µc
100.0a»2
c2,000.0
b»2c
3.0
aÁc 1.0 £ 10¡3
bÁc 1.0 £ 10¡3
a!c 1.0 £ 10¡3
b!c 1.0 £ 10¡3
TECHNOMETRICS, MAY 2004, VOL. 46, NO. 2
Dow
nloa
ded
by [
Nor
th C
arol
ina
Stat
e U
nive
rsity
] at
15:
19 2
8 M
ay 2
014
160 C. SHANE REESE ET AL.
(a) (b)
(c) (d)
(e) (f)
Figure 3. Comparison of Posterior Distributions Conditional on Different Sources of Information: (a) ¯0 , Intercept; (b) ¯1, Air Temperature;(c) ¯2 , Flow Rate; (d) ¯3 , Fluid Velocity; (e) ¯4 , Interaction Between Flow Rate and Fluid Velocity; and (f) ¾ 2 . The different lines indicate inclusionof different data sources. ( physical only; physical + 3 computers; physical + computer 1; physical + computer 2; physical +computer 3.)
sources do not necessarily contain the same amount of informa-tion (although in our example they do have the same number ofobservations).
In addition to posterior distributions for ¯ and ¾ 2, our mod-eling approach allows us to estimate the bias terms. As an il-lustration, Figures 4(a) and 4(b) present the location and scale
(a) (b)
Figure 4. Comparison of (a) Location Bias and (b) Scale Bias Predictive Distributions for Three Different Computer Models of the Fluidized BedProcess. [(a) computer 1; computer 2; computer 3; (b) computer 1; computer 2; computer 3.]
TECHNOMETRICS, MAY 2004, VOL. 46, NO. 2
Dow
nloa
ded
by [
Nor
th C
arol
ina
Stat
e U
nive
rsity
] at
15:
19 2
8 M
ay 2
014
INTEGRATED ANALYSIS OF COMPUTER AND PHYSICAL EXPERIMENTS 161
Table 7. Comparison of Con� dence and Credible Intervals
bias predictive distributions for each of the computer models.Note that these distributions are integrated over the distribu-tion of individual-speci�c locations and scale bias terms. Oneappealing feature of these plots is that they indicate a new ap-proach to computer model validation relative to the physicalobservations.Those models that have most mass over 0 are lesslocationbiased for the physical experimentaldata. For example,the bias is more concentrated around 0 for the third computermodel than for the other two computer models. These plots alsoreveal the uncertainty associated with the bias terms, a featurethat cannot easily be inferred from a casual examination of thedata. Note that the third model is the computer model that at-tempts to account for more phenomena.Figure 4(b) reveals thatall three computer models tend to underestimate the variabil-ity in the physical experimental data. Scale bias terms greaterthan 1 (because the scale bias is parameterized as 1=kCi ) indi-cate underestimation of variability.
Table 7 contains the maximum likelihood estimates (MLEs),95% con� dence intervals (calculated from only the physical ex-perimental data), and the posterior mean and 95% highest pos-terior density (HPD) intervals calculated using the integratedcomputer and physical experimental data for ¯ and ¾ 2 . Re-call that an HPD interval is the shortest interval in the posteriordistribution containing 95% of the posterior probability.Noticethat the HPD intervals are shorter (sometimes signi� cantly so)than the 95% con� dence intervals, re� ecting the additional in-formation that has been incorporated into the analysis.
3.3.1 Expert Opinion Data. Although no expert opinionswere available for use in the � uidized bed example, it is inter-esting to observe the impact of such data on the results. Forpurely illustrative purposes, suppose that eight expert opinionswere elicited for use in the � uidized bed example. The expertopinions are given in Table 8, where T2;o denotes the expectedsteady-state outlet air temperature, q:9 is the correspondingsub-
jective :9 quantile on the outlet air temperature, and m.e/o is the
equivalent “worth” of the opinion (see Sec. 2.2).Figure 5 contains two posteriordistributions,one distribution
for the regression coef� cient for � ow rate (¯2) and one for theerror variance (¾ 2). The solid line is the posterior distributionconditional on the arti� cial expert opinion with one computermodel and the physical experimental data. The dotted line isthe posterior distribution with only the physical experimentaldata and one computer model. Due to estimation of locationand scale biases for both the computer data and the arti� cial ex-pert opinion data, only a small gain in information results fromadding the expert opinion data. No inference from these pos-terior distributions should be made, because the expert opiniondata were generated for illustration purposes only.
(a)
(b)
Figure 5. Comparison of Posterior Distributions for (a) ¯2 , Flow Rateand (b) ¾ 2 . The solid line represents the posterior distribution condi-tional on the arti� cial expert opinion with one computer model and thephysical experimentaldata; the dotted line, the posterior distributionwithonly the physical experimental data and one computer model. ( EOincluded; EO excluded.)
TECHNOMETRICS, MAY 2004, VOL. 46, NO. 2
Dow
nloa
ded
by [
Nor
th C
arol
ina
Stat
e U
nive
rsity
] at
15:
19 2
8 M
ay 2
014
162 C. SHANE REESE ET AL.
Table 9. Hyperparameter Values for Sensitivity Analysis
Bayesian analyses that contain many parameters have the po-tential to rely heavily on prior distributionsand prior parameterchoice. To assess the impact of our choices of prior parameters(and hyperparameters), we conducted a small sensitivity analy-sis. To address hyperparameter sensitivity,we designed a 210¡5
fractional factorial design in which we chose “high” and “low”values that we deemed feasible. The values that we chose aregiven in Table 9.
Marginal posterior distributions for the regression coef� -cients (¯0; : : : ; ¯4) and the error variance (¾ 2) are shown inFigure 6. Because the posterior distributions do not lie ex-
(a) (b)
(c) (d)
(e) (f)
Figure 6. Sensitivity Analysis for Selected Hyperparameters in the RBHM Formulation: (a) ¯0 , Intercept; (b) ¯1, Air Temperature; (c) ¯2 , FlowRate; (d) ¯3 , Fluid Velocity; (e) ¯4, Interaction Between Flow Rate and Fluid Velocity; and (f) ¾ 2 . The different lines indicate a different factorialcombination in the sensitivity analysis, and the thick solid line indicates the posterior at the original settings. The “rug” at the bottom of each pictureis the frequentist con�dence interval based on the physical data only � t to the �ve-parameter linear model.
TECHNOMETRICS, MAY 2004, VOL. 46, NO. 2
Dow
nloa
ded
by [
Nor
th C
arol
ina
Stat
e U
nive
rsity
] at
15:
19 2
8 M
ay 2
014
INTEGRATED ANALYSIS OF COMPUTER AND PHYSICAL EXPERIMENTS 163
actly on one another, there is clearly some sensitivity to priorspeci� cation. The differences in the posterior distributions arenot signi� cant, however. The only clear deviations are 6 ofthe 32 fractional factorial combinations for ¾ 2 and 2 of the 32fractional factorial combinations for ¯0 . These produce sig-ni� cant departure from the posterior distributions presented inSection 3.3. We note that each of these stems from a prior dis-tribution that includes nearly no mass around the posterior dis-tribution; that is, they represent infeasible prior distributions.This indicates that care should be taken when specifying priorparameters on the variability.Sensitivity is observed only whenpriors are completely misspeci� ed.
5. DISCUSSION AND CONCLUSIONS
When expert opinion is elicited, an equivalentnumber of ob-servations, m.e/
oi , is also stated that re� ects its worth in termsof a number of equivalent physical observations. This parame-ter is not required for the computer experimental data, becausethis information is captured in the prior parameters µc; »2
c ;Ác,and !c . These parameters control the prior information aboutthe location and scale biases for the computer experimentaldata. If the biases are known exactly (a point mass prior),then each computer observation counts as exactly one physi-cal observation—no information must be used to estimate thebiases, and it can all be used to estimate ¯ and ¾ 2. If these pa-rameters are used to specify a very diffuse (“noninformative”)prior with close to in� nite variances, then each computer obser-vation counts for only a tiny fraction of a physical observation.If the parameters specify an informative prior, then the com-puter observations account for some intermediate fraction of aphysical observation.
The model that we have used in our example treats each com-puter model independently. In the extreme, this implies thatif the three models were identical, then we would count eachobservation three times the fraction of a physical observationimplied by the prior distributions. We can change this by mod-eling correlations between the computer models. There are twoobvious ways to do this. The simplest way is to add a hier-archical structure on the hyperparameters (µc, »c , Ác, and !c)of the various computer models. As discussed in Section 2.3,this induces correlations in the unconditional distributions ofthe computer observations.A second is to model the entire vec-tor of observations from the three computer models directly asa multivariate normal and to specify an appropriate covariancestructure. This choice would be especially appropriate in thecase in which we had precise information about the differencesin the physics modeled by the individual computer models. Forthis example, we have insuf� cient knowledge about the precisesimilarities/differences between the three computer models topermit the use of any of them.
We have not imposed the requirement that the computed (ormeasured) value of the response variable be considered at thesame factor values in both experiments. We only require thatthere exist some common set of factors (either all or at leastsome) for both experiments. Although the example does notfully illustrate this, it is an important feature in the generalmodel. As the analysis proceeds by using information from onetype of experiment to update the distribution of the parameters,
if there are no data at a particular design point for a particu-lar experiment, then the distribution for the parameter remainsunchanged, except for correlations that may exist in the para-meters.
As with any Bayesian analysis, there is sensitivity to thespeci� cation of the prior distributions for the hyperparameters.Fortunately, however, the sensitivity is particularly acute onlywhen the priors are completely misspeci� ed. Although some ofthe hyperparameter selections in Section 3.2 are somewhat ar-bitrary, they illustrate the kinds of discussions that the analystwould engage in with the data owner to arrive at “reasonable”hyperparameter distributions. If at all possible, we prefer dif-fuse but informative prior distributionsusing expert input.
In this example we included all three sets of computer data,even though we believed that the models were successively im-proved. We made this choice for two reasons: � rst, we believethat by appropriate modeling of biases, there is information inall of the codes that should not be discarded; and second, it isoften of interest to characterize the biases of each code relativeto the physical data.
We have presented an RBHM that can be used to combinedata from both computerand physicalexperiments.When avail-able, expert opinion data are also used to “sharpen” the initialinformative, but rather diffuse prior distributions. Appropriatebiases are introduced as a way to account for differences inthese data sources. Sample results indicate that signi� cantlymore precise estimates of the regression coef� cients and er-ror variance are obtained by means of this method. In addition,the methodology can be used to recursively estimate those un-known biases of particular interest. Biases that are not particu-larly interesting can be marginalized (i.e., averaged out of theanalysis using appropriate priors). Obviously, not all problemsinvolving combinationof computer models and physical exper-iments are well suited to combination through statistical (re-sponse surface) models. In our example, however, the approachis well suited to the data collected, and the biases seem to re-� ect the actual differences between the computer models andthe physical data.
The methodology can also be used to combine various otherkinds of experimental information.Similarly, information frommore than two physical and/or computer experiments can alsobe combined using the RBHM simply by considering an appro-priate bias structure for each data source and by increasing thenumber of stages in the analysis accordingly.
ACKNOWLEDGMENTS
The authors thank Valen Johnson for his valuable comments.They also thank the editor, an associate editor, and the refereesfor their suggestions, which improved the manuscript.
APPENDIX: COMPUTATIONAL DETAILS
A.1 Markov Chain Monte Carlo
Suppose that we are interested in making statistical inferenceabout a parameter (possibly vector valued) 2. We characterizeour information (or lack of information) about the distributionof 2 D fµ1; µ2; : : : ; µng as ¼.2/ (prior distribution). Data are
TECHNOMETRICS, MAY 2004, VOL. 46, NO. 2
Dow
nloa
ded
by [
Nor
th C
arol
ina
Stat
e U
nive
rsity
] at
15:
19 2
8 M
ay 2
014
164 C. SHANE REESE ET AL.
collected and represented by the likelihood or by f .xj2/. Inany Bayesian analysis, inference on the parameters depends onthe calculated posterior distribution
¼.2jx/ D¼.2/f .xj2/R
2¼.2/f .xj2/ d2
: (2)
In many situations, the denominator of (2) is not a well-knownintegral and must be calculated numerically by, for example,MCMC. Let 2¡v be 2 with the vth element removed. A suc-cessive substitution implementation of the MCMC algorithmproceeds as follows:
1. Initialize 2.0/ and set t D 1.2. Set v D 1.3. Generate an observation µ .t/
v from the distribution of[µvj2.t¡1/
¡v ], replacing recently generated elements of
2.t¡1/¡v with elements of 2
.t/¡v if they have been generated.
4. Increment v by 1 and go to step 3 until v D n.5. If v D n increment t by 1 go to step 2.
Under conditions outlined by Hastings (1970), as t ! 1 thedistribution of fµ .t/
1 ; : : : ; µ.t/n g tends to the joint posterior distri-
bution of 2, as desired.Typical implementation of the algorithm generates an initial
“large” number of iterations (called the “burn-in”) until the ob-servations have converged. The burn-in samples are discarded,and the observations generated thereafter are used as obser-vations from the posterior distribution of 2. Nonparametricdensity estimators (Silverman 1986) can then be used to ap-proximate the posterior distribution.
A.2 Metropolis–Hastings
Some complete conditional distributions may not be avail-able in closed form; that is, it may be dif� cult to sample from[µvj2.t¡1/
¡v ] / g.µv/. Obtaining observationsfrom such distribu-tions is facilitated by implementinga Metropolis–Hastings step(Hastings 1970) for step 3 in the algorithmgiven in Section A.1.This is dif� cult, because the distribution is known only up to aconstant. The procedure is as follows:
1. Initialize µ.0/vold and set j D 0.
2. Generate an observation µ. j/vnew from a candidate distribu-
tion q.µ. j/vold ; µ
. j/vnew /, where q.x; y/ is a probability density in
y with mean x.3. Generate a uniform.0; 1/ observation u.4. Let
µ . jC1/vnew
D(
µ. j/vnew if u · ®
¡µ
. j/vold ; µ
. j/vnew
¢
µ. j/vold otherwise,
where ®.x; y/ D minf g.y/q.y;x/g.x/q.x;y/;1g.
5. Increment j and go to step 2.
The candidate distribution can be almost any distribution(Gilks, Richardson, and Spiegelhalter 1996), although a sym-metric distribution such as the normal results in a simpli� cationof the algorithm, and is called a Metropolis step (as opposedto a Metropolis–Hastings step). A common choice for q.x;y/
is a normal distribution with mean x and some variance whichallows the random deviates to be a representative sample fromthe entire complete conditional distribution. A rule of thumbgiven by Gilks et al. (1996) suggests that the variance in q.x;y/
be one-third of the sample variance of the observed data.
[Received June 2000. Revised October 2003.]
REFERENCES
Agliari, A., and Parisetti, C. C. (1988), “A g Reference Informative Prior:A Note on Zellner’s g-Prior,” The Statistician, 37, 271–275.
Berry, D. A. (1996), Statistics: A Bayesian Perspective, New York: DuxburyPress.
Broemeling, L. (1985), Bayesian Analysis of Linear Models, New York: MarcelDekker.
Chipman, H., Hamada, M., and Wu, C. F. J. (1997), “A Bayesian Variable-Selection Approach for Analyzing Designed Experiments With ComplexAliasing,” Technometrics, 39, 372–381.
Craig, P. S., Goldstein, M., Rougier, J. C., and Seheult, A. H. (2001), “BayesianForecasting for Complex Systems Using Computer Simulators,” Journal ofthe American Statistical Association, 96, 717–729.
Currin, C., Mitchell, T. J., Morris, M. D., and Ylvisaker, D. (1991), “BayesianPrediction of Deterministic Functions, With Applications to the Design andAnalysis of Computer Experiments,” Journal of the American Statistical As-sociation, 86, 953–963.
Dewettinck, K., Visscher, A. D., Deroo, L., and Huyghebaert, A. (1999), “Mod-eling the Steady-State Thermodynamic Operation Point of Top-Spray Flu-idized Bed Processing,” Journal of Food Engineering, 39, 131–143.
Draper, D., Gaver, D. P., Goel, P. K., Greenhouse, J. B., Hedges, L. V.,Morris, C. N., Tucker, J. R., and Waternaux, C. M. (1992), “Selected Sta-tistical Methodology for Combining Information (CI),” in Combining Infor-mation: Statistical Issues and Opportunities for Research, eds. D. Cochranand J. Farrally, Washington, DC: National Academy Press, Chap. 4.
Draper, N. R., and Smith, H. (1998), Applied Regression Analysis, New York:Wiley.
Gelman, A., Carlin, J. B., Stern, H. S., and Rubin, D. B. (1995), Bayesian DataAnalysis, London: Chapman & Hall.
Gilks, W. R., Richardson, S., and Spiegelhalter, D. J. (1996), Markov ChainMonte Carlo in Practice, London: Chapman & Hall.
Hastings, W. K. (1970), “Monte Carlo Sampling Methods Using MarkovChains and Their Applications,” Biometrika, 57, 97–109.
Hedges, L. V., and Olkin, I. (1987), Statistical Methods for Meta Analysis, NewYork: Wiley.
Kennedy, M. C., and O’Hagan, A. (2001), “Bayesian Calibration of ComputerModels” (with discussion), Journal of the Royal Statistical Society, Ser. B,63, 425–464.
McKay, M. D., Beckman, R. J., and Conover, W. J. (1979), “A Comparisonof Three Methods for Selecting Values of Input Variables in the Analysis ofOutput From a Computer Code,” Technometrics, 21, 239–245.
Meyer, M. A., and Booker, J. M. (1990), Eliciting and Analyzing Expert Judg-ment, Washington, DC: U.S. Nuclear Regulatory Commission.
MRuller, P., Parmigiani, G., Schildkraut, J., and Tardella, L. (1999), “A BayesianHierarchical Approach for Combining Case-Control and Prospective Stud-ies,” Biometrics, 55, 858–866.
Sacks, J., Welch, W. J., Mitchell, T. J., and Wynn, H. P. (1989), “Design andAnalysis of Computer Experiments,” Statistical Science, 4, 409–423.
Santner, T. J., Williams, B. J., and Notz, W. I. (2003), The Design and Analysisof Computer Experiments, New York: Springer-Verlag.
Shapiro, S. (1994), “Meta-Analysis/Shmeta-Analysis,” American Journal ofEpidemiology, 140, 771–791.
Silverman, B. W. (1986), Density Estimation for Statistics and Data Analysis,London: Chapman & Hall.
Welch, W. J., Buck, R. J., Sacks, J., Wynn, H. P., Mitchell, T. J., andMorris, M. D. (1992), “Screening, Predicting, and Computer Experiments,”Technometrics, 34, 15–25.
Zeckhauser, R. (1971), “Combining Overlapping Information,” Journal of theAmerican Statistical Association, 66, 91–92.
Zellner, A. (1986), “On Assessing Prior Distributions and Bayesian Regres-sion Analysis With g-Prior Distributions,” in Bayesian Inference and Deci-sion Techniques, eds. P. Goel and A. Zellner, New York: Elsevier Science,pp. 233–243.
