Applying Meta-modeling for extended

CGE-modeling: Sampling techniques and

potential application

Ding Jindjin@ae.uni-kiel.de

jinding.merlin@gmail.comJohannes Hedtrich

Johannes.Hedtrich@ae.uni-kiel.deChrisitan Henning

chenning@ae.uni-kiel.deUniversity of Kiel

Faculty of Agricultural Economics

April 14, 2018

Abstract

Apart from the computational time and expenses of the CGE model,the discussion of elasticity parameter estimation and various closure rulesas well as the difficulty of combining the results with other analysis ap-proaches always poses obstacles ahead of us, therefore we are motivated toapply the meta-modeling technique in order to tackle these problems froma new perspective, test its applicability and performance in the frame-work of the Senegal-CGE model and even compare the CGE models. Themeta-modeling technique includes three essential components, which aresimulation models, meta-models, and design of experiments. Our findingsshow that the meta-models possess a decent prediction capacity and themarginal effects differ distinctly among the sectors. However, we have notdetected significant variability of the marginal effects within each sectorseparately. We plan to include closure rules in the follow-up research.

Keywords: CGE modeling; Elasticities; Closure Rules; Meta-modeling; Meta-models; DOE.

1

1 Introduction

CGE-applications are workhorse models in applied economic policy analysis,i.e. the development economic literature or modeling climate and energy poli-cies. However, beyond its prominent application CGE-model approaches arealso heavily criticized. On the one hand, while the general equilibrium modelhas the advantages in terms of internal consistency and allowing for cleareridentification of causality, the application of a CGE model requires simplifyingassumptions that are open to challenge. Moreover, empirical results derivedfrom the CGE-model application are very sensitive to specific model specifi-cations, that are often only weakly empirically justified, e.g. assumed closurerules and assumed elasticity parameters. Thus, many results, e.g. growth-poverty linkages, that are derived from a CGE model are in fact plagued byhigh model uncertainty implying a limited potential to generate robust policy-relevant messages. A drawback of existing approaches is that they focus ongrowth-poverty linkages and neglect policy-growth linkages, i.e. from the view-point of a government economic growth does not fall from heaven, but ratherhas to be generated using scare budget resources. A good case in point areanalyses of development policies, where policy impacts on poverty are modeledvia induced policy-growth and growth-poverty linkages. However, the formeris generally modeled following an ad hoc approach assuming exogenous policyimpacts on sectoral technical progress. To overcome theoretical shortcomings ofad hoc CGE-approaches we suggest a combined approach incorporating econo-metric approaches to assess policy-growth linkages that are integrated into theCGE-approach modeling growth-poverty linkages. However, estimation of in-tegrated econometric and CGE-modeling approaches are often tedious.Finally,CGE-model approaches are often applied to provide scientific expertise to advisethe government in political practice. Hence, it would be necessary to incorpo-rate general equilibrium models into overall decision-making models. However,given the size and complexity of CGE-models integration of these approachesinto an overall decision-making modeling approach is rather difficult and oftennumerically not tractable.

Therefore, in the context of such a situation, we suggest application of meta-modeling as a potential solution of the application problems of standard CGE-models in advanced policy modeling frameworks and we are motivated to beginwith tackling the problem of elasticity parameters and closure rules.

Meta-models, compared with standard simulation models such as the CGEmodel, incorporate several charming attributes: meta-models are fast to ana-lyze. A probable property shared by all simulation models might be that theyrequire a large amount of computational time and expenses, which rise exponen-tially with the growth of model complexity. Meta-models, on the other hand,are by nature mathematical approximation equations, therefore after they arefit and validated, it’s easy and fast for users to exploit them and combine themwith other approaches; meta-models are relatively easy to understand. Prob-lem understanding is also an overarching goal of implementing meta-models.The interpretation of CGE model results is always tricky as there exist usually

2

transmitting channels which distribute the impact to a number of aspects andcomplicate the understanding and interpretation of the CGE model. In com-parison, meta-models such as lower-order polynomial meta-model, have explicitforms which are readily comprehensible; Meta-models are flexible to construct.We can treat the discussion of elasticity parameter estimation and closure rulesfrom a totally new perspective by integrating the elasticity parameters into themeta-models and estimating them under each possible closure rule; Meta-modelsenable us to combine them with other approaches directly. It is extremely prob-lematic and sometimes even impossible to integrate CGE models into otheranalysis approaches. For example, we want to apply the Bayesian model selec-tion method in order to select which CGE model has the highest probability ofbeing the true one and this process is impossible to be carried out by using CGEmodels. However, under such circumstances, using meta-models as surrogatesof CGE models provides us an opportunity to solve this problem.

The meta-modeling methodology aims at generating the valid meta-modelswhich are considered to be the surrogate models of the CGE models such thatthe comparison between various meta-models is equivalent to the comparisonbetween various CGE models.

All in all, the application of the meta-modeling technique enables us to un-derstand the CGE model better, analyze it faster and easier, tackle the problemof elasticity parameters and closure rules with a new method and use it for moreresearch purposes.

2 Literature Review

CGE-modeling is a common workhorse in development economics and policyanalysis. It has been widely used to model climate and energy policies as well.(Bourguignon [2003]; Lofgren et al. [2002]; Fan [2008]). However, in spite ofits prominent applications, CGE-modeling has been heavily criticized becauseempirical results derived from CGE-models are very sensitive to specific modelspecifications, that are often only weakly justified, e.g. assumed closure rulesand assumed elasticity parameters. (Lofgren and Robinson [2008]; Hazledineet al. [1992]; Arndt et al. [2002]) Thus, many results, e.g. growth-poverty link-ages, that are derived from a CGE model are in fact plagued by high modeluncertainty implying a limited potential to generate robust policy-relevant mes-sages. (Lofgren and Robinson [2008]). Meta-modeling technique has been ex-tensively used in field such as engineering, natural science, production designand etc. (Srivastava et al. [2004]; Noordegraaf et al. [2003]; Kleijnen and Stan-dridge [1988]) The basic approach is to construct approximation models of thesimulation models in order to generate surrogate models that are accurate andreliable enough to replace the original ones with the purpose of understand-ing the simulation models better and combining the simulation models withother analysis methods.(Kleijnen and Sargent [2000]; Kleijnen [2008]) Buildingapproximation models include two essential components: design of experimentsand meta-models, the former is used to produce the simulation sample (Kleijnen

3

et al. [2005]; Giunta et al. [2003]; Eriksson et al. [2000]) while the latter is usedto determine the form of the surrogate models. (Simpson et al. [2001]; Wang[2007]). The application of meta-modeling technique enables us to circumventthe discussion about the rationality of closure rules and elasticity parametersand go directly into the exploration of true CGE model and integrate CGEmodels into other research analyses. To the best of our knowledge, despite therich applications of the meta-modeling technique in other areas, it has not beenused in company with CGE models. Thus, this paper aims at building a bridgebetween the two methods.

3 Meta-modeling

The meta-modeling technique includes three essential components: the simula-tion model, the meta-model, and the experimental design.(Kleijnen and Sargent[2000]) The meta-model is a mathematical approximation equation that we as-sume and use to approximate the Input/Output behavior of the simulationmodel (the Senegal-CGE model in this paper). The experimental design, alsoknown as Design of Experiments or DOE for short, is a method to produce thesimulation sample (simulation inputs) from the design space. The simulationmodel, the foundation of this technique, is used to take in the simulation inputs(I) and generate the simulation outputs (O). In practice, it is treated as a blackbox, in other words, it is used as a simulation machine and we focus mainlyon the Inputs/Outputs instead of what is happening inside. To sum up, thesimulation inputs will be absorbed by the black box and the simulation outputswill be produced, thus we can use them to fit and validate the meta-model.

3.1 Meta-models

Meta-models aim at approximating the Input/Output relationships of simula-tion models. The term meta-model was popularized and developed by JackKleijnen(Kleijnen [1975]), but the term and concept were both originated byRobert Blanning(Blanning [1974]; Blanning [1975]). Meta-models are usuallyused to model the behavior of another model and they are also termed surro-gate models or response surface models. In the history of meta-models, theyare applied to approximate both the stochastic simulation and the deterministicsimulation.

A meta-model is a mathematical function that takes some simulation modeldesign parameters as inputs and produces an approximation of simulation out-puts. Examples of model design parameters are actually any parameter of in-terest which are considered to have the possibility to exert an impact on theoutput. In the Senegal-CGE model, for example, we have three types of modeldesign parameters: policy indicators(e.g., technical progress shocks), productionelasticities and trade elasticities. Examples of simulation outputs in our caseare the welfare of small-scale farmers, urban consumer welfare and etc.

There are many types of meta-models in the literature and for our current

4

research purpose, we will focus on two types of them, which are the lower-orderpolynomial and kriging meta-models.

3.1.1 Lower-Order Polynomial Metamodel

Lower-order polynomial meta-models are originally developed for the analysis ofphysical experiments(Box and Wilson [1951]) and they have been used effectivelyfor building approximations in a variety of applications. There are differentforms of this type but the most commonly used forms are first-order polynomialand second-order polynomial meta-models.

A second-order polynomial meta-model has the functional form:

y = β0 +k∑

i=1

βixi +

k∑i=1

βiix2i +

∑i

∑j

βijxixj + ε, (1)

where xi and xj are the model design parameters and β′s are the correspondingcoefficients and ε is the error term which is often assumed to be a white noiseprocess.

The corresponding coefficients β′s are estimated using the ordinary least-squares regression and the estimates are computed as follows:

β = [X ′X]−1X ′y, (2)

where X is the model design matrix and y is the simulation output. We canalso perform other standard statistical analysis of the estimates.

Lower-order polynomial meta-models are attractive because they are easyto construct, understand and analyze. Besides, they work well in modelinglocal and linear behavior of the simulation model but if the simulation modelis nonlinear or irregular, they might fail in approximating the behavior and wemust resort to other meta-model types.

3.1.2 Kriging Metamodel

Kriging meta-models are originally developed for applications in geostatistics(Cressie and Chan [1989]), a kriging model postulates a combination of a poly-nomial model and departures of the form:

y =

k∑i=1

βifi(x) + Z(x), (3)

where fi(x) is the polynomial model and Z(x) is assumed to be a realization ofa stochastic process with mean zero and spatial correlation function given by:

Cov[Z(xi), Z(xj)] = σ2R(xi, xj), (4)

where σ2 is the variance of this process and R is assumed to be the correlationfunction of this process. A variety of correlation functions can be chosen, such

5

as linear correlation function, exponential correlation function and Gaussiancorrelation function. Besides, as Z(x) is also assumed to be a stationary processso the covariances R(xi, xj) are dependent only on the distance between theinput combinations xi and xj . The corresponding coefficients are estimatedusing the maximum likelihood estimation method.

The Kriging meta-mdels are more flexible and can be used to model nonlinearor irregular behaviors of the simulation model.

3.2 Design of Experiments(DOE)

Design of experiments (Eriksson et al. [2000]), or DOE for short, is a samplingtechnique which we can apply to sample the model design space in order togenerate the simulation sample. For example, we have k quantitative designparameters and each of them has n different values, which means that if wewant to run all the possible scenarios, we would end up with nk simulation runsand it could probably be a number that we are not able to handle. Therefore,we need a technique with which we can generate a workable sample while at thesame time this sample must possess desirable properties and enough informationfor the follow-up analysis.

There is a large number of experimental designs in the literature, but forour current purpose we will discuss two types of DOE, the Central CompositeDesign and the Latin Hypercube Sample Design.

3.2.1 Central Composite Design

The Central Composite Design or CCD is a classical fractional factorial exper-imental design which spreads the sample points at three different places of thedesign space: (i) the vertices of the design space; (ii) the center of the designspace; (iii) the star points which are placed along the axes but outside the designspace (Giunta et al. [2003]).

A two-variable CCD contains the following sample points:

Figure 1: A Central Composite Design for n = 2.

6

The central composite design guarantees that the estimates of the coefficientsof a second-order polynomial metamodel are unbiased. The number of samplepoints of a CCD follows the formula 2n + 2n + C0, where n is the number ofvariables, 2n is the number of sample points at the vertices, 2n is the number ofstar points and C0 is the number of center points. In a central composite design,we can not control the number of sample points once n is fixed except that C0

is an arbitrary number which we can alter. This means when the number ofvariables n grows, the sample points that we need to estimate the metamodelalso increases exponentially.

3.2.2 Latin Hypercube Sample Design

Latin Hypercube Sample Design, or LHS for short, is a space-filling design whicharranges the sample points as spread-out as possible across the design space inorder to collect information inside the design space. Besides, LHS has anotherattribute that we can control the number of sample points based on practicalconcerns.

The Latin hypercube sample design works as follows (McKay et al. [1979];Stocki [2005]): suppose we have n variables and we need p sample points to fitour metamodel. Then the intervals of every variable are divided into p subin-tervals and one value is chosen out of every subinterval based on the probabilitydensity within that subinterval for each variable. Next, the p values of x1 ispaired randomly with the p values of x2, then this established pair of x1 andx2 is again paired at random with the p values of x3, and this process will becontinued until the p n-tuplets are formed which are exactly the p sample pointsthat we need for the simulation.

We can have a look at the following example with n = 2 and p = 4:

Figure 2: A Latin Hypercube Sample Design.

As a member of the space-filling design family, the latin hypercube designaims at placing the sample points as spread-out as possible across the designspace. There are many criteria and optimality rules regarding generation ofnice latin hypercube samples from which we list the following ones, such as (Rpackage “lhs”):

7

1. Random LHS: draws a latin hypercube sample from a set of uniform dis-tributions for use in creating a latin hypercube design. This sample istaken in a random manner without regard to optimization.

2. Improved LHS: draws a latin hypercube sample from a set of uniformdistributions for use in creating a latin hypercube design. This sampleis drawn based on the idea of optimizing the sample with respect to anoptimum euclidean distance between design points.

3. Maximin LHS: draws a latin hypercube sample from a set of uniformdistributions for use in creating a latin hypercube design. This sampleis drawn based on the idea of optimizing the sample by maximizing theminimum distance between design points (maximin criteria).

4. Genetic LHS: draws a latin hypercube sample from a set of uniform dis-tributions for use in creating a latin hypercube design. This sample isdrawn based on the idea of optimizing the sample with respect to the Soptimality criterion through a genetic type algorith. S optimality seeks tomaximize the mean distance from each design point to all the other pointsin the design space, so the points are as spreat-out as possible.

5. Optimum LHS: draws a latin hypercube sample from a set of uniformdistributions for use in creating a latin hypercube design. This sampleis drawn using the Columnwise Pariwise (CP) algorithm to generate anoptimal design with respect to the S optimality criterion.

3.3 CGE Models

The simulation model we use in this paper is a standard dynamic CGE modelusing the Senegal data for calibration. For a detailed description of CGE model,see Lofgren et al. [2002].

3.4 Methodology

The general meta-modeling flow (Figure 3) can be described as folows:

1. The simulation model (the Senegal-CGE model) is treated as a black boxand we assume a meta-model for it.

2. The design of experiments is applied to generate the inputs and they areused to produce the outputs.

3. The simulaton inputs and outputs are collected in order to fit and validatethe meta-model. If the criteria are met, we can use the meta-model forother research purposes.

Table 1 lists the detailed meta-modeling process which contains the following7 steps (Barton [2015]):

8

Figure 3: The Meta-modeling Flow.

Step Activity

1 Meta-modeling Purposes

2 Inputs and Outputs Indentification

3 Meta-model Types

4 Experimental Design based on Meta-model Types

5 Conducting Simulation Runs and Collecting Outputs

6 Fitting and Validation of Meta-model Adequacy

7 Use Meta-model for Other Research Purposes

Table 1: Meta-modeling Process

3.4.1 Meta-modeling Purposes

Our purpose of implementing meta-models is to understand the Senegal-CGEmodel better and use meta-models in the follow-up research.

3.4.2 Inputs and Outputs Indentification

The Senegal-CGE model contains a number of variables (inputs) and for ourcurrent purpose, we take three categories of them into consideration (in totalwe have 20 variables of interests):

1. Policy indicators. In our case it is the technical progress shock in eightaggregated sectors: crop, export, livestock, fish, agribusiness, industry,service and public. All of them range from 1% to 10%.

2. Production elasticities (factor subsitution) in eight aggregated sectors:

9

crop, export, livestock, fish, agribusiness, industry, service and public.All of them range from 1.5 to 6.

3. Trade elasticities (Armington transformation) in agricultural and nona-gricultural sectors. The trade elasticities in agricultural sector range from0.5 to 3.3 and the trade elasticities in nonagricultural sector range from0.9 to 4.1.

The Senegal-CGE model has seven outputs, which are: z1 (Small HouseholdIncome), z2 (Poverty Reduction Index), z3 (General Public Services), Z4 (Wel-fare of Agribusiness), z5 (Urban Consumer Welfare), z6 (Welfare of AgriculturalExport Sectors), z7 (Environmental Protection). In this paper, we will analyzez1, z2, and z5 in order to test the performance of meta-models and the impactsof reduced-form meta-models.

3.4.3 Meta-model Types

In section 3.1, we have introduced two commonly used meta-model types: thelower-order polynomial meta-model and the Kriging meta-model. In additionto these two types, there are many other meta-model types to choose from, in-cluding radial basis functions, neural networks, and regression trees. See Chenet al. [2006] for a review. Because of the simplicity of construction and compre-hension, lower-order polynomial meta-models are always a good place to start.

3.4.4 Experimental Design

In section 3.2, we have introduced two experimental designs: the central com-posite design and Latin hypercube design. There have been researches on theconnection between the meta-models and experimental designs as well as theconnection between the nature of simulations and experimental designs. (Simp-son et al. [2001]) In this paper, because of the practical considerations such ascomputational time and the fact that we come across a deterministic Senegal-CGE model, LHS is a proper begining point.

3.4.5 Fitting and Validation

We collect the simulation inputs using the DOE and the simulation outputs byrunning the simulation scenarios. Then we use them to fit the meta-model. Thisprocess is not tricky because we use the standard OLS approach to estimate thecoefficients. If the fitting is satisfactory, usually it is determined by the R2

adj ,we can move forward to validating the meta-model. There are various kinds ofcriteria which can determine the validation adequacy and we use the root meansquared prediction error(rmse) and the maximal absolute relative error(mare):

10

rmse =

√√√√ 1

n

n∑i=1

(yi − yi)2 (5)

mare = max|yi − yiyi| ∀i, (6)

where n is the number of observations, yi is the true output and yi is thecorresponding predicted output.

Besides, in order to be more certain of the validation adequacy, we applyboth the in-sample validation and out-of-sample validation, therefore we needto split the sample into two parts. We have a sample with the size of 2000and divide it into two subsamples, which are the training sample and the testsample, each with the size of 1000. The training sample is used to perform thein-sample validation while the test sample is used to perform the out-of-samplevalidation.

With respect to the in-sample validation, we use the cross-validation method.Firstly, the first observation is deleted and the meta-model is fitted using therest 999 observations. Then the output of the deleted observation is predictedusing the newly fitted meta-model. Secondly, the second observation is deletedand the meta-model is fitted using the rest 999 observations. Then the outputof the deleted observation is predicted using the newly fitted metamodel. Thesame process will be applied to each observation of the training sample and 1000predicted outputs will be computed. With respect to the out-of-sample valida-tion, we use the training sample to fit the meta-model and then use the fittedmeta-model to predict the outputs using the observations of the test sample.Thus we can calculate the two statistics for both the in-sample validation andout-of-sample validation.

3.4.6 Use Meta-model for Other Research Purposes

The CGE models are criticized for many reasons from which we will discuss twoprominent ones: elasticity parameters and closure rules.

On the one hand, the CGE models are calibrated by the elasticity parametersin order to fit the data, namely, for each different set of elasticities, we havea completely different CGE model. In one case, some elasticity parameterscannot be estimateds and researchers usually solve this problem by assumingthem to be certain values, which leads to the fact that the results become verysensitive. In anoter case, the estimation and determination of some elasticitiesare not trivial because they require a large amount of econometric modeling,therefore in practice, most researchers recycle the estimates of others, thoughoften modifying them for one reason or another or “......selecting these valuesby consulting econometric and other model-based studies......” (Lofgren andRobinson [2008]). Besides, empirically, researchers will perform the sensitivityanalysis by altering the elasticities either to a high value or to a low value andcomparing the result with that of the original value.

11

On the other hand, the CGE models aim to capture all the impacts, whichmight be distributed through the whole economy, of a shock such as tax rise ortariff cut. And the models are ‘closed’ by requiring that the total supply of anobject must be equal to the total demand, with whatever adjustments neededto achieve it. In practice, however, there does not exist a true general modeland the choice of closure rules is usually arbitrary, which will definitely affectthe results.

All in all, both the elasticity parameters and the closure rules have impactson the final results but we normally don’t have information on which of themare correct or rational, therefore in this paper, we would like to tackle thisproblem from a completely new perspective with the help of the meta-modelingtechnique.

With respect to the elasticity parameters, the simulation sample at our handcontains a sample size of 2000 and each of them contains a set of elasticityparameters. Therefore the simulation sample is equivalent to 2000 CGE modelsin the background. For each CGE model, we generate a reduced-form meta-model for it. From such an angle, we free ourselves from the discussion on therationality of elasticity parameters, instead, we accept the fact that we haveno idea which CGE model is the true one but we can use our data to find it(this is the topic of our follow-up paper). In this paper, we want to test if thepolicy indicators are sensitive to different CGE models because if the answer wasyes, it would make a great sense that we endeavor to find the true CGE modelbut if the answer was no, that means the option of CGE models (elasticitiesparameters) in our case does not merit high attention and efforts.

With respect to the closure rules, we use the same approach to generate thereduced-form meta-models under each closure rule and apply other methods tofind the best one that matches the data.

In this paper, we will only discuss the analysis of elasticity parameters. Thestudy of closure rules will be left to another paper.

The general polynomial meta-model in our case takes policy indicators, pro-duction elasticities and trade elasticities as independent variables:

y = α0 +∑i

αitpi +∑j

βjθj

+∑i

∑i′=i+1

αii′tpitpi′ +∑i

∑j

βijtpiθj +∑j

∑j′=j+1

γjj′θjθj′

+∑i

αiitp2i +

∑j

γjjθ2j ,

(7)

where tpi’s are the technical progress shocks and θ’s are the elasticity parame-ters. α’s, β’s and γ’s are the corresponding coefficients.

The reduced-form meta-model takes only the policy indicators as indepen-dent variables while the production and trade elasticities are considered to be

12

fixed values in the reduced-form metamodel:

y =(α0 +∑j

βj θj +∑j

∑j′=j+1

γjj′ θj θj′ +∑j

∑j

γjj θ2j )

+∑i

(αi +∑j

βij θj)tpi

+∑i

∑i′=i+1

αii′tpitpi′ +∑i

αiitp2i ,

(8)

where the general settings remain the same with the only exception that in thereduced-form meta-models θ’s are fixed values and thus they are grouped intoother coefficients.

In such a sense, we are using the general meta-model we have fitted and val-idated to construct the reduced-form meta-models. Furthermore, we also vali-date these reduced-form meta-models by using the reduced-form meta-models topredict the outputs based on all observations and determine their performancevia the two aforementioned criteria rmse and mare. The fitting and validationprocess is the first barrier on our way to using this technique, if we had goodresults of it, we are ready to move to the second barrier, which is, to what extentcan meta-models affect the variables that we are interested in. We quantify thisimpact by the marginal effects of the sector-specific technical progress shockson the outputs:

∂y

∂tpi=∑i

(αi +∑j

βij θj)tpi

+∑

i′=i+1

αii′tpi′ +∑i

αiitpi

(9)

We expect to detect significant deviations of the marginal effects for differentsectors because that could lead us to the conclusion that reduced-form meta-models are impacting the variables of interests (technical progress shocks) andit is thus meaningful to make selections among them by using other methodssuch as Bayesian model selection.

4 Results

4.1 Fitting and Validation

4.1.1 General Meta-model

We apply the methodology to three outputs Z1(welfare of small-scale farmers),Z2(poverty reduction) and Z5(urban consumer welfare).

Firstly, let’s have a look at the fitting performance. The R2adj are 0.998,

0.9998 and 0.998 respectively, which means that the fitting works quite well forthe three outputs.

13

Secondly, let’s have a look at the validation performance which is summa-rized in Table 2.

In-Sample rmse In-Sample mare Out-of-Sample rmse Out-of-Sample mare

Z1 0.821 0.0035 0.808 0.0034

Z2 0.008 0.0004 0.012 0.0014

Z5 2.792 0.0027 2.750 0.0051

Table 2: Validation Performance

In practice, there is not a lower threshold for rmse but the value of both thein-sample and out-of-sample validation can be viewed as very low, meaning agood validation performance. While for mare, a recommended lower thresholdis 0.1 and we can easily see that the values for both cases are much lower thanthis threshold, which gives us an interpretation that our meta-model has donea good job in modeling the relationship between the inputs and outputs.

Besides, we can have a look at Figure 4 which shows the predicted outputsversus true outputs for both the training sample and the test sample. The factthat there are not clear deviations of the predicted values from the true values(almost a perfect fit) leads us again to the conclusion that the meta-modelworks well in modeling the behavior of the simulation model (the Senegal-CGEmodel) and can be used as a surrogate of the simulation model in the followinganalysis. In other words, if we intend to achieve other research purposes withthe help of the simulation model, we can now use the meta-model to replacethe simulation model. This will not only make the analysis faster and easierbut will also make some previously-not-feasible approach feasible, such as theBayesian Model Selection approach.

4.1.2 Reduced-form Meta-model

Since the general meta-model has been validated, we can furthermore validateall the reduced-form metamodels by using each reduced-form meta-model topredict the outputs based on each set of technical progress shocks. Moreover,we can compare the rmse and mare of the predictions from all reduced-formmeta-models in order to quantify the performance.

mean rmse min rmse mean mare min mare

Z1 4.477 3.308 0.014 0.010

Z2 0.052 0.039 0.002 0.001

Z5 14.611 10.628 0.012 0.008

Table 3: Reduced-form Meta-model Performance

With respect to Z1, the average and minimal rmse are 4.477 and 3.308while the average and minimal mare are 0.014 and 0.010. In comparison with

14

(a) In-Sample(Z1) (b) Out-of-Sample(Z1)

(c) In-Sample(Z2) (d) Out-of-Sample(Z2)

(e) In-Sample(Z5) (f) Out-of-Sample(Z5)

Figure 4: Prediction Performance

the validation performance of the general meta-model, we can readily see thatthe reduced-form meta-models have much larger rmse and mare than thatof the general meta-model meaning that the general meta-model has betterprediction capability. However, we could have foreseen this result because wehave constructed the reduced-form meta-models on the basis of the generalmeta-model such that they have less explanatory variables which decrease theircapacity of prediction. Nevertheless, we can still come to the conclusion that thereduce-form meta-models are accepted based on the validation results. Figure 5displays the distribution of rmse which proves that the prediction performanceof reduced-form meta-models are in general still quite good.

With respect to Z2 and Z5, we arrive at similar conclusions which can alsobe shown by Figure 6 and 7 that the reduced-form meta-models work well in

15

(a) Histogram (b) Boxplot

Figure 5: RMSE of Reduced-form Meta-models for Z1

(a) Histogram (b) Boxplot

Figure 6: RMSE of Reduced-form Meta-models for Z2

predicting the outputs meaning that we can use the reduced-form meta-modelsto perform the analysis in the next step.

4.2 Testing the Impact of Reduced-form Meta-models

In practice, we find out that the coefficients of the main effects are much largerthan the coefficients of the interaction effects and quadratic effects, therefore, tosimplify the calculation, we measure the marginal effects of technical progressshocks using only the coefficients of the main effects. Figure 8,10 and 12 show thedistribution of the sector-specific marginal effects of technical progress shockson the outputs Z1, Z2 and Z5 respectively and figure 9, 11 and 13 are the corre-sponding histogram of the marginal effects. Thus, we want to use two measuresto determine if there is variability in the sector-specific marginal effects, namely,the interquartile range and standard deviation. By definition, the interquartilerange of a box-whisker plot includes the middle 50% of the data and the largerthe interquartile range, the more variable the data set is. From figure 8,10 and12 we can come to the conclusion that for all the three outputs, the interquartileranges of marginal effects of every sector are extremely narrow meaning thatthe sector-specific marginal effects are quite stable in this case, however, we cansee that for different sectors, the marginal effects differ obviously, which means

16

(a) Histogram (b) Boxplot

Figure 7: RMSE of Reduced-form Meta-models for Z5

that sectors have quite distinct impacts on the outputs. Moreover, from figure9, 11 and 13 we can detect that the marginal effects of each sector for all thethree outputs is distributed normally or approximately normally, and thus the95% of the distribution is within two standard deviations from the mean. Inour case, the standard deviation of the marginal effects of each sector for allthe three outputs are relatively low, mostly varies between 0.5 and 2, therefore,we can conclude that the marginal effects are not variable. To summarize, theimpacts from elasticity parameters in this case are stable and not obvious.

The elasticity parameters are very important in CGE modeling (Lofgrenet al. [2002] and Fan [2008]). Some of them cannot be directly estimated whilesome of them need complex econometric models in order to be estimated, thusresearchers often extract them from literature and use them to start the simula-tions. Afterward, for the sake of comparison, they usually perform the sensitiv-ity analysis by giving the parameters high and low values so as to compare theresults and draw conclusions. In our demonstration, we use a completely differ-ent method to test the impact of reduced-form meta-models which is equivalentto the impact of elasticity parameters. Although we have not detected obviouseffects, the methodology merits research and study. Besides, in the follow-upresearch, we plan to incorporate closure rules into the analysis in order to see ifthey can make a difference.

17

Figure 8: Marginal Effects of Technical Progress Shocks on Output Z1

(a) Crop (b) Export (c) Livestock

(d) Fish (e) Agribusiness (f) Industry

(g) Service (h) Public

Figure 9: Histogram of Marginal Effects of Technical Progress Shocks on OutputZ1

18

Figure 10: Marginal Effects of Technical Progress Shocks on Output Z2

(a) Crop (b) Export (c) Livestock

(d) Fish (e) Agribusiness (f) Industry

(g) Service (h) Public

Figure 11: Histogram of Marginal Effects of Technical Progress Shocks on Out-put Z2

19

Figure 12: Marginal Effects of Technical Progress Shocks on Output Z5

(a) Crop (b) Export (c) Livestock

(d) Fish (e) Agribusiness (f) Industry

(g) Service (h) Public

Figure 13: Histogram of Marginal Effects of Technical Progress Shocks on Out-put Z5

20

5 Conclusion

A highly discussed and criticized aspect of CGE modeling is the elasticity pa-rameters, researchers usually extract them from literature because of eitherthe complex and tedious estimation process or the impossibility of estimationand use them directly in the simulations. Moreover, due to the complexityof CGE models, it is difficult to combine them with other approaches such asBayseian Model Selection or incorporate them into other models such as thedecision-making model. Therefore, we apply the meta-modeling methodologyto generate valid surrogates of the CGE models which are calibrated by variousgroups of elasticity parameters and test the impact of them on the outputs. Inour demonstration, we have shown how to apply the meta-modeling technique.Although the results in our case present that the reduced-form meta-models donot have large impacts with respect to elasticity parameters, the valid surro-gates of CGE models open the door to many possibilities. The appplication ofmeta-modeling allows us to tackle the problems from a totally new angle andgives us the opportunity to solve them.

21

References

Channing Arndt, Sherman Robinson, and Finn Tarp. Parameter estimationfor a computable general equilibrium model: a maximum entropy approach.Economic Modelling, 19(3):375–398, 2002.

Russell R Barton. Tutorial: simulation metamodeling. In Proceedings of the2015 Winter Simulation Conference, pages 1765–1779. IEEE Press, 2015.

Robert W Blanning. The sources and uses of sensitivity information. Interfaces,4(4):32–38, 1974.

Robert W Blanning. The construction and implementation of metamodels.simulation, 24(6):177–184, 1975.

Francois Bourguignon. The impact of economic policies on poverty and incomedistribution: evaluation techniques and tools. World Bank Publications, 2003.

G. E. P. Box and K. B. Wilson. On the experimental attainment of optimumconditions. Journal of the Royal Statistical Society. Series B (Methodological),13(1):1–45, 1951. ISSN 00359246. URL http://www.jstor.org/stable/

2983966.

Victoria CP Chen, Kwok-Leung Tsui, Russell R Barton, and MartinMeckesheimer. A review on design, modeling and applications of computerexperiments. IIE transactions, 38(4):273–291, 2006.

Noel Cressie and Ngai H Chan. Spatial modeling of regional variables. Journalof the American Statistical Association, 84(406):393–401, 1989.

L Eriksson, E Johansson, N Kettaneh-Wold, C Wikstrom, and S Wold. Design ofexperiments. Principles and Applications, Learn ways AB, Stockholm, 2000.

Shenggen Fan. Public expenditures, growth, and poverty: Lessons from develop-ing countries, volume 51. Intl Food Policy Res Inst, 2008.

Anthony A Giunta, Steven F Wojtkiewicz, Michael S Eldred, et al. Overviewof modern design of experiments methods for computational simulations. InProceedings of the 41st AIAA aerospace sciences meeting and exhibit, AIAA-2003-0649, 2003.

Tim Hazledine et al. A critique of computable general equilibrium models fortrade policy analysis. Technical report, 1992.

Jack PC Kleijnen. A comment on blanning’s metamodel for sensitivity analysis:the regression metamodel in simulation. Interfaces, 5(3):21–23, 1975.

Jack PC Kleijnen. Design and analysis of simulation experiments, volume 20.Springer, 2008.

22

Jack PC Kleijnen and Robert G Sargent. A methodology for fitting and vali-dating metamodels in simulation. European Journal of Operational Research,120(1):14–29, 2000.

Jack PC Kleijnen and Charles R Standridge. Experimental design and regressionanalysis in simulation: An fms case study. European Journal of OperationalResearch, 33(3):257–261, 1988.

Jack PC Kleijnen, Susan M Sanchez, Thomas W Lucas, and Thomas M Cioppa.State-of-the-art review: a users guide to the brave new world of designingsimulation experiments. INFORMS Journal on Computing, 17(3):263–289,2005.

Hans Lofgren and Sherman Robinson. Public spending, growth, and povertyalleviation in sub-saharan africa: A dynamic general equilibrium analysis.Public expenditures, growth, and poverty: lessons from developing countries,2008.

Hans Lofgren, Rebecca Lee Harris, and Sherman Robinson. A standard com-putable general equilibrium (CGE) model in GAMS, volume 5. Intl FoodPolicy Res Inst, 2002.

M. D. McKay, R. J. Beckman, and W. J. Conover. A comparison of threemethods for selecting values of input variables in the analysis of output froma computer code. Technometrics, 21(2):239–245, 1979. ISSN 00401706. URLhttp://www.jstor.org/stable/1268522.

Antonie Vonk Noordegraaf, Mirjam Nielen, and Jack PC Kleijnen. Sensitivityanalysis by experimental design and metamodelling: Case study on simulationin national animal disease control. European Journal of Operational Research,146(3):433–443, 2003.

Timothy W Simpson, Dennis KJ Lin, and Wei Chen. Sampling strategies forcomputer experiments: design and analysis. International Journal of Relia-bility and Applications, 2(3):209–240, 2001.

A Srivastava, Kurt Hacker, Kemper Lewis, and TW Simpson. A method for us-ing legacy data for metamodel-based design of large-scale systems. Structuraland Multidisciplinary Optimization, 28(2-3):146–155, 2004.

Rafal Stocki. A method to improve design reliability using optimal latin hyper-cube sampling. Computer Assisted Mechanics and Engineering Sciences, 12(4):393, 2005.

Chunhua Wang. A dynamic stochastic frontier production model with time-varying efficiency: Comment. Applied Economics Letters, 14(6):415–417,2007. doi: 10.1080/13504850500447455. URL http://dx.doi.org/10.1080/

13504850500447455.

23