Proceedings of the 2000 Winter Simulation ConferenceJ. A. Joines, R. R. Barton, K. Kang, and P. A. Fishwick, eds.

VERIFICATION, VALIDATION, AND ACCREDITATION OF SIMULATION MODELS

Robert G. Sargent

Simulation Research GroupDepartment of Electrical Engineering and Computer Science

College of Engineering and Computer ScienceSyracuse University

Syracuse, NY 13244, U.S.A.

-ltr

-

rsey

-n

-e

-ntsceA

ed

e

aeofe

al,s

inedfdeIfiesndatary

ss.

saninsr

l-e

:id-

ABSTRACT

This paper discusses verification, validation, andaccreditation of simulation models. The different approaches to deciding model validity are presented; how modeverification and validation relate to the model developmenprocess are discussed; various validation techniques adefined; conceptual model validity, model verification, operational validity, and data validity are described; ways todocument results are given; a recommended procedurepresented; and accreditation is briefly discussed.

1 INTRODUCTION

Simulation models are increasingly being used in problemsolving and in decision making. The developers and useof these models, the decision makers using information drived from the results of the models, and people affected bdecisions based on such models are all rightly concernewith whether a model and its results are “correct.” This concern is addressed through model verification and validatioModel verification is often defined as “ensuring that thecomputer program of the computerized model and its implementation are correct,” and is the definition adopted herModel validation is usually defined to mean “substantiationthat a computerized model within its domain of applicability possesses a satisfactory range of accuracy consistewith the intended application of the model” (Schlesinger eal. 1979) and is the definition used here. A model sometimebecomes accredited through model accreditation. Model acreditation determines if a model satisfies a specified modaccreditation criteria according to a specified process.related topic is model credibility. Model credibility is con-cerned with developing in (potential) users the confidencthey require to use a model and the information derivefrom that model.

A model should be developed for a specific purpos(or application) and its validity determined with respect to

50

e

is

-

d

.

.

t

-l

that purpose. If the purpose of a model is to answervariety of questions, the validity of the model needs to bdetermined with respect to each question. Numerous setsexperimental conditions are usually required to define thdomain of a model’s intended applicability. A model maybe valid for one set of experimental conditions and invalid inanother. A model is considered valid for a set of experimentconditions if its accuracy is within its acceptable rangewhich is the amount of accuracy required for the model’intended purpose. This usuallyrequires that the model’soutput variables of interest (i.e., the model variables usedanswering the questions that the model is being developto answer) be identified and that their required amount oaccuracy be specified. The amount of accuracy requireshould be specified prior to starting the development of thmodel or very early in the model development process.the variables of interest are random variables, then propertand functions of the random variables such as means avariances are usually what is of primary interest and are whis used in determining model validity. Several versions ofmodel are usually developed prior to obtaining a satisfactovalid model. The determination of whether a model is validor not, i.e., model verification and validation, is usually aprocess and is part of the total model development proce

It is often too costly and time consuming to determinethat a model isabsolutelyvalid over the complete domainof its intended applicability. Instead, tests and evaluationare conducted until sufficient confidence is obtained thatmodel can be considered valid for its intended applicatio(Sargent 1982, 1984 and Shannon 1975). Figure 1 contathe relationships of cost (a similar relationship holds fothe amount of time) of performing model validation andthe value of a model to the user as a function of modeconfidence. The cost of model validation is usually quite significant, especially when extremely high model confidencis required.

The remainder of this paper is organized as followsSection 2 discusses the basic approaches used in dec

Sargent

st

a

r

os

hot

n

es

di

g

lIl

elsV

t

ored.sssretsen

(s).

elfbee

g(4)e

oshess.g

re

.,d

s

d;

p-

-el

).

elml.

100%0% Model Confidence

Value

Modelof

toUser

Cost

Value

Cost

Figure 1: Model Confidence

ing model validity; Section 3 defines validation techniqueSections 4, 5, 6, and 7 contain descriptions of data validiconceptual model validity, model verification, and operational validity, respectively; Section 8 describes ways odocumenting results; Section 9 gives a recommended vidation procedure; Section 10 contains a brief descriptioof accreditation; and Section 11 has the summary.

2 VALIDATION PROCESS

Three basic approaches are used in deciding whethesimulation model is valid or invalid. Each of the approacherequires the model development team to conduct verificatiand validation as part of the model development procewhich is discussed below. The most common approachfor the development team to make the decision as to whetthe model is valid. This is a subjective decision basedthe results of the various tests and evaluations conducas part of the model development process.

Another approach, often called “independent verification and validation” (IV&V), uses a third party to decidewhether the model is valid. The third party is independeof both the model development team and the model sposor/user(s). (A third party is also usually used for modaccreditation.) There are two common ways that IV&V iconducted. One way is to conduct IV&V concurrently withmodel development. The other way is to conduct IV&Vafter the model has been completely developed by the modevelopment team. IV&V is often used when a large costassociated with the problem the simulation model is beinused for and/or to help in model credibility.

In the concurrent way of conducting IV&V, the mod-el development team receives input regarding verificatioand validation from the IV&V team as the model is beindeveloped. Thus, the development of a model should nprogress beyond each stage of development if the modenot satisfying the verification and validation requirements.the IV&V is conducted after the model has been completedeveloped, the evaluation performed can range from simpevaluating the verification and validation conducted by thmodel development team to a complete verification anvalidation effort. Wood (1986) describes experiences ovthis range of evaluation by a third party on energy modeOne conclusion that Wood makes is that a complete IV&

51

;y,-fl-n

asns,iserned

-

tn-l

elsg

n

otisfylyedr.

evaluation is extremely costly and time consuming for whais obtained. This author’s view is that if a third party isto be used, it should beduring the model developmentprocess. If a model has already been developed, this authbelieves that a third party should usually only evaluate thverification and validation that has already been performe

The last approach for determining whether a model ivalid is to use a scoring model (see, e.g., Balci (1989), Ga(1993), and Gass and Joel (1987)). Scores (or weights) adetermined subjectively when conducting various aspecof the validation process and then combined to determincategory scores and an overall score for the simulatiomodel. A simulation model is considered valid if its overalland category scores are greater than some passing scoreThis approach is infrequently used in practice.

This author does not believe in the use of a scoring modfor determining validity because (1) the subjectiveness othis approach tends to be hidden and thus appears toobjective, (2) the passing scores must be decided in som(usually subjective) way, (3) a model may receive a passinscore and yet have a defect that needs correction, andthe score(s) may cause overconfidence in a model or bused to argue that one model is better than another.

We now discuss how model verification and validationrelate to the model development process. There are twcommon ways to view this relationship. One way usesome type of detailed model development process, and tother uses some type of simple model development proceBanks, Gerstein, and Searles (1988) reviewed work usinboth of these ways and concluded that the simple way moclearly illuminates model validation and verification. Thisauthor recommends the use of a simple way (see, e.gSargent (1981) and Sargent (1982)), which is presentenext.

Consider the simplified version of the modeling procesin Figure 2. Theproblem entityis the system (real or pro-posed), idea, situation, policy, or phenomena to be modeletheconceptual modelis the mathematical/logical/verbal rep-resentation (mimic) of the problem entity developed for aparticular study; and thecomputerized modelis the con-ceptual model implemented on a computer. The concetual model is developed through ananalysis and model-ing phase, the computerized model is developed througha computer programming and implementation phase, andinferences about the problem entity are obtained by conducting computer experiments on the computerized modin the experimentation phase.

We now relate model validation and verification to thissimplified version of the modeling process (see Figure 2Conceptual model validityis defined as determining that thetheories and assumptions underlying the conceptual modare correct and that the model representation of the probleentity is “reasonable” for the intended purpose of the modeComputerized model verificationis defined as ensuring that

Sargent

n

ffih

den

iniddfw

es

nfan

”caahl

-on

-)o

t

-or

e

tsin

.

.t

eed

,

r

he

ts

-

ModelConceptual

ModelConceptual

ModelComputerized

Operational

Experimentation

ComputerizedModel

ProblemEntity

Computer Programming

Data

Modelingand

Analysis

and Implementation

Validity

ValidityValidity

Verification

Figure 2: Simplified Version of the ModelingProcess

the computer programming and implementation of the coceptual model is correct.Operational validity is definedas determining that the model’s output behavior has sucient accuracy for the model’s intended purpose over tdomain of the model’s intended applicability.Data validityis defined as ensuring that the data necessary for mobuilding, model evaluation and testing, and conducting thmodel experiments to solve the problem are adequate acorrect.

Several versions of a model are usually developedthe modeling process prior to obtaining a satisfactory valmodel. During each model iteration, model validation anverification are performed (Sargent 1984). A variety o(validation) techniques are used, which are described beloNo algorithm or procedure exists to select which techniquto use. Some attributes that affect which techniques to uare discussed in Sargent (1984).

3 VALIDATION TECHNIQUES

This section describes various validation techniques (atests) used in model verification and validation. Most othe techniques described here are found in the literature,though some may be described slightly differently. They cabe used either subjectively or objectively. By “objectively,we mean using some type of statistical test or mathematiprocedure, e.g., hypothesis tests and confidence intervA combination of techniques is generally used. These tecniques are used for validating and verifying the submodeand overall model.

Animation: The model’s operational behavior is displayed graphically as the model moves through time. Fexample, the movements of parts through a factory duria simulation are shown graphically.

52

-

-e

el

d

.se

d

l-

lls.-

s

rg

Comparison to Other Models:Various results (e.g.,outputs) of the simulation model being validated are compared to results of other (valid) models. For example, (1simple cases of a simulation model may be compared tknown results of analytic models, and (2) the simulationmodel may be compared to other simulation models thahave been validated.

Degenerate Tests:The degeneracy of the model’s be-havior is tested by appropriate selection of values of theinput and internal parameters. For example, does the average number in the queue of a single server continue tincrease with respect to time when the arrival rate is largethan the service rate?

Event Validity: The “events” of occurrences of thesimulation model are compared to those of the real systemto determine if they are similar. An example of events isdeaths in a fire department simulation.

Extreme Condition Tests:The model structure andoutput should be plausible for any extreme and unlikelycombination of levels of factors in the system; e.g., if in-process inventories are zero, production output should bzero.

Face Validity: “Face validity” is asking people knowl-edgeable about the system whether the model and/or ibehavior are reasonable. This technique can be useddetermining if the logic in the conceptual model is correctand if a model’s input-output relationships are reasonable

Fixed Values:Fixed values (e.g., constants) are used forvarious model input and internal variables and parametersThis should allow the checking of model results agains(easily) calculated values.

Historical Data Validation: If historical data exist (orif data are collected on a system for building or testing themodel), part of the data is used to build the model andthe remaining data are used to determine (test) whether thmodel behaves as the system does. (This testing is conductby driving the simulation model with either samples fromdistributions or traces (Balci and Sargent 1982a, 1982b1984b).)

Historical Methods: The three historical methods ofvalidation arerationalism, empiricism, and positive eco-nomics.Rationalism assumes that everyone knows whethethe underlying assumptions of a model are true. Logicdeductions are used from these assumptions to develop tcorrect (valid) model. Empiricism requires every assump-tion and outcome to be empirically validated. Positiveeconomics requires only that the model be able to predicthe future and is not concerned with a model’s assumptionor structure (causal relationships or mechanism).

Internal Validity: Several replications (runs) of astochastic model are made to determine the amount of (internal) stochastic variability in the model. A high amountof variability (lack of consistency) may cause the model’sresults to be questionable and, if typical of the problem

Sargent

o

mo1)nl’smps

vgrsg

noipsgfi-re

acamed

ceif

etsn

ellyt,seorthed

tanal

edIn

titygr.orotot

te,a-

ctlye

canndesersis.

ealheh-hep-

e-ityity,lesoingn,re,tes

eecte

la-se,ndionandtitynd

entity, may question the appropriateness of the policysystem being investigated.

Multistage Validation:Naylor and Finger (1967) pro-posed combining the three historical methods of rationalisempiricism, and positive economics into a multistage prcess of validation. This validation method consists of (developing the model’s assumptions on theory, observatiogeneral knowledge, and function, (2) validating the modeassumptions where possible by empirically testing theand (3) comparing (testing) the input-output relationshiof the model to the real system.

Operational Graphics:Values of various performancemeasures, e.g., number in queue and percentage of serbusy, are shown graphically as the model moves throutime; i.e., the dynamic behaviors of performance indicatoare visually displayed as the simulation model moves throutime.

Parameter Variability–Sensitivity Analysis: Thistechnique consists of changing the values of the input ainternal parameters of a model to determine the effect upthe model’s behavior and its output. The same relationshshould occur in the model as in the real system. Thoparameters that are sensitive, i.e., cause significant chanin the model’s behavior or output, should be made sufciently accurate prior to using the model. (This may requiiterations in model development.)

Predictive Validation: The model is used to predict(forecast) the system behavior, and then comparisonsmade between the system’s behavior and the model’s foreto determine if they are the same. The system data may cofrom an operational system or from experiments performon the system. e.g., field tests.

Traces: The behaviors of different types of specifientities in the model are traced (followed) through thmodel to determine if the model’s logic is correct andthe necessary accuracy is obtained.

Turing Tests: People who are knowledgeableabout the operations of a system are asked if thcan discriminate between system and model outpu(Schruben (1980) contains statistical tests for use with Turitests.)

4 DATA VALIDITY

Even though data validity is often not considered to bpart of model validation, we discuss it because it is usuadifficult, time consuming, and costly to obtain sufficienaccurate, and appropriate data, and is frequently the reathat attempts to validate a model fail. Data are needfor three purposes: for building the conceptual model, fvalidating the model, and for performing experiments withe validated model. In model validation we are concernonly with the first two types of data.

ng

53

r

,-

s,

,

ersh

h

dnsees

reste

y.

g

ond

To build a conceptual model we must have sufficiendata on the problem entity to develop theories that cbe used to build the model, to develop the mathematicand logical relationships in the model that will allow itto adequately represent the problem entity for its intendpurpose, and to test the model’s underlying assumptions.addition, behavioral data are needed on the problem ento be used in the operational validity step of comparinthe problem entity’s behavior with the model’s behavio(Usually, this data are system input/output data.) If behavidata are not available, high model confidence usually cannbe obtained, because sufficient operational validity cannbe achieved.

The concern with data is that appropriate, accuraand sufficient data are available, and if any data transformtions are made, such as disaggregation, they are correperformed. Unfortunately, there is not much that can bdone to ensure that the data are correct. The best thatbe done is to develop good procedures for collecting amaintaining data, test the collected data using techniqusuch as internal consistency checks, and screen for outliand determine if they are correct. If the amount of datalarge, a data base should be developed and maintained

5 CONCEPTUAL MODEL VALIDATION

Conceptual model validity is determining that (1) ththeories and assumptions underlying the conceptumodel are correct, and (2) the model representation of tproblem entity and the model’s structure, logic, and matematical and causal relationships are “reasonable” for tintended purpose of the model. The theories and assumtions underlying the model should be tested using mathmatical analysis and statistical methods on problem entdata. Examples of theories and assumptions are linearindependence, stationary, and Poisson arrivals. Exampof applicable statistical methods are fitting distributions tdata, estimating parameter values from the data, and plottthe data to determine if they are stationary. In additioall theories used should be reviewed to ensure they weapplied correctly; for example, if a Markov chain is useddoes the system have the Markov property, and are the staand transition probabilities correct?

Next, each submodel and the overall model must bevaluated to determine if they are reasonable and corrfor the intended purpose of the model. This should includdetermining if the appropriate detail and aggregate retionships have been used for the model’s intended purpoand if the appropriate structure, logic, and mathematical acausal relationships have been used. The primary validattechniques used for these evaluations are face validationtraces. Face validation has experts on the problem enevaluate the conceptual model to determine if it is correct areasonable for its purpose. This usually requires examini

Sargent

aghheedein

tee

erno

ulontioifein

ri-onhteeroryprehegtey.r-wra

lay

easinicnn

thhel-roe

i-cen

-t

el,

ed

d

lye,

-d

,n

-

erd.f

t-

the flowchart or graphical model, or the set of model equtions. The use of traces is the tracking of entities throueach submodel and the overall model to determine if tlogic is correct and if the necessary accuracy is maintainIf errors are found in the conceptual model, it must brevised and conceptual model validation performed aga

6 MODEL VERIFICATION

Computerized model verification ensures that the compuprogramming and implementation of the conceptual modare correct. The major factor affecting verification is whetha simulation language or a higher level programming laguage such as FORTRAN, C, or C++ is used. The usea special-purpose simulation language generally will resin having fewer errors than if a general-purpose simulatilanguage is used, and using a general purpose simulalanguage will generally result in having fewer errors thana general purpose higher level language is used. (The usa simulation language also usually reduces the programmtime required and the flexibility.)

When a simulation language is used, verification is pmarily concerned with ensuring that an error free simulatilanguage has been used, that the simulation languagebeen properly implemented on the computer, that a tes(for correctness) pseudo random number generator has bproperly implemented, and that the model has been pgrammed correctly in the simulation language. The primatechniques used to determine that the model has beengrammed correctly are structured walk-throughs and trac

If a higher level language has been used, then tcomputer program should have been designed, developand implemented using techniques found in software enneering. (These include such techniques as object-oriendesign, structured programming, and program modularitIn this case verification is primarily concerned with detemining that the simulation functions (such as the time-flomechanism, pseudo random number generator, anddom variate generators)and the computer model have beenprogrammed and implemented correctly.

There are two basic approaches for testing simution software: static testing and dynamic testing (Fairle1976). In static testing the computer program is analyzto determine if it is correct by using such techniquesstructured walk-throughs, correctness proofs, and examing the structure properties of the program. In dynamtesting the computer program is executed under differeconditions and the values obtained (including those geerated during the execution) are used to determine ifcomputer program and its implementations are correct. Ttechniques commonly used in dynamic testing are tracinvestigations of input-output relations using different vaidation techniques, internal consistency checks, and repgramming critical components to determine if the sam

54

-

.

.

rl

-ft

n

ofg

asden-

o-s.ed,i-d

)

n-

-

d

-

t-ees,

-

results are obtained. If there are a large number of varables, one might aggregate some of the variables to reduthe number of tests needed or use certain types of desigof experiments (Kleijnen 1987).

It is necessary to be aware while checking the correctness of the computer program and its implementation thaerrors may be caused by the data, the conceptual modthe computer program, or the computer implementation.

For a detailed discussion on model verification, seeWhitner and Balci (1989).

7 OPERATIONAL VALIDITY

Operational validity is concerned with determining that themodel’s output behavior has the accuracy required for thmodel’s intended purpose over the domain of its intendeapplicability. This is where most of the validation testingand evaluation takes place. The computerized model is usein operational validity, and thus any deficiencies found maybe due to an inadequate conceptual model, an improperprogrammed or implemented conceptual model (e.g., duto programming errors or insufficient numerical accuracy)or due to invalid data.

All of the validation techniques discussed in Section 3are applicable to operational validity. Which techniques andwhether to use them objectively or subjectively must be decided by the model development team and other interesteparties. The major attribute affecting operational validityis whether the problem entity (or system) is observablewhere observable means it is possible to collect data othe operational behavior of the program entity. Table 1gives a classification of the validation approaches for operational validity. “Comparison” means comparing/testingthe model and system input-out behaviors, and “explormodel behavior” means to examine the output behavioof the model using appropriate validation techniques anusually includes parameter variability-sensitivity analysisVarious sets of experimental conditions from the domain othe model’s intended applicability should be used for bothcomparison and exploring model behavior.

Table 1: Operational Validity ClassificationOBSERVABLE NON-OBSERVABLE

SYSTEM SYSTEM

SUBJECTIVE • COMPARISON USING • EXPLOREAPPROACH GRAPHICAL DISPLAYS MODEL BEHAVIOR

• EXPLORE MODEL • COMPARISON TOBEHAVIOR OTHER MODELS

OBJECTIVE • COMPARISON • COMPARISONAPPROACH USING TO OTHER

STATISTICAL MODELS USINGTESTS AND STATISTICALPROCEDURES TESTS AND

PROCEDURES

To obtain ahigh degree of confidence in a model andits results, comparisons of the model’s and system’s inpu

Sargent

tamtesian

eer

edhgnstythore

oatinhe7o

nphivacacivai

.i

se

output behaviors for several different sets of experimenconditions are usually required. There are three basic coparison approaches used: (1) graphs of the model and sysbehavior data, (2) confidence intervals, and (3) hypothetests. Graphs are the most commonly used approach,confidence intervals are next.

7.1 Graphical Comparison of Data

The behavior data of the model and the system are graphfor various sets of experimental conditions to determinif the model’s output behavior has sufficient accuracy foits intended purpose. Three types of graphs are ushistograms, box (and whisker) plots, and behavior grapusing scatter plots. (See Sargent (1996a) for a thoroudiscussion on the use of these for model validation.) Aexample of a box plot is given in Figure 3, and exampleof behavior graphs are shown in Figures 4 and 5. A varieof graphs using different types of (1) measures such asmean, variance, maximum, distribution, and time seriesa variable, and (2) relationships between (a) two measuof a single variable (see Figure 4) and (b) measures of twvariables (see Figure 5) are required. It is important thappropriate measures and relationships be used in validaa model and that they be determined with respect to tmodel’s intended purpose. SeeAnderson and Sargent (19for an example of a set of graphs used in the validationa simulation model.

120

100

60

40

80

System Model

Figure 3: Box Plot

These graphs can be used in model validation in differeways. First, the model development team can use the grain the model development process to make a subjectjudgment on whether a model possesses sufficient accurfor its intended purpose. Second, they can be used in the fvalidity technique where experts are asked to make subjectjudgments on whether a model possesses sufficient accurfor its intended purpose. Third, the graphs can be usedin Turing tests. Another way they can be used is in IV&VWe note that independence of data is not required (asrequired for most formal statistical approaches) in the uof these graphs. See Sargent (1996a) for details.

r-ees,

55

l-msd

d

:sh

efs

tg

4)f

ts

eyeecys

s

Figure 4: Reaction Time

Figure 5: Disk Access

7.2 Confidence Intervals

Confidence intervals (c.i.), simultaneous confidence intevals (s.c.i.), and joint confidence regions (j.c.r.) can bobtained for the differences between the means, varianc

Sargent

i-.i.ac

aoresa-eysn

-hd

elngon

fesohedlsndthofeoe

lne

nri-tasoa

y-

-

-.

s.

e

k

n

em

ntalueofity

,y

4).ics

lldelle

ticd,

ticr’s

and distributions of different model and system output varables for each set of experimental conditions. These cs.c.i., and j.c.r. can be used as the model range of accurfor model validation.

To construct the model range of accuracy, a statisticprocedure containing a statistical technique and a methof data collection must be developed for each set of expeimental conditions and for each variable of interest. Thstatistical techniques used can be divided into two group(1) univariate statistical techniques and (2) multivariate sttistical techniques. The univariate techniques can be usto develop c.i., and with the use of the Bonferroni inequalit(Law and Kelton 1991), s.c.i. The multivariate techniquecan be used to develop s.c.i. and j.c.r. Both parametric anonparametric techniques can be used.

The method of data collection must satisfy the underlying assumptions of the statistical technique being used. Tstandard statistical techniques and data collection methoused in simulation output analysis (Banks, Carson, and Nson 1996, Law and Kelton 1991) can be used for developithe model range of accuracy, e.g., the methods of replicatiand (nonoverlapping) batch means.

It is usually desirable to construct the model range oaccuracy with the lengths of the c.i. and s.c.i. and the sizof the j.c.r. as small as possible. The shorter the lengthsthe smaller the sizes, the more useful and meaningful tmodel range of accuracy will usually be. The lengths anthe sizes (1) are affected by the values of confidence levevariances of the model and system output variables, asample sizes, and (2) can be made smaller by decreasingconfidence levels or increasing the sample sizes. A tradeneeds to be made among the sample sizes, confidence levand estimates of the length or sizes of the model rangeaccuracy, i.e., c.i., s.c.i., or j.c.r. Tradeoff curves can bconstructed to aid in the tradeoff analysis.

Details on the use of c.i., s.c.i., and j.c.r. for operationavalidity, including a general methodology, are contained iBalci and Sargent (1984b). A brief discussion on the usof c.i. for model validation is also contained in Law andKelton (1991).

7.3 Hypothesis Tests

Hypothesis tests can be used in the comparison of meavariances, distributions, and time series of the output vaables of a model and a system for each set of experimenconditions to determine if the model’s output behavior haan acceptable range of accuracy. An acceptable rangeaccuracy is the amount of accuracy that is required ofmodel to be valid for its intended purpose.

The first step in hypothesis testing is to state the hpotheses to be tested:

H0: Model is valid for the acceptable range of accuracy under the set of experimental conditions.

56

,y

ld-

:

d

d

es-

r

,

efls,f

s,

l

f

H1: Model is invalid for the acceptable range of accuracy under the set of experimental conditions

Two types of errors are possible in testing hypotheseThe first, or type I error, is rejecting the validity of a validmodel and the second, or type II error, is accepting thvalidity of an invalid model. The probability of a type errorI, α, is calledmodel builder’s risk, and the probability ofthe type II error,β, is calledmodel user’s risk(Balci andSargent 1981). In model validation, the model user’s risis extremely important and must be kept small. Thusbothtype I and type II errors must be carefully considered wheusing hypothesis testing for model validation.

The amount of agreement between a model and a systcan be measured by a validity measure,λ, which is chosensuch that the model accuracy or the amount of agreemebetween the model and the system decreases as the vof the validity measure increases. The acceptable rangeaccuracy can be used to determine an acceptable validrange, 0≤ λ ≤ λ∗.

The probability of acceptance of a model being validPa , can be examined as a function of the validity measure busing an Operating Characteristic Curve (Johnson 199Figure 6 contains three different operating characteristcurves to illustrate how the sample size of observationaffectPa as a function ofλ. As can be seen, an inaccuratemodel has a high probability of being accepted if a smasample size of observations is used, and an accurate mohas a low probability of being accepted if a large sampsize of observations is used.

Figure 6: Operating Characteristic Curves

The location and shape of the operating characteriscurves are a function of the statistical technique being usethe value ofα chosen forλ = 0, i.e.,α∗, and the samplesize of observations. Once the operating characteriscurves are constructed, the intervals for the model userisk β(λ) and the model builders riskα can be determinedfor a givenλ∗ as follows:

α∗ ≤ model builder’s riskα ≤ (1− β∗)0 ≤ model user’s riskβ(λ) ≤ β∗.

Sargent

k

d

ineefd6

uo4ht

ca

a

e

m

g

me

e--l

’

-

-

-

nsy

ion

,

ng

y

r

-

e

e

e.e

-

n

Thus there is a direct relationship among the builder’s rismodel user’s risk, acceptable validity range, and the sampsize of observations. A tradeoff among these must be main using hypothesis tests in model validation.

Details of the methodology for using hypothesis testscomparing the model’s and system’s output data for modvalidations are given in Balci and Sargent (1981). Examplof the application of this methodology in the testing ooutput means for model validation are given in Balci anSargent (1982a, 1982b, 1983). Also, see Banks et al. (199

8 DOCUMENTATION

Documentation on model verification and validation is usually critical in convincing users of the “correctness” of amodel and its results, and should be included in the simlation model documentation. (For a general discussiondocumentation of computer-based models, see Gass (198Both detailed and summary documentation are desired. Tdetailed documentation should include specifics on the tesevaluations made, data, results, etc. The summary domentation should contain a separate evaluation table for dvalidity, conceptual model validity, computer model verifi-cation, operational validity, and an overall summary. SeTable 2 for an example of an evaluation table of conceptumodel validity. (See Sargent (1994, 1996b) for exampleof two of the other evaluation tables.) The columns of thtable are self-explanatory except for the last column, whicrefers to the confidence the evaluators have in the resuor conclusions, and this is often expressed as low, mediuor high.

9 RECOMMENDED PROCEDURE

This author recommends that, as a minimum, the followinsteps be performed in model validation:

1. Have an agreement madeprior to developing themodel between (a) the model development teaand (b) the model sponsors and (if possible) thusers, specifying the basic validation approach ana minimum set of specific validation techniques tobe used in the validation process.

2. Specify the amount of accuracy required of thmodel’s output variables of interest for the model’s intended application prior to starting the development of the model or very early in the modedevelopment process.

3. Test, wherever possible, the assumptions and thories underlying the model.

4. In each model iteration, perform at least face validity on the conceptual model.

5. In each model iteration, at least explore the modelbehavior using the computerized model.

57

,lee

ls

).

-

-n).)es,u-ta

el

s

hlts

,

d

e-

-

s

6. In at least the last model iteration, make comparisons, if possible, between the model and systembehavior (output) data for several sets of experimental conditions.

7. Develop validation documentation for inclusion inthe simulation model documentation.

8. If the model is to be used over a period of time, develop a schedule for periodic review of the model’svalidity.

Models occasionally are developed to be used more thaonce. A procedure for reviewing the validity of these modelover their life cycles needs to be developed, as specified bstep 8. No general procedure can be given, as each situatis different. For example, if no data were available on thesystem when a model was initially developed and validatedthen revalidation of the model should take place prior toeach usage of the model if new data or system understandihas occurred since its last validation.

10 ACCREDITATION

The DoD has moved to accrediting simulation models. Thedefine accreditation in DoDD 5000.59 as “the official certi-fication that a model or simulation is acceptable for use foa specific application.” The evaluation for accreditation isusually conducted by a third (independent) party, is subjective, and often includes not only verification and validationbut items such as documentation and how user friendly thsimulation is. The acronym VV&A is used for Verification,Validation, and Accreditation.

11 SUMMARY

Model verification and validation are critical in the devel-opment of a simulation model. Unfortunately, there is noset of specific tests that can easily be applied to determinthe “correctness” of the model. Furthermore, no algorithmexists to determine what techniques or procedures to usEvery new simulation project presents a new and uniquchallenge.

There is considerable literature on verification and validation. Articles given in the limited bibliography canbe used as a starting point for furthering your knowl-edge on model verification and validation. For a fairlyrecent bibliography, see the following UHL on the web:<http://manta.cs.vt.edu/biblio/> .

REFERENCES

Anderson, H. A. and R. G. Sargent. 1974. An investigatiointo scheduling for an interactive computer system,IBMJournal of Research and Development, 18 (2):125–137.

Sargent

Table 2: Evaluation Table for Conceptual Model ValidityCategory/Item Technique(s) Justification for Reference to Result/ Confidence

Used Technique Used Supporting Report Conclusion In Result• Theories • Face validity• Assumptions • Historical• Model • Accepted

representation approach• Derived from

empirical data• Theoretical

derivation

StrengthsWeaknesses

Overall evaluation for Overall Justification for ConfidenceComputer Model Verification Conclusion Conclusion In Conclusion

,.

-

el,

e,

r

l

y

Balci, O. 1989. How to assess the acceptability and credibility of simulation results,Proc. of the 1989 WinterSimulation Conf., 62–71.

Balci, O. 1998. Validation, verification, and accreditationProc. of the 1998 Winter Simulation Conf., 41–48.

Balci, O. and R. G. Sargent. 1981. A methodology for costrisk analysis in the statistical validation of simulationmodels,Comm. of the ACM, 24(4):190–197.

Balci, O. and R. G. Sargent. 1982a. Validation of multi-variate response simulation models by using hotelling’two-sampleT 2 test,Simulation, 39(6):185–192.

Balci, O. and R. G. Sargent. 1982b. Some examplessimulation model validation using hypothesis testingProc. of the 1982 Winter Simulation Conf.,620–629.

Balci, O. and R. G. Sargent. 1983. Validation of multivariateresponse trace-driven simulation models,Performance83, ed. Agrawada and Tripathi, North Holland, 309–323.

Balci, O. and R. G. Sargent. 1984a. A bibliography on thcredibility assessment and validation of simulation anmathematical models,Simuletter, 15(3):15–27.

Balci, O. and R. G. Sargent. 1984b. Validation of simulationmodels via simultaneous confidence intervals,AmericanJournal of Mathematical and Management Science,4(3):375–406.

Banks, J., J. S. Carson II, and B. L. Nelson. 1996.Discrete-event system simulation, 2nd Ed., Prentice-Hall, En-glewood Cliffs, N.J.

Banks, J., D. Gerstein, and S. P. Searles. 198Modeling processes, validation, and verification ocomplex simulations: A survey,Methodology and Val-idation, Simulation Series, Vol. 19, No. 1, The Societyfor Computer Simulation, 13–18.

DOD simulations: Improved assessment procedures wouincrease the credibility of results. 1987. U. S. GeneraAccounting Office, PEMD-88-3.

5

-

,

-

s

of,

ed

8.f

ldl

Fairley, R. E. 1976. Dynamic testing of simulation softwareProc. of the 1976 Summer Computer Simulation Conf,Washington, D.C., 40–46.

Gass, S. I. 1983. Decision-aiding models: Validation, assessment, and related issues for policy analysis,Oper-ations Research, 31(4):601–663.

Gass, S. I. 1984. Documenting a computer-based modInterfaces, 14(3):84–93.

Gass, S. I. 1993. Model accreditation: A rationale andprocess for determining a numerical rating,EuropeanJournal of Operational Research, 66(2):250–258.

Gass, S. I. and L. Joel. 1987. Concepts of model confidencComputers and Operations Research, 8, 4, pp. 341–346.

Gass, S. I. and B. W. Thompson. 1980. Guidelines fomodel evaluation: An abridged version of the U.S.general accounting office exposure draft,OperationsResearch, 28(2):431–479.

Johnson, R. A. 1994. Miller and Freund’s probabilityand statistics for engineers, 5th edition, Prentice-Hall,Englewood Cliffs, N.J.

Kleijnen, J. P. C. 1987. Statistical tools for simulationpractitioners, Marcel Dekker, New York.

Kleijnen, J. P. C. 1999. Validation of models: Statisticatechniques and data availability,Proc. of 1999 WinterSimulation Conf., 647–654.

Kleindorfer, G. B. and R. Ganeshan. 1993. The philosophof science and validation in simulation,Proc. of 1993Winter Simulation Conf., 50–57.

Knepell, P. L. and D. C. Arangno. 1993.Simulation val-idation: A confidence assessment methodology, IEEEComputer Society Press.

Law, A. M. and W. D. Kelton. 1991.Simulation modelingand analysis, 2nd Ed., McGraw-Hill.

Naylor, T. H. and J. M. Finger. 1967. Verification ofcomputer simulation models,Management Science,14(2):B92–B101.

8

Sargent

i

,

ts

e

-

-

-

ds

-

,

s

en

l

l-

d

Oren, T. 1981. Concepts and criteria to assess acceptabilof simulation studies: A frame of reference,Comm. ofthe ACM, 24(4):180–189.

Rao, M. J. and R. G. Sargent. 1988. An advisory systemfor operational validity,Artificial Intelligence and Sim-ulation: The Diversity of Applications, ed. T. Hensen,Society for Computer Simulation, San Diego, CA, 245–2250.

Sargent, R. G. 1979. Validation of simulation modelsProc. of the 1979 Winter Simulation Conf., San Diego,CA,497–503.

Sargent, R. G. 1981. An assessment procedure and a secriteria for use in the evaluation of computerized modeland computer-based modeling tools, Final TechnicaReport RADC-TR-80-409.

Sargent, R. G. 1982. Verification and validation of sim-ulation models, Chapter IX inProgress in Modellingand Simulation, ed. F. E. Cellier, Academic Press,London,159–169.

Sargent, R. G. 1984. Simulation model validation,Simula-tion and Model-Based Methodologies: An IntegrativeView, ed. Oren, et al., Springer-Verlag.

Sargent, R. G. 1985. An expository on verification andvalidation of simulation models,Proc. of the 1985Winter Simulation Conf., 15-22.

Sargent, R. G. 1986. The use of graphic models in modvalidation,Proc. of the 1986 Winter Simulation Conf.,Washington, D.C., 237–241.

Sargent, R. G. 1988. A tutorial on validation and verificationof simulation models,Proc. of 1988 Winter SimulationConf., 33–39.

Sargent, R. G. 1990. Validation of mathematical models, Proc. of Geoval-90: Symposium on Validation ofGeosphere Flow and Transport Models, Stockholm,Sweden, pp. 571–579.

Sargent, R. G. 1991. Simulation Model Verification and Validation,Proc. of 1991Winter Simulation Conf., Phoenix,AZ, 37–47.

Sargent, R. G. 1994. Verification and validation of simulation models,Proc. of 1994 Winter Simulation Conf.,Lake Buena Vista, FL, 77–87.

Sargent, R. G. 1996a. Some subjective validation methousing graphical displays of data,Proc. of 1996 WinterSimulation Conf., 345–351.

Sargent, R. G. 1996b. Verifying and validating simulationmodels,Proc. of 1996 Winter Simulation Conf., 55–64.

Sargent, R. G. 1999. Validation and verification of simulation models,Proc. of 1999 Winter Simulation Conf.,39–48.

Schlesinger, et al. 1979. Terminology for model credibilitySimulation, 32(3):103–104.

Schruben, L. W. 1980. Establishing the credibility of sim-ulations,Simulation, 34(3):101–105.

59

ty

of

l

l

Shannon, R. E. 1975.Systems simulation: The art and thescience, Prentice-Hall.

Whitner, R. B. and O. Balci. 1989. Guidelines for selectingand using simulation model verification techniques,Proc. of 1989 Winter Simulation Conf., Washington,D.C., 559–568.

Wood, D. O. 1986. MIT model analysis program: Whatwe have learned about policy model review,Proc. ofthe 1986 Winter Simulation Conf., Washington, D.C.,248–252.

Zeigler, B. P. 1976.Theory of Modelling and Simulation,John Wiley and Sons, Inc., New York.

AUTHOR BIOGRAPHY

ROBERT G. SARGENT is a Research Professor and Pro-fessor Emeritus at Syracuse University. He received hieducation at The University of Michigan. Dr. Sargenthas served his profession in numerous ways and has beawarded the TIMS (now INFORMS) College on SimulationDistinguished Service Award for longstanding exceptionaservice to the simulation community. His current researchinterests include the methodology areas of both modeing and discrete event simulation, model validation, andperformance evaluation. Professor Sargent has publisheextensively and is listed inWho’s Who in America. Hisemail and web addresses are<rsargent@syr.edu> and<www.cis.syr.edu/srg/ rsargent/> .