Date post: | 09-Apr-2018 |
Category: |
Documents |
Upload: | vinayak-gururaj |
View: | 249 times |
Download: | 0 times |
of 36
8/8/2019 7 Verification Validation
1/36
1
Verification and Validationof Simulation Models
Dr. A. K. Dey
8/8/2019 7 Verification Validation
2/36
2
Verification and Validation
of Simulation Models
Verification: concerned with building the model
right. It is utilized in the comparison of the
conceptual model to the computer representationthat implements that conception.
It asks the questions: Is the model implemented
correctly in the computer? Are the input
parameters and logical structure of the modelcorrectly represented?
8/8/2019 7 Verification Validation
3/36
3
Verification and Validation
of Simulation Models
Validation: concerned with building the rightmodel. It is utilized to determine that a model is
an accurate representation of the real system.Validation is usually achieved through thecalibration of the model, an iterative process ofcomparing the model to actual system behaviorand using the discrepancies between the two, andthe insights gained, to improve the model. Thisprocess is repeated until model accuracy is judgedto be acceptable.
8/8/2019 7 Verification Validation
4/36
4
Model building, verification & validation
First step Observe the real system Observe interactions among their various components Collect data on their behaviour
Second step Construct a conceptual model: a collection of assumptions
about the components and the structure of the system, plushypotheses about the values of the model input parameters
It is comparison of the of the real system to the conceptual
model Third step
Implementation of a operational model by using asimulation software
8/8/2019 7 Verification Validation
5/36
5
Verification of Simulation
Models
Many commonsense suggestions can be given for
use in the verification process.
1. Have the code checked by someone other than
the programmer.
2. Make a flow diagram which includes each
logically possible action a system can take whenan event occurs, and follow the model logic for
each action for each event type.
8/8/2019 7 Verification Validation
6/36
6
Verification of Simulation
Models
3. Closely examine the model output for
reasonableness under a variety of settings of the
input parameters. Have the code print out a widevariety of output statistics.
4. Have the computerized model print the input
parameters at the end of the simulation, to be
sure that these parameter values have not beenchanged inadvertently.
8/8/2019 7 Verification Validation
7/36
7
Verification of Simulation
Models
5. Make the computer code as self-documenting as
possible. Give a precise definition of every
variable used, and a general description of thepurpose of each major section of code.
These suggestions are basically the same ones
any programmer would follow when debugging acomputer program.
8/8/2019 7 Verification Validation
8/36
8
Calibration and Validation
of Models
R e a l
S y s t e m
In i t ia l
M o d e l
F i rs t rev is ion
o f m o d e l
S e c o n d
rev is iono f m o d e l
R e v i s e
R e v i s e
R e v i s e
C o m p a r e m o d e l
to rea l ity
C o m p a r e
r e v is e d m o d e l
to real i ty
C o m p a r e 2 n d
r e v is e d m o d e l
to real i ty
8/8/2019 7 Verification Validation
9/36
9
Validation of Simulation Models
As an aid in the validation process, Naylor and Fingerformulated a three-step approach which has been
widely followed:
1. Build a model that has high face validity
Should appear to be reasonable to model users and otherknowledgeable (about the real system) people
Users may be asked if the model behaves in the expected
manner if the input values are changed (sensitivity analysis)
If there are many input variables, select critical variables for
tests
8/8/2019 7 Verification Validation
10/36
10
Validation of Simulation Models
2. Validate model assumptions: Two classes of assumptions- Structuraland Data
Structural: Number of lines, One line for each server or one line fro multiple
servers, Customers could change lines if one line is moving faster, No.
of tellers could be fixed or variable These should be verified by actual observations at appropriate time
periods also by knowing the policies of the organization Data
Collection of reliable data and correct statistical analysis Examples: Inter arrival time during peak and slack periods, service
times for corporate and personal accounts Homogeneity tests and correlation tests should be carried out
Homogeneity: Do data sets collected at two different peak hours have come fromthe same population? If so, the two sets can be combined
Correlation: As soon as the analyst is assured of dealing with a random sample (i.e.,correlation is not present) the statistical analysis can begin
8/8/2019 7 Verification Validation
11/36
11
Validation of Simulation Models
Analysis consists of three steps Identify an appropriate probability distribution
Estimate the parameters of the hypothesized distribution
Validate the assumed statistical model by a goodness-of-fit test: Chi Square or
K S tests and by graphical method
Use of goodness-of-fit test is an important part of validation of
assumptions
3. Compare the model input-output transformations to
corresponding input-output transformations for the real system.
8/8/2019 7 Verification Validation
12/36
12
Validation of
Model Assumptions
Structuralinvolves questions of how the system operate
(Example1)
Data assumptions should be based on the collection of reliabledata and correct statistical analysis of the data.
Customers queueing and service facility in a bank (one line ormany lines)
1. Interarrival times of customers during several 2-hourperiods of peak loading (rush-hour traffic)
2. Interarrival times during a slack period
3. Service times for commercial accounts
4. Service times for personal accounts
8/8/2019 7 Verification Validation
13/36
13
Validation of
Model Assumptions
The analysis of input data from a random sampleconsists of three steps:
1. Identifying the appropriate probability distribution
2. Estimating the parameters of the hypothesizeddistribution
3. Validating the assumed statistical model by agoodness-of fit test, such as the chi-square or
Kolmogorov-Smirnov test, and by graphicalmethods.
8/8/2019 7 Verification Validation
14/36
14
Validating Input-Output
Transformation
(Example) : The Fifth National Bank of Jaspar
The Fifth National Bank of Jaspar, as shown in thenext slide, is planning to expand its drive-in
service at the corner of Main Street. Currently,there is one drive-in window serviced by one teller.Only one or two transactions are allowed at thedrive-in window, so, it was assumed that eachservice time was a random sample from some
underlying population. Service times {Si, i = 1,2, ... 90} and interarrival times {Ai, i = 1, 2, ... 90}
8/8/2019 7 Verification Validation
15/36
15
S T O P
Main
Street
M a in S t r e e t
Y ' a ll C o m e
B a c k
W e l c o m e
to J a s p a r
J a s p a r
B a n k
Validating Input-Output
Transformation
Drive-in window at the
Fifth National Bank.
8/8/2019 7 Verification Validation
16/36
16
Validating Input-Output
Transformation
were collected for the 90 customers who arrivedbetween 11:00 A.M. and 1:00 P.M. on a Friday.This time slot was selected for data collectionafter consultation with management and the tellerbecause it was felt to be representative of atypical rush hour. Data analysis led to theconclusion that the arrival process could bemodeled as a Poisson process with an arrival
rate of 45 customers per hour; and that servicetimes were approximately normally distributedwith mean 1.1 minutes and
8/8/2019 7 Verification Validation
17/36
17
Validating Input-Output
Transformation
standard deviation 0.2 minute. Thus, the model
has two input variables:
1. Interarrival times, exponentially
distributed (i.e. a Poisson arrivalprocess) at rate = 45 per hour.
2. Service times, assumed to be N(1.1,
(0.2)2)
8/8/2019 7 Verification Validation
18/36
18
Validating Input-Output
Transformation
M
O
D
EL
Poisson arrivalsX11 , X12,...rate = 45/hour
Service timesX21 , X22,...
N(D2,0.22
)
One teller
D1 = 1
Mean service time
D2 = 1.1 minutes
One line
D3 = 1
Random
variables
Decision
variables
black
box
Tellers utilization
Y1 =
Average delay
Y2
Maximum line length
Y3
Input variables Model Output variables
Model input-output transformation
8/8/2019 7 Verification Validation
19/36
19
Validating Input-Output
Transformation
The uncontrollable input variables aredenoted by X, the decision variables by D,and the output variables by Y. From the
black box point of view, the model takesthe inputs X and D and produces theoutputs Y, namely
(X, D) f Y
or
f(X, D) = Y
8/8/2019 7 Verification Validation
20/36
20
Validating Input-Output
Transformation
Input Variables Model Output Variables, Y
D = decision variables Variables of primary interest
X = other variables to management (Y1, Y
2, Y
3)
Poisson arrivals at rate Y1= tellers utilization
= 45 / hour Y2= average delay
X11, X
12,.... Y
3= maximum line length
Input and Output variables for model of current bank operation (1)
8/8/2019 7 Verification Validation
21/36
21
Validating Input-Output
Transformation
Input Variables Model Output Variables, Y
Service times, N(D2,0.22) Other output variables of
X21 , X22 ,..... secondary interest
Y4 = observed arrival rate
D1 = 1 (one teller) Y5 = average service time
D2 = 1.1 min Y6 = sample standard deviation
(mean service time) of service times
D3
= 1 (one line) Y7
= average length of line
Input and Output variables for model of current bank operation (2)
8/8/2019 7 Verification Validation
22/36
22
Validating Input-Output
Transformation
Statistical Terminology Modeling Terminology Associated Risk
Type I : rejecting H0
Rejecting a valid model when H0 is true
Type II : failure to reject H0 Failure to reject an
when H0 is false invalid model
Note: Type II error needs controlling increasing will decrease and vice versa, given a fixed sample size.Once is set, the only way to decrease is to increasethe sample size.
(Table 1) Types of error in model validation
8/8/2019 7 Verification Validation
23/36
8/8/2019 7 Verification Validation
24/36
24
Validating Input-Output
Transformation
Z2 = 4.3 minutes, the model responses, Y2. Formally,
a statistical test of the null hypothesis
H0 : E(Y2) = 4.3 minutes
versus ----- (Eq 1)H1 : E(Y2) 4.3 minutes
is conducted. If H0 is not rejected, then on the
basis of this test there is no reason to consider
the model invalid. If H0 is rejected, the currentversion of the model is rejected and the modeleris forced to seek ways to improve the model, asillustrated in Table 3.
8/8/2019 7 Verification Validation
25/36
25
Validating Input-Output
Transformation
As formulated here, the appropriate statistical test is
the t test, which is conducted in the following
manner:
Step 1. Choose a level of significance and asample size n. For the bank model, choose
= 0.05, n = 6
Step 2. Compute the sample mean Y2 and the
sample standard deviation S over the n replications.
Y2 = {1/n} Y2i = 2.51 minutes
n
1i=
8/8/2019 7 Verification Validation
26/36
26
and
S = { (Y2i - Y2)2 / (n - 1)}1/2 = 0.82 minute
where Y2i , i = 1, .., 6, are shown in Table 2.
Step 3. Get the critical value oft from Table A.4. For
a two-sided test such as that in Equation 1, use
t /2, n-1 ; for a one-sided test, use t , n-1 or -t , n-1 as
appropriate (n -1 is the degrees of freedom). From
Table A.4, t0.025,5 = 2.571 for a two-sided test.
Validating Input-Output
Transformation
n
1i=
8/8/2019 7 Verification Validation
27/36
27
Validating Input-Output
Transformation
Step 4. Compute the test statistic
t0 = (Y2 - 0) / {S / n} ----- (Eq 2)where 0 is the specified value in the null
hypothesis, H0 . Here 0 = 4.3 minutes, so thatt0 = (2.51 - 4.3) / {0.82 / 6} = - 5.34
Step 5. For the two-sided test, if |t0| > t /2, n-1 , reject H0. Otherwise, do not reject H0. [For the one-sided
test with H1: E(Y2) > 0, reject H0 if t > t , n-1 ; withH1 : E(Y2) < 0 , reject H0 if t < -t , n-1]
8/8/2019 7 Verification Validation
28/36
8/8/2019 7 Verification Validation
29/36
29
Validating Input-Output
Transformation
Y4
Y5
Y2=avg delay
Replication (Arrivals/Hours) (Minutes) (Minutes)
1 51 1.07 5.37
2 40 1.11 1.98
3 45.5 1.06 5.294 50.5 1.09 3.82
5 53 1.08 6.74
6 49 1.08 5.49
sample mean 4.78standard deviation 1.66
(Table 3) Results of six replications of the REVISED Bank Model
8/8/2019 7 Verification Validation
30/36
30
Validating Input-Output
Transformation
Step 1. Choose = 0.05 and n = 6 (sample size).
Step 2. Compute Y2 = 4.78 minutes, S = 1.66
minutes ----> (from Table 3)
Step 3. From Table A.4, the critical value is
t0.025,5 = 2.571.
Step 4. Compute the test statistic
t0 = (Y2 - 0) / {S / n} = 0.710.Step 5. Since | t |< t0.025,5= 2.571, do not reject H0 , and
thus tentatively accept the model as valid.
8/8/2019 7 Verification Validation
31/36
8/8/2019 7 Verification Validation
32/36
32
Validating Input-Output
Transformation
Type II error ( ) versus for given samplesize n. Table A.9 is for a two-sided ttest while
Table A.10 is for a one-sided ttest. Suppose that
the modeler would like to reject H0 (modelvalidity) with probability at least 0.90 if the true
means delay of the model, E(Y2), differed from
the average delay in the system, 0 = 4.3
minutes, by 1 minute. Then is estimates by = | E(Y2) - 0 | / S = 1 / 1.66 =
0.60
8/8/2019 7 Verification Validation
33/36
33
Validating Input-Output
Transformation
For the two-sided test with = 0.05, use of TableA.9 results in
( ) = (0.6) = 0.75 for n = 6
To guarantee that ( ) 0.10, as was desiredby the modeler, Table A.9 reveals that a samplesize of approximately n = 30 independentreplications would be required. That is, for asample size n = 6 and assuming that the
population standard deviation is 1.66, theprobability of accepting H0 (model validity) , when
in fact the model is invalid
8/8/2019 7 Verification Validation
34/36
34
Validating Input-Output
Transformation
(| E(Y2) - 0 | = 1 minute), is = 0.75, which isquite high. If a 1-minute difference is critical, and
if the modeler wants to control the risk of
declaring the model valid when model predictionsare as much as 1 minute off, a sample size of n =
30 replications is required to achieve a power of
0.9. If this sample size is too high, either a higher
risk (lower power), or a larger difference ,must be considered.
8/8/2019 7 Verification Validation
35/36
35
8/8/2019 7 Verification Validation
36/36
36