1

Validation & Verification

Chapter 10

2

VALIDATION & VERIFICATION

• Very Difficult• Very Important• Conceptually distinct, but performed

simultaneously• You must be sure your model is correct • Your client must be confident that your

model is correct• Should be an integral part of model

building

3

VALIDATION

• Goals – Produce a model that represents system

behavior closely enough to be a substitute for the system for experimentation• Analyzing & predicting performance

– Increase credibility of model to managers & decision makers

4

DEFINITIONS

• Verification

–Ensures that the model developed is correctly implemented in the software

• Validation

–Ensures that the model accurately represents the real-world system

5

Validation & Verification Process

• An integral part of model design & implementation process – not separate

• Most methods are informal or heuristic in nature

• Experience in model development, simulation programming, & the system are essential

6

MODEL BUILDING

1. Observation of real system1. Collect data

2. Talk to members of system

2. Construct conceptual model1. Assumptions about components & structure

of system

2. Hypothesis – values of input parameters

3. Implement operational model1. Usually using simulation software

**Not linear process, iterative!!

7

SUGGESTIONS FOR VERIFICATION

1. Operational model checked by simulation software expert – not developer

2. Flow diagram for each possible action

3. Examine output for reasonableness under various inputs – use wide variety of output statistics

4. Print input at end of run to ensure stability

5. DOCUMENTATION!!!

6. Ensure animation of model is correct

8

SUGGESTIONS FOR VERIFICATION

7. Use debugger of interactive run controller (IRC) – advantages

1. Can monitor simulation progress & display results

2. Can focus on single line or process

3. Can observe model components & variables

4. Can pause; reassign values

8. GUI – always recommended

** Basic Software Engineering Principles

9

Suggestion 3

• Examine output for reasonableness– Calculate expected results– Vary input– Ask users to review

• Examples

Suppose multiple servers & only look at throughput. Maybe too many went to one server & too few to the other.

If priority queue, are they actually processed in correct order.

10

Suggestion 3 (cont’d.)

• Utilize standard output from simulation environments

• Current Count & Total Count are important variables

• Consider predictions– Mathematical (e.g. Utilization)– Experts

• Perform a Trace– Snapshots– By hand

11

CALIBRATION & VALIDATION

• Validation – comparing model to system

• Calibration – iterative process of comparing model to real system & adjusting the model – repeat!

• Comparisons– Subjective – experts review– Objective – use of data & results

12

VALIDATION

• Never truly completely validated

–Model never totally represents the real system

• Be sure model is not “fit” to one set of data

13

3-Step Approach to Validation

Naylor & Finger [1967]

1. Build a model with high face validity

2. Validate model assumptions

3. Compare model I/O transformations to corresponding I/O transformations for the real system

14

Step 1. FACE VALIDITY

• Construct a model that seems valid to the users/experts knowledgeable with system

• Include users in calibration – builds perceived credibility

• Sensitivity Analysis – change one or more input value & examine change in results – Are results consistent with real system?– Choose most critical variables to reduce

cost of experimentation

15

Step 2. Validation of Assumptions

• 2 categories of assumptions– Structural assumptions– Data Assumptions

• Structural Assumptions– Involves how system operates – Includes simplifications & abstractions of

reality

• Data Assumptions– Based on data collection & statistical

analysis

16

Step 2. Validation of Assumptions (contd)

• Review – Analysis of Data– Identify probability distribution

– Estimate parameters of distribution

– Perform goodness-of-fit test

• Chi Square, Kolmogorov-Smirnov tests

• Test homogeneity of data– Are two independently collected sets of

data come from the same parent population?

17

Step3. Validating I/O Transformations

• Ultimate Test of a Model– Ability to predict the future behavior of the

real system

• Model viewed as an I/O Transformation

• Can also us historical data to test prediction ability

• Note: If main purpose of simulation changes, model should be revalidated

18

Step 3. Validating (cont’d)

• Models are often used for comparing alternate system designs– Minor changes in parameters

• IA rate, # servers

– Minor change in statistical distribution– Major change in logical structure of

subsystem• Queue discipline

– Major design change• Manual to automated system

19

Using Historical Input Data

• An alternative to randomly generated data – don’t mix different data sets

• File, Spreadsheet, or Database– {A1, A2,…,An} & {S1, S2,…Sn}– Feed data into the FEL

• Compare output to what happened in the real system

• May be able to use technology to collect historical data for use

20

I/O Validation – Turing Test

• What is the Turing Test?• Generate 5 “fake” reports from

simulation & mix with 5 real reports; ask experts if they can distinguish fake from real

• If cannot, then pass Turing Test!

21

Validation Techniques

In order of cost-to-value ratio – Van Horn (1969, 1971)

1. Develop model with high face validity by including experts, previous research, studies, observation, experience

2. Test input data for homogeneity, randomness, goodness-of-fit

3. Turing test – use knowledgeable people

22

Validation Techniques (cont’d)

4. Compare model & system output using statistical tests

5. After model development, collect new data & repeat steps 2 to 4

6. Build new system or redesign old one, collect data on new system & use to validate model (not recommended)

7. Do little or no validation. Implement. (not recommended)

23

Conclusion

• Do not become obsessed with validation & verification to the detriment of progress; causing excessive cost

• Modeler should select techniques most valuable and appropriate to the particular system being modeled and the goals of the project

• Validate & Verify to assure Accuracy & Credibility