Statistical Analysis of Output from

Terminating Simulations

Statistical Analysis of Output from

Terminating Simulations

• Time frame of simulations

• Strategy for data collection and analysis

• Confidence intervals

• Comparing two scenarios using Output Analyzer

• Searching for an optimal scenario with OptQuest

Motivation• Run a simulation (once) — what does it mean?

• Was this run “typical” or not?

• Variability from run to run (of the same model)?

• Need statistical analysis of output data

• From a single model configuration

• Compare two or more different configurations

• Search for an optimal configuration

• Statistical analysis of output is often ignored

• This is a big mistake – no idea of precision of results

• Not hard or time-consuming to do this – it just takes a little planning and thought, then some (cheap) computer time

Time Frame of Simulations

• Terminating: Specific starting, stopping conditions

– Run length will be well-defined (and finite)

• Steady-state: Long-run

– Over theoretically infinite time frame

– Initial conditions for the simulation don’t matter

– A steady-state simulation has to stop at some point

and as you might guess these runs can be pretty long

Strategy for Data Collection and Analysis

• For terminating case, make some number n (IID) of independent replications:

– To do this, just open the Run > Setup > Replication Parameters dialog box and enter the value of n you want for the number of replications.

– Check both boxes for Initialize Between Replications

To cause both the system state variables and the statistical accumulators to be cleared at the end of each replication.

Separate results for each replication –Category by Replication report

for 10 replications

Replication Number served Finished Time(hours) Average delay in

queue (minutes)Average Queue Length

1 469 8.243 1.0452 0.9869

2 482 8.045 1.279 1.2735

3 509 8.026 1.5341 1.6151

4 530 8.049 1.5592 1.7045

5 510 8.125 2.4832 2.5874

6 548 8.15 4.7143 5.2616

7 484 8.103 3.7054 3.6731

8 490 8.36 2.62 2.55

9 511 8.2 1.33 1.37

10 483 8.148 1.09 1.07

– Consider a bank with 5 tellers, one FIFO queue, open 9am

to 5pm, flush out before stopping.

– Inter-arrivals ~ expo (mean = 1 min.), service times ~ expo

(mean = 4 min.)

– Simulating 10 days (replications) starting with no

customers, the summary measures are as follows.

Note that the “variability” of summary measures across replications.

Confidence Intervals for Terminating

Systems

Replication Number served Finished Time(hours) Average delay in queue

(minutes)Average Queue Length

1 469 8.243 1.0452 0.9869

2 482 8.045 1.279 1.2735

3 509 8.026 1.5341 1.6151

4 530 8.049 1.5592 1.7045

5 510 8.125 2.4832 2.5874

6 548 8.15 4.7143 5.2616

7 484 8.103 3.7054 3.6731

8 490 8.36 2.62 2.55

9 511 8.2 1.33 1.37

10 483 8.148 1.09 1.07

Sample Mean 2.13604 2.20921

Standard deviation 1.239555412 1.362130309

Half width 0.886662919 0.974341627

Min 1.0452 0.9869

Max 4.7143 5.2616

Half Width, Number of Replications

• Prefer smaller confidence intervals — precision

• Notation:

• Confidence interval:

• Half-width =

• Can’t control t or s

• Must increase n — how much?

• Depends on how one defines precision

Confidence interval =95% ,n=10 replications

s(waiting time)=1.23

Half-width =0.886

point estimate=2.13

Confidence Interval=2.13 0.886

Confidence Intervals for

Terminating Systems– If > 1 replication, Arena uses cross-repl. data as

above

– Possibly most useful part: 95% confidence interval on expected values

Half Width, Number of Replications

Want this to be “small,” say < h where h is prespecified

• Set half-width = h, solve for n

• Not really solved for n (t, s depend on n)

• Approximation:

Replace t by z, corresponding normal critical value

Pretend that current s will hold for larger samples

Half Width, Number of Replications

• Get

12

s = sample standard

deviation from “initial”

number n0 of replications

For confidence interval =95%,z=1.96,h=10% ( )=0.213

n 1.96 1.23 = 128 replications

After running the model for 128 replications

n grows as h decreases

Half Width, Number of Replications

• Easier but different approximation:

h0 = half width from “initial” number n0 of replications

n 10 0.886 = 173 replications

Warm Up Period

Steady-State : beyond which the output appears not to be

drifting any more statistically.

Arena’s companion package “Output Analyzer” can be

used to get a plot of any model parameter for the

replication.

In our example I want a plot for Total WIP and for Total

Number in Queue for 10 replications.

Using statistic module:

Save the output file as .dat

Warm Up Period

• Open Output Analyzer

• Click on plot icon

• Search for the file you save choose for replication All

Warm Up Period

Total WIP

Total Number in Queue

Warm Up Period

Comparing Two Scenarios

• Two scenarios:

1)Base scenario is Bank Example

2) New scenario is that we hire 6 tellers instead of 5 tellers.

replications Scenario 1(waiting time in queue)W1 Scenario 2(waiting time in queue)W2 Z=W1-W2

1 1.0452 0.3651 0.6801

2 1.279 0.2145 1.0645

3 1.5341 0.4643 1.0698

4 1.5592 0.3585 1.2007

5 2.4832 0.1066 2.3766

6 4.7143 0.4175 4.2968

7 3.7054 0.6298 3.0756

8 2.62 0.7119 1.9081

9 1.33 0.5928 0.7372

10 1.09 0.7317 0.3583

Average 1.67677

Standard deviation 1.244945066

confidence intetval 1.68 0.8905

Comparing Two Scenarios

• Reasonable but not-quite-right idea

– Make confidence intervals on expected outputs from

each scenario, see if they overlap; look at total wait

time in queue.

– At 95% t-paired Confidence interval [0.78,2.57]

– Since the interval does not contain 0 , this implies that

the two systems are sensibly different .In addition

system 2 seems to be better (as the customer stays

lower time in the system)

– But this doesn’t allow for a precise, efficient statistical

conclusion

Comparing Two Scenarios Using

Output Analyzer

• Output Analyzer is a separate application that operates

on .dat files produced by Arena

– Launch separately from Windows, not from Arena

• To save output values (Expressions) of entries in

Statistic data module (Type = Output) – enter

filename.dat in Output File column

• Did for both scenarios

Comparing Two Scenarios Using Output

Analyzer

• Analyze > Compare Means menu option

– Add data files … “A” and “B” for two scenarios

– Select “Lumped” for Replications field

– Title, confidence level, accept Paired-t Test, do not

Scale Display since two output performance

measures have different units

Compare Means via Output Analyzer

• Results:

• Point estimate = 1.68

• At 95% t-paired Confidence interval [0.78,2.57]

Comparing Two Scenarios Using Output

Analyzer over 100 Replications

• Point estimate = 1.61

• At 95% t-paired Confidence interval [1.37,1.84]

Searching for an Optimal Scenario with

OptQuest (not included in student

version of Arena)

• Seek input controls minimizing Total Cost while

keeping Percent Rejected ≤ 5

– Explore all possibilities – add resources in any

combination

– New rules:

26 number of trunk lines 50

Total number of new employees of all five types 15

OptQuest(Model 5-3)

• Formulate as an optimization problem:Minimize Total Cost

Subject to

26 MR(Trunk Line) 50

0 New Sales + New Tech 1 + New Tech 2 + New Tech 3 + New Tech All 15

Percent Rejected 5

OptQuest requires finite Replication Length

• Tools > OptQuest for Arena

• New Optimization or Browse for saved one (.opt)

• Tree on left, expand for Controls and Responses

• File >OptQuest Optimization

Objective function is a simulation-model output

Constraints

on input

control

(decision)

variables

Constraint on another output

OptQuest

In this toolbar we will

input our optimization

problem parameters

Searching for an Optimal Scenario with

OptQuest – Controls, Responses

• Controls → Resources → Trunk Line

– Integer, Lower Bound = 26, Suggested Value = 29,

Upper Bound = 50

• Controls → User Specified → New Sales

– Integer, Lower Bound = 0, Suggested Value = 3,

Upper Bound = 15

– Click on “Included” to collect selections at top or

bottom

• Responses → User Specified → Output

– Check Percent Rejected, Total Cost

Slide 27 of 31

Searching for an Optimal Scenario with

OptQuest – Constraints, Objective

• Constraints

– Add button, then each of first five controls, “+”, then

“<= 15”

– Add button, then Percent Rejected, then “<= 5”

• Objectives

– Add button, Total Cost, Minimize radio button

• Options

– Stopping rules

– Tolerance for regarding results as “equal”

– Replications per simulation

– Solutions log file location

Searching for an Optimal Scenario with

OptQuest – Running

• or Run > Start or F5

– Optimization branch on tree to watch progress,

scenarios so far, best scenario so far

• Can’t absolutely guarantee a true optimum

– But usually finds far better configuration than possible by hand

Output