Confidence Interval Estimation in System Dynamics Models

Gokhan Dogan*

MIT Sloan School of Management

System Dynamics Group

*Special thanks to John Sterman for his support

Motivation

Calibration

Manual Calibration Automated Calibration (e.g. Vensim, Powersim)

Automated Calibration

02468

10121416

Time

Actual Data

Model Output

Motivation

• Once model parameters are estimated with automated calibration,

next step: Estimate confidence intervals!

• Questions:

-Are there available tools at software packages?

-Do these methods have any limitations?

-Are there alternative methods?

Why are confidence intervals important

Parameter Estimate

θ

95% Confidence

Interval

0 Parameter Estimate

θ

95% Confidence

Interval

0

We reject the claim that the parameter value is equal to 0

(with 95% probability)

We can’t reject the claim that the parameter value is equal

to 0 (with 95% probability)

How can we estimate confidence intervals?

Likelihood Ratio Method

Bootstrapping

Used in the System Dynamics Software (Vensim) /Literature

The method we suggest for System Dynamics models

Both methods yield approximate confidence intervals!

Likelihood Ratio Method

• The likelihood ratio method is used in system dynamics software packages (Vensim) and literature (Oliva and Sterman, 2001).

• It relies on asymptotic theory (large sample assumption).

However

Likelihood Ratio Method (as it is used at software packages) assumes:

At system dynamics models:

-Large Sample -It is not always possible to have large sample

-No feedback (autocorrelation)

-There are many feedback loops

-Normally distributed error terms

-Error terms are not always normally distributed

Bootstrapping

• Introduced by Efron (1979) and based on resampling. Extensive survey in Li and Maddala (1996).

• It seems more appropriate for system dynamics models because- It doesn’t require large sample- It is applicable when there is feedback (autocorrelation) - It doesn’t assume normally distributed error terms

Drawbacks of bootstrapping

• The software packages do not implement it.

• It is time consuming.

Bootstrapping

ERROR TERMS

8

4

0

-4

-8

1 13 25 36 48Time (Week)

Error Terms : 15289pres

MODEL_vs_HISTORICAL_DATA

20

15

10

5

0

1 13 25 36 48Time (Week)

Historical Data : 15289pres casesModel Output : 15289pres cases

Compute the Error Terms

Fit the model and estimate parameters

Bootstrapping uses resampling

ERROR TERMS

8

4

0

-4

-8

1 13 25 36 48Time (Week)

Error Terms : 15289pres

Nonparametric: Reshuffle Them and

Generate many many new error term sets

using the autocorrelation information

Parametric: Fit a distribution and

Generate many many new error term sets

using the autocorrelation and distribution information

Resampling the Error Terms

• If we know that:- The error terms are autocorrelated- Their variance is not constant (heteroskedasticity)- They are not normally distributed=> We can use this information while resampling the error terms

• Flexibility of bootstrapping stems from this stage

ERROR TERMS

8

4

0

-4

-8

1 13 25 36 48Time (Week)

Error Terms : 15289pres

ERROR TERMS

20

10

0

-10

-20

1 13 25 36 48Time (Week)

Residuals : subject1_ar1_inf_btstrp5

ERROR TERMS

6

3

0

-3

-6

1 13 25 36 48Time (Week)

Residuals : subject1_ar1_inf_btstrp4

ERROR TERMS

8

4

0

-4

-8

1 13 25 36 48Time (Week)

Residuals : subject1_ar1_inf_btstrp2

. . .

MODEL OUTPUT

20

15

10

5

0

1 13 25 36 48Time (Week)

Model Output : 15289pres cases

HISTORICAL DATA

20

15

10

5

0

1 13 25 36 48Time (Week)

Actual Orders : subject1_ar1_inf_btstrp2

HISTORICAL DATA

20

15

10

5

0

1 13 25 36 48Time (Week)

Actual Orders : subject1_ar1_inf_btstrp4

HISTORICAL DATA

20

15

10

5

0

1 13 25 36 48Time (Week)

Actual Orders : subject1_ar1_inf_btstrp5

. . .

+ =

FABRICATED ERROR TERMS

FABRICATED “HISTORICAL” DATA

HISTORICAL DATA

20

15

10

5

0

1 13 25 36 48Time (Week)

Actual Orders : subject1_ar1_inf_btstrp2

HISTORICAL DATA

20

15

10

5

0

1 13 25 36 48Time (Week)

Actual Orders : subject1_ar1_inf_btstrp4

HISTORICAL DATA

20

15

10

5

0

1 13 25 36 48Time (Week)

Actual Orders : subject1_ar1_inf_btstrp5

. . .

FABRICATED “HISTORICAL” DATA

Fit the model and estimate parameters

Fit the model and estimate parameters

Fit the model and estimate parameters

Parameter Estimate

Parameter Estimate

Parameter Estimate

500

Parameter Estimates

Distribution of a model parameter

Experiments

• We had experimental time series data from 240 subjects.

• Subjects were beer game players.

• For each subject we had 48 data points, so we estimated parameters and confidence intervals using 48 data points.

Model (Same as Sterman 1989)

• Ot = Max[0, θLRt + (1–θ)ELt + α(S' – St –βSLt) + error termt]

Parameters to be estimated are θ, α, β, S‘

Individual Results

θ=0.95

10

95% Confidence Intervals for θ

10

θ=0.95

0.77

0.01

95% CI

95% CI

Likelihood Ratio Method

Bootstrapping

Individual Results

β =0.01

0.20

95% Confidence Intervals for β

95% CI

Likelihood Ratio Method

Bootstrapping

β =0.01

0.20 95% CI

Significantly Different From 0!!!

Overall Results

  Theta Alpha Beta S-Prime

Likelihood Ratio Method 0.19 0.11 0.11 13.20

Bootstrapping 0.67 0.30 0.52 973.59

Average 95% Confidence Interval Length

  Theta Alpha Beta S-Prime

Likelihood Ratio Method 0.10 0.08 0.06 2.32

Bootstrapping 0.84 0.24 0.48 10.10

Median of 95% Confidence Interval Length

Overall Results

Theta Alpha Beta S-Prime

Bootstrapping CI wider than Likelihood Ratio Method CI

97.76% 98.81% 100% 98.56%

Percentage of Subjects for whom the bootstrapping confidence interval is wider than the likelihood ratio method confidence interval

Likelihood Ratio Method vs Bootstrapping

• Likelihood Ratio Method: Is easy to compute Very fast BUT depends on

assumptions that are usually violated by system dynamics models

Yields very tight confidence intervals

• Bootstrapping: Is NOT easy to compute Takes longer time DOES NOT depend on

assumptions that are usually violated by system dynamics models

Yields larger confidence intervals. Usually more conservative.