Performance Metrics For Evaluating LNG Vapor …psc.tamu.edu/files/symposia/2009/presentations/2...

Mary Kay O’Connor Process Safety Center2009 International Symposium

College Station, Texas

- 1 -

Performance MetricsPerformance Metrics For EvaluatingFor Evaluating

LNG Vapor Dispersion ModelsLNG Vapor Dispersion Modelsby

Frank A. Licari, PE, CSPUnited States Department of Transportation

Pipeline and Hazardous Materials Safety AdministrationPipeline Safety Office

Washington, DC





- 2 -

AgendaAgenda

• Historical Perspective of Metrics

• Novel Performance Metric - MSWC

• Methodology to Calculate MSWC

• Example Calculations

• Error Analyses & Their Importance

• Conclusions



- 3 -

Traditional Metrics ValidateTraditional Metrics Validate Vapor Dispersion Model PerformanceVapor Dispersion Model Performance

Model Comparisons – Hanna et al [2]



- 4 -

Historical Perspective of MetricsHistorical Perspective of Metrics

1980s Havens & Spicer DEGADIS

1993 Hanna et al Comparative Study

2001 Carissimo et al SMEDIS Validation

2004 Chang & Hanna Model Performance



- 5 -

Historical Perspective of MetricsHistorical Perspective of Metrics

• Are Valuable Tools To Validate Dispersion Models

• Traditional Statistical Methodologies:

– characterize strengths of models and

– identify their best applications

• Yet, Past Metrics Don’t Describe:

– extra separation distance that protects the public or

– additional confidence in a model’s predictions



- 6 -

New Metric DescribesNew Metric Describes ModelModel’’s Inherent Safety & Confidences Inherent Safety & Confidence



- 7 -

Novel Performance Metric Novel Performance Metric -- MSWCMSWC

• Margin of Safety With Confidence

– is a statistical tool

– quantifies model performance

– allows models to be compared

– describes model’s minimum margin of safety

– accurately describes confidence level of model predictions for 30+ data pairs



- 8 -

Novel Performance Metric Novel Performance Metric -- MSWCMSWC

• Margin of Safety for One Prediction Is

• Margin of Safety of a Vapor Dispersion Model Is a Range of Values Due to:

– atmospheric conditions

– local terrain

– test error

– modeling assumptions

– computational error

i

i

OP

iMs



- 9 -

Methodology to Calculate Methodology to Calculate MSWCMSWCLNG Test &

Atmospheric Stability (Pred/ObsRatio)

Burro 8 - E 0.716

1.462

0.798

0.683

Burro 9 - C 1.885

1.629

1.669

Maplin 29 - D 0.775

0.803

1.137

0.972

1.231

1.424

1.204

Maplin 39 - D 0.541

1.147

1.139

2.111

1.554

1.672

2.319

Table 1 – Excerpt of Havens 1992 Gas Concentration Ratios [5]

iMs • DEGADIS Dispersion Model Predictions

• Gas Concentration Data from LNG Field Tests

• 21 Data Ratios

• Range = .541 to 2.319

• = 1.28

x

x



- 10 -

Methodology to Calculate Methodology to Calculate MSWCMSWC

Histogram of in Table 1iMs



- 11 -

Methodology to Calculate Methodology to Calculate MSWCMSWCLNG Test &


Burro 8 - E 0.716

1.462

0.798

0.683

Burro 9 - C 1.885

1.629

1.669

Maplin 29 - D 0.775

0.803

1.137

0.972

1.231

1.424

1.204

Maplin 39 - D 0.541

1.147

1.139

2.111

1.554

1.672

2.319

Table 1 – Excerpt of Havens 1992 Gas Concentration Ratios [5]

iMs

= .49

= 1.0

= -.57

Confidence Level = 72%

desiredMs

MSWC = 1.0 with 72% Confidence

1

)Ms( 2i

1

n

xS

n

iMs

Ms

desired

SxMs

scoreZ



- 12 -Figure 1 - Determines and Confidence Level

= 1.281.0 =

desiredMs

desiredMs

desiredMs

scoreZ

72%



- 13 -

MSWC MSWC Explains HowExplains How

desiredMs

• Model’s Inherent Margin of Safety Is 1.0 or More

• 72 % of Gas Concentration Predictions Equal or Exceed LNG Field Trial Observations

• Models May Be Evaluated By Comparing Their

– inherent margins of safety (or safety buffers)

– confidence level (bias to over or under predict)



- 14 -

MSWCMSWC Explains ModelExplains Model’’s Accuracys Accuracy & Shapes Evaluation Decision& Shapes Evaluation Decision



- 15 -

MSWCMSWC Example for Distance PredictionsExample for Distance Predictions

• Explains Distance Prediction Concepts for Siting LNG Facilities

• Describes Importance of Societal Risk Preferences

• Calculates MSWC for 2 Geographic Regions

• Compares Regional Decisions to Accept DEGADIS Distance Predictions



- 16 -

Distance Prediction ConceptsDistance Prediction Concepts

100% LFL

50% LFL

• Site property line so hazards of flammable gas during an LNG spill remain in facility

• Gas concentration at 100 % of the lower flammability limit (LFL) is min. distance

• 50% LFL is NFPA 59A required distance

Distance to LNG Facility’s Property Line

Property Line



- 17 -

MSWCMSWC Example Example -- Societal Risk PreferencesSocietal Risk Preferences

• Property Line of LNG Facility May Extend to 100% LFL

• Region A Prefers Safety Buffer & = 1.5

• Region R Prefers No Safety Buffer & = 1.0

• Each Region Expects DEGADIS Predictions to Have High Confidence Levels

AdesiredMs

RdesiredMs



- 18 -

MSWCMSWC AA ExampleExample of Distance Predictionsof Distance PredictionsLNG Test & Atmospheric

Stability

Flammability Limit (LFL)

Observed

Distance (m)

Predicted

Distance (m)

iMs (Pred/Obs

Ratio)

Burro 8 - E 50% 700 550 0.786 Burro 9 - C 50% 480 700 1.458

Maplin 29 - D 50% 280 300 1.071 Maplin 39 - D 50% 230 400 1.739

Burro 8 - E 100% 360* 360* 1.000* Burro 9 - C 100% 240* 450* 1.875*

Maplin 29 - D 100% 150* 180* 1.200* Maplin 39 - D 100% 125* 220* 1.760*

MSWCA = 1.5 with 37% Confidence

Table A.1 – for Distances at 50 & 100 Percent LFLiMs

*data extrapolated from Figures 3 through 6 [5]

x MsS AdesiredMs= 1.36 = .40 = 1.5 = .34A

scoreZ



- 19 -

MSWCMSWC R R ExampleExample of Distance Predictionsof Distance PredictionsLNG Test & Atmospheric

Stability

Flammability Limit (LFL)

Observed

Distance (m)

Predicted

Distance (m)

iMs (Pred/Obs

Ratio)

Burro 8 - E 50% 700 550 0.786 Burro 9 - C 50% 480 700 1.458

Maplin 29 - D 50% 280 300 1.071 Maplin 39 - D 50% 230 400 1.739

Burro 8 - E 100% 360* 360* 1.000* Burro 9 - C 100% 240* 450* 1.875*

Maplin 29 - D 100% 150* 180* 1.200* Maplin 39 - D 100% 125* 220* 1.760*

MSWCR = 1.0 with 81% Confidence

Table A.1 – for Distances at 50 & 100 Percent LFLiMs


x MsS RdesiredMs= 1.36 = .40 = 1.0 = -.89 R

scoreZ



- 20 -Figure 2 - & Shape an Evaluation DecisionRMSWC AMSWC

= 1.5

1.36

of 37%

1.081%



- 21 -

FindingsFindings From Distance Prediction ExampleFrom Distance Prediction Example

desiredMs

• Siting a LNG Facility Property Line at 100% LFL Reduces Safety Buffer to Zero

• Desired, Minimum Margin of Safety Shapes Region’s Acceptance Decision

• Region A May Reject Model; It Overpredicts by Factor of 1.5 with 37% Confidence

• Region R May Accept Model, If Its Constituents Believe Safety Buffer is Unnecessary



- 22 -

What Is Size What Is Size of Modelof Model’’s Safety Buffer?s Safety Buffer?



- 23 -

MSWCMSWC DD of Exclusion Zone Predictionsof Exclusion Zone Predictions

desiredMs

• Describe Margin of Safety for an Exclusion Zone Prediction

• Calculate MSWC for Exclusion Zone Predictions

• Compare MSWC for Exclusion Zone Predictions to Distance Predictions



- 24 -

Exclusion Zone Creates Safety BufferExclusion Zone Creates Safety Buffer

desiredMs

100% LFL

50% LFL

During LNG Spill, Exclusion Zone at 50% LFL Separates Public from

Hazards of Flammable Gas at 100% LFL

Property Line



- 25 -

Part 193 Requires Margin of SafetyPart 193 Requires Margin of Safety

desiredMs

• 49CFR Part 193 & NFPA 59A (2001 edition) Establish LNG Facility Property Line at 50% LFL

• Safety Buffer Protecting Public Is Inherent Margin of Safety of 50 vs. 100% LFL

• Margin of Safety for DEGADIS Prediction of Exclusion Zone Distance Is:

LFLi

LFLi

OP

%100

%50DiMs



- 26 -

MSWC MSWC DD Example Example -- Exclusion Zone DistancesExclusion Zone Distances

MSWCD = 1.5 with 88% Confidence

Table A.1 – for Distances at 50 & 100 Percent LFLDiMs


x MsS AdesiredMs= 2.41 = .78 = 1.5 = -1.17A

scoreZ

LNG TestsAtmospheric

Stability

100% LFL Observed

Distance (m)*

50% LFL Predicted

Distance (m)(Pred/Obs

Ratio)

Burro 8 E 360 550 1.528

Burro 9 C 240 700 2.917

Maplin 29 D 150 300 2.000

Maplin 39 D 125 400 3.200

DiMs



- 27 -

MSWC MSWC DD Compared To Previous ExamplesCompared To Previous Examples

desiredMs

Examples Prediction

Inherent Margin

of SafetyConfidence Level (%)

Safety Buffer

MSWC D Exclusion Zone Distance 2.41 1.5 88 robust

MSWC R Distance 1.36 1.0 81 none

MSWC A Distance 1.36 1.5 37 inadequate

x

Societal Preferences Shape Model’s Acceptance



- 28 -

What Is ErrorWhat Is Error In In MSWCMSWC’’’’s Confidence Level?s Confidence Level?



- 29 -

MSWCMSWC Error Analyses & Their ImportanceError Analyses & Their Importance

desiredMs

• Sample Sizes in Previous Examples Are Small

• Calculations Contain Some Statistical Error

• Large Datasets with 30+ Minimize Error

• Error Analyses Characterize MSWC’s Accuracy

iMs



- 30 -

Analysis of Analysis of EE GG, Standard Error of Mean, Standard Error of MeanLNG Test &


Burro 8 - E 0.716

1.462

0.798

0.683

Burro 9 - C 1.885

1.629

1.669

Maplin 29 - D 0.775

0.803

1.137

0.972

1.231

1.424

1.204

Maplin 39 - D 0.541

1.147

1.139

2.111

1.554

1.672

2.319

Table A.3 – Excerpt of Havens 1992 Gas Concentration Ratios [5]

= 1.28 = .49

= 1.0 = -.57

= 1.725 at 90% = 21

= .184

or = = 1.10 or 1.46GG Ex min max

)(2/ GMsGG

n

StE G

Gt 2/

GMsSGxGdesiredMs G

scoreZ

Gn

GiMs



- 31 -

GMSWC

= 1.28= .18

1.0 =

= 1.46

1.1

of 83%

of 58%

Figure A.6 - With Min. & Max. Confidence Limits



- 32 -

Confidence Level Accuracy for Small SampleConfidence Level Accuracy for Small Sample

desiredMs

• Error in Confidence Level Is Estimated:

-.20 or -.95

• & Respectively Indicate Confidence Levels of 58 & 83%

• MSWCG Is 1.0 with 72% Confidence with Approximate Errors of -14 and +11%

GMs

GG

s SE

scoreG

max scoreG

min core Zor ZZ

Gmin coreZs

Gmax scoreZ



- 33 -

Effective Performance Metrics Effective Performance Metrics Ensure Prudent Evaluation DecisionsEnsure Prudent Evaluation Decisions



- 34 -

ConclusionsConclusions

• Uncertainties Must Be Reconciled As Predictions & Observations Are Correlated

• Model Predictions Vary By Factor of 2 Due to “Natural & Stochastic Variability” [2]

• Screening Predictions & Observations Guides Model Validation Process [3]



- 35 -

ConclusionsConclusions

• Geometric Mean Bias & Geometric Variance Graphs Readily Compare Model Performance [2]

• Fractional Results ( % between .5 & 2) Identify Best Applications for Models [4]

• Larger Validation Datasets Favor New Performance Metrics Like MSWC

Together All Metrics Balance Evaluation Decisions

iMs



- 36 -

Thank You!Thank You!

Frank A. Licari, PE, CSPPhone: (202) 366-5162Email: [email protected]://www.phmsa.dot.gov/pipeline

Questions



- 37 -

Backup SlidesBackup Slides



- 38 -

Why Is Confidence Level 37%?Why Is Confidence Level 37%?

• Inherent Margin of Safety Is Zero

• Model’s Bias to Overpredict Is Low

• Hanna Concluded Good Models Are Within Factor of .5 to 2



- 39 -

Confidence Level Accuracy For Large SampleConfidence Level Accuracy For Large Sample

desiredMs

• Confidence Level Error Is Estimated By:

• Standard Error of Is:

• For 30+ , &

Yields Confidence Level Error for Large Sample

nZ

SMs

Ms

21 2/

max

nZ

SMs

Ms

21 2/

min

iMs

minin ty variabili scoreZ

Ms

MsdesiredMsMs

maxMs

MsdesiredMs

or

MsS

Msx MsMsin ty variabili score

Z

Date post:	26-Sep-2018
Category:	Documents
Upload:	vuongtu
View:	212 times
Download:	0 times

Performance Metrics For Evaluating LNG Vapor …psc.tamu.edu/files/symposia/2009/presentations/2...

Documents