LNG Model Evaluation Protocol & Validation Database Update
Presented by:
James Stewart
U.K. Health & Safety Executive and
Daniel Gorham
Fire Protection Research Foundation
2017 NFPA Conference & Expo
© British Crown Copyright, HSE 2017
• Session Learning Objectives
• LNG Facilities and NFPA 59A
• The LNG Model Evaluation Protocol (MEP)
- What is the LNG MEP and what is its purpose?
• The LNG Model Validation Database
• Updating the LNG MEP and Database
- Why have the LNG MEP and Database been updated?
- What changes have been made? - How does this affect the process of gaining approval for alternate vapor dispersion models for use in LNG siting applications?
• Reviewing Updated Versions of Approved Models
• Key Messages
Overview
© British Crown Copyright, HSE 2017
Session Learning Objectives
The aims of this NFPA Conference & Expo 2017
Education Session are to:
• Provide information on how U.S. energy needs have lead to
increased attention for LNG facilities and NFPA 59A
• Explain the purpose of the LNG Model Evaluation Protocol
(MEP) and the LNG Model Validation Database
• Describe the updates and changes that have recently been
made to the LNG MEP and Database
© British Crown Copyright, HSE 2017
Existing LNG Import/Export Terminals
Source: http://www.beaumontenterprise.com/news/article/10B-LNG-export-terminal-planned-for-Sabine-Pass-4506985.php
Source: https://https://en.wikipedia.org/wiki/LNG_carrier
Source: http://www.lngworldnews.com/ferc-updates-u-s-lng-terminals-summaries/
© British Crown Copyright, HSE 2017
Proposed LNG Export Terminals
Source: Federal Energy Regulation Commission (FERC) https://www.ferc.gov/industries/gas/indus-act/lng.asp
© British Crown Copyright, HSE 2017
History of NFPA 59A
AGA begins
work on LNG
1960
1964
Draft submitted
to NFPA as basis for
new standard
Standard prepared by Sectional Committee
on Utility Gas tentatively adopted
1966
1967
First official
edition of NFPA 59A adopted
Committee on LNG established to
develop standard with scope broader than
utility gas plant applications
1969
1971
First edition of NFPA 59A developed
under broadened
scope
© British Crown Copyright, HSE 2017
Purpose
To provide minimum fire protection, safety, and related requirements for the location, design, construction, security, operation, and maintenance of LNG plants.
Source: http://soldoza.com/participation-on-lng-tanks/tk-3001-lng-tank-barcelona-port/
© British Crown Copyright, HSE 2017
Applies to:
(1) Facilities that liquefy natural gas
(2) Facilities that store, vaporize, transfer, and handle LNG
(3) The training of all personnel involved with LNG
(4) The design, location, construction, maintenance, and operation of all LNG facilities
Does not apply to:
(1) Frozen ground containers
(2) Portable storage containers stored or used in buildings
(3) All LNG vehicular applications, including fueling of LNG vehicles (NFPA 52 and 30A)
Scope and Application
© British Crown Copyright, HSE 2017
LNG Facilities and NFPA 59A
LNG Facility Siting Requirements
• NFPA 59A requires that any LNG container or transfer system
must have a dispersion exclusion zone
• The size of such zones must be determined using an
approved vapor dispersion model, one of:
• 49 CFR 193 Subpart B, states that alternative models may be
used, subject to the approval of the “Administrator”
• In the case of LNG facilities the administrator is the Pipelines
and Hazardous Materials Safety Administration (PHMSA)
DEGADIS 2.1
FEM3A
PHAST v6.6/6.7
FLACS v9.1 r2
© British Crown Copyright, HSE 2017
LNG Facilities and NFPA 59A
LNG Facility Siting Requirements
• PHMSA state that alternative models may be used in LNG
siting applications provided that the models “…take into
account the same physical factors as the approved
models, are validated by experimental test data, and
receive the Administrator’s approval…”
• The LNG Model Evaluation Protocol should be used as a
means of evaluating alternative models
• The LNG MEP, and the accompanying LNG Model Validation
Database, have recently been updated to take account of
additional requirements imposed by PHMSA
© British Crown Copyright, HSE 2017
LNG Model Evaluation Protocol
What is the LNG Model Evaluation Protocol (MEP)?
• The LNG MEP is a document outlining the process that
should be used to assess the suitability of vapor dispersion
models for evaluating hazard distances in LNG siting
applications
What is the purpose of the LNG MEP?
• The purpose of the LNG MEP is to provide a comprehensive
evaluation methodology for determining the suitability of
models to accurately simulate the dispersion of vapors
emanating from accidental spills of LNG
© British Crown Copyright, HSE 2017
LNG Model Evaluation Protocol
What does the LNG MEP involve?
• The LNG MEP comprises three distinct phases:
Scientific Assessment
Model Verification
Model Validation
• Each stage must be completed during an evaluation of a model
against the LNG MEP
• The outputs are then recorded in a Model Evaluation Report
• This process must be followed when seeking approval to use
an alternate vapor dispersion model for LNG siting applications
© British Crown Copyright, HSE 2017
LNG Model Evaluation Protocol
LNG MEP: Scientific Assessment
• Involves critically reviewing the physical, mathematical and
numerical basis of a model by someone independent of the
model development
Key details of the model should be available for scientific assessment
Model produces output suitable for assessment against MEP SPM
Model should be based on accepted/published science
Model accepts a credible source term
Model accounts for the effects of wind speed
Model accounts for the effects of surface roughness on dispersion
Model accounts for the effects of atmospheric stability on dispersion
Model accounts for passive dispersion
Model accounts of gravity-driven spreading
Model accounts for the effects of buoyancy on dilution
Numerical methods are based on accepted/published practice practice
Model is suitable for
evaluation against MEP
Model contains necessary
physics with appropriate
mathematical
implementation
Model is based on suitable
numerical methods
© British Crown Copyright, HSE 2017
LNG Model Evaluation Protocol
LNG MEP: Model Verification
• Model verification is the process of checking that a computer
implementation of the model accurately represents its
mathematical basis
• In the LNG MEP, verification is treated as a passive element of
the scientific assessment, rather than an exercise in its own
right
• Evidence of model verification should be provided to
demonstrate that the model being evaluated has been suitably
verified
• PHMSA makes the ultimate decision about model suitability
and limitations
© British Crown Copyright, HSE 2017
LNG Model Evaluation Protocol
LNG MEP: Model Validation
• Model validation is the most significant stage of the LNG MEP
• Scientific assessment and model verification give an indication
of model suitability, whereas validation assesses how well a
model actually performs for scenarios of practical interest
• Model validation for the LNG MEP involves comparing model
predictions to experimental data for a range of dense gas
dispersion scenarios
• The experiments included in the model validation database
were selected to test the key physical processes involved in
the dispersion of LNG
© British Crown Copyright, HSE 2017
LNG Model Validation Database
LNG Model Validation Database
• The Database is an Excel spreadsheet used
during model validation for the LNG MEP
• It contains data from 33 experiments taken
from 8 different series of tests
• Model predictions of gas concentrations at
specified locations are entered into the
Database for each experiment
• Max. arc-wise concentrations and Statistical
Performance Measures (SPM) are then
automatically calculated by the spreadsheet
and then used for quantitative assessment
of the model
© British Crown Copyright, HSE 2017
LNG Model Validation Database
LNG Model Validation Database
• Details of each experiment are
given in separate Excel worksheets
• These details comprise:
- Basic test information
- Details of the substance released
- Details of the conditions of release
- Atmospheric conditions
• The Database also lists sensor
locations and averaging
times/windows for each experiment
© British Crown Copyright, HSE 2017
LNG Model Validation Database
LNG MEP: Model Validation
• The Database produces graphs of predicted vs. measured
maximum arc-wise gas concentration for each experiment
• Two of the worksheets give tabulated values of SPM, color-
coded to show whether or not the suggested quantitative
performance measures are met, e.g. for Maplin Sands trial 27
© British Crown Copyright, HSE 2017
LNG Model Validation Database
LNG MEP: Model Validation
• The final worksheet of the Database contains scatter plots of
predicted vs. measured concentrations to give a visual
representation of model performance
• Five such plots are produced
to show model performance
for:
- All Trials
- Field Trials
- Wind Tunnel Trials
- Unobstructed Trials
- Obstructed Trials
© British Crown Copyright, HSE 2017
LNG Validation Database Guide
Guide to the LNG Model Validation Database Version 12
• The Database is described in:
Stewart et al. (2016), Guide to the Model Validation Database Version 12
© British Crown Copyright, HSE 2017
LNG Validation Database Guide
Guide to the LNG Model
Validation Database Version 12
• The Database guide describes the
Model Validation Database and
how it should be used as part of
the LNG MEP
• It gives details of each series of
experiments and describes the
experimental uncertainties
• The methods used in the Excel
spreadsheet to automatically
calculate the SPM are also
outlined
© British Crown Copyright, HSE 2017
Updating the LNG MEP & Database
Aims of the update:
• To add new features to the Database and amend the MEP to
incorporate the additional requirements given in the PHMSA
Advisory Bulletin PHMSA-2010-0226
• To incorporate previously-omitted experimental data and to
correct errors within the Database
• To automate the calculation of SPMs and maximum arc-wise
gas concentrations in the Database to reduce user effort
during model validation
• To provide more extensive documentation and guidance on
using the Database and MEP
© British Crown Copyright, HSE 2017
Updating the LNG MEP & Database
Deliverables from the LNG MEP & Database Update
• The 2nd Edition of the LNG Model Evaluation Protocol
M. J. Ivings et al. (2016), Evaluating Vapor Dispersion Models for Safety
Analysis of LNG Facilities, 2nd edition
• The LNG Model Validation Database version 12
• An updated guide to the LNG Model Validation Database
J. R. Stewart et al. (2016), Guide to the LNG Model Validation Database Version 12
• The two reports are available from the NFPA research reports
webpage1 and the Database is available as an Excel spreadsheet
on request
1 http://www.nfpa.org/news-and-research/fire-statistics-and-reports/research-reports/hazardous- materials/gases/lng-model-evaluation-protocol-and-validation-database-update
© British Crown Copyright, HSE 2017
Updating the LNG MEP & Database
Requirements of the PHMSA Advisory Bulletin
PHMSA-2010-0226
• Maximum arc-wise gas concentrations
- Method for determining these is now prescribed
• Experimental and modeling uncertainty
- Discussion of experimental uncertainty incorporated into the Database Guide
• Additional SPM:
- Concentration Safety Factor (CSF)
- Concentration Safety Factor to LFL (CSFLFL)
- Distance Safety Factor to LFL (DSFLFL)
© British Crown Copyright, HSE 2017
Updating the LNG MEP & Database
Maximum Arc-wise Gas Concentrations
• Comparison of measured and predicted maximum arc-wise
gas concentrations forms a key part of the LNG MEP model
validation exercise
• Ensuring all models are compared in a like-for-like manner is
fundamental for consistent model evaluation
• PHMSA has prescribed the method which should be used
when applying for approval to use an alternate vapor
dispersion model for use in LNG siting applications
© British Crown Copyright, HSE 2017
Updating the LNG MEP & Database
Maximum Arc-wise Gas Concentrations
“The maximum arc wise concentration should be based on
the location of the experimental sensor data that produced
the maximum arc wise concentration relative to the cloud
centerline” (PHMSA-2010-0226)”
© British Crown Copyright, HSE 2017
Updating the LNG MEP & Database
Maximum Arc-wise Gas Concentrations:
What’s the difference?
• Difference between the PHMSA-prescribed method and the
common1 alternative is subtle but potentially significant
1 The method often used is that of Hanna et al. (1993) where the max. arc-wise concentration at a given downwind distance is
taken as the max. at any position along the arc at the height of the lowest experimental sensor
Predicted max. arc-wise concentration Predicted max. arc-wise concentration Cloud centerline
Cloud centerline
Sensors
Gas source
‘Common’ Method PHMSA Method
© British Crown Copyright, HSE 2017
Maximum Arc-wise Gas Concentrations:
What are the benefits of the PHMSA approach?
Updating the LNG MEP & Database
• The method is conservative, it
will typically result in larger
exclusion zones for LNG sites
• It accounts for cases where the
plume bypasses the gas
sensors
• It encourages the development
of plume meandering models
© British Crown Copyright, HSE 2017
Experimental and Modeling Uncertainty
• Following guidance given in the PHMSA Advisory Bulletin
PHMSA-2010-0226, sensitivity analyses must be undertaken
during the model validation stage of the LNG MEP
• The updated Database Guide gives an overview of the
experimental and modeling uncertainties for each test series
included in the Database
• These uncertainties are based on the review of the DEGADIS
model by FERC and on experimental analysis reports
• This information should be used to guide sensitivity analysis
studies conducted as part of the LNG MEP process
Updating the LNG MEP & Database
© British Crown Copyright, HSE 2017
Additional Statistical Performance Measures (SPM)
• The Advisory Bulletin PHMSA-2010-0226 requires calculation
of three additional SPM from the model validation data:
- Concentration Safety Factor (CSF): 𝐶𝑆𝐹 =𝑐𝑝
𝑐𝑚
- Concentration Safety Factor to LFL (CSFLFL): 𝐶𝑆𝐹𝐿𝐹𝐿 =𝑐𝑝
𝐿𝐹𝐿
- Distance Safety Factor to LFL (DSFLFL): 𝐷𝑆𝐹𝐿𝐹𝐿 =𝑋𝑝,𝐿𝐹𝐿
𝑋𝑚,𝐿𝐹𝐿
• These parameters are automatically calculated within the
Database to ensure that a consistent approach is used to
evaluate all models
Updating the LNG MEP & Database
© British Crown Copyright, HSE 2017
Updating the LNG MEP & Database
Additional Experimental Data
• Time-varying concentration data has been processed to produce
point-wise concentrations for the Maplin Sands (1980) trials
© Shell
© British Crown Copyright, HSE 2017
Updating the LNG MEP & Database
Additional Experimental Data
• Point-wise concentration data has been added for the Thorney
Island (1982) experiments
© British Crown Copyright, HSE 2017
Updating the LNG MEP & Database
Additional Experimental Data
• Point-wise concentrations at
multiple sensor heights
• Highlighting of sensor locations
where negligible concentrations
were measured
• Negligible (below 0.1 % v/v)
concentrations excluded from
SPM calculations
e.g. Thorney Island 45
© British Crown Copyright, HSE 2017
Updating the LNG MEP & Database
Corrections to the Database
• Adjustment of sensor height for
one of the downwind
measurement locations for
CHRC Trials B and C
• Height used in earlier version of
Database coincided with
location of the dike
CHRC A, Havens and Spicer (2006)
© British Crown Copyright, HSE 2017
Updating the LNG MEP & Database
Corrections to the Database
• Corrected ambient and release conditions to be consistent with
original experimental reports
• Adjusted surface roughness height to coincide with inflow
profiles from experiment
• Corrected geometry sizes and positions for obstructed scenarios
© British Crown Copyright, HSE 2017
Updating the LNG MEP & Database
Further Database Changes
• Visual representation of concentration sensor locations
• Transformed sensor locations for use with models in which the
wind is aligned with the grid
© British Crown Copyright, HSE 2017
Updating the LNG MEP & Database
Further Database
Changes
• Cloud widths are calculated
in cases where the gas
cloud is not bifurcated in the
cross-wind direction
• Plots (see right) are created
for the Database user to
assess cloud shape
• The user then indicates
whether or not the cloud is
bifurcated
Long Time-averaged Predicted Arc-wise Concentration Profiles (at 1 m elevation)
Note that only concentration values at the lowest sensor height are shown here and are included in any cloud
width calculations
0.0
1.0
2.0
3.0
4.0
5.0
6.0
7.0
8.0
9.0
10.0
-50.0 -25.0 0.0 25.0 50.0
Co
nce
ntr
atio
n (%
)
Cross-wind distance (m)
57 m Arc Measured Predicted
0.0
1.0
2.0
3.0
4.0
5.0
6.0
7.0
-100 -50 0 50 100
Co
nce
ntr
atio
n (%
)
Cross-wind distance (m)
140 m Arc Measured Predicted
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
-200 -100 0 100 200
Co
nce
ntr
atio
n (%
)
Cross-wind distance (m)
400 m Arc Measured Predicted
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.0 50.0 100.0 150.0 200.0
Co
nce
ntr
atio
n (%
)
Cross-wind distance (m)
800 m Arc Measured Predicted
© British Crown Copyright, HSE 2017
Updated Versions of Models
Reviewing Updated Versions of Approved Models
• A new process has been proposed for the expedited review
of updated versions of models already approved by PHMSA
• The approval of a model for use in LNG siting applications is
only valid for the exact version of the model evaluated
• However, models are frequently updated to fix errors or to
include additional features or model improvements
• PHMSA intend to provide the option of an expedited review
for updated versions of approved models so as to avoid
unduly impeding the use of new code developments
© British Crown Copyright, HSE 2017
Updated Versions of Models
Reviewing Updated Versions of Approved Models
• The new process is based on the comparison of the updated
model to the previously-approved version
• The extent to which the updated model is assessed is based
on whether the changes are deemed to be ‘major’ or ‘minor’
• Here, a ‘minor’ change is meant to address four issues:
- Bug fixes in the software
- Changes to the Graphical User Interface (GUI)
- Addition of compatible hardware support
- Modification of sub-models not relevant to LNG dispersion
• A ‘major’ change makes advances to the scientific basis of
the model for modelling LNG dispersion
© British Crown Copyright, HSE 2017
Updated Versions of Models
Reviewing Impact of Model Updates: Minor Changes
• Assessing an updated model that has undergone only minor
changes requires the model to be run against a subset of the
LNG validation database experiments
• This subset includes the following 6 tests:
Burro 8 Thorney Island 47
Coyote 5 CHRC B
Falcon 1 BA Hamburg DAT223
• Point-wise concentration predictions for these tests are then
compared to the results from the approved version of the model
• The two versions of the model should use identical setups for
each of the tests
© British Crown Copyright, HSE 2017
Updated Versions of Models
Reviewing Impact of Model Updates: Minor Changes
• The next step is to determine if any changes in model predictions
are ‘significant’
• In this context a change in predicted concentration is considered
to be significant if there is:
A 1% relative difference in predictions from the two versions of the
model where the absolute predicted concentration is > 𝟏𝟎% v/v
Or
A 0.1% v/v absolute difference in predictions from the two versions of
the model where the absolute predicted concentration is ≤ 𝟏𝟎% v/v
© British Crown Copyright, HSE 2017
Updated Versions of Models
Reviewing Impact of Model Updates: Minor Changes
• If there are no significant differences between the model
predictions for the validation subset then the following items
should be submitted to PHMSA as a record of the review:
- A short report to describe the changes to the model since
the previous version was approved
- Point-wise concentration predictions for the subset of
validation test cases
- Model input files for the subset of validation test cases
• If there are significant differences as a result of model changes,
the ‘minor’ changes must be treated as ‘major’ changes and the
model should be reviewed accordingly
© British Crown Copyright, HSE 2017
Updated Versions of Models
Reviewing Impact of Model Updates: Major Changes
• For models that have been updated to include a major change,
e.g. to a LNG dispersion related sub-model, the process for
evaluating the updated version of the model is more rigorous
• In this scenario the model must be evaluated against the three
stage LNG MEP process, i.e.
Scientific Assessment – to assess the impact of the model updates
Model Verification – to ensure that the model is verified to an equal or
higher standard than the previous version of the model
Model Validation – against the entire LNG model validation database with
significant differences in model predictions highlighted
and explained
© British Crown Copyright, HSE 2017
Updated Versions of Models
Reviewing Impact of Model Updates: Major Changes
• On completion of an evaluation of an updated model with major
changes from the approved version, the following should be
submitted to PHMSA:
- A change-log report to describe changes to the model since it was
previously approved by PHMSA. This report should include results
of the scientific assessment, verification and validation of the
updated model in accordance with the LNG MEP
- Model input files for the LNG MEP model validation exercise
- Access to the updated and currently-approved versions of the
model
© British Crown Copyright, HSE 2017
Updated Versions of Models
Major or Minor update?
Run subset of validation
cases
Are any differences significant?
Evaluation against full LNG MEP
Submit following to PHMSA: 1. Change-log report 2. Model input files 3. Access to new and currently
approved versions of model
Submit following to PHMSA: 1. Short report 2. Point-wise concentration
predictions 3. Model input files
Major Minor
No
Yes
© British Crown Copyright, HSE 2017
Key Messages
Summary
• Updated versions of the LNG Model Evaluation Protocol
(MEP), Model Validation Database and Database Guide are
available from the NFPA
• When seeking approval to use alternative vapor dispersion
models for LNG siting applications from PHMSA, the latest
version of the MEP and Database must be used
• A streamlined process to gain approval of an updated
version of a model currently approved for determining LNG
facility dispersion exclusion zones has been defined
• PHMSA are currently working towards using this new
approach
© British Crown Copyright, HSE 2017
References
• PHMSA Advisory Bulletin, PHMSA-2010-0226 http://phmsa.dot.gov/pv_obj_cache/pv_obj_id_B1E12F1E74C27BEAB343DEB90D621DF5BB340
700/filename/ADB-10-07%20LNG%20Facilities.pdf
• DEGADIS 2.1 evaluation for LNG MEP https://www.ferc.gov/industries/gas/indus-act/lng/degadis-report.pdf
• PHAST v6.6 and v6.7 evaluation for LNG MEP http://www.regulations.gov/#!docketDetail;D=PHMSA-2011-0075
• FLACS 9.1r2 evaluation for LNG MEP https://www.regulations.gov/#!docketDetail;D=PHMSA-2011-0101
• Hanna et al. (1993) http://www.dtic.mil/cgi-bin/GetTRDoc?AD=ADA270095
• Havens J. and Spicer T. (2006) Vapor dispersion and thermal hazard
modelling, Final topical report to Gas Technology Institute under sub-contract
K100029184, October 2006.
© British Crown Copyright, HSE 2017
HSE's Health and Safety Laboratory (HSL) is a multi-disciplinary laboratory whose work delivers high quality science to meet the needs of industry and government in the UK and overseas. Our commercial customers can commission services and research using our state-of-the-art scientific laboratory in Buxton, U.K. as well as analytical expertise from other parts of HSE’s science base.
http://www.hse.gov.uk/
http://www.hsl.gov.uk/
This presentation and the work it describes were undertaken by the Health and Safety Laboratory under contract to Oak Ridge National Laboratory. Its contents, including any opinions and/or conclusion expressed or recommendations made, do not necessarily reflect policy or views of the Health and Safety Executive.
CEUs: To receive CEUs for this session, scan your badge at the back of the room before leaving Evaluation: Complete a session evaluation on the mobile app. (Search app store for ‘NFPA 2017 C&E.’) Handouts: Handouts will be available via the mobile app and at nfpa.org/conference Recordings: Audio recordings of all sessions will be available free of charge via NFPA Xchange.
2017 NFPA Conference & Expo