1

Modeling Complex Systems – How Much Detail is Appropriate?

David W. Esh

US Nuclear Regulatory Commission

2007 GoldSim User Conference, October 23-25, 2007, San Francisco CA

2

Overview

• Background

• Model development process

• Model complexity

• Model abstraction

• Examples

• Conclusions

3

Background

• The issue of how much detail to include in models of complex systems is not new.

• 14th century philosophers were considering different approaches to explain the world around them.

• Decisions regarding model complexity apply to all fields of study.

• Modern tools and computational capabilities present unique opportunities.

4

Model – representation of essential aspects of a system (existing or planned)

5

Model Development Process: Key Questions

• Why are you using a model?

• What is the purpose of your model?

• Who is your audience?

• What are your resources?

6

Model Development Process: Key Questions

• Why are you using a model?

– Developing understanding (integrating, generalizing, testing)

– Directing research (identify data gaps, propose new lines of research)

– Representing reality (prohibitively costly or can’t observe)

• What is the purpose of your model?

– Is the decision controversial?

– Is it high risk? ($, safety, etc.)

7

Model Development Process: Key Questions

• Who is your audience?

– Technical, lay person, policy

– High competency, low competency, mix

• What are your resources?

– Now and future

– Computational

– Time

– For collection of additional information

8

Model Development Process: Example

Site Assessment

Site Selection Site Selection and Characterizationand Characterization

NRC would require a Performance Assessment to:•Provide site and design data•Describe barriers that isolate waste•Evaluate features, events, and processes that affect safety

•Provide technical basis for models and inputs•Account for variability and uncertainty•Evaluate results from alternative models, as needed

What is Performance Assessment?What is Performance Assessment?• Systematic analysis of what could

happen at a site

Collect Data

Combine Models and Estimate Effects

Develop Conceptual Models

Develop Numerical and

Computer Models

Performance Assessment:

a learning process

Site Characteristics

Design andWaste Form

Overview of Performance Assessment

Why use it?Why use it?• Complex system• Systematic way to evaluate data• Internationally accepted approach

How is it conducted?How is it conducted?• Collect data• Develop scientific models• Develop computer code• Analyze results

What is assessed?What is assessed?• What can happen?• How likely is it?• What can result?

9

Model Complexity

Goals:

• Simple is better (all things equal)

• Broader scope

• Systematic approach

Metrics:

• Accuracy

• Explanatory Power

• Reliability and Validity

“Theories should be as simple as possible, but no simpler.”

10

• Can improve model fit (But does it improve explanatory power?)

• Can identify the need for enhancements

• Increases difficulty in understanding

• Increases difficulty in working with it

• Increases computational burden

Model Complexity

So how do I decide?

11

Model Complexity – How Much?

Complexity and EffortComplexity and Effort

Comparison of featuresComparison of featuresMass balance (watershed)Mass balance (watershed)

GIS based analysisGIS based analysisModel comparisonsModel comparisonsAnalogsAnalogs

Long-term field experimentsLong-term field experimentsIsotopic studiesIsotopic studies

12

Model Complexity – How Much?

• No complete methodologies (generally)

– Iteration (+/- interactions)

– Statistical analysis of results

– Visualization (data and output)

– Metamodels

• Most modelers put too much in to manage the risk of leaving something out

• If complexity is not inexorably linked with accuracy, there may exist an opportunity to simplify

13

Model Complexity – How Much?

1 Prices go up, farmers produce more (too much)2 Prices go down, farmers produce less3 Repeat

12

3

14

• Models provide information to think about, they don’t do your thinking for you

• Decision makers need to reason about the issues

• Model abstraction approaches can and should be used

Model Complexity – How Much?

ComplexityE

ffor

t

P(decision)

15

Model Abstraction Example

NUREG/CR-6884 Model Abstraction Techniques for Soil-Water Flow and Transport

16

Model Abstraction

• Need to start with a broad model space – allows exploratory analysis essential to abstraction

• Reduce complexity – maintain validity

• Show the abstraction represents the complex model

Benefits• Less $• Fewer inputs• Easier to integrate• Easier to interpret

Types (not exhaustive)• Drop unimportant parts• Replace with simpler part• Coarsen ranges of values• Group parts together

17

Model Abstraction:Example Benefit

Uncertainty analysis

• Simpler model yielder stronger results (6 variables identified compared to 3)

• Allowed focused refinement of model

• Complexity can have many unintended consequences

Variable DescriptionImport

ance Factor

Grout_deg_start

Time at which degradation of the wasteform can begin 0.98

Nm

MacMullin number. The effective diffusion coefficient is a product of Nm and the molecular diffusion coefficient.

0.93

Degraded_grout_Kh

Hydraulic conductivity for degraded region of the wasteform. 0.36

TransFactor_indoor

Factor to account for shielding of radiation when an individual is inside a residence.

0.29

Se_solubilitySolubility of Se in the pore fluid of the wasteform. 0.21

Kd_waste_Sr_ox

Distribution coefficient for Sr in the oxidized region of wasteform. 0.11

Vent_light_activity

Breathing rate for an individual during light activity. 0.11

SZ_dispersivity_factor

Used with the transport length in the saturated zone to develop the saturated zone dispersivity.

0.10

Kd_Waste_Eu

Distribution coefficient for Eu in the intact portion of the wasteform. 0.08

18

Conclusions

• Methodologies to address the level of model complexity continue to evolve

• Model abstraction can have many benefits when done properly

• Simple is better (all things being equal)