Beyond Probability - A Pragmatic Approach to Uncertainty Quantification in Engineering

7/29/2019 Beyond Probability - A Pragmatic Approach to Uncertainty Quantification in Engineering

1/88

Beyond ProbabilityA pragmatic approach to uncertainty quantification in engineering

Scott Ferson, Applied Biomathematics, [email protected]

NASA Statistical Engineering Symposium, Williamsburg, Virginia, 4 May 2011

2010 Applied Biomathematics


2/88

Wishful thinking

Using inputs or models because they are

convenient, or because you hope theyre true


3/88

Kansai International Airport

30 km from Kobe in Osaka Bay

Artificial island made with fill

Engineers told planners itd sink [6, 8] m

Planners elected to design for 6 m

Its sunk 9 m so far and is still sinking

(The operator of the airport denies these media reports)


4/88

Variability = aleatory uncertainty

Arises from natural stochasticity

Variability arises fromspatial variation

temporal fluctuations

manufacturing or genetic differences

Not reducible by empirical effort


5/88

Incertitude = epistemic uncertainty

Arises from incomplete knowledge

Incertitude arises fromlimited sample size

mensurational limits (measurement error)

use of surrogate data

Reducible with empirical effort


6/88

Suppose

A is in [2, 4]

B is in [3, 5]

What can be said about the sumA+B?

4 6 8 10

The right answer for

engineering is [5,9]

Propagating incertitude


7/88

They must be treated differently

Variability should be modeled as randomness

with the methods of probability theory

Incertitude should be modeled as ignorance

with the methods of interval analysis

Imprecise probabilities can do both at once


8/88

Incertitude is common in engineering

Periodic observationsWhen did the fish in my aquarium die during the night?

Plus-or-minus measurement uncertaintiesCoarse measurements, measurements from digital readouts

Non-detects and data censoringChemical detection limits, studies prematurely terminated

Privacy requirementsEpidemiological or medical information, census data

Theoretical constraintsConcentrations, solubilities, probabilities, survival rates

Bounding studiesPresumed or hypothetical limits in what-if calculations


9/88

Wishful thinking

Pretending you know the

Value

Distribution function

Dependence

Model

when you dont is wishful thinking

Uncertainty analysis makes a prudent analysis


10/88

Wishfulthinking

Prudentanalysis

Failure

Success

Dumb luck

NegligenceHonorable

failure

Good

engineering


11/88

Traditional uncertainty analyses

Interval analysis

Taylor series approximations (delta method) Normal theory propagation (ISO/NIST)

Monte Carlo simulation

Stochastic PDEs

Two-dimensional Monte Carlo


12/88

Untenable assumptions

Uncertainties are small

Distribution shapes are known Sources of variation are independent

Uncertainties cancel each other out

Linearized models good enough

Underlying physics is known and modeled


13/88

Need ways to relax assumptions

Hard to say what the distribution is precisely

Non-independent, orunknown dependencies Uncertainties that may not cancel

Possibly large uncertainties

Model uncertainty


14/88

Probability bounds analysis (PBA)

Sidesteps the major criticisms

Doesnt force you to make any assumptionsCan use only whatever information is available

Bridges worst case and probabilistic analysis

Distinguishes variability and incertitude

Acceptable to both Bayesians and frequentists


15/88

Probability box (p-box)

0

1

1.0 2.0 3.00.0 X

Cumulativeprobability

Interval bounds on a cumulative distribution function


16/88

Uncertain numbers

Nota uniform

distribution


0 10 20 30 400

1

10 20 30 400

1

10 20 300

1

Probability

distribution

Probability

box Interval


17/88

Uncertainty arithmetic

We can do math on p-boxes

When inputs are distributions, the answersconform with probability theory

When inputs are intervals, the results agreewith interval (worst case) analysis


18/88

Calculations

All standard mathematical operations Arithmetic (+, , , , ^, min, max)

Transformations (exp, ln, sin, tan, abs, sqrt, etc.)

Magnitude comparisons (, , )

Other operations (nonlinear ODEs, finite-element methods)

Faster than Monte Carlo

Guaranteed to bound the answer

Optimal solutions often easy to compute


19/88

Probability bounds analysis

Special case of imprecise probabilities

Addresses many problems in risk analysisInput distributions unknown

Imperfectly known correlation and dependency

Large measurement error, censoringSmall sample sizes

Model uncertainty


20/88

Better than sensitivity analysis

Unknown distribution is hard for sensitivity

analysis since infinite-dimensional problem

Analysts usually fall back on a maximum

entropy approach, which erases uncertainty

rather than propagates it

Bounding seems very reasonable, so long as

it reflects all available information


21/88

Example: uncontrolled fire

F=A &B & C&D

Probability of ignition source

Probability of abundant fuel presence

Probability fire detection not timely

Probability of suppression system failure


22/88

Imperfect information

CalculateA&B&C&D, with partial information:

As distribution is known, but not itsparameters

Bs parameters known, but not its shape

C has a small empirical data set

D is known to be aprecise distribution

Bounds assuming independence?

Without any assumption about dependence?


23/88

A = {lognormal, mean = [.05,.06], variance = [.0001,.001])

B = {min = 0, max = 0.05, mode = 0.03}

C= {sample data = 0.2, 0.5, 0.6, 0.7, 0.75, 0.8}

D = uniform(0, 1)

0 0.1 0.2 0.30

1

A

0 0.02 0.04 0.060

1

B

0 1

0

1

D

0 1

0

1

CCDF

CDF


24/88

Resulting answers

0 0.02 0.04 0.06

0

10 0.01 0.02

0

1


Cumulativepro

bability

All variables

independent

Make no assumption

about dependencies


25/88

Summary statisticsIndependent

Range [0, 0.011]

Median [0, 0.00113]

Mean [0.00006, 0.00119]

Variance [2.9 10 9, 2.1 10 6]

Standard deviation [0.000054, 0.0014]

No assumptions about dependence

Range [0, 0.05]Median [0, 0.04]

Mean [0, 0.04]

Variance [0, 0.00052]

Standard deviation [0, 0.023]


26/88

How to use the results

When uncertainty makes no difference(because results are so clear), bounding gives

confidence in the reliability of the decision

When uncertainty swamps the decision

(i) use other criteria within probability bounds, or

(ii) use results to identify inputs to study better


27/88

Justifying further empirical effort

If incertitude is too wide for decisions, and

bounds are best possible, more data is needed

Strong argument for collecting more data


28/88

Advantages

Computationally efficient

No simulation or parallel calculations needed

Fewer assumptions

Not just different assumptions,fewerof them

Distribution-free probabilistic risk analysis

Rigorous results

Built-in quality assurance

Automatically verified calculation


29/88

Disadvantages

P-boxes dont say what outcome is most likely

Hard to get optimal bounds on non-tail risks

Some technical limits (e.g., sensitive to

repeated variables, tricky with black boxes)

A p-box may not express the tightest possible

bounds given all available information

(although it often will)


30/88

Software

UC add-in for Excel (NASA, beta 2011)

RAMAS Risk Calc 4.0 (NIH, commercial)

Statool (Dan Berleant, freeware)

Constructor (Sandia and NIH, freeware)

Pbox.r library for R

PBDemo (freeware)

Williamson and Downs (1990)


31/88


32/88


33/88


34/88


35/88


36/88


37/88


38/88


39/88


40/88


41/88


42/88


43/88


44/88


45/88

Diverse applications

Superfund risk analyses

Conservation biology extinction/reintroduction

Occupational exposure assessment

Food safety

Chemostat dynamics

Global climate change forecasts

Safety of engineered systems

Engineering design


46/88

Case study:

Spacecraft design undermission uncertainty


47/88

Mission

Deploy satellite carrying a large optical sensor

W

L

D

X Y

Z


48/88

Wertz and Larson (1999) Space Mission Analysis and Design (SMAD). Kluwer.


49/88


50/88

Attitude control

Command data systems

Configuration

CostGround systems

Instruments

Mission design

Power

Program management

Propulsion

Science

Solar arraySystems engineering

TelecommunicationsSystem

TelecommunicationsHardware

Thermal control

Typical subsystems

Attitude control

Command data systems

Configuration

CostGround systems

Instruments

Mission design

Power

Program management

Propulsion

Science

Solar arraySystems engineering

TelecommunicationsSystem

TelecommunicationsHardware

Thermal control


51/88

Demonstrations

Calculations within a single subsystem (ACS)

Calculations within linked subsystems

Attitudecontrol

Power

Solararray


52/88

Attitude control subsystem (ACS)

3 reaction wheels Design problem: solve forh

Required angular momentum

Needed to choose reaction wheels

Mission constraints

torbit = 1/4 orbit time

slew = max slew angle

tslew

= min maneuver time

Inputs from other subsystems

I,Imax,Imin = inertial moment Depend on solar panel size, which

depends on power needed, so on h

h tot torbit

slew

4 slew

tslew2

I

tot slew dist

dist g sp m a

g

3

2 RE H3

Imax Imin sin 2

sp LspFs

cAs 1 q cos i

m

2MD

RE H3

a

1

2La CdAV2


53/88

Attitude control input variablesSymbol Unit Variable Type Value SMAD

Cd unitless Drag coefficient p-box range=[2,4]

mean=3.13

3.13

La m Aerodynamic drag torque moment p-box range=[0,3.75]

mean=0.25

0.25

Lsp m Solar radiation torque moment p-box range=[0,3.75]

mean=[0.25]

0.25

Dr A m2 Residual dipole interval [0,1] 1

i degrees Sun incidence angle interval [0,90] 0

kg m3

Atmospheric density interval [3.96e-12,9.9e-11] 1.98e-11

degrees Major moment axis deviation from nadir interval [10,19] 10

q unitless Surface reflectivity interval [0.1,0.99] 0.6

Imin kg m2 Minimum moment of inertia interval [4655] 4655

Imax kg m2 Maximum moment of inertia interval [7315] 7315

m3 s-2 Earth gravity constant point 3.98e14 3.98e14

A m2 Area in the direction of flight point 3.752 3.752

RE km Earth radius point 6378.14 6378.14H km Orbit altitude point 340 340

Fs W m-2 Average solar flux point 1367 1367

slew degrees Maximum slewing angle point 38 38

c m s-1 Light speed point 2.9979e8 2.9979e8

M A m2 Earth magnetic moment point 7.96e22 7.96e22

tslew s Minimum maneuver time point 760 760

As m2

Area reflecting solar radiation point 3.752

3.752

torbit s Quarter orbit period point 1370 1370


54/88

Coefficient of drag, Cd

Cd (unitless)

1 2 3 4 50

1SMAD


55/88

Aerodynamic drag torque moment,La

La (m)

-1 0 1 2 3 40

1SMAD


56/88

Required angular momentum h

0.02 0.030

0.5

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.70

0.5

1

0 200 400 600 8000

0.5

1

distsleworbitth

4655torbit (sec)

1370

= ( + )

h (N m sec) slew (N m) dist (N m)


57/88

Value of information: pinching

Initial result

After pinching(atmospheric density)

(kg m-3

) h (N m sec)

0 200 400 600 800 10000

1

0 10 100

1


58/88

Three linked subsystems

Attitudecontrol

Power

Solararray


59/88

Variables passed iteratively

Minimum moment of inertiaImin

Maximum moment of inertiaImax

Total torque tot

Total powerPtot

Solar panel areaAsa


60/88

Analysis of calculations

Need to check that original SMAD values and all

Monte Carlo simulations are enclosed by p-boxes

Need to ensure iteration through links doesntcause runaway uncertainty growth (or reduction)

Four parallel analyses SMADspoint estimates

Monte Carlo simulation

P-boxes but without linkage among subsystems

P-boxes with fully linked subsystems


61/88

Supports of results

tot (N m)0.0 0.2 0.4 0.6 0.8 1.0

Ptot (W)

1000 1500 2000

Asa

(m2)0 10 20 30 40 50 60

Probability bounds

PBA but unlinkedMonte Carlo simulation

SMADpoint estimates


62/88

Case study findings

Different answers are consistent Point estimates match the SMAD results

P-boxes span the points and the Monte Carlo intervals

Calculations workable No runaway inflation (or loss) of uncertainty

Easier than with Monte Carlo

Practical and interesting results Uncertainty can affect engineering decisions

Reducing uncertainty about (by picking a launch date)

strongly reduces design uncertainty


63/88

Accounting for epistemic and aleatory

uncertainty in early system design

NASA SBIR Phase 2 Final Report

Applied Biomathematics

100 North Country Road

Setauket, NY 11733

Phone: 1-631-751-4350

Fax: 1-631-751-3435

Email: [email protected]

www.ramas.com

Order Number: NNL07AA06C

July 2009

For more information, consult

SBIR project report to NASA,

July 2009


64/88

Uses for probability bounds analysis

Uncertainty propagation

Risk assessment

Sensitivity analysis (for control and study)

Reliability theory

Engineering design

Validation

Decision theory Regulatory compliance

Finite element modeling

Differential equations


65/88

Differential equations


66/88

Uncertainty usually explodes

Time

x

The explosion can be traced

to numerical instabilities


67/88

Uncertainty

Artifactual uncertainty

Too few polynomial terms

Numerical instabilityCan be reduced by a better analysis

Authentic uncertainty

Genuine unpredictability due to input uncertainty

Cannot be reduced by a better analysis

Only by more information, data or assumptions


68/88

Uncertainty propagation

We wantthe prediction to break down if

thats what should happen

But we dont want artifactual uncertainty

Numerical instabilities

Wrapping effectDependence problem

Repeated parameters


69/88

Problem

Nonlinear ordinary differential equation (ODE)

dx/dt=f(x, )

with uncertain and uncertain initial statex0

Information about andx0 comes asInterval ranges

Probability distributions

Probability boxes


70/88

Model

Initial states (bounds)

Parameters (bounds)

VSPODEMark Stadherr et al. (Notre Dame)

List of constants

plus remainder

Taylor models

Interval Taylor series


71/88

Example ODE

dx1/dt= 1x1(1x2)

dx2/dt= 2x2(x11)

What are the states at t= 10?

x0 = (1.2, 1.1)T

1 [2.99, 3.01]

2 [0.99, 1.01]

VSPODE

Constant step size h = 0.1, Order of Taylor model q = 5,

Order of interval Taylor series k= 17, QR factorization


72/88

VSPODE tells how to computex1

1.916037656181642 10

21 + 0.689979149231081 1

120 +

-4.690741189299572 10

22 + -2.275734193378134 1

121 +

-0.450416914564394 12

20 + -29.788252573360062 1

023 +

-35.200757076497972 11

2

2

+ -12.401600707197074 12

2

1

+-1.349694561113611 13

20 + 6.062509834147210 1

024 +

-29.503128650484253 11

23 + -25.744336555602068 1

222 +

-5.563350070358247 13

21 + -0.222000132892585 1

420 +

218.607042326120308 10

25 + 390.260443722081675 1

124 +

256.315067368131281 12 23 + 86.029720297509172 13 22 +15.322357274648443 1

421 + 1.094676837431721 1

520 +

[ 1.1477537620811058, 1.1477539164945061 ]

where s are centered forms of the parameters; 1 = 1 3, 2 = 2 1

Input p boxes


73/88

1 2

p-box

0

1

Pro

bability

0.99 1 1.012.99 3 3.01

precise

Input p-boxes

R lt


74/88

Results

Probability

1.12 1.14 1.16 1.180

1

0.87 0.88 0.89 0.9

x1 x2

p-box

precise


75/88

Still repeated uncertainties

1.916037656181642 10

21 + 0.689979149231081 1

120 +

-4.690741189299572 10

22 + -2.275734193378134 1

121 +

-0.450416914564394 12

20 + -29.788252573360062 1

023 +

-35.2007570764979721

1

2

2 + -12.4016007071970741

2

2

1 +

-1.349694561113611 13

20 + 6.062509834147210 1

024 +

-29.503128650484253 11

23 + -25.744336555602068 1

222 +

-5.563350070358247 13

21 + -0.222000132892585 1

420 +

218.607042326120308 10

25 + 390.260443722081675 1

124 +

256.315067368131281 12 23 + 86.029720297509172 13 22 +15.322357274648443 1

421 + 1.094676837431721 1

520 +

[ 1.1477537620811058, 1.1477539164945061 ]


76/88

Subinterval reconstitution

Subinterval reconstitution (SIR)

Partition the inputs into subintervals

Apply the function to each subintervalForm the union of the results

Still rigorous, but often tighter

The finer the partition, the tighter the unionMany strategies for partitioning

Apply to each cellin the Cartesian product


77/88

Discretizations

2.99 3 3.01

0

1


78/88

0.87 0.88 0.89 0.90

1

1.12 1.14 1.160

1

x1 x2

Contraction from SIR

Probability

Best possible bounds

reveal the authenticuncertainty


79/88

Monte Carlo is more limited

Monte Carlo cannot propagate incertitude

Monte Carlo cannot produce validated results

Though can be checked by repeating simulation

Validated results from distributions can be obtained

by modeling inputs with (narrow) p-boxes and

applying probability bounds analysis

Results converge to narrow p-boxes obtained from

infinitely many Monte Carlo replications


80/88

Results

Probability bounds analysis with VSPODEare useful for bounding solutions ofnonlinear ODEs

They rigorously propagate uncertainty

about in the form of

IntervalsDistributions

P-boxes

Initial states

Parameters


81/88

Paper inAIChE Journal[American

Institute of Chemical Engineers],

February 2011 (on line May 2010)


82/88

PBA relaxes assumptions

Everyone makes assumptions, but not all sets

of assumptions are equal:

Linear Normal Independence

Montonic Unimodal Known correlation

Any function Any distr ibution Any dependence

PBA doesnt require unwarranted assumptions


83/88

Wishful thinking

Analysts often make convenient assumptions

that are not really justified:

1. Variables are independent of one another

2. Uniform distributions model gross incertitude

3. Distributions are stationary (unchanging)4. Distributions are perfectly precisely specified

5. Measurement uncertainty is negligible


84/88

You dont have to think wishfully

A p-box can discharge a false assumption:

1. Dont have to assume any dependence at all

2. An interval can be a better model of incertitude

3. P-boxes can enclose non-stationary distributions4. Can handle imprecise specifications

5. Measurement data with plus-minus, censoring


85/88

Rigorousness

Automatically verified calculations

The computations are guaranteed to enclosethe true results (so long as the inputs do)

You can still be wrong, but the methodwont be the reason if you are


86/88

Take-home messages

Using bounding, you dont have to pretend

you know a lot to get quantitative results

Probability bounds analysis bridges worst

case and probabilistic analyses in a way thats

faithful to both and makes it suitable for use

in early design


87/88

Acknowledgments

Larry Green, LaRC

Bill Oberkampf, Sandia

Chris Paredis, Georgia Tech

NASA

National Institutes of Health (NIH)Electric Power Research Institute (EPRI)

Sandia National Laboratories


88/88

End

Date post:	14-Apr-2018
Category:	Documents
Upload:	jakubgramol
View:	219 times
Download:	0 times

Beyond Probability - A Pragmatic Approach to Uncertainty Quantification in Engineering

Documents