Probabilistic Approach to Design under Uncertainty

Dr. Wei Chen

Associate Professor

Integrated DEsign Automation Laboratory

(IDEAL)Department of Mechanical Engineering

Northwestern University

Outline

• Uncertainty in model-based design• What is probability theory?• How does one represent uncertainty?• What is the inference mechanism?• Connection between probability theory and

utility theory• Dealing with various sources of uncertainty

in model-based design• Summary

• Model (lack of knowledge) • Parametric (lack of knowledge, variability)• Numerical• Testing data

Types of Uncertainty in Model-Based Design

Problem faced in design under uncertainty• To choose from among one set of possible design

options X, where each involves a range of uncertain outcomes Y

• To avoid making an “illogical choice”

• Probability theory is the mathematical study of probability.• Probability derives from fundamental concepts of set theory

and measurement theory.

Basic Concepts of Probability Theory

Event –subset of a sample space e.g., A {e2 and e3} –experiments result in two different faces

Probability P(e1)=P(e2)=P(e3)=P(e4)=0.25 P(A)= P(e2)+P(e3)=0.5

P(null) = 0 P()=1

Sample Space

Event A

e3e2 e1

e4

Example: Flip two coins Sample space – set of all possible outcomes of a random experiment under uncertainty

Outcomes {e1=HH, e2=HT, e3=TH, e4=TT}

• Three axioms of probability measure– 0 P(A) 1; P()=1; P(Ai)=P(Ai) Ai are disjoint events

• Arithmetic of probabilities– Union, Intersection, and Conditional probabilities

• Random variable is a function that assigns a real number to each outcome in the sample space

• Probability density function & arithmetic of moments of a random variable, e.g.,

E[XY]=E[X]E[Y] if X and Y are independent

• Convergence (law of large numbers) and central limit theorem

Mathematics in Probability Theory

Example: define x = total number of heads among the two tossesPossible values {X=0}={TT}; {X=1}={HT, TH}, {X=2}={HH} P{X=1}=0.5

Probabilistic Design Metrics in Quality Engineering

Robustness

0

Probability Density (pdf)

Performance y

Target M

Bias

Minimizing the effect of variations without eliminating the causes

yy y Performance g

R=Area = Prob{g(x)c}

Reliability

pdf

To assure proper levels of “safety” for the system designed

C

• Frequentist – Assign probabilities only to events that are random based on

outcomes of actual or theoretical experiments– Suitable for problems with well-defined random experiments

• Bayesian – Assign probabilities to propositions that are uncertain according to

subjective or logically justifiable degrees of belief in their truth

Example of proposition: “there was life on Mars a billion years ago”

– More suitable for design problems: events in the future, not in the past; all design models are predictive.

– More popular among decision theorists

Philosophies of Estimating Probability

• In the absence of data (experiments), we have to guess– A probability guess relies on our experience with “related”

events

• Once data is collected, inference relies on Bayes theorem– Probabilities are always personal degrees of belief

– Probabilities are always conditional on the information currently available

– Probabilities are always subjective

• “Uncertainty of probability” is not meaningful.

Bayesian Inference

Bernardo, J.M. and Smith, A. F., Bayesian Theory, John Wiley, New York, 2000.

Bayes’ Theorem

H - HypothesisD - Data

Bayes’ theorem provides •A solution to the problem of how to learn from data•A form of uncertainty accounting•A subjective view of probability

( | ) ( )( | )

( )

P D H P HP H D

P D

P (D | H) = L(H)

Max. Likelihood. Est.

Data

H

Belief about H before obtaining data, prior P

P (H)

HPrior mean

Belief about H after obtaining data, posterior P

P (H | D)

HPosterior mean

Updatedby data

Formalism of Bayesian Statistics

• Offers a rationalist theory of personalistic beliefs in contexts of uncertainty with axioms clearly stated

• Establishes that expected utility maximization provides the basis for rational decision making

• Not descriptive, i.e., not to model actual behavior.

• Prescriptive, i.e., how one should act to avoid undesirable behavioural inconsistency

• Three basic elements of decision– the alternatives (options) X– the predicted outcomes (performance) Y– decision maker’s preference over the outcomes, expressed as an

objective function f in optimization

• Utility theory– Utility is a preference function built on the axiomatic basis

originally developed by von Neumann and Morgenstern (1947)– Six axioms (Luce and Raiffa, 1957; Thurston, 2006) Completeness of complete order Transitivity Monotonicity Probabilities exist and can be quantified Monotonicity of Probability Substitution-independence

Connection of Probability Theory and Utility Theory

In agreement to employing probability to model uncertainty

• Without uncertainty – objective function f = V(Y) = V(Y(X))

V - value function, e.g. profit

• With uncertainty – objective function f =

E(U) - expected utility. The preferred choice is the alternative (lottery) that has the higher expected utility.

Decision Making – Ranking Design Alternatives

pdf (V)

V (e.g. profit)

A B

U (V)

V

1

0worst best

Risk neutral

Risk averse

Risk prone

dV)V(pdf)V(U)U E(

Issues in Model-Based Design

Chen, W., Xiong, Y., Tsui, K-L., and Wang, S., “Some Metrics and a Bayesian Procedure for Validating Predictive Models in Engineering Design”, DETC2006-99599, ASME Design Technical Conference.

• How should we provide probabilistic quantification of uncertainty associated with a model?

• How should we deal with model uncertainty (reducible) and parameter uncertainty (irreducible) simultaneously?

• How should we make a design decision with good confidence?

Bayesian Approach for Quantifying the Uncertainty of Predictive Model

( ) ( ) ( )

( ) ( ) ( )

e r

m

Y Y

Y

x x x

x x x

( )eY x( )rY x

( ) x

( ) x

- Physical observation

- Computer model output

- Bias function (between and ) ( )rY x

( )mY x- True but unknown real performance

( )mY x

- Random error in physical experiment

ˆ ( )mY x

Computer experiments

Physical experiments

Observations (data) of ( ) x

Metamodel of

Uncertainty is accounted for by

( ) ( ) ( )r mY Y x x x

Bias-Correction

ˆ ( )rY x and UQ

( ) x

ˆ( ) x

[ ( ) ( )] ( ) ( )e mY Y x x x x

Bayesian Approach

and UQ

Bayesian posteriorof ( ) x

( )mY x

mean: zero variance:

Model assumption

( ) x

( ) x Gaussian process

-

-

Gaussian process (I.I.D.)mean: covariance: ( ) ( )T

x f x 2

1

( ) exp ( )p

i j k i k j kk

R x x

x x

2 2

2R .

More about the Bayesian Approach

Known parameters (deterministic)k

Priors distribution of parameters (nondeterministic)2 ( )IG 2 2( )N b V 2 ( )IG

Estimated from data, by MLE or Cross validation

1( ( ) ( ))e

e e e Tny y y x x

or

Physical experimentData

Computer experiment 1( ( ) ( ))e

m m m Tny y y x x

1ˆ ˆ( ( ) ( ))

e

m m m TnY Y y x x

1 1( ) ( ) ( ) ( ))e e

e m e m Tne n ny y y y δ x x x x

2( ) ( ( ) ( ))e me m e m e mT n x y y x x

Posterior distribution of parameters

Posterior distribution of ( ) x

2

That is, the posterior of is a non-central t process

(omitted here)

( ) x

Predictive model and uncertainty quantification

Design validation metrics

Design objective function and uncertainty quantification

No

Givencomputer model

Yes

ˆ ( )rY x

Parameteruncertainty

Specified confidence

level Pth

Design decisionExpected Utility Optimization

Design validity requirements satisfied

(MD < Pth)?

Integrated Framework for Handling Model and Parameter Uncertainties

ˆ ( )f x

DM

Computer experiments

Physical experiments

Sequential experiment design

1 2( ) ( ) ( )r rY Yf w w x x x

( )f x95% PI

Realizations of ( )rY x

x

A robust design objective (smaller-is-better) is used to determine the optimal solutions.

( )f x

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 153.5

54

54.5

55

55.5

56

56.5

57

57.5

58

58.5

x

y

Yrpredction

95% PI

realization of Y r

( )rY x ( )f xUncertainty of

w1, w2 : weighting factors

Uncertainty of

Apley et al. (2005) developed analytical formulations to approximately quantify the mean and variance of . In this example, Monte Carlo Simulation is employed.( )f x

x is a design variable and a noise variable

Uncertainty Quantification of Design Objective Function with Parameter Uncertainty

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 153.5

54

54.5

55

55.5

56

56.5

57

57.5

58

58.5

x

f

fpredction

95% PI

( )f x

( )f x( )rY x

Mean of

0

1

,

( *) ( *) ( )i d i

K

D iX

M P f f

x x

x x x

Three types of design validation metrics (MD) – f is small-the-better

0,

1( *) ( *) ( )

i d i

D iX

M P f fN

x x

x x x

0,

( *) min ( *) ( )i d i

D iX

M P f f

x x

x x x

Type 2: Additive Metric

Type 3: Worst-Case Metric

Validation Metrics

Type 1: Multiplicative Metric

averaging

Probabilistic measure of whether a candidate optimal design is better than other design choices with respect to a particular design objective

Larger confidence Smaller confidence

MD is intended to quantify the confidence of choosing x* as the optimal design among all design candidates or within design region .d

f2f f

2f f f

x*

x

f2f f

2f f f

x*

xx1 x2

x1 x2

Summary

• Prediction is the basis for all decision making, including engineering design.

• Probability is a belief (subjective), while observed frequencies are used as evidence to update the belief.

• Probability theory and the Bayes theorem provide a rigorous and philosophically sound framework for decision making.

• Predictive models in design should be described as stochastic models.

• The impact of model uncertainty and parameter uncertainty can be treated separately in the process of improving the predictive capability.

• Probabilistic approach offers computational advantages and mathematical flexibility.