Challenges In Uncertainty, Calibration, Validation and ... · Epistemic & Aleatory uncertainty....

1GE – Aviation & Global Research© 2011 General Electric Company - All Rights Reserved

Challenges In Uncertainty, Calibration, Validation and Predictability of Engineering Analysis Models

Dr. Liping WangGE Global ResearchManager, Probabilistics LabNiskayuna, NY

2011 UQ WorkshopUniversity of Minnesota, MNJune 02, 2011

Team MembersGE Global Research: Arun Subramaniyan, Nataraj Chennimalai,

Xingjie Fang, Giridhar Jothiprasad, Martha Gardner, Amit Kale

GE Aviation: Don Beeson, Gene Wiggs, and John Nelson


Outline

•Motivation

• History of Development

– How far are we along the path?

• GE capabilities

• Technical Challenges & Solutions

• Future Direction

• Summary


Input

Factors (Xs)

Motivation

• Why Model Calibration, Validation, Prediction & Uncertainty Quantification?

• What has been accomplished? �Literature Review �GE Experience

• Possible technical solutions and future direction

Output

Factors (Ys)Uncertainty:

• Aleatory – Random and usually modeled by

probability distributions.

Methods include probability theory

and classical statistics

• Epistemic – Lack of knowledge. Methods include

fuzzy logic or evidence/possibility

theory

X1X2

Etc.

.

.

Etc.

Deterministic

Simulation

X1

X2

Etc.

Input

Factors (Xs)

Y2

Etc.

Output

Factors (Ys)

Y1

Model parametersModel discrepancy

.

θ

δδδδ

Uncertainty Quantification (UQ)


•One task at a time (calibration, validation, prediction & uncertainty quantification) - since the 1980s

�Calibration - data matching, or inverse problems, or parameter estimation

- applied to heat transfer, fluid mechanics, solid mechanics, etc.

�Verification & Validation (V&V) - introduced by DoD, AIAA, ASME, National Labs …

�Prediction -well-established physics models, calibrated empirical models, and

meta-models (Response Surface, Kriging, Gaussian Process, Radial Basis Function, etc)

�Uncertainty quantification - Monte Carlo, First Order Second Moment, moments

based, polynomial chaos, etc

•All tasks simultaneously - first introduced by Kennedy and O’Hagan in 2001 (Bayesian framework)

History of Development

– How far are we along the path?


Kennedy & O’Hagan (2001)

y(xi)= η(xi, θ) + δ(xi) + ε(xi), i=1,2,…,n

Observations from Observations from the physical systemthe physical system Output of a simulator, Output of a simulator,

with design inputs (with design inputs (xx) and ) and calibration parameters (calibration parameters (θθ))

Discrepancy between Discrepancy between the simulator and the the simulator and the physical systemphysical system

Observation Observation (measurement system) (measurement system) errorerror

• Build & calibrate Gaussian Process (GP) models for both ηηηη and δ...δ...δ...δ...

� Specify beliefs about θ , δ through prior probability distributions � Use Markov Chain Monte Carlo (MCMC) to obtain parameter estimates

• Similar approaches by Higdon et al. and Liu et al.

•Kennedy & O’Hagan Hybrid Model Formulation:

Most implementations are for single output

Given data

( | ) ( )( | )

( | ) ( )

f y ff y

f y f

θ θθ

θ θ=

∫

( | )f yθ

pdf

Posterior

θ

( )f θ

pdfPriorθ

Likelihood function

( ) ( | )L f yθ θ=

θ

•What is Bayesian Statistics?


Multiple Outputs

•Implementation by Los Alamos National Lab (LANL) - Higdon, William et al.

� Principal Components Analysis (PCA) for dimension reduction & efficiency improvements

� Correlated outputs

),(...),(),( 11 θθθηηη

xwkxwkx pp++=

)(...)()( 11 xvdxvdx pp δδδ ++=

ηpkkk ,...,, 21 and are the principal componentsδpddd ,...,, 21

w & v are the GP models for simulator and model correction

More Applicable to Real Problems with LANL Implementation


Maximum Likelihood Estimation (MLE)

• Alternative approach to Bayesian (Xiong, Chen, Tsui and Apley)

• Investigated three possible formulations

• Implementation only for single output

• Sensitivity analysis prior to MLE optimization to avoid numerical instability

εθη +=Θ ),(),( xxy

εδθη ++=Θ )(),(),( xxxy

εδθη ++=Θ )())(,(),( xxxxy

best


Model Inadequacy Correction & Prediction

(No Calibration)

• Capture model inadequacy with no model calibration

(Wang et al. and Chen et al.)

• Closed form Bayesian posterior

• Solve GP hyper-parameters using MLE

• Improved efficiency for high dimensional design space

• Useful for well established physical models where calibration is not necessary or performed previously

εδη ++= )()()( xxxy


V&V and Model Validation Metrics

Most Common

Desired State

• Current and desired state

of validation metrics(Oberkampf et al. 2004)

• Quantitative Metrics using

classical hypothesis testing,

Bayes factor, frequentist’s

metrics, and area metrics

• Quantitative Metrics using

Kennedy & O’Hagan Bayesian

Framework (Chen et al.)

• Preliminary elements of model validation

(Paez, Swiler, Mayes, Miller, et al.

– 2009 International Modal Analysis

Conference, Orlando, FL)

• Epistemic uncertainty (Paez & Swiler, Paez)

Analysis

Modelers

Customers Stakeholders

Experimentalists

Validation Analysts


GE Capabilities

• Deterministic Inverse modeling since 2003�Methods development and implementation

�Efficient transient data matching using PCA based hybrid

metamodels & zoning techniques

�Partial probabilistic data matching to update standard deviations

•Widely used across GE businesses

Materials Design Acceleration – Material Modeling

GEnx – 29 XsGP7000 – 100XsGE90 – 78Xs

Analysis time savings >50%Data mismatch reduced by half

Transient Analysis and Performance

23 Xs, 35 TCs simultaneously

Others:Transient cycle models 3D transient clearances Undercowl heat transfer Empirical model tuning

Heat Transfer and Fluid Systems


Probabilistic Inverse modeling (Bayesian Hybrid Modeling) since 2006

� Built on Kennedy & O’Hagan Bayesian Method and LANL Implementation

� Efficiency improvement (~2X), flexibility, robustness …

� Investigated possible formulations

� Key drivers of model inadequacy & insight to possible model improvements

� Validated with multiple benchmark problems

GE Capabilities

εθη

εδη

δη

η

+=

++=

+=

=

),()(

)()()()(

)()()(

)()(

xxy

xxxxy

xxxy

xxy

εδθη

εδθη

εδθη

εδθη

+=

++=

++=

++=

))(,,()(

)())(,()(

))(,()(

)()(),()(

xxxy

xxxxy

xxxy

xxxxy Kennedy & O’Hagan


Demonstration with challenging engineering problems

GE Capabilities

Corrected flow

PT_ra

tio

1201151101051009590858075706560555045

1.9

1.8

1.7

1.6

1.5

1.4

1.3

1.2

1.1

10.9

1201151101051009590858075706560555045

1.9

1.8

1.7

1.6

1.5

1.4

1.3

1.2

1.1

10.9

Missing vertical parts of high speed lines

Test data : y(x)

Test data uncertainty: ε(x)

BRM model: η(x,θ)

Design parameters: x

Model parameters: θ

104 104.5 105 105.5 106 106.5 107 107.5 108 108.5 1091.54

1.56

1.58

1.6

1.62

1.64

1.66

1.68

1.7

1.72

1.74

Test

Hybrid Modeling

Calibrated Simulator only

105% speed-line

104 104.5 105 105.5 106 106.5 107 107.5 108 108.5 1091.54

1.56

1.58

1.6

1.62

1.64

1.66

1.68

1.7

1.72

1.74

Test

Hybrid Modeling


105% speed-line

Performance Maps at Speed = 105%

δ(x)

Model Discrepancy & Updatingδ(x)

HybridModeling

-1 -0.5 0 0.5 1

Mean=0.1797Std=0.39367

θ (calibrated)

Test

HM Mean

90% CI

ConfidenceBounds

Corrected flow

PT_ra

tio

Corrected flow

PT_ra

tio

1201151101051009590858075706560555045

1.9

1.8

1.7

1.6

1.5

1.4

1.3

1.2

1.1

10.9

1201151101051009590858075706560555045

1.9

1.8

1.7

1.6

1.5

1.4

1.3

1.2

1.1

10.9

Missing vertical parts of high speed lines

Test data : y(x)

Test data uncertainty: ε(x)

BRM model: η(x,θ)

Design parameters: x

Model parameters: θ

104 104.5 105 105.5 106 106.5 107 107.5 108 108.5 1091.54

1.56

1.58

1.6

1.62

1.64

1.66

1.68

1.7

1.72

1.74

Test

Hybrid Modeling


105% speed-line

104 104.5 105 105.5 106 106.5 107 107.5 108 108.5 1091.54

1.56

1.58

1.6

1.62

1.64

1.66

1.68

1.7

1.72

1.74

Test

Hybrid Modeling


105% speed-line

Performance Maps at Speed = 105%

δ(x)

Model Discrepancy & Updatingδ(x)

HybridModeling

-1 -0.5 0 0.5 1

Mean=0.1797Std=0.39367

θ (calibrated)

Test

HM Mean

90% CI

ConfidenceBounds

Corrected flow

PT_ra

tio

95 100 105 1101.35

1.4

1.45

1.5

1.55

1.6

1.65

1.7

1.75

Test


Hybrid Modeling

105%

100%

95%

95 100 105 1101.35

1.4

1.45

1.5

1.55

1.6

1.65

1.7

1.75

Test


Hybrid Modeling

105%

100%

95%

98 100 102 104 106 108 1101.4

1.45

1.5

1.55

1.6

1.65

1.7

1.75

1.8

Test


Hybrid Modeling

100%

105%

98 100 102 104 106 108 1101.4

1.45

1.5

1.55

1.6

1.65

1.7

1.75

1.8

Test


Hybrid Modeling

100%

105%

2 Speed Lines 3 Speed Lines

95 100 105 1101.35

1.4

1.45

1.5

1.55

1.6

1.65

1.7

1.75

Test


Hybrid Modeling

105%

100%

95%

95 100 105 1101.35

1.4

1.45

1.5

1.55

1.6

1.65

1.7

1.75

Test


Hybrid Modeling

105%

100%

95%

98 100 102 104 106 108 1101.4

1.45

1.5

1.55

1.6

1.65

1.7

1.75

1.8

Test


Hybrid Modeling

100%

105%

98 100 102 104 106 108 1101.4

1.45

1.5

1.55

1.6

1.65

1.7

1.75

1.8

Test


Hybrid Modeling

100%

105%

2 Speed Lines 3 Speed Lines

-0.4 -0.2 0 0.2 0.4 0.6 0.8 10

50

100

150

200

250

300

350

400

450

Mean=0.036642

Std=0.1968

-0.4 -0.2 0 0.2 0.4 0.6 0.8 10

50

100

150

200

250

300

350

400

450

Mean=0.036642

Std=0.1968

Matches Test Data Well for Single & Multiple Speed Lines


GE Capabilities GE Capabilities

13

Transient AIR Model (Benchmark problem)

• 6 calibration parameters

• 9 Outputs (Ys)

• 52 time points in each transient

(> 3000 DoE points)

• Only 52 DoE (simulation) points used

for Hybrid modeling

BHM calibrates transient model accurately with very little data

ηηηη(t, θθθθ)

y(t)= ηηηη(t, θθθθ) + δδδδ(t)

δδδδ(t)Calibrated Model

Calibrated &

Discrepancy Adjusted

Model

Discrepancy


Demonstration with challenging engineering problems

GE Capabilities

Demonstration with IPE Status Match, TOW Uncertainty Severity Modeling, Cycle Deck Performance …

IPE Status Match TOW Uncertainty

Cycle Deck

0 0.2 0.4 0.6 0.8 1500

550

600

650

700

750

800

ZT

49

calibrated simulator

0 0.2 0.4 0.6 0.8 1

ZPCN12

discrepancy-adjusted

0 0.2 0.4 0.6 0.8 1-8

-6

-4

-2

0

2

4

6

8

10discrepancy

-0.04 -0.03 -0.02 -0.01 0 0.01 0.02 0.03 0.040

50

100

150

200

250

300

350

DE42DD

His

togra

m C

oun

ts

θ

Posterior Distribution

of HPT Efficiency

Fan Speed

Exh

aust Gas Temperature

Calibrated Simulator Discrepancy-Adjusted Discrepancy

Fan Speed

Exh

aust Gas Temperature

Calibrated Simulator Discrepancy-Adjusted Discrepancy

• Improved calibration results by capturing model discrepancy• More confidence in solution with probabilistic estimation

• Characterizing model discrepancy and uncertainty in severity models • Main effects able to point high thrust severity for improvement of

current models


Extension to Model Validation

GE Capabilities

Acoustic fluctuations (p’)

Flame heat release fluctuations (q’)





Xs

θθθθs

δδδδ(x)

εεεε(x)

Combustion Dynamics

GE90 Fan Blade Row Model

-40 -30 -20 -10 0 10 20 30 40 50 600

5

10

15

20

25

30

δδδδ(x) at untested points

-50 -40 -30 -20 -10 0 10 20 30 40 500

5

10

15

20

25

θ (calibrated)

Co

mp

uta

tio

n v

. E

xp

eri

men

t

-2.5

-2

-1.5

-1

-0.5

0

0.5

δδδδ(x)

Freq1

Calibrated Simulator Model Discrepancy δδδδ(x)

δδδδ(x) at tested points

-40 -30 -20 -10 0 10 20 30 40 50 600

5

10

15

20

25

30

δδδδ(x) at untested points

-50 -40 -30 -20 -10 0 10 20 30 40 500

5

10

15

20

25

θ (calibrated)

Co

mp

uta

tio

n v

. E

xp

eri

men

t

-2.5

-2

-1.5

-1

-0.5

0

0.5

δδδδ(x)

Freq1

Calibrated Simulator Model Discrepancy δδδδ(x)

δδδδ(x) at tested points

Enabler for Probabilistic Validation Metrics at Tested & Untested Points


Going-forward�Continue improving MCMC speed issues for high dimensional (>100Xs)

and solving challenging engineering data matching applications

� Model Validation for All Engineering Models (3-year program)

(Flexible to all models based on data availability, affordable & accurate,

account for all types of uncertainty, probabilistic quantitative metrics)

GE Capabilities

TTPTVrVthKω

combustor exit

purge flow

geom etrytip clearance,

core shift, film

hole d rilling

Xs

TTPTVrVthKω

combustor exit

purge flow

geom etrytip clearance,

core shift, film

hole d rilling

Xs

θθθθsεεεε(x)







Xs

θθθθsδδδδ(x)

εεεε(x)

δδδδ(x) = y(x) – (ηηηη(x, θθθθ) + εεεε(x))δδδδ(x) = y(x) – (ηηηη(x) + εεεε(x)) δδδδ(x) = y(x) – (ηηηη(x, θθθθ) + εεεε(x))

Aero

Mechanical

2011 Hot Gas Path Heat Transfer 2012 Combustion Dynamics 2013 All Engineering Models

Complicated Physics, Unknown Uncertainty, High Dimension … Challenges Remain!


Technical Challenges and Solutions

• Curse of dimensionality�Large number of calibration parameters. MCMC speed issues.

�Transient data matching

�PCA/Sensitivity, sparse matrix inversion, adaptive convergence criteria for MCMC, sequential MC or other optimization techniques …

• Source of Uncertainty�Epistemic & Aleatory uncertainty.

�Probability, statistics, fuzzy logic or evidence/possibility theory

• Model inadequacy, uncertainty and characterization�Identifiability of parameter calibration and model inadequacy

Use as much knowledge as you can on the prior. Carefully choose

the range. Uncertainty quantification of experiments.

�Better understand model inadequacy and key drivers

�Post-processing to model discrepancy terms. Bayesian model comparison for possible suggestion to model improvement



• Lack of data and extrapolation�Limited test & simulation data

�No overlap between simulation & test data (extrapolation)

�Scientific knowledge, designer’s belief (prior)

• Confounding Effects�High-dimensionality coupled with lack of data

�Challenging mathematical issues

�Scientific knowledge, designer’s belief (prior)

• Model validation and quantitative metrics�Account for all source of uncertainty (epistemic & aleatory)

�Flexible for all analysis models - empirical, physics (no calibration) & metamodels … based on data availability (complete, partial and extrapolation)

�Affordable & accurate

�Confidence and probabilistic metrics



• Multiple sets of experimental data�Mixed datasets – multiple speed lines, multiple engine data …

�Multiple modes of posterior distributions

�Numerical and speed issues

• Measurement error and uncertainty�Uncertainty quantification for both epistemic and aleatory uncertainty

�Statistical analysis

�Outlier detection (sensor failure)

• Multi-physics & multi-fidelity models�Direct simulations are prohibitively expensive

�Vast scale difference among the lowest atomic to the highest macroscopic scale

�Difficult to establish criteria & strategies when switching design space from one scale to another


Summary

• Advancements in model calibration, validation, prediction and uncertainty quantification in the past three decades

• Much research and many publications from industry, government, academia

• GE has been very active

• Technical challenges remain and are being worked

Date post:	05-Jul-2020
Category:	Documents
Upload:	others
View:	10 times
Download:	4 times

Challenges In Uncertainty, Calibration, Validation and ... · Epistemic & Aleatory uncertainty....

Documents