Post on 27-Jun-2018
transcript
1 July 19, 2012
Bayesian Methods in Probability of Detection Estimation and Model-assisted Probability of
Detection (MAPOD) Evaluation John C. Aldrin*, Jeremy S. Knopp, Harold A. Sabbagh†
Nondestructive Evaluation Branch (AFRL/RXLP)
Materials and Manufacturing Directorate Air Force Research Laboratory
Wright-Patterson AFB, Ohio, USA
*Computational Tools, Gurnee, Illinois, USA † Victor Technologies LLC, Bloomington, Indiana, USA
Review of Progress in Quantitative NDE
Denver, CO, USA July 19, 2012
2 July 19, 2012
Outline
• Background
• Demonstrations:
– Multiparameter Regression Models
– Integration of Physics-based Models
– Hierarchical Models in POD (MAPOD) Evaluation
• Discussion:
– Software Tools
– Challenges / Future Work
3 July 19, 2012
What is Probability of Detection (POD) and Model Assisted POD (MAPOD)?
• Probability of detection (POD) of a certain discontinuity as a function of some size metric given a defined inspection technique and target population.
• We define “MAPOD” as the collection of approaches that use models of inspections as some portion of the inputs that are processed to yield an estimate of POD.
4 5 6 7 8 9 2 3 4 5 6 7 8 9
4
5
6
7
8
9
2
3
4
5
6
7
8
9
m h 1 8 2 3
E X A M P L E 1 â v s a . x l s
0 . 6
0 . 8
resp
onse
(arb
itrar
y un
its)
discontinuity size (arbitrary units)
noise in the absence of
discontinuity
data
decision threshold
FALSE CALLSβ0
β1
ε
εββ ++= aa lnln 10
^
11
0 and )ln(
whereβδσ
ββ
µ =−
= thy
−
Φ−=σ
µaaPOD ln1)(
:Φ cumulative normal distribution function
Evaluating Reliability Using Simulated and Empirical Data: • To mitigate cost of validation study, one must better assess the
critical sources of error and variation on reliability performance • Hoppe [2009] presented historical case highlighting benefit of improving
the measurement model through including crack length and depth in fit •
•
• Physics-based models provide opportunity for reducing experimental samples and cost
Mitigating Cost of POD Study through Improved Model Accuracy
Objective: Explore Case Studies to Assess Impact of Measurement Model Quality on POD Estimation and Sample Number
εββ ++= 110ˆ aa
εβββ +++= 22110ˆ aaa
Increase Model Accuracy
Reduces
Residuals in Model Fit
Improves Bounds on Parameter Estimates (POD)
Impacts Experimental
Sampling Requirements ( ) εββ ++= 2110 ,ˆ aafa
Model-Assisted POD Model Building Process [MIL-HNBK 1823A, Appendix H (2009)]
Uncertainty Propagation
Model Error
Input Parameter Variability
(Distributions)
Stochastic Models
Model ‘Calibration‘
Revise Model Estimates Using Bayesian Methods
Confidence Bounds (Limited Samples)
Objective: Leverage Bayesian Method in MAPOD Evaluation
Assess Key Factors (Joint PDFs) using Bayesian Methods
Approach: Integrate Modeling and Simulations with Empirical Studies
• Bayesian methods are necessary to incorporate empirical data with NDE models (prior information)
• Application of Bayes’ rule: • : prior probability of θ • : conditional probability (likelihood)
of new evidence (data), x , given θ • : posterior probability of θ given
new evidence x • Posterior distribution can be evaluated, providing
a refinement to the original prior distribution through numerical methods such as Markov Chain Monte Carlo (MCMC) simulation
)()()(
)(x
xx
PPP
Pθθ
θ =
)(θP)( θxP
)( xθP
Models and Simulation
Experiments
posterior
prior
θ
likelihood
Integration (Bayesian Approach)
50 100 150 200
50
100
150
-200
0
200
400
600
-0.35 -0.3 -0.25 -0.2 -0.15 -0.1 -0.05 0 0.05-0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
dR (Ω)
dX ( Ω
)
exp: l=1.0 mmexp: l=1.5 mmexp: l=2.0 mmexp: l=2.5 mmVIC: l=1.0 mmVIC: l=1.5 mmVIC: l=2.0 mmVIC: l=2.5 mm
Bayesian Methods for in POD / MAPOD Evaluation
General References: 1. Gelman, A., Carlin, J.B., Stern, H.S., Rubin, D.B., Bayesian Data Analysis, 2003. 2. Lunn, D.; Spiegelhalter, D.; Thomas, A.; Best, N. (2009). "The BUGS project:
Evolution, critique and future directions". Statistics in Medicine 28: 3049–3067 3. Christensen, R., W. Johnson, and A. Branscum, Bayesian Ideas and Data Analysis:
An Introduction for Scientists and Statisticians,” CRC Press, 2010. NDE References: 1. Meeker, W.Q. and L.A. Escobar, "Introduction to the Use of Bayesian Methods for
Reliability Data," Statistical Methods for Reliability Data, Wiley, 1998, pp. 343-368. 2. Leemans, D.V, and Forsyth, D., “Bayesian Approaches to Using Field Test Data in
Determining the Probability of Detection,” Materials Evaluation, 2004. – early demonstration of Bayesian methods in hit-miss POD evaluation
3. Annis, C., http://www.statisticalengineering.com/ – discusses Bayesian updating, confidence vs. credible bounds,
4. Wang, Y., “Advanced statistical methods for analysis of NDE data ,” Dissertation, 2006. (Advisor: W.Q. Meeker.)
5. Thompson, R.B., A Bayesian Approach to the Inversion of NDE and SHM Data”, Rev. Prog. Quant. Nondestr. Eval, Vol 29, 2010, pp. 679-686.
Bayesian Methods in MAPOD/POD Evaluation – Prior Work
Bayesian Methods in MAPOD/POD Evaluation - Recent Work
NDE References (cont.): 6. Li, Meeker and Hovey, “Joint Estimation of NDE Inspection Capability and Flaw-
size Distribution for In-service Aircraft Inspections,” RNDE, 2012. – Evaluate noise interference model POD and crack distribution
7. Kanzler, D., Muller, C., Pitkanen, J., Ewert, U.,“Bayesian Approach for the Evaluation of the Reliability of Non-Destructive Testing Methods,” WCNDT 2012.
Related Recent Work for AFRL: 1. Statistical Analysis of Hit/Miss Data using Bayes Factors (Model Selection)
[Knopp and Zeng, 2012, submitted for publication] 2. Application of Gaussian Process Models for Quantifying the Accuracy and
Capability of Nondestructive Sensing Methods for Damage Characterization – Victor Technologies Phase I SBIR [Aldrin et al., 2012; ASNT Fall conference ]
Objectives of Presentation: • Present Bayesian methods for POD evaluation with NDE measurement
models of increasing complexity: – Multivariate models – Physics-based models
• Explore Bayesian methods for stochastic model parameter estimation
Demo 1: Eddy Current Inspection of Surface-breaking Cracks in Ti-6Al-4V
Identify Controlling Factors: • Crack Characteristics
– Length and Depth (aspect ratio) – Width (cracks, EDM notches) – Stress state across crack face (closure) – Crack morphology
• Material Properties – Conductivity – Material noise (anisotropy, grain structure) – Surface condition (roughness, residual stress, coldwork)
• Part Geometry (assume locally flat) • Probe (frequency fixed at 2.0 MHz)
– liftoff – tilt – dimensions, windings (probe to probe variability)
• Scan resolution (fixed) • DAQ hardware (Agilent Impedance Analyzer, Nortec 19eII) • Calibration process - isolate liftoff direction in response
- set full screen height for known notch (0.10“) • Human data interpretation - use quantitative metrics in evaluation
x y z
50 100 150 200
50
100
150
-200
0
200
400
600
1.0 mm crack
response in Ti-6Al-4V
-0.35 -0.3 -0.25 -0.2 -0.15 -0.1 -0.05 0 0.05-0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
dR (Ω)
dX ( Ω
)
exp: l=1.0 mmexp: l=1.5 mmexp: l=2.0 mmexp: l=2.5 mmVIC: l=1.0 mmVIC: l=1.5 mmVIC: l=2.0 mmVIC: l=2.5 mm
Input Parameters in Study: • a1 = Crack length
– primary variable for POD • a2 = Crack depth (width)
– dependent variable on a1 crack length
– relationship define by function a2(a1) = a4 * a1
– a4 is the aspect ratio and defined as an random variable
• a3 = Liftoff – uncontrolled parameter
during study – estimation of liftoff
could improve POD performance (to verify)
0.06 0.08 0.1 0.120
200
400
600
liftoff2 2.5 3
0
200
400
600
aspect ratio
0 1 2 30
200
400
600
length0 0.5 1 1.5
0
200
400
600
800
width
Simulated POD Studies: EC Inspection of Cracks in Ti-6Al-4V
EC Response wrt Crack Length (variation for a2 and a3 included)
• Example: Eddy Current Inspection of Surface-breaking Cracks in Ti-6Al-4V
• Case Study for Multivariate POD Model Evaluation (ahat-vs-a1-and-a2):
• POD Analysis Implemented in POD Toolkit [TRI/Austin]: • R link to WinBUGS used for Bayesian Analysis
• Compare different model fits and confidence bounds approaches – Analysis 0: Neglect a2 – Analysis 1: Regression (MLE) fit,
Delta method for confidence bds. – Analysis 2: Regression (MLE) fit,
Monte Carlo for confidence bds. – Analysis 2: Bayesian (MCMC) fit
for model and confidence bds. – Good agreement between three
multivariate POD model fits
Analysis 0 Analysis 1 Analysis 2 Analysis 3 survreg() survreg() glm()/MC BayesMCMC
neglect a2 use a1, a2 use a1, a2 use a1, a2 B0 -0.05780 -0.05986 -0.05986 -0.05983 B1 5.39532 2.77503 2.77503 2.77668 B2 0.00000 6.65178 6.65178 6.64630
Delta 0.02538 0.02001 0.02001 0.02061 Threshold 0.10000 0.10000 0.10000 0.10000
var11 0.00003 0.00002 0.00002 0.00002 var22 0.00912 0.11971 0.12346 0.12687 var33 0.00000 0.73494 0.75790 0.78247 a50 0.02925 0.03204 0.03204 0.03203 a90 0.03529 0.03719 0.03721 0.03732
a90/95 0.03616 0.03720 0.03837 0.03733
εβββ +++= 22110ˆ aaa
Bayesian Methods for POD / MAPOD Evaluation (1)
-0.35 -0.3 -0.25 -0.2 -0.15 -0.1 -0.05 0 0.05-0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
dR (Ω)
dX ( Ω
)
exp: l=1.0 mmexp: l=1.5 mmexp: l=2.0 mmexp: l=2.5 mmVIC: l=1.0 mmVIC: l=1.5 mmVIC: l=2.0 mmVIC: l=2.5 mm
xyz
),0(~ 2εσε N
Bayesian Methods - Software
Bayesian Analysis Software Options: • OpenBUGS (WinBUGS): Comprehensive tool for MCMC simulation
• Define ‘model’ as separate file
Bayesian Methods - Software
Bayesian Analysis Software Options: • OpenBUGS (WinBUGS) http://www.openbugs.info
• Comprehensive tool for MCMC simulation • Define ‘model’ as separate file • Guide to Running WinBugs and OpenBugs from R
• http://www.stat.columbia.edu/~gelman/bugsR/ • Interface with R / Matlab for Bayesian POD Evaluation
• R: function (x1, y1, a.hat.decision, model.file, winbugs.path) • Very difficult to embed numerical model results in ‘model’
• Matlab - DRAM - Delayed Rejection Adaptive Metropolis [http://www.helsinki.fi/~mjlaine/dram/] • Provides means to embed Matlab function calls in MCMC • Facilitates integration of physics-based (surrogate) models • Not as general and robust as OpenBUGS
• Matlab - Statistics Toolbox • Pymc: Python Toolkit for Markov Chain Monte Carlo sampling
Demo 2: EC Inspection of Fastener Sites for Fatigue Cracks
• C-5 Wing Splice Fatigue Crack Specimens: – Two layer specimens are 14" long and 2" wide, – 0.156" top layer, 0.100" bottom layer – 90% fasteners were titanium, 10% fasteners were steel – Fatigue cracks position at 6 and 12 o’clock positions – Crack length ranged from 0.027" – 0.169“ (2nd layer) – vary: location of cracks – at both 1st and 2nd layer
• AFRL/UDRI Acquired Data (Hughes, Dukate, Martin)
A1-16C0
100 200 300 400 500 600 700
20406080
100120 0
2
4
A1-16C0
100 200 300 400 500 600 700
20406080
100120
-2
0
2
0.110" 0.107"0.110" 0.107"
x
z
ba
2nd layer – corner crack
zb
a
1st layer – corner crack
0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18-0.05
0
0.05
0.1
0.15
a1 (in.)
a hat
0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18-0.1
-0.05
0
0.05
0.1
a1 (in.)
resi
dual
s
• Example: Eddy Current Inspection of Cracks at Fastener Sites • Case Study for Physics-based Model Evaluation:
• • where f () is a function call for a physics-based model (i.e. VIC-3D)
• Bayesian POD Analysis Performed in Matlab + R: • MCMC library in Matlab used for Bayesian Analysis • Matlab Provides Option for Integration of Model Function in Bayesian Fit
• Compare Ahat-vs-a fit (MLE, Wald bounds) and a Physics-based Model Fit
Bayesian Methods for POD / MAPOD Evaluation (2)
( ) εββ ++= 2110 ,ˆ aafa
zb
a
0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18-0.05
0
0.05
0.1
0.15
a1 (in.)
a hat
0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18-0.04
-0.02
0
0.02
0.04
a1 (in.)
resi
dual
s
Ahat-vs-a fit (MLE, Wald bounds) Physics-based model fit (Bayes/MCMC)
Physics-based Model Fit Provides
Better Match and
Residuals Are Reduced
),0(~ 2εσε N
0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.180
0.5
1
a1 (in.)
PO
D
POD: physics based model, Bayes/MCMC
0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18-0.05
0
0.05
0.1
0.15
a1 (in.)
a hat
0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18-0.05
0
0.05
0.1
0.15
a1 (in.)
a hat
0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18-0.04
-0.02
0
0.02
0.04
a1 (in.)
resi
dual
s
• Example: Eddy Current Inspection of Cracks at Fastener Sites • Case Study for Physics-based Model Evaluation:
• • where f () is a function call for a physics-based model (i.e. VIC-3D)
• Bayesian POD Analysis Performed in Matlab + R: • MCMC library in Matlab used for Bayesian Analysis • Matlab Provides Option for Integration of Model Function in Bayesian Fit
• Compare Ahat-vs-a fit (MLE, Wald bounds) and a Physics-based Model Fit
Bayesian Methods for POD / MAPOD Evaluation (2)
( ) εββ ++= 2110 ,ˆ aafa
zb
a
0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.180
0.5
1
a1 (in.)
PO
D
Ahat-vs-a fit (MLE, Wald bounds) Physics-based model fit (Bayes/MCMC)
Result is More Accurate Representation of the Data in
the POD Model Fit
),0(~ 2εσε N
0.8 0.85 0.9 0.95 1 1.05 1.1 1.15 1.2 1.250
2000
4000
6000
1/theta3 (aspect ratio: a/b)
0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18-0.05
0
0.05
0.1
0.15
a1 (in.)a ha
t
• Example: Eddy Current Inspection of Cracks at Fastener Sites • Case Study for Physics-based Model Evaluation:
• • where f () is a function call for a physics-based model • β0 , β1 = model calibration parameters • β2 = random variable associated with crack aspect ratio (b/a) • β3 = random variable associated with liftoff variation
• Results: Fit POD Model and Estimate of Variation in Aspect Ratio [use non-informative priors]
• Issues with ‘Naïve’ Approach: • Need true estimate of variance for
crack aspect ratio random variable → Use hierarchical models
• Address correlated / confounded parameters in estimation problem → Use informative priors and
constraints from expert opinion
Bayesian Methods for POD / MAPOD Evaluation (3)
( ) εββββ ++= 32110 ,;ˆ afa
zb
a),0(~ 2εσε N
0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.160
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
a1 (in.)
a hat
layer 1layer 2
0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18-0.05
0
0.05
0.1
0.15
a1 (in.)
a hat
0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18-0.1
-0.05
0
0.05
0.1
a1 (in.)
resi
dual
s
• Example: Eddy Current Inspection of Cracks at Fastener Sites • Challenge: Address non-constant variance wrt flaw size • Case Study for Physics-based Model Evaluation:
• • where f () is a function call for a physics-based model • β0 , β1 = model calibration parameters • β2 = random variable associated with crack aspect ratio (b/a)
Hierarchical Models for Estimating Variance of a Random Variable
( ) εβββ ++= 2110 ;ˆ afa
zb
a
),0(~ 2εσε N
experimental results simulated examples
β0 = 0.0 β1 = 1.0
µ_β2 = 0.75 σ_β2 = 0.12 σ_ε = 0.0
1st
2nd
0 0.05 0.1 0.15 0.2-0.1
0
0.1
0.2
0.3
a1 (in.)
a hat
• Example: Eddy Current Inspection of Cracks at Fastener Sites • Challenge: Address non-constant variance wrt flaw size • Hierarchical NDE Measurement Models:
•
• where f () is a function call for a physics-based model • β0 , β1 = model calibration parameters • η = random variable (varying-slope model) • σ2
η = variance in slope parameter • β2 = random variable associated
with crack aspect ratio (b/a) • σ2
η = variance in slope parameter
• Simple Test Case: Fit data from model with varying slope >> noise.
Hierarchical Models for Estimating Variance of a Random Variable
( ) aaa ˆ110ˆ εηββ +++=
zb
a
),0(~ 2ˆˆ aa N σε
1st 2nd
( ) aafa ˆ2110 ;ˆ εβββ ++=
ηεη = ),0(~ 2ηη σε N ),(~ 2
2 22 ββ σµβ N);,0(~ 2ˆˆ aa N σε
physics-based model statistical model
• A. Gelman, J. B. Carlin, H. S. Stern, and D. B. Rubin, Bayesian Data Analysis, 2003. • A. Gelman and J. Hill, Data Analysis Using Regression and Multilevel/Hierarchical Models, 2007.
• Simple Test Case: Fit data from model with varying slope >> noise. • Hierarchical NDE Measurement Models: • Results: Ns = 100
Hierarchical Models for Estimating Variance of a Random Variable
( ) aaa ˆ110ˆ εηββ +++= ),0(~ 2ˆˆ aa N σε
ηεη = ),0(~ 2ηη σε N
Ns = 100 β0 β1 ση σε True Value 0.0000 1.0000 0.3000 0.00100 WinBUGS
Mean -0.00024 1.0312 0.2580 0.00139 95% Credible Bds (-0.00119,0.00364) (0.9848,1.0810) (0.2286,0.2923) (0.00103,0.00210)
Matlab Mean -0.00027 1.0361 0.2768 0.01166
95% Credible Bds (-0.00418,0.00071) (0.9862,1.0863) (0.2593,0.2974) (0.01012,0.01346)
0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.20
0.05
0.1
0.15
0.2
0.25
a1 (in.)
a hat
0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2-0.1
0
0.1
0.2
0.3
a1 (in.)
a hat
• Simple Test Case: Fit data from model with varying slope >> noise. • Hierarchical NDE Measurement Models: • Results: Ns = 1000
Hierarchical Models for Estimating Variance of a Random Variable
( ) aaa ˆ110ˆ εηββ +++= ),0(~ 2ˆˆ aa N σε
ηεη = ),0(~ 2ηη σε N
Ns = 1000 β0 β1 ση σε True Value 0.0000 1.0000 0.3000 0.00100 WinBUGS
Mean -0.00011 1.0145 0.2966 0.00113 95% Credible Bds (-0.00052,0.00027) (0.9952,1.0330) (0.2853,0.3084) (0.00101,0.00139)
• Simple Test Case: Fit data from model with varying slope > noise. • Hierarchical NDE Measurement Models: • Results: Ns = 100
• Future Work: Perform Bayesian Evaluations with Physics-based Models Addressing Random Variable Parameter Estimation (in Matlab)
0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.20
0.05
0.1
0.15
0.2
a1 (in.)
a hat
Hierarchical Models for Estimating Variance of a Random Variable
( ) aaa ˆ110ˆ εηββ +++= ),0(~ 2ˆˆ aa N σε
ηεη = ),0(~ 2ηη σε N
Ns = 100 β0 β1 ση σε True Value 0.0000 1.0000 0.1000 0.00500 WinBUGS
Mean 0.00193 0.9716 0.1014 0.00545 95% Credible Bds (0.00007,0.00382) (0.9371,1.007) (0.0795,0.1248) (0.00432,0.00677)
Bayesian Methods - Challenges Uncertainty Quantification (UQ) community developing Bayesian framework
for the use of computational models with observational data • SAMSI program on UQ (2012): • SIAM UQ conference 2012: http://www.siam.org/meetings/uq12/
Key Insight / Research Directions: • 1) Must include model discrepancy and not treat it as random error.
• Calibrating (inverting, tuning) a wrong model gives parameter estimates that are wrong (not equal to their true physical values) [O’Hagan, 2012]
• Gaussian Process (GP) models typically used to fit model discrepancy [Kennedy/O’Hagan 2002].
• 2) Use of prior information in Bayesian framework can greatly help. • To learn about model parameters in the presence of discrepancy, better
prior information is needed [Bayarri, 2012] • Elicitation of expert opinion is an active research topic [O’Hagan, 2012]
• 3) Should leverage model form uncertainty (assessment) approaches. • To identify best models and address limitations cited by UQ community
[Grandhi et al, Wright State University]
24 July 19, 2012
Acknowledgements
• This work was partially supported by the U.S. Air Force Research Laboratory under UTC Prime Contract, FA8650-10-D-5210
• Charles Annis, Statistical Engineering
• David Forsyth, TRI/Austin
• Eric Lindgren, AFRL
Bayesian POD Evaluation Examples and Code Coming Soon www.computationaltools.com