Modern Applications of ExperimentalUncertainty Analysis
Glenn SteeleDepartment of Mechanical Engineering
Mississippi State University
and
Hugh ColemanPropulsion Research Center
Department of Mechanical and Aerospace Engineering
University of Alabama in Huntsville
www.uncertainty-analysis.com
Outline of Presentation
• History • Current Standards• Regression Uncertainty• Code Verification and Validation• Uncertainty in the Design Process
Methodology Used in Kline and McClintock
Total Uncertainty
)X,...,X,r(Xr J21=
2
XJ
2
X2
2
X1
2r J21
UXr...U
XrU
XrU
∂∂
++
∂∂
+
∂∂
=
Methodology Used in PTC 19.1-1985
95% Confidence Uncertainty
99% Confidence Uncertainty
[ ] 21
22RSS S)(tBU +=
StBUADD +=
THE ISO GUM
Thede factointernational standard
Methodology Used in GUM
Combined Standard Uncertainty
Expanded Uncertainty (At a Given % Confidence)
)X,....,X,r(Xr J21=
∑ ∑ ∑=
−
= += ∂∂
∂∂
+
∂∂
=J
1i
1J
1i
J
1ikki
kii
22
i
2c )X,u(X
Xr
Xr2)(Xu
Xr(r)u
(r)ukU c%% =
The GUM expresses uncertainty estimates, u(Xi), based on their source
• TYPE A evaluation (of uncertainty) – method of evaluation of uncertainty by the statistical analysis of series of observations.
• TYPE B evaluation (of uncertainty) – method of evaluation of uncertainty by means other than the statistical analysis of series of observations.
• Book Number D04598 Price $95.00
• ASME Customer Service Dept Box 2900 Fairfield NJ 07007-2900 [email protected]
AIAA S-071A-1999
www.aiaa.org
Methodology Used in Engineering Standards
For
then
and
)X,....,X,r(Xr J21=
2J
1iX
i
2r i
SXrS ∑
=
∂∂
=
∑ ∑ ∑=
−
= += ∂∂
∂∂
+
∂∂
=J
1i
1J
1i
J
1ikXX
ki
2
Xi
2r kii
BXr
Xr2B
XrB
and
or
where
21
2r
2r
95 S2B2U
+
=
[ ]21
2r
2r95 PBU +=
rr 2SP =
SYSTEMATIC ERROR (β) AND SYSTEMATIC UNCERTAINTY (B)
A useful approach to estimating the magnitude of a systematic error is to assume that the systematic error for a given case is a single realization drawn from some statistical parent distribution of possible systematic errors, as shown below:
If the parent distribution is Gaussian, then the systematic uncertainty B corresponds to the “2S” limit of a 95% confidence interval.
•In the 1985 version of the ASME Standard, the 1st edition of Coleman & Steele, and the current AIAA Standard this is called the “Bias Limit” (hence the symbol “B”).
•Since the mid-1990’s, it has been generally agreed to call the 95% confidence limit BX the “systematic uncertainty.” This is the usage in the 2nd edition of Coleman & Steele and in the 1998 version of the ASME Standard.
•Thus, a 95% estimate → Systematic Uncertainty, BX
a 68% estimate → Standard deviation of bias error distribution,
and so for a Gaussian bias error distributionXBS
/2BS XBX=
In keeping with the nomenclature of the ISO Guide, uc is called the combined standard uncertainty. For the data reduction expression
uc is given by
where
S +
S 2 + S = u
2i
2i
J
1 = i
Bki
J
1i+ = k
1 - J
1 = i
2B
2i
J
1 = i
2c
iki
θ
θθθ
∑
∑∑∑
( )J21 X,...,X,Xrr =
ii x
r∂∂
=θ
To obtain an uncertainty Ur (termed the expanded uncertainty in the ISO Guide) at some specified confidence level (95%, 99%, etc), the combined standard uncertainty uc must be multiplied by a coverage factor, k%,
u k = U c%%
The ISO Guide recommends that the appropriate value for k% is the t value for the specified confidence level at the degrees of freedom for the result, νr.
The effective number of degrees of freedom νr for determining the t-value is given (approximately) by the so-called Welch-Satterthwaite formula as
[ ]
[ ] [ ]( )νθνθ
θθν
iii
i
B4
BiS4
ii
J
1=i
2B
2i
2i
2i
J
1=i
2
r
/)S (+ /)S (
S+S = ∑
∑
where1 - N = iSiν
≈
SS
21
i
ii
B
B-2
B∆
ν
• If the large sample assumption is made so that t = 2, then the 95% confidence expression for Ur becomes
• Recalling the definition of systematic uncertainty, the (2SBi) factors are equal to the 95% confidence systematic uncertainties Bi.
• The (2Si) factors correspond to the 95% confidence random uncertainties Pi, when all Ni ≥10.
( ) ( )
)S (2 +
S2 2 + S 2 = U
2i
2i
J
1 = i
B2
ki
J
1 + i = k
1- J
1 = iB
22i
J
1 = i
2r
iki
θ
θθθ
∑
∑∑∑
The 95% confidence random uncertainty interval around a single reading of a variable X.
For the large sample case with νr ≥ 9 and all Ni ≥ 10, we can define the systematic uncertainty (bias limit) of the result as
and the random uncertainty (precision limit) of the result as
so that
Bθθ 2 + B θ = B ikki
J
1 + i = k
1- J
1 = i
2i
2i
J
1 = i
2r ∑∑∑
)P( = P 2i
2i
J
1 = i
2r θ∑
P + B = U 2r
2r
2r
EXAMPLE
)TT(mcQ 12p −=
2
T1
2
T2
2
cp
2
mTT21
2
T1
2
T2
2
cp
2
m2Q
12
p21
12p
PTQP
TQ
PcQP
mQB
TQ
TQ2
BTQB
TQB
cQB
mQU
∂∂
+
∂∂
+
∂∂
+
∂∂
+∂∂
∂∂
+
∂∂
+
∂∂
+
∂∂
+
∂∂
=
The GUM and Engineering Standards Uncertainty Analysis Methodologies are Identical
GUM
• Considers source of uncertainty – Type A or Type B uncertainties
PTC 19.1-1998, etc.
• Considers effect of uncertainty on variable Xi -Systematic or Random uncertainties
International Presentation of Uncertainty
• Show source and effect for uncertainties –B B,XA,XB,BA,BB,XA,XB,XA,X SSSSorPPB
XX
Uncertainties and
Regressions
Introduction
“When experimental information is represented using a regression, what is the uncertainty that should be associated with the use of that regression?”
(X1,Y1)
(X2,Y2)
(X3,Y3)
cmXY +=
Y
Y(Xnew)
Xnew X
Consider a 1st order least squares regression
cmXY new +=
( )2
11
2
1 11
−
−=
∑∑
∑ ∑∑
==
= ==
N
ii
N
ii
N
i
N
ii
N
iiii
XXN
YXYXNm
( )
( )2
11
2
1 111
2 )(
−
−=
∑∑
∑ ∑∑∑
==
= ===
N
ii
N
ii
N
i
N
iii
N
ii
N
iii
XXN
YXXYXc
Classical Random Uncertainty• The statistic that defines the standard deviation for a
straight-line curvefit is the standard error of regression defined as
• For a value of Xnew, the (large sample) random uncertainty associated with the Ynew obtained from the curvefit is
• where
( )2/1
2
121
−−
−= ∑
=
cmXYN
S ii
N
iY
21
22 )(12
−+=
XXYY S
XXN
SP
N
XS
N
ii
i
N
iXX X
2
12
1
−=∑
∑ =
=
• Key assumptions:– the random uncertainty in Y is constant over the range of the
curvefit– there is no random uncertainty in the X variable– there is no systematic uncertainty in either variable
• In many (if not most) instances, these assumptions do not hold and there is uncertainty in Ynew that is notaccounted for….
Consider the Range of Regression Uncertainty Cases of Engineering Interest
• Uncertainty of regression coefficients: Um, Uc
• Uncertainty of Y value from regression: UY(Xnew)• some or all (Xi,Yi) data pairs from different
experiments• all (Xi,Yi) data pairs from same experiment• Xnew from same apparatus• Xnew from different apparatus • Xnew with no uncertainty
• Regression variables as functions: (Xi ,Yi ) not measured directly
•
A Comprehensive Methodology that Covers All of the Cases:
• Brown, Coleman, and Steele, “A Methodology for Determining the Experimental Uncertainty in Regressions,” J. of Fluids Engineering, Vol. 120, No. 3, 1998, pp. 445-456.
• Presented in detail and with examples in Chapter 7 of Coleman and Steele.
• Methodology: Treat regression expression as a data reduction equation
• and apply the uncertainty propagation equations
),,...,,...,,( 11 newNN XYYXXYY =
B rX
B rX
rX
Brii
J
iik i
J
i
J
kik
2
1
22
11
1
2=
+
= = +=
−
∑ ∑∑∂∂
∂∂
∂∂
P rX
P rX
rX
Pri
ii
J
ik i
J
i
J
kik
22
2
1 11
12=
+
= = +=
−
∑ ∑∑∂∂
∂∂
∂∂
( )
( )
( )2
11
2
1 111
2
2
11
2
1 11)(
)(
−
−+
−
−=
∑∑
∑ ∑∑∑
∑∑
∑ ∑∑
==
= ===
==
= ==
N
ii
N
ii
N
i
N
iii
N
ii
N
iii
newN
ii
N
ii
N
i
N
ii
N
iiii
new
XXN
YXXYXX
XXN
YXYXNXY
Monte Carlo Simulations Performed• 1st order regression coefficients
– studied effect of sample size• 1st order regression Y uncertainty• Polynomial regression Y uncertainty• Functions as Regression Variables
– 1st Order and Polynomial• Type of dominant uncertainty• Percent of reading type uncertainties• Percent of full scale type uncertainties
All Simulations Indicated This Methodology Provides Appropriate Uncertainty Intervals
(X1,Y1)
(X2,Y2)
(X3,Y3)Y
Y(Xnew)
Xnew X
Y X m X X X Y Y Y X c X X X Y Y Ynew new( ) ( , , , , , ) ( , , , , , )= +1 2 3 1 2 3 1 2 3 1 2 3
∑∑
∑∑∑ ∑∑
∑ ∑ ∑
∑∑
==
= =
−
= +==
=
−
= +=
==
+
+
+
+
+
+
+
+
+
+
=
3
1
3
1
22
22
3
1
3
1
13
1
3
1
3
1
22
3
1
13
1
3
1
22
3
1
223
1
22
2
22
22
2
iY
inewiX
inewnewnew
i kYX
kii ikXX
kiiX
i
i i ikYY
kiY
i
iX
iiY
iY
inewinewnewnew
kikii
kii
ii
BYYYB
XYYPYBY
BYY
XYB
XY
XYB
XY
BYY
YYB
YY
PXYP
YYU
XXXX XXXX ∂∂
∂∂
∂∂
∂∂
∂∂
∂∂
∂∂
∂∂
∂∂
∂∂
∂∂
∂∂
∂∂
∂∂
∂∂
∂∂
Y X m X X X Y Y Y X c X X X Y Y Ynew new( ) ( , , , , , ) ( , , , , , )= +1 2 3 1 2 3 1 2 3 1 2 3
Note: (1) that the first summation on the RHS produces an
identical PY estimate as the classical method, and (2) that the derivatives with respect to Xi and Yi are
functions of Xnew
38
Reporting Uncertainty UY of Y Value from the Regression
X
Y
Y(X)=mX+c
Y(X)+UY(X)
Y(X)-UY(X)
UY(X)
X
UY
• Since UY is a function of Xnew , would have to carry along entire data set to calculate a value for UY (Xnew) each time we wanted one!!!
• SOLUTION: Report the uncertainty as an equation
UY- regress = f(X)
developed by curvefitting a set of (X, UY) points generated from the uncertainty propagation expression, then combine that with the uncertainties associated with Xnew to obtain overall uncertainty in the Y from the regression:
• Some subtleties associated with this are discussed in detail in Chapter 7.
( ) [ ]2X
2X
2
new
2regressY
2Y newnew
PBX
YUU +
∂
∂+= −
Detailed Example:Using a venturi in an experiment to determine a
flow rate.
• Perform a calibration and curvefit the data:
• Substitute into the 1st equation and solve for Q to obtain the equation used in the test:
4new2
newd
Dd1
∆PdK)(ReCQ
−
=
)Re,...Y(Xa),...Y(Xa(Re)C N11N10d +=
• That equation for Q is
• and the expressions for the uncertainty in the value of Q obtained from the equation are
4new
new
1
4new2
0
Dd1
∆Pπυ
Kd4a1
Dd1
∆PdKa
Q
−
−
−
=
0.1162Re102Re101Re105U new52
new103
new16
regressQ −×+×−×= −−−−
2∆P
2
new
2regressQ
2Q new
P∆P
QUU
∂
∂+= −
0.30
0.35
0.40
0.45
0.50
0.55
0.60
0.65
0.70
0.75
60,000 70,000 80,000 90,000 100,000 110,000 120,000 130,000 140,000 150,000 160,000
Reynolds Number
UQ
-Reg
ress
(gpm
)
0.1162Re102Re101Re105U new52
new103
new16
regressQ −×+×−×= −−−−
2∆P
2
new
2regressQ
2Q new
P∆P
QUU
∂
∂+= −
Uncertainty Analysis
and the
Verification and Validation (V&V) of Simulations
Brief Overview
• Coleman, H.W. and Stern, F., 1997, "Uncertainties in CFD Code Validation," ASME J. Fluids Eng., Vol. 119, pp. 795-803. (Also see “Authors’ Closure,” ASME J. Fluids Eng., Vol. 120, September 1998, pp. 635-636.)
• Fred Stern – Tutorial III – Thurs May 31– 4-6 pm “Code Verification and Validation”
Suppose we have a simulation result. How good is it? The V&V process helps answer that question.
Consider the comparison between a simulation result and experimental data….
The uncertainties determine
(a) the scale at which meaningful comparisons can be made
(b) the lowest level at which validation is possible; i.e., the “noise level”
Thus, these uncertainties must be considered as part of the V&V process.
r(X) + U (X)r
r(X) - U (X)r
Predicted r(X)r
X
The V&V Process• Preparation: Specification of objectives, validation
variables, validation set points, validation levels required, etc.
• Verification: → Are the equations solved correctly?
→ grid convergence studies, etc
• Validation: → Are the correct equations being solved?
→ comparison with benchmark experimental data
• Documentation
Consider a Validation Comparison:
Experimental result, D
Comparison error, E
Simulation result, S
E = D - S = δD - δS
δD → error in data
δS → error in simulation
• The simulation error δS is composed of– errors δSN due to the numerical solution of the equations– errors δSPD due to the use of previous data (properties, etc.)– errors δSMA due to modeling assumptions
δS = δSN + δSPD + δSMA
• Therefore, the comparison error E can be written as
E = D - S = δD - δS
orE = δD - δSN - δSPD - δSMA
• Consider the error equation
E = δD - δSN - δSPD - δSMA
• When we don’t know the value of an error δi, we estimate an uncertainty interval ±Ui that bounds δi
• The uncertainty interval ±UE which bounds the comparison error E is given by (assuming no correlations among the errors)
or222
22
22
SDSDE UUUSEU
DEU +=
∂∂
+
∂∂
=
22222SMASPDSNDE UUUUU +++=
E = δD - δSN - δSPD - δSMA
• The interval ±UE bounds E with 95% confidence; however, in reality we know of no a priori approach for estimating USMA, which precludes making an estimate of UE . In fact, one objective of a validation effort is to identify and estimate δSMA or USMA.
• So, we define a validation uncertainty UV given by
The interval ±UV would contain E 95 times out of 100 if δSMA were zero. The verification process provides an estimate for USN .
UUUUUU SPDSNDSMAEV
222222 ++=−=
22222SMASPDSNDE UUUUU ++= +
• The validation uncertainty UV is the key metric in the validation process. It is the “noise level” imposed by the uncertainties UD , USN , and USPD ; thus, it is the lowest level at which validation can be achieved.
• It can be argued that one cannot discriminate when |E| < UV ; that is, one cannot evaluate the effectiveness of proposed model “improvements” since changes in δSMA cannot be distinguished. If |E| <UV, then validation has been achieved at the UV level, which is the best that can be done considering the existing uncertainties.
• On the other hand, if |E| » UV , then one could argue that probably δSMA ≈ E.
UUUUUU SPDSNDSMAEV
222222 ++=−=
• Another important metric is the required level of validation, Ureqd, which is set by program objectives.
• Thus, the three important quantities in evaluating the results of a validation effort are E, UV , and Ureqd . In comparing E and UV andUreqd there are six combinations:
1. E < VU < reqdU
2. E < reqdU < VU
3. reqdU < E < VU
4. VU < E < reqdU
5. VU < reqdU < E
6. reqdU < VU < E
1. E < VU < reqdU
2. E < reqdU < VU
3. reqdU < E < VU
→ In cases 1, 2 and 3, |E| < UV ; validation is achieved at the UV
level; and E is below the noise level, so attempting to decrease the error due to the modeling assumptions, δSMA , is not feasible from an uncertainty standpoint.
→ In case 1, validation has been achieved at a level below Ureqd, so validation is successful from a programmatic standpoint.
4. VU < E < reqdU
5. VU < reqdU < E
6. reqdU < VU < E
→ In cases 4, 5 and 6, |E| > UV , so E is above the noise level and using the sign and magnitude of E to estimate δSMA is feasible from an uncertainty standpoint. If |E| >>
UV , then E corresponds to δSMA and the error from the modeling assumptions is determined unambiguously.
→ In case 4, |E| < Ureqd , so validation is successful at the |E| level from a programmatic standpoint.
Marine Propulsor Thrust Coefficient Validation
CFDSHIP-IOWA code (Stern et al. (1996)); marine-propulsorflow data (Chen, 1996; Jessup, 1994)
D S E % UV % UD % USN % USN / UD
K t 0.146 0.149 -2.1 3.2 2.0 2.5 1.3
Ship Wave Profile ValidationTraditional Comparison
Ship Wave Profile ValidationColeman-Stern Comparison
UNCERTAINTY IN THE DESIGN PROCESS
• With limited resources and time, where are these resources and time best spent to produce the optimum design?
• Given the design process, available resources, and available time, will the design meet program goals?
Sample design process:
1. 1-D Meanline Code (Step 1)2. 2-D/3-D Steady Codes (Step 2)3. Baseline Design (Step 3)4. 3-D Steady/Unsteady Codes (Step 4)5. Design II (Step 5)6. Cold-flow Testing/Code Validation (Step 6)7. Design III (Step 7)8. Prototype Manufacture (Step 8)9. Hot-fire Testing (Step 9)10.Final Design (Step 10 or n-2)11.Final Product Manufacture (Step 11 or n-1)12.Flight Test/Design Validation/Certification (Step 12 or n)
What is the Issue?How good does the design have to be?• For a given design process, determine the overall
uncertainty in the design.How do the individual steps in the design process
interact to produce the final design?• Determine how the uncertainty estimates for each step in
the design process propagate through the design process to produce the overall uncertainty in the design.
What are the critical steps in the design process?• Use uncertainty techniques to identify the critical steps
and improve these steps to insure that the design meets the program goals.
EXAMPLEUncertainty in Design Modeling
Pump head required is determined from the model
where
∑=
+
π+∆+
ρ∆
=pipesN
1i4i
ii
ii
22p D
KDLf
Q8ZgPw
2
i
i9.0
i10
i
D7.3Re74.5log
25.0f
∈+
=
Example – Oil Transfer System
Uncertainty Percentage Contributions
0102030405060
visco
sity
f mod
el K1 D2 L2Delt
a Zroug
hnes
s L1 K2 D1 D3 L3 K3de
nsity
Variables
Perc
enta
ge
