Verification and Validation of CFD Simulations
F. Stern1, R. Wilson1, H. Coleman2, and E. Paterson1
1. Iowa Institute of Hydraulic Research, University of Iowa; 2. Propulsion Research Center, University of Alabama in Huntsville
Tutorial2001 Fluids Engineering Division Summer Meeting
May 29 - June 1, 2001, New Orleans, Louisiana
Sheraton New Orleans Hotel
OutlineBackgroundApproachOverall Verification and Validation MethodologyVerification ProceduresValidation ProceduresExample for RANS CFD CodeConclusions
2
BackgroundDiscussion and methodology for estimating errors and uncertainties in CFD simulations has reached a certain level of maturity– Editorial policies and early recognition of importance and need
for “quality control” (ASME JFE, 1993, Vol. 115, pp. 339-340)
– Increased attention and recent progress on common terminology through published guidelines (AIAA, 1998)
– Advocacy and detailed methodology in recent textbook (Roache, 1998)
– Numerous case studies, including a special AIAA J. issue (e.g.,Mehta, 1998)
3
BackgroundProgress accelerated in response to the urgent need for achieving consensus on concepts and definitions and useful methodology and procedures– CFD is applied to increasingly complex geometry and physics
and integrated into the engineering design process
– Realization simulation-based design
– Other uses of CFD such as simulating flows for which experiments are difficult
In spite of progress and urgency, various viewpoints have not converged and current approaches fall short of providing practical methodology and procedures
4
ApproachComprehensive, pragmatic approach to verification and validation (V&V) methodology and procedures for estimating errors and uncertainties for industrial CFDAlready developed CFD code without requiring source code Specified objectives, geometry, conditions, and available benchmark experimental dataDeveloped for RANS CFD codes, but should also be applicable to boundary-element methods and some aspects LES and DNS
5
Approach
Definitions of errors and uncertainties consistent with experimental uncertainty analysisConcepts, definitions, and equations derived for simulation errors and uncertainties provide overall mathematical frameworkNumerical error treated both as deterministic or stochasticGeneralized Richardson extrapolation for J input parameters and correction factorsUse of quantitative estimates for errors and uncertainties constitute new V&V approach
6
Approach
Previous work on verification (Stern et al., 1996) extended and and combined with subsequent work on validation (Coleman and Stern, 1997)– Stern, F., Wilson, R.V., Coleman, H., and Paterson, E.,
“Verification and Validation of CFD Simulations,” Iowa Institute of Hydraulic Research, The University of Iowa, IIHR Report No. 407, September 1999 (in review ASME JFE)
Nearly two years experience through International Towing Tank Conference (ITTC) community and Gothenburg 2000 Workshop on CFD in Ship Hydrodynamics (Larsson et al., 2000)
7
Overall V&V Methodology Concepts and Definitions
Error, δ - difference between a simulation value (or an experimental value) and the truth
Error estimate, δ* - estimate of both sign and magnitude of error δUncertainty estimate, U - estimate of magnitude(but not sign) of error such that interval ±U contains true value 95 times out of 100
8
Overall V&V Methodology Concepts and Definitions
Simulation error, δS - difference between simulation S and truth T postulated as comprised of addition of modeling and numerical errors
The uncertainty equation corresponding to the error equation is
SNSMS TS δδδ +=−=
222SNSMS UUU +=
9
Overall V&V Methodology Concepts and Definitions
For certain conditions, the numerical error δSN can be considered as
– where δ*SN is an estimate of sign and magnitude of δSN and εSN is the error in that estimate
The corrected simulation value SC (numerical benchmark) is
With error and uncertainty equations
SNSNSN εδδ += *
SNSMSNC TSS εδδ ++=−= *
SNSMCS TSC
εδδ +=−= 222NSSMS CC
UUU +=
10
Overall V&V Methodology Concepts and Definitions
Verification - assessment of numerical uncertainty USN and when conditions permit, estimating the sign and magnitude of the numerical error δ*SNitself and the uncertainty in that estimate
Validation - assessment of modeling uncertainty USM by using benchmark data and, when conditions permit, estimating the sign and magnitude of the modeling error δSM itself
NSCU
11
Overall V&V Methodology Verification
Numerical errors decomposed into contributions from iteration number grid size, time step, and other parameters
Similarly,∑+==
∗∗∗ J
jjISN 1
δδδ
SNSM
J
jjIC TSS εδδδ ++=∑+−=
=
∗∗ )(1
∑+==
J
jjINS CCC
UUU1
222
∑=
+=+++=J
jjIPTGISN
1δδδδδδδ
∑=
+=+++=J
jjIPTGISN UUUUUUU
1
2222222
12
Overall V&V Methodology Validation
The comparison error E is defined by
– with δSM decomposed into errors from modeling assumptions δSMA and use of previous data δSPD
The uncertainty UE in the comparison error is
Standard methodology and procedures available for estimating UD (Coleman and Steele, 1999)
( )SNSPDSMADSDSDE δδδδδδ ++−=−=−=
22222SNSPDSMADE UUUUU +++=
13
2 2 2STE SPD SNU U U= +
Overall V&V Methodology Validation
Ideally, like to postulate if |E|<UE, validation achieved; however, no known method for estimating USMA
More stringent alternative is, |E|<UV, where UVis validation uncertainty
– USTE is the total estimated simulation uncertainty
If |E|<UV then validation achieved at UV levelIf |E|>>UV then
2 2 2 2 2 2 2 2V E SMA D SPD SN D STEU U U U U U U U= − = + + = +
SMAE δ≈
14
Overall V&V Methodology Validation
Definition of comparison error
S + U
E
UD
Ux
r
X
D
S
STE
15
Overall V&V Methodology Validation
Corrected comparison error
Corrected validation uncertainty
– SC and EC can either be larger or smaller than S and E
– and should be smaller than UE and UV
( )SNSPDSMADCC SDE εδδδ ++−=−=
222222NSSPDDSMAEV CCC
UUUUUU ++=−=
CEUCVU
16
Verification ProceduresConvergence Studies
Verification procedures based on
Parameter convergence studies– use multiple (m) solutions by varying the kth input
parameter ∆xk while holding all other parameters constant– input parameters (step sizes) defined such that for
finest resolution– uniform parameter refinement
• Not required
Solutions corrected for iterative errors∑
≠=
∗∗∗ ++=−=J
kjjjkCIkk mmmkmm
SSS,1
ˆ δδδ
)(1
∑=
∗∗ ++=J
jjICSS δδ
2312// kkkkk xxxxr ∆∆=∆∆=
0→∆ kx
17
Verification ProceduresConvergence Studies
Convergence studies require a minimum of m=3solutions corresponding to fine, medium, and coarse values for the kth input parameter
Solution changes ε for medium-fine and coarse-medium solutions and their ratio Rk are defined by
( )321
ˆ,ˆ,ˆkkk SSS
21 2 1ˆ ˆ
k k kS Sε = −
32 3 2ˆ ˆ
k k kS Sε = −
21 32k k kR ε ε=
18
Verification ProceduresConvergence Studies
Converging condition: 0 < Rk < 1, monotonic convergence and generalized RE is used to estimate Uk or and
Oscillatory condition: Rk < 0, solutions exhibit oscillations, which may be erroneously identified as convergent or divergent
– uncertainties estimated using upper (SU) and lower (SL) bounds of solution oscillation and require more than 3 solutions
Diverging condition: Rk > 1, solutions exhibit divergence and errors and uncertainties can not be estimated
CkU
)(21
LUk SSU −=
kδ ∗
19
Verification ProceduresMonotonic Convergence: Generalized RE
Generalized Richardson Extrapolation (RE) for convergent condition– Modified and numerical error equations show error terms
are in the form of a power series expansion in input parameter
– Subtraction of multiple solutions eliminates terms and provides equations for SC, p(i)
k, and g(i)k (assuming p(i)
k, and g(i)k
independent ∆xk)
– Since each term contains 2 unknowns, m=2n+1 solutions required (i.e., for n=1, m=3 and for n=2, m=5, etc.)
∑∑≠=
∗
=
+∆+=J
kjjj
n
i
ik
pkCk m
ik
mmgxSS
,11
)()(
)(ˆ δ
∗mkδ
∗mj
δ
20
Verification ProceduresMonotonic Convergence: Generalized RE
For m=3 solutions, only leading term estimated
– for sufficiently small ∆xk solutions in the asymptotic range such that higher-order terms negligible
– achieving asymptotic range for practical geometries and conditions not possible and m>3 undesirable from resource point of view
21
1 1 1k k
kk RE p
krε
δ δ∗ ∗= =−
( )( )
32 21ln /
lnk k
kk
pr
ε ε=
21
Verification ProceduresEstimating Errors and Uncertainties using Generalized RE
with Correction FactorsResults for analytical benchmarks (1D wave and 2D Laplace equations) show that when solutions not in asymptotic range correct form but pkpoorly estimatedAnalysis results suggests concept of correction factors Ck– provide quantitative metric to determine proximity of
solutions to asymptotic range– account for effects of higher-order terms– use for defining and estimating errors and uncertainties
REδ ∗
22
Verification ProceduresEstimating Errors and Uncertainties using Generalized RE
with Correction FactorsMultiplication by Ck provides estimate accounting for effects higher order terms
Ck based on leading and first two terms
– where pkest, qkest are improved estimates of orders of accuracy (e.g., modified equation or simplified geometry with similar grid expansion)
– Ck<1 or Ck>1 indicates that the leading-order term over predicts or under predicts the error, respectively
11)1(
−−
=estk
k
pk
pk
k rrC
( ) )1)(()1)(/(
)1()1)(/( 12231223)2(
−−
−−+
−−
−−=
estkestkestk
kestk
kk
estkestkestk
kestk
kk
qk
qk
pk
pk
pk
pk
qk
pk
pk
qk
k rrrrr
rrrrr
Cεεεε
21
1 1( )
1)k k
kk k RE k p
k
C Cr
εδ δ∗ ∗= =
−
∗1kδREδ ∗
23
Verification ProceduresEstimating Errors and Uncertainties using Generalized RE
with Correction FactorsFor Ck<1 or Ck>1 and lacking confidence, Uk is estimated, but not and Ukc
For Ck≈1 and having confidence, and Ukc are estimated
– In the limit of the asymptotic range, Ck=1, = , and Ukc=0
∗∗ −+=11
)1(kk REkREkk CCU δδ
21 21
1 1 1 1k k th
k kk k RE k p p
k k
C Cr r
ε εδ δ∗ ∗
= = = − −
1(1 )
C kk k REU C δ ∗= −
kδ ∗
kδ ∗
1kδ ∗1kREδ ∗
24
Verification ProceduresEstimating Errors and Uncertainties using Generalized RE
with Factors of SafetyRoache (1998) proposes factor of safety approach
Can be extended for estimate of corrected simulation numerical uncertainty
In this approach, fixed percentage of three-gird error estimate used to define uncertainty: FS=1.25 for careful grid study otherwise =3– Results for analytical benchmark show FS overly
conservative compared to Ck approach
1kk S REU F δ ∗=
( )1
1C kk S REU F δ ∗= −
25
Verification ProceduresFundamental and Practical Issues
Fundamental Issues– Convergence power series expansion– Assumption p(i)
k, and g(i)k independent ∆xk
– Estimating based on theoretical values or solutions for simplified geometry and conditions with stretched grids
– Ck vs. FS or other approachesPractical Issues– For complex flows with relatively coarse girds, solutions far from
asymptotic range such that while some variables convergent other variables may be oscillatory or even divergent
– pk shows variability between different variables same grid study and same variables different grid studies
– More than 3 grids required– Selection parameter refinement ratio– Nature asymptotic range for practical applications unknown– Interpretation results important since limited experience for
guidance
estkp
26
Validation ProceduresSix combinations of |E|, UV, and Ureqd (program validation requirement)1. |E| < UV < Ureqd2. |E| < Ureqd < UV3. Ureqd < |E| < UV4. UV < |E| < Ureqd5. UV < Ureqd < |E| 6. Ureqd < UV < |E| In cases 1, 2, and 3, validation achieved at UV levelIn cases 4, 5, and 6, validation not achieved at UV. If E>>UVthen
In cases 1 and 4, validation successful programmatically Similar conclusions for corrected simulation results
SMAE δ≈
27
Example for RANS CFD Code
CFDSHIP-IOWASeries 60 cargo/container ship– ITTC benchmark data (Toda et al., 1992)– Conditions
• Froude number Fr = 0.316• Reynolds number Re = 4.3x106
– V&V for resistance CT (integral variable) and wave profile ζ (point variable)
28
Example for RANS CFD CodeGrid Studies
Grid refinement ratio m=4 grids with systematic grid refinement in each coordinate direction enables two separate grid studies: grids 1-3 (GS1) and grids 2-4 (GS2)
2Gr =
Table 1. Grid dimensions and y+ values for grid refinement studies.
Grid Grid
Dimensions
Total Number
of points y+
1 287x78x43 876,211 0.7
2 201x51x31 317,781 1
3 144x36x22 114,048 1.4
4 101x26x16 42,016 2
29
Example for RANS CFD CodeGrid Studies
Grids and wave contours
30
Example for RANS CFD CodeVerification for Resistance
Verification performed for iterative and grid convergence
Limiting order of accuracy estimated as formal order of accuracyof the CFD code
Iterative convergence negligible (i.e., at least one order of magnitude smaller grid convergence)
Grid 1 UI=0.07%S1
Grid 2 UI=0.02%S1
Grid 3 UI=0.03%S1
Grid 4 UI=0.01%S1
S1=solution on finest grid
2 2 2SN I GU U U= +
2.0est thk kp p= =
31
Example for RANS CFD CodeVerification for Resistance
Iteration history: solution change; forces; and magnified CT last two periods: fine grid; 4 order of magnitude drop in residuals; UI=.07%S1
Iteration
Res
idua
l
0 5000 10000 15000 2000010-7
10-6
10-5
10-4
UVWP
(a)
Iteration12000 14000 16000 18000
0
0.002
0.004
0.006
0.008
(b)
CFCPCT
Iteration12000 14000 16000 18000
0.00504
0.00505
0.00506
SL=5.046x10-3
SU=5.053x10-3
(c)
32
Example for RANS CFD CodeVerification for Resistance
Monotonic convergence; variability pG and counter expectation
Table 2. Grid convergence study for total CT, pressure CP, and frictional CF resistance (x10-3) for Series 60.
Grid S4 (grid 4) S3 (grid 3) S2 (grid 2) S1 (grid 1) Data
CT
ε 6.02 5.39
-10% 5.11
-5.2% 5.05
-1.2% 5.42
CP
ε 1.88 1.61
-14%
1.60 -0.6%
1.60 0.0%
CR = 2.00
CF
ε 4.14 3.69
-11% 3.51
-4.9% 3.45
-1.7% 3.42 ITTC
% SG.
Table 3. Verification of CT (x10-3) for Series 60. Study RG pG CG
1 (grids 1-3)
0.21 4.4 3.7
2 (grids 2-4)
0.44 2.3 1.3
% SG
33
Example for RANS CFD CodeVerification for Resistance
CF=70%CT and within 1%CF(ITTC)For GS1, CP grid independent and CF convergent with pG>pth
For GS2, both convergent with pG>pth
Fact that CP and CF converge with different pG and depend different physics helps explain variability
Table 4 Verification of CP and CF (x10-3) for Series 60. CP CF Study
RG pG CG RG pG CG
1 (grids 1-3)
0.00 - - 0.33 3.2 2.0
2 (grids 2-4)
0.04 9.5 26 0.40 2.6 1.5
% S1
34
Example for RANS CFD CodeVerification for Resistance
Error and uncertainty values reasonable in consideration number grid pointsFS approach less conservative, which is opposite results for analytical benchmark
Table 5. Errors and uncertainties for CT (x10-3) for Series 60. CT uncorrected CT corrected Grid
GU (CG) GU (FS) *Gδ
CGU (CG) CGU (FS) SC
1 2.1% 0.5% 1.2% 0.9% 0.1% 4.99 2 6.7% 5.6% 5.5% 1.1% 1.1% 4.83
% S1
35
Example for RANS CFD CodeVerification for Resistance
Next finer grid requires 2.4M grid pointsExpect but with UI similar order magnitudeFrom resource point of view, accept present Scand for finest gridBased on overall verification: four solutions display monotonic convergence with notwithstanding variability pG which precludes complete confidenceAdditional solutions desirable for knowledge asymptotic range for practical applications
2kr =
CG GU U→
cGU
0Gδ ∗ >
36
Example for RANS CFD CodeValidation for Resistance
Uncorrected solution– For GS1, and UD>USN
– For GS2, so CT validated at 7%D and USN >UD
Corrected Solution– For GS1 and GS2, and
Table 6. Validation of uncorrected total resistance for Series 60. Grid E% UV% UD% USN%
1 6.8 3.1 2.5 1.9
2 5.7 6.7 2.5 6.3
%D.
Table 7. Validation of corrected total resistance for Series 60. Grid CE %
CVU % DU % CS NU %
1 7.9 2.6 2.5 0.8 2 11 2.7 2.5 1.0
%D.
7%V SMAE U E Dδ> ⇒ ≈ =
VE U<
8%cc V c SMAE U E Dδ>> ⇒ = =
cS N DU U<<
37
Example for RANS CFD CodeV&V for Wave Profile
Wave height at free-surface hull intersectionConvergence ratio and order of accuracy defined using L2 norms of and
As with CT, UI<<UG such that USN=UG
21kε32kε
21 322 2/G G GR ε ε=
( )32 21 22ln /
ln( )G G
GG
pr
ε ε=
38
Example for RANS CFD CodeVerification for Profile Average Wave Profile
For both GS1 and GS2, monotonic convergenceUncertainties GS1=1/2GS2Trends pG consistent expectationUncertainty values reasonable in consideration number of grid points
Table 8. Profile-averaged verification results for wave profile for Series 60.
Study RG pG CG GU CGU
1 (grids 1-3)
0.64 1.3 0.56 2.0% 0.9%
2 (grids 2-4)
0.68 1.1 0.47 4.1% 2.2%
%ζmax .
39
Example for RANS CFD CodeValidation for Profile Average Wave Profile
Uncorrected solution– For GS1, not validated at E=5.2%ζmax but margin small– For GS2, nearly validated at 5.6%ζmax
Corrected Solution– Not validated but margins relatively small and cS N DU U<<
Table 9. Profile-averaged validation results for uncorrected wave profile for Series 60. Grid E% UV% UD% USN%
1 5.2 4.2 3.7 2.0
2 5.6 5.5 3.7 4.1
%ζmax .
Table 10. Profile-averaged validation results for corrected wave profile for Series 60. Grid CE %
CVU % DU % CS NU %
1 5.6 3.8 3.7 0.9 2 6.6 4.3 3.7 2.2
%ζmax .
40
Example for RANS CFD CodeV&V for Wave Profile
Regions not validated indicate under prediction crests and troughs
x/L
ζ/L
0 0.25 0.5 0.75 1
-0.01
0
0.01 Grid 1 (287x71x43)Grid 2 (201x51x31)Grid 3 (144x36x22)Grid 4 (101x26x16)Toda et al. (1992)
x/L
ζ/L
0 0.25 0.5 0.75 1
-0.01
0
0.01
(a)x/L
E
0 0.25 0.5 0.75 1
-0.2
-0.1
0
0.1
0.2 E=D-S+UV
-UV
(b)
x/L
E C
0 0.25 0.5 0.75 1
-0.2
-0.1
0
0.1
0.2 EC=D-SC
+UV
-UV
(c)
x/L
E
0 0.25 0.5 0.75 1
-0.2
-0.1
0
0.1
0.2 E=D-S+UV
-UV
(d)
x/LE C
0 0.25 0.5 0.75 1
-0.2
-0.1
0
0.1
0.2 EC=D-SC
+UV
-UV
(e)
41
Example for RANS CFD CodeOverall V&V Conclusions for CT and Wave Profile
CT and ζ not validated due to 8%D and 6%ζmaxmodeling errorsImprove modeling assumptions for dynamic sinkage and trim, free surface boundary conditions, and turbulence for validation at 3%D and 4%ζmaxlevelsReduction level validation UV requires reduction UD
42
Conclusions
V&V methodology and procedures successful in assessing levels of verification and validation or modeling errors. For practical applications many issues– Solutions far from the asymptotic range
– Analysis and interpretation results important in assessing variability for order of accuracy, levels of verification, and strategies for reducing numerical and modeling errors and uncertainties
43
Conclusions
Future work on verification should focus on both fundamental and practical issues, as previously discussedV&V methodology and procedures should facilitate– Documented V&V studies for transition CFD codes to
design– Sufficient number of documented solutions should enable
accreditation of CFD code for a certain range of applications
44
Analytical Benchmarks
1D Wave equation
Exact analytical solution
( ) ( )
0),(:
exp0,:
0)()()(
2
0
=−∞
−=
=∂∂
+∂∂
===
tABC
BxAxAIC
xAc
tAALMLTL AMT
( )2
0( , ) expx ct
A x t AB
−= −
x
S(x,
t)
-1 0 1 20
0.5
1
1.5
c=1
t=0t=1
45
Analytical Benchmarks
Simulation error and uncertainty
Corrected Simulation error and uncertainty
Verification
S SNS Aδ δ= − = 2 2S SNU U=
2 2C CS S NU U=
CS C SNS Aδ ε= − =
CC C S NE A S U= − <SNE A S U= − <
46
Analytical Benchmarks
Verification results 1st order solution 1D wave equation
∆x=∆t/2 10-510-410-310-20.4
0.5
0.6
0.7
0.8
0.9
1
1.1
1.2(b)
(A-S)/δ*RE
C(1)SN
C(2)SN
∆x=∆t/2
Erro
r
Ord
er,p
10-510-410-310-2
0
0.05
0.1
0.15
0.2
1
2(a) A-S
δ*RE
C(1)SNδ*
REC(2)
SNδ*RE
p
∆x=∆t/2
Erro
r
10-510-410-310-2
10-3
10-2
10-1
100
|A-S|
U(1)SN=|CKδRE|+|(1-CK)δRE|
U(2)SN=|CKδRE|+|(1-CK)δRE|
USN=FSδRE, FS=1.25
(d)
∆x=∆t/2
Erro
r
10-510-410-310-210-5
10-4
10-3
10-2
10-1 |A-SC|
U(1)SCN=|(1-CK)δRE|
U(2)SCN=|(1-CK)δRE|
USCN=(FS-1)|δRE|, FS=1.25
(c)