AIAA-2001-0921 THE CONNECTION BETWEEN THE COMPLEX-STEP DERIVATIVE APPROXIMATION AND ALGORITHMIC DIFFERENTIATION Joaquim R. R. A. Martins, * Peter Sturdza, † and Juan J. Alonso ‡ Department of Aeronautics and Astronautics Stanford University, Stanford, CA Abstract This paper presents improvements to the complex- step derivative approximation method which in- crease its accuracy and robustness. These improve- ments unveil the connection to algorithmic differen- tiation theory. The choice between these two meth- ods then hinges on a trade-off between ease of im- plementation and execution efficiency. Automatic implementations for Fortran and C/C++ are pre- sented and their relative merits are discussed. These new methods were successfully implemented in two very large multidisciplinary programs and the re- sulting sensitivities are shown to be as accurate as the analyses. Accuracy and ease of implementation make these tools very attractive options for sensitiv- ity analysis. Introduction Sensitivity analysis is an important field of engineer- ing research. One of its main applications is for de- sign optimization methods that use gradient-based optimizers. The calculation of sensitivity informa- tion is usually the costliest step in the optimization cycle and the optimization process often fails due to inaccurate sensitivity information. The complex-step derivative approximation is a method for estimating sensitivities that has been shown to have a relatively easy implementation while yielding very accurate results. 1 The poten- tial of this technique has been recognized and it has been used in CFD sensitivity analysis by Anderson 2 and in an MDO environment by Newman. 3 The de- tails and automatic implementation of this method * Graduate Student, AIAA Student Member † Graduate Student, Not an AIAA Member ‡ Assistant Professor, AIAA Member Copyright c 2001 by the authors. Published by the American Institute of Aeronautics and Astronautics, Inc., with permis- sion. have been described by the authors. 1, 4 In this paper we present further research on the complex-step derivative approximation, starting with some of the problems that we have found in the original implementation. In the process of solv- ing these problems, we realized that the complex- step method could be improved, leading to a form of algorithmic differentiation. Algorithmic differen- tiation — also known as computational differentia- tion or automatic differentiation — is a well known method based on the systematic application of the differentiation chain rule to computer programs. 5 The first objective of this paper is to show how the complex-step derivative approximation is related to algorithmic differentiation, further contributing to the understanding of this relatively new method. On the implementation side, we will continue to focus on automatic implementations, discussing the trade- offs between complex step and algorithmic differen- tiation for the cases of Fortran and C/C++. Fi- nally, computational results corresponding to these two programming languages will be presented. The Connection Complex-Step Method Basics Finite-differencing formulae are common for esti- mating derivatives. These formulae may be de- rived by truncating a Taylor series expansion about a given point x. When estimating sensitivities us- ing these formulae we are faced with the “step size dilemma”, i.e. the desire to choose a small step size to minimize truncation error while avoiding the use of a step so small that errors due to subtractive can- cellation become dominant. The complex-step derivative approximation is an equally simple first derivative estimate for real func- tions and may be obtained using complex calculus. If f is a real function of real variables and it is an- alytic, we can expand it in a Taylor series about a 1 American Institute of Aeronautics and Astronautics
Transcript
AIAA-2001-0921
THE CONNECTION BETWEEN THE COMPLEX-STEP DERIVATIVEAPPROXIMATION AND ALGORITHMIC DIFFERENTIATION
Joaquim R. R. A. Martins, ∗ Peter Sturdza, † and Juan J. Alonso ‡
Department of Aeronautics and AstronauticsStanford University, Stanford, CA
Abstract
This paper presents improvements to the complex-step derivative approximation method which in-crease its accuracy and robustness. These improve-ments unveil the connection to algorithmic differen-tiation theory. The choice between these two meth-ods then hinges on a trade-off between ease of im-plementation and execution efficiency. Automaticimplementations for Fortran and C/C++ are pre-sented and their relative merits are discussed. Thesenew methods were successfully implemented in twovery large multidisciplinary programs and the re-sulting sensitivities are shown to be as accurate asthe analyses. Accuracy and ease of implementationmake these tools very attractive options for sensitiv-ity analysis.
Introduction
Sensitivity analysis is an important field of engineer-ing research. One of its main applications is for de-sign optimization methods that use gradient-basedoptimizers. The calculation of sensitivity informa-tion is usually the costliest step in the optimizationcycle and the optimization process often fails due toinaccurate sensitivity information.
The complex-step derivative approximation is amethod for estimating sensitivities that has beenshown to have a relatively easy implementationwhile yielding very accurate results.1 The poten-tial of this technique has been recognized and it hasbeen used in CFD sensitivity analysis by Anderson2
and in an MDO environment by Newman.3 The de-tails and automatic implementation of this method
∗Graduate Student, AIAA Student Member†Graduate Student, Not an AIAA Member‡Assistant Professor, AIAA Member
In this paper we present further research onthe complex-step derivative approximation, startingwith some of the problems that we have found inthe original implementation. In the process of solv-ing these problems, we realized that the complex-step method could be improved, leading to a formof algorithmic differentiation. Algorithmic differen-tiation — also known as computational differentia-tion or automatic differentiation — is a well knownmethod based on the systematic application of thedifferentiation chain rule to computer programs.5
The first objective of this paper is to show how thecomplex-step derivative approximation is related toalgorithmic differentiation, further contributing tothe understanding of this relatively new method. Onthe implementation side, we will continue to focuson automatic implementations, discussing the trade-offs between complex step and algorithmic differen-tiation for the cases of Fortran and C/C++. Fi-nally, computational results corresponding to thesetwo programming languages will be presented.
The Connection
Complex-Step Method Basics
Finite-differencing formulae are common for esti-mating derivatives. These formulae may be de-rived by truncating a Taylor series expansion abouta given point x. When estimating sensitivities us-ing these formulae we are faced with the “step sizedilemma”, i.e. the desire to choose a small step sizeto minimize truncation error while avoiding the useof a step so small that errors due to subtractive can-cellation become dominant.
The complex-step derivative approximation is anequally simple first derivative estimate for real func-tions and may be obtained using complex calculus.If f is a real function of real variables and it is an-alytic, we can expand it in a Taylor series about a
1
American Institute of Aeronautics and Astronautics
real point x using an imaginary step ih. Solvingthis series for the first derivative yields the followingapproximation:
∂f
∂x=
Im [f (x + ih)]h
+ h2 f ′′′(x)3!
+ . . . (1)
The result is an O(h2) estimate of the derivative off which is not subject to subtractive cancellationerror, since it does not involve a difference opera-tion. This constitutes a tremendous advantage overthe finite-difference approach. For more details onthe theory and implementation of this method, thereader is encouraged to consult previously publishedwork.1
Improving the Complex-Step Method
When using the complex-step derivative approxima-tion, in order to effectively eliminate truncation er-rors, it is typical to use a step that is many ordersof magnitude smaller than the real part of the cal-culation. For most operations, having real and com-plex parts that differ by many orders of magnitudedoes not pose a problem, as the real and imaginaryparts effectively remain separate. With 64 bit float-ing point precision, for example, a complex step of10−20 results in no truncation errors if the real partof the computation is of order one.
However, there are some complex functions wherea small imaginary part may vanish. Consider, forexample, the definition of the inverse sine functionas used in the previous implementation:1
arcsin(z) = −i log[iz +
√1− z2
]. (2)
This function does not maintain accurate derivativesif the complex step is too small because it mixes theimaginary and real parts of the numbers during itscomputation. If z = x + ih where x = O(1) andh = O(10−20), then in the following operation,
iz + z = (x− h) + i (x + h) , (3)
h disappears when using finite-precision arithmeticand the the derivative information is lost.
Similarly, in Equation (2), iz is added to a termwith the same magnitude as z. If small enough, theimaginary part, which carries the derivative infor-mation, disappears when using finite precision arith-metic resulting in a value of zero for the derivative.
Even though such an operation that exchangesreal and imaginary parts would not appear explicitlyin a real-valued code, it may be inadvertently intro-duced in the definition of complex functions. There-fore, real arithmetic codes that have been modified
to use the complex-step method may be vulnerableto this weakness.
Another potential problem may arise when the fol-lowing complex definition of sine is used,
sin(z) =eiz − e−iz
2i. (4)
If z = x + ih, then the imaginary part of the aboveequation is
Im [sin(x + ih)] = cos(x)eh − e−h
2. (5)
This term becomes a problem when h is very smallsince subtraction of two very similar numbers — eh
and e−h — leads to subtractive cancellation. Thisformula is not used in the numeric libraries used bythe authors for the calculation of sine and cosine,but it may appear in the definitions of tangent andhyperbolic tangent whose complex definitions mustoften be supplied by the user. Rather than usingthe definition given in Equation (4), the followingexpression is preferred:
Note that this expression, for small h, simplifies to
sin(x + ih) ≈ sin(x) + ih cos(x). (7)
The real part reduces to the function itself, and theimaginary part to its derivative multiplied by thestep size.
The application of this approach to the inversesine function definition expressed in Equation (2)can yield an excellent replacement for that equation.By setting z = x + ih and applying the approxima-tion for small h, we obtain
arcsin(x + ih) ≡ arcsin(x) + ih√
1− x2. (8)
This is a much improved definition of the inversesine for the purpose of calculating the function sen-sitivity. Not only is the original difficulty with smallstep sizes solved, but truncation error is eliminatedentirely. The real part is the inverse sine functionitself, and the imaginary part, when divided by h, isthe exact derivative of the function for any value ofh.
It is obvious from a Taylor series expansion thatsuch an improved function definition can be foundfor any analytic complex function, f , by linearizingabout h = 0, i.e.,
f(x + ih) ≡ f(x) + ih∂f(x)
∂x. (9)
2
American Institute of Aeronautics and Astronautics
The end result is a sensitivity calculation methodthat is essentially equivalent to algorithmic differen-tiation. This means that the complex-step method,as the step size is reduced, converges to a form ofalgorithmic differentiation that is very simple to im-plement for programming languages with intrinsiccomplex variable types that permit function over-loading.
Algorithmic Differentiation
As mentioned previously, algorithmic differentiationhas been in use for a long time.5,6 The method isbased on the application of the chain rule of differ-entiation to each operation in the program flow. Foreach intermediate variable in the algorithm, a vari-ation due to one input variable is carried through.As a simple example, suppose we want to differen-tiate the multiplication operation, f = x1x2, withrespect to x1. Table 1 compares how the differen-tiation would be performed using either algorithmicdifferentiation or the complex-step method.
As we can see, algorithmic differentiation storesthe derivative value in a separate set of variableswhile the complex step carries the derivative infor-mation in the imaginary part of the variables. It isshown that in this case, the complex-step methodperforms one extra operation — the calculation ofthe term h1h2 — which, for the purposes of calculat-ing the derivative is superfluous. The complex-stepmethod will nearly always include these superfluouscomputations (which can also be identified in Equa-tion (6).) The extra computations correspond to thehigher order terms in the Taylor series expansion ofEquation (1).
Although this example involves only one opera-tion, both methods work for an algorithm involv-ing an arbitrary sequence of operations by propa-gating the variation of one input forward through-out the code. This means that in order to calculaten derivatives, the differentiated code must be exe-cuted n times. This particular form of algorithmicdifferentiation is called the forward mode. It calcu-lates the derivatives of all the outputs with respectto one input. The other mode — the reverse mode5
— has no equivalent in the complex-step method.When using the reverse mode, the code is executedforwards and then backwards to calculate derivativesof one output with respect to n inputs. The totalnumber of operations is independent of n, but thememory requirements may be prohibitive, especiallyfor the case of large iterative algorithms.
Table 1: The differentiation of the multiplication op-eration f = x1x2 with respect to x1 using algorith-mic differentiation and the complex-step derivativeapproximation.
recently become widespread, there are some issuesthat seem unresolved. However, now that this con-nection to algorithmic differentiation has been es-tablished, we can look at the extensive research thathas been done on the subject of algorithmic differ-entiation for some answers.
Important issues include how to treat singular-ities,7 differentiability problems due to if state-ments,8 and the convergence of iterative solvers,9,10
all of which have been addressed by the algorithmicdifferentiation research community.
The singularity issue — i.e. what to do when thederivative is infinite — is handled automatically bythe complex-step method, at the expense of someaccuracy. For example, the computation of
√x + ih
differs substantially from√
x+ ih 12√
xas x vanishes,
but this has not produced noticeable errors in thelarge algorithms that we tested.
Regarding the issue of if statements, in rare cir-cumstances, modification of the original algorithmis necessary as its differentiability may be compro-mised by piecewise function definitions.
Finally, concerning convergence of iterative codes,the experience of the authors has been that in allcases the imaginary part converges at a rate similarto the real part, although somewhat lagged. Fur-thermore, for engineering applications where the it-erative process converges to a differentiable function,it is our experience that the derivative also convergesto the correct result.
Implementations
The implementation of any of the derivative calcu-lation methods, for practical purposes, should be asautomated as possible. Changing the source codemanually is not only an extremely tedious task, butis also likely to result in the introduction of codingerrors in the program.
3
American Institute of Aeronautics and Astronautics
There are two main possibilities for implementingalgorithmic differentiation: by source code transfor-mation or by using derived datatypes and operatoroverloading.
To implement algorithmic differentiation bysource transformation, the whole code must be pro-cessed with a parser and all the derivative calcu-lations are introduced as extra lines of code. Theresulting extended code is greatly enlarged and itbecomes practically unreadable. This fact consti-tutes an implementation disadvantage as it becomesimpractical to debug this new extended code. Onehas to work with the original source, and every timeit is changed (or if different derivatives are desired)one must rerun the parser before compiling the newversion.
In order to use derived types, we need languagesthat support this feature, such as Fortran 90 orC++. A new type of structure must be created thatcontains both the value and its derivative. All theexisting operators must then be re-defined for thenew type. The new operator would have exactly thesame behaviour as before for the value part of thenew type, but use the derivative of the operator tocalculate the derivative portion. This results in avery elegant implementation since very few changesare required in the original code.
In the following section, implementations of thesemethods will be described in detail for Fortran andC/C++. Some notes for other programming lan-guages will also be included.
Fortran
Many tools for automatic algorithmic differentiationof Fortran programs by source transformation al-ready exist. Some of these are ADIFOR, TAMC,DAFOR, GRESS, Odysse and PADRE2.11 As ex-plained above, extending the original source code isa method that greatly compromises the code’s main-tainability.
There are also tools for automatic algorithmic dif-ferentiation of Fortran programs by using deriveddatatypes and operator overloading such as AD01,ADOL-F, IMAS and OPTIMA90.11 They have beenextensively developed and provide the user a lot offunctionality.
However, all of these require the user to make aseries of changes to the source code by hand. Theseinclude selective changes to variable type declara-tions, the addition of use statements, as well as themodification of input, output and variable initializa-tion statements. When dealing with a large program
with a multitude of subroutines, this approach maybe time consuming and prone to errors.
The automatic Fortran differentiation tool pre-sented in this paper is an improved version of thecomplex-step method published previously.1 It iscomposed of a a Python script that makes the neces-sary changes to the source code and Fortran 90 mod-ule that contains overloaded function definitions.
The Python script has been enhanced by addingsupport for a wider range of platforms and compilersas well as support for MPI based parallel implemen-tations. Further flexibiliy was added to the script byaddressing file input and output issues.
The new version of the complexify module hasbeen improved by using the new function defini-tons discussed in the previous section, i.e., defini-tions that use the original real function for the realpart and the derivative of the function multiplied byh for the imaginary part, rather than the complexfunction definition.
These new definitions only apply to functions thatare not defined for complex arguments in standardFortran, such as tangent, inverse trignometric func-tions and hyperbolic functions. The reason for notusing these improved definitions for all functions isthat Fortran 90 does not permit the redefinition offunctions that already exist.
The way around this restriction would be to cre-ate a new type of variable, thus allowing the use ofour own function definitions. This is the approachadopted by the algorithmic differentiation methodsthat use derived datatypes. The drawbacks with thisapproach are that there are far more changes re-quired to the original source code and it is no longercompatible with some old Fortran constructs.
This tool for implementing the complex-stepmethod represents, in our opinion, a good compro-mise between ease of implementation and algorithmefficiency. While pure algorithmic differentiationis numerically more efficient, the method used inthis paper requires far fewer changes to the origi-nal source code resulting in improved maintainabil-ity. Furthermore, practically all the changes are per-formed automatically by the use of the script. Allthe necessary files, together with detailed instruc-tions are available on the first author’s webpage.4
C/C++
The C/C++ implementations of the complex-stepmethod and algorithmic differentiation are muchmore straightforward than the Fortran implemen-tations. Two different C/C++ implementations are
4
American Institute of Aeronautics and Astronautics
Table 2: Run times (in seconds) for a boundary layer solution with no derivatives; with derivatives via thecomplex-step method; and with derivatives using an algorithmic differentiation method.
presented and used in this paper and are availableon the web.4
The first method is analogous to the Fortran im-plementation, i.e., it uses complex variables andoverloaded complex functions and operators. Aninclude file, complexify.h, defines a new variabletype called cmplx and all the functions that are nec-essary in the implementation of the complex-stepmethod. The inclusion of this file and the replace-ment of double or float declarations with cmplx isnearly all that is required.
The remaining work involves dealing with inputand output routines. The usual casting of the in-puts to cmplx and printing of the outputs using thereal() and imag() functions works well. For ASCIIfiles or terminal input and output, the use of theC++ iostream library is also possible. In this case,the “À” operator will automatically read in a realnumber and properly cast it to cmplx, or read in acomplex number in the Fortran parenthesis format,e.g., “(2.3,1.e-20)”. The “¿” operator will out-put in the parenthesis format.
The second method is a version of algorithmic dif-ferentiation that is basically a realization of Equa-tion (9). It can be implemented by including afile called derivify.h and by replacing declara-tions of double with declarations of surreal. Thederivify.h file redefines all relational operators,the basic arithmetic formulae, trigonometric func-tions, and other formulae in the math library whenapplied to the surreal variables. These variablescontain “value” and “derivative” parts analogous tothe real and imaginary parts of complex variables.It works just as the complex step version does, ex-cept that the step size may be set to 1 since there isno truncation error.
One feature available to the C++ programmer isworth mentioning: templates. Templates make itpossible to write source code that is independentof variable type declarations. This approach in-
volves considerable work with complicated syntaxin function declarations and requires at least a bitof object-oriented programming. There is no need,however, to modify the function bodies themselvesor to change the flow of execution, even for pure Cprograms.
The distinct advantage is that the variable typescan be decided at run time, so the very same exe-cutable can run either the real-valued, the complexstep or the algorithmic differentiation version. Thisconsiderably simplifies version control and debug-ging since updating or debugging one version of thecode updates and debugs the others — the sourcecode is literally the same for all three versions of theprogram.
An execution speed comparison of the complexstep and pure algorithmic differentiation method ona finite-difference boundary layer code is shown inTable 2. The rows labeled “Real,” “Complex” and“Surreal” contain execution times in seconds, and,in parentheses, the ratios of the latter two methodsto the real-valued execution times. This programuses C++ templates to implement the three meth-ods, so the size of the executable can increase con-siderably. When comparing to the real-valued code,the executables here are 5 to 7 times larger. How-ever, there was no time or memory penalty due tothe use of templates.
The basic result is that, as is often stated, thecomplex variable version of the program takes aboutthree times longer to run than the real-valued ver-sion, and the algorithmically differentiated codetakes about twice as long as the real-valued pro-gram. Interestingly, the SGI platform is much moreefficient at computing derivatives than the other sys-tems, either due to a better compiler, its 64 bit hard-ware architecture, or both.
It should be noted that there are much moresophisticated algorithmic differentiation packagesavailable for C/C++.11 The one presented here is
5
American Institute of Aeronautics and Astronautics
meant as a simple improvement to the complex-stepmethod that is equally easy to implement. Otherpackages include ADIC,12 an implementation mir-roring ADIFOR, and ADOL-C,13 a free package thatuses operator overloading and can operate in forwardand reverse modes as well as compute higher orderderivatives.
Other Programming Languages
In addition to the Fortran and C/C++ implemen-tations described above, some work was done withother programming languages.
The complex-step method works well under Mat-lab with practically no changes to the user’s sourcecode.1
A simple implementation of the complex-stepmethod for Python was also developed for the pur-poses of this paper. In order to do this, one mustimport cmath, a standard module for complex arith-metic. Then, just as in Fortran 77 or C, functionssuch as c abs, c atan2 and c max must be defined.Since Python supports operator overloading, a moreelegant algorithmic differentiation method similar tothat used in C++ also seems possible.
In general, algorithmic differentiation can be im-plemented in any programming language that sup-ports derived datatypes and operator overloading.For languages that do not have these features, thecomplex-step method can be used wherever complexarithmetic is supported.
Results
Two-Dimensional Flow Solver
The automatic implementation of the complex-stepmethod for FLO82 — a two-dimensional finite vol-ume solver for the Euler equations — has beendecribed in previous work.1 Although the resultsthat were presented therein are correct, they didnot achieve the full working precision of the algo-rithm because of an error in the compilation. There-fore, we will show results from the same solver thatwere obtained using the latest implementation of thecomplex-step method.
Since the flow solver is an iterative algorithm, itis useful to compare the convergence of a given re-sult with that of its derivative, which is containedin its complex part. This comparison is shown inFigure 1 for the drag coefficient and its derivativewith respect to the freestream Mach number. The
50 100 150 200 250 300 350 400 450 500
10−15
10−10
10−5
100
Iterations
Rel
ativ
e E
rror
, ε
CD
∂ CD
/ ∂ M∞
Figure 1: Convergence of CD and ∂CD/∂M∞ for the2D flow solver.; ε = |f−fref |
|fref | .
drag coefficient converges to the algorithm’s preci-sion, which is almost machine zero. The sensitivityis shown to lag slightly in the convergence. Thisis expected, since the calculation of the sensitivityof a given quantity is dependent on the value ofthat same quantity. After the drag coefficient re-sult achieves the algorithm’s precison, the sensitiv-ity catches up after about 20 additional iterations.
10−30
10−20
10−10
10−15
10−10
10−5
100
Rel
ativ
e E
rror
, ε
Step Size, h
Complex−Step Finite−difference
Figure 2: Sensitivity estimate errors for ∂CD/∂M∞given by finite-difference and the complex step fordecreasing step size; ε = |f−fref |
|fref | .
The next plot, Figure 2, shows a comparison be-tween the estimates of ∂CD/∂M∞ given by both thecomplex-step and the forward-difference methods for
6
American Institute of Aeronautics and Astronautics
various step sizes.The forward-difference estimate initially improves
since the O(h) truncation error decreases with stepsize. However, as the step is reduced below a value ofabout 10−10, subtractive cancellation errors becomean issue and the estimates become unreliable. Whenthe interval h is so small that no difference exists inthe output — for steps smaller than 10−17 — thefinite-difference estimates eventually yield zero.
The complex-step estimate improves quadraticallyas the step size is decreased. The estimate is prac-tically insensitive to variations of step size below anh of the order of 10−10, value for which the estimateachieves the accuracy of the flow solver. Althoughthe complex step size can be made extremely small,there is a lower limit (h = 10−321) below which un-derflow starts to occur and the estimate results inNaN.
When comparing the best accuracy of theseapproaches, we see that finite-differencing onlyachieves a fraction of the accuracy of the complex-step approximation.
Three-Dimensional Aero-Structural Solver
The tools that we have developed to implement thecomplex-step method automatically were also testedon a much larger application: a high-fidelity aero-structural solver that is part of an MDO frameworkwhich was created to solve wing design optimizationproblems.14 Since an analytic method for calculat-ing sensitivities is being developed for this analysiscode, the complex-step method will be an extremelyuseful reference for validation.
Figure 3: Aero-structural model and solution of atransonic jet configuration, showing a slice of thegrid and the wing’s internal structure.
Within the aero-structural solver, the aerody-
namic analysis is performed by Reuther et al.’sSYN107-MB15 — a multiblock Euler flow solver —which also calculates the sensitivity of the drag andother aerodynamic quantities with respect to winggeometry perturbations using an adjoint method.These perturbations are the shape design variablesand consist of “bump” functions distributed on thetop and bottom surfaces of the wing.15
The wing structure is modeled with FESMEH,16
a finite element solver which includes the followingwing structural components: wing skins, spars andribs. In addition to solving for displacements andstresses, the structural solver is able to calculate thederivatives of these quantities with respect to ele-ment dimensions analytically. The coupling of theaerodynamic and structural analysis codes has pre-viously been developed by the authors and uses aconsistent and conservative scheme.14
The aero-structural model and a sample solutionare shown in Figure 3. The internal structure of thewing and a slice of the 3D CFD grid are shown.
100 200 300 400 500 600 700 80010
−8
10−6
10−4
10−2
100
Iterations
Ref
eren
ce E
rror
, ε
CD
∂ C
D / ∂ b
1
Figure 4: Convergence of CD and ∂CD/∂b1 for the3D aero-structural solver; ε = |f−fref |
|fref | .
To validate the complex-step results for thissolver, we chose the derivative of the drag coeffi-cient to the shape perturbations, accounting for thestructural displacements. There is a three by threegrid of perturbations on each of the wing’s surfacesmaking up for a total of 18 design variables.
A plot of the convergence of the drag coefficient isalso shown for this case (see Figure 4), together withthe convergence of the sensitivity corresponding tothe first shape design variable, b1.
As with the case of the two-dimensional flowsolver, the derivative converges at the same rate as
7
American Institute of Aeronautics and Astronautics
the coefficient and it lags slightly, taking about 100additional iterations to achieve the maximum preci-sion. The maximum precision of the derivative is ob-served to be slightly lower than the precision of thecoefficient. When looking at the number of digitsthat are converged, the coefficient consistently con-verged to six digits, while the derivative convergedto five or six digits. This might be explained by theincreased round-off errors of complex arithmetic,17
which does not affect the real part when such smallstep sizes are used.
10−15
10−10
10−5
10−6
10−5
10−4
10−3
10−2
10−1
100
Rel
ativ
e E
rror
, ε
Step Size, h
Complex−Step Finite−difference
Figure 5: Sensitivity estimate errors for ∂CD/∂b1
given by finite-difference and the complex step fordifferent step sizes; ε = |f−fref |
|fref | .
Figure 5 shows a plot analogous that of Figure 2.In this case the finite-difference result has an ac-ceptable precision for only one step size (h = 10−2).Again, the complex-step method yields accurate re-sults for step sizes ranging from h = 10−2 to h =10−200.
The results corresponding to the complete shapesensitivity vector are shown in Figure 6. Althoughmany different sets of finite-difference results wereobtained, only the best set is plotted here. The plotshows no discernible difference between the two setsof results.
Again, we would like to emphasize that while therewas considerable effort involved in obtaining rea-sonable finite-difference results by trying differentstep sizes, no such studies were necessary with thecomplex-step method.
2 4 6 8 10 12 14 16 18
−0.05
0
0.05
0.1
0.15
∂ C
D /
∂ b i
Shape variable, i
Complex−Step, h = 1×10−20 Finite−Difference, h = 1×10−2
Figure 6: Comparison of the estimates for the shapesensitivities of the drag coefficient, ∂CD/∂bi.
Supersonic Viscous/Inviscid Solver
The following example illustrates how the complex-step method can be applied to an analysis fromwhich it is very difficult to extract accurate finitedifference gradients. It is a complicated code thatuses input and output file manipulations to com-bine five computer programs including an iterativeEuler solver and boundary layer solutions featuringtransition prediction. This code was developed forsupporting design work of the supersonic NaturalLaminar Flow aircraft concept.18
In this framework, Python is used as the glue thatjoins the many programs that constitute the functionanalysis with an optimizer that requires values andderivatives of the objective and constraints. Gra-dients are computed with the complex-step methodand with algorithmic differentiation in the Fortran,C++ and Python programming languages. The se-quence of calculations for each function evaluationis as follows:
• The Python wrapper program is given a list ofdesign variables by the optimizer. It then be-gins by computing, using complex variables, ageometry description for the grid generator.
• CH GRID, written by D. Saunders and J.Reuther at NASA Ames, automatically gener-ates a 3D grid for either a wing alone or a wing-body. It is a Fortran 90 program which wascomplexified with the script described in thispaper.
• CFL3D19 provides the supersonic Euler flow so-
8
American Institute of Aeronautics and Astronautics
Figure 7: Trapezoidal wing with 5% thick biconvexairfoils, 32◦ leading edge sweep, 22.5 ft root chord,5.45 ft tip chord and 18.94 ft semispan.
lution. Starting with Version 6, CFL3D comeswith the complex-step method already imple-mented.
• The Euler solution is post-processed by a C++program that was complexified following the de-scription in this paper.
• A quasi-3D boundary-layer solver18 is then exe-cuted on the wing at six spanwise stations, topand bottom, by the Python wrapper (viscous-inviscid iterations are not required at the su-personic flight conditions of this NLF aircraft).It can compute laminar and turbulent bound-ary layers and also has the capability to predictthe transition location. This boundary-layersolver uses the C++ algorithmic differentiationapproach.
• The Python wrapper merges the 12 boundarylayer solutions to produce a three-dimensionalskin-friction drag result to which it adds the in-viscid drag computed by CLF3D and computesother quantities to evaluate constraints (such aswing structural weight constraints).
• The above steps are repeated for each of thedesign and constraint variables, and then thePython wrapper passes the values and deriva-tives to the optimizer.
10−20
10−15
10−10
10−5
100
10−8
10−6
10−4
10−2
100
102
Step Size, h
Rel
ativ
e E
rror
, ε
Finite DifferenceComplex−Step
Figure 8: Convergence of gradients as step size isdecreased. ε = |f−fref |
|fref | .
The results presented here are gradients of theskin-friction drag coefficient with respect to the rootchord of the trapezoidal wing depicted in Figure 7at Mach 2.0, 40,000 ft and 4◦ angle of attack. Theother design variables are the tip chord, the root andtip thickness to chord ratios, the leading edge sweepand the tip twist. Twist and thickness vary linearlyspanwise, and the area is held constant. Therefore,when the root chord varies, the trailing edge sweepand wingspan change.
A comparison of finite central difference resultswith the complex-step gradients is shown in Fig-ures 8-11. The first one, computed with a laminarboundary layer, shows the rather poor quality finitedifference gradients of this analysis.
There are several properties of this analysis thatmake it difficult to extract useful finite differencedata. The most obvious is transition. It is difficultto truly smooth the movement of the transition frontwhen transition prediction is computed on a dis-cretized domain. Since transition has such a large ef-fect on skin friction, this difficulty is expected to ad-versely affect finite difference gradients of drag. Ad-ditionally, the discretization of the boundary layerby computing it along 12 arcs that change curvatureand location as the wing planform is perturbed issuspected to cause some noise in the laminar solu-tion as well (these arcs, their purpose, and furtherdetails of the boundary layer solution and transitionprediction are described elsewhere.18)
A plot of the gradient itself in Figure 9 shows thatfinite difference and complex-step results do agreefor step sizes larger than 0.001 feet. Figure 10 de-
9
American Institute of Aeronautics and Astronautics
10−10
10−8
10−6
10−4
10−2
100
−3
−2
−1
0
1
2
x 10−5
Step Size, h
dCd f /
dCr
Finite DifferenceComplex−Step
Figure 9: Gradient of skin friction drag coefficientwith respect to wing root chord computed with var-ious step sizes (laminar boundary layer).
picts the drag coefficient as it varies with pertur-bations in root chord. The slope computed withthe complex-step at 22.5 ft chord is in good agree-ment with the function evaluations in the vicinity.Figure 11 shows the expected noise increase for thesame conditions when transition to turbulent flowis computed. The complex-step slope, however, re-mains very reasonable — not too far from a leastsquares fit of the data.
Conclusions
New insights into the complex-step method havebeen presented. Solutions to several subtle problemsin its implementation have been clarified, and its re-lationship to traditional algorithmic differentiationmethods has been exposed. This enables the appli-cation of a substantial body of knowledge — thatof the algorithmic differentiation community — tothe complex-step method, answering many impor-tant questions.
The complex-step technique may still be the eas-iest of any derivative calculation method to imple-ment, particularly in legacy codes and in codes thatmix programming languages. Furthermore, the re-sulting complexified code is no harder to debug andmaintain than the original.
The example implementations and their results il-lustrate these points clearly and put forward twoexcellent uses for the method: validating a more so-phisticated gradient calculation scheme and provid-
22.495 22.5 22.505
4.3728
4.3729
4.373
4.3731
4.3732
4.3733
4.3734
4.3735
4.3736
x 10−4
Root Chord (ft)
Cd f
Function EvaluationsComplex−Step Slope
Figure 10: Skin friction drag coefficient as a functionof wing root chord with a laminar boundary layer.
ing accurate and smooth gradients for analyses thataccumulate substantial computational noise.
Acknowledgements
The authors would like to thank Dr. James Reutherfor his invaluable help, particularly with SYN107-MB and the aero-structural solver. He was unfortu-nately demoted from the top of the first page to thissection because of bureaucratic constraints.
References
[1] Martins, J. R. R. A., I. M. Kroo, and J. J.Alonso “An Automated Method for SensitivityAnalysis using Complex Variables” Proceedingsof the 38th Aerospace Sciences Meeting, Reno,NV, January 2000. AIAA Paper 2000-0689.
[2] Anderson, W. K., J. C. Newman, D. L. Whit-field, E. J. Nielsen, “Sensitivity Analysis forthe Navier-Stokes Equations on UnstructuredMeshes Using Complex Variables”, AIAA Pa-per No. 99-3294, Proceedings of the 17th Ap-plied Aerodynamics Conference, 28 Jun. 1999.
[3] Newman, J. C. , W. K. Anderson, D. L. Whit-field, “Multidisciplinary Sensitivity DerivativesUsing Complex Variables”, MSSU-COE-ERC-98-08, Jul. 1998.
[4] http://aero-comlab.stanford.edu/jmartins
[5] Griewank, A., Evaluating Derivatives, SIAM,Philadelphia, 2000.
10
American Institute of Aeronautics and Astronautics
22.495 22.5 22.505
3.0222
3.0222
3.0222
3.0222
3.0222
3.0223
3.0223
x 10−3
Root Chord (ft)
Cd f
Function EvaluationsComplex−Step Slope
Figure 11: Skin friction drag coefficient as a functionof wing root chord with transition to turbulent flowin the boundary layer.
[6] Bischof, C., A. Carle, G. Corliss, A. Grienwank,P. Hoveland, “ADIFOR: Generating DerivativeCodes from Fortran Programs”, Scientific Pro-gramming, Vol. 1, No. 1, 1992, pp. 11-29.
[7] Bischof, C., G. Corliss, A. Grienwank, “ADI-FOR Exception Handling”, Argonne TechnicalMemorandum, MCS-TM-159, 1991.
[8] Beck, T. and H. Fischer, “The if-problem inautomatic differentiation”, Journal of Compu-tational and Applied Mathematics, Vol. 50, pp.119-131, 1994.
[9] Griewank, A., C. Bischof, “Derivative Conver-gence for Iterative Equation Solvers” Optimiza-tion Methods and Software, Vol. 2, pp. 321-355,1993.
[10] Beck, T., “Automatic Differentiation of itera-tive processes” Journal of Computational andApplied Mathematics, Vol. 50, pp. 109-118,1994.
[12] Bischof, C., L. Roh, A. Mauer, “ADIC – AnExtensible Automatic Differentiation Toolfor ANSI-C,” to appear in Software: Prac-tice and Experience, also ANL/MCS-P626-1196(ftp://info.mcs.anl.gov/pub/tech reports/reports/P626.ps.Z)
[13] Griewank, A., et al., “ADOL-C: A Packagefor the Automatic Differentiation of AlgorithmsWritten in C/C++,” ACM TOMS, Vol. 22(2),June 199, pp. 131-167, Algorithm 755.
[14] Reuther, J., J. J. Alonso, J. R. R. A. Mar-tins, and S. C. Smith, “A Coupled Aero-Structural Optimization Method for CompleteAircraft Configurations”, Proceedings of the37th Aerospace Sciences Meeting, AIAA Paper1999-0187. Reno, NV, January 1999.
[15] Reuther, J., J. J. Alonso and A. Jameson,“Constrained Multipoint Aerodynamic ShapeOptimization Using an Adjoint Formulationand Parallel Computers: Part I”, Journal ofAircraft, 36(1):51-60, 1999.
[16] Holden, M. E., “Aeroelastic Optimization usingthe Collocation Method”, PhD Thesis, Stan-ford, May 1999.
[17] Olver, F. W. J., “Error Analysis of ComplexArithmetic”, Computational Aspects of Com-plex Analysis, pp. 279-292, 1983.
[18] Sturdza, P., V. M. Manning, I. M. Kroo, andR. R. Tracy, “Boundary Layer Calculationsfor Preliminary Design of Wings in SupersonicFlow,” AIAA Paper 99-3104, Proceedings ofthe 17th Applied Aerodynamics Conference, 28June 1999.
[19] Krist, S. L., R. T. Biedron, and C. L.Rumsey, CFL3D User’s Manual (Version 5.0)NASA/TM-1998-208444, June 1998.
11
American Institute of Aeronautics and Astronautics
