The NASA Langley MDO method evaluationstudy seeks to arrive at a set of guidelines for usingpromising MDO methods by accumulating and an-alyzing computational data for such methods. Thedata are collected by conducting a series of re-producible experiments. In the first phase of thestudy, three MDO methods were implemented in theiSIGHT* framework and used to solve a set of ten rel-atively simple problems. In this paper, we commenton the general considerations for conducting methodevaluation studies and report some initial results ob-tained to date. In particular, although the results arenot conclusive because of the small initial test set,preliminary numbers suggest that the performanceof the methods tends to be consistent with their pre-dicted theoretical properties.
Multidisciplinary Design Optimization (MDO)problems are optimization problems that describecomplex coupled engineering systems. The systemsare composed of physically interacting subsystems de-scribed by disciplinary analyses, each of which pos-sesses a certain degree of autonomy but depends onother subsystems via a number of couplings, alsoknown as interdisciplinary variables.
We distinguish MDO formulations from optimiza-tion algorithms in the following way. MDO formula-tions belong to an area that studies MDO problemdefinitions, including problem decomposition and in-tegration. To analyze an MDO formulation, one con-siders such attributes as consistency, equivalence to
other formulations, optimality conditions, and sen-sitivity of solutions to various perturbations. Opti-mization algorithms are used to solve a particularMDO formulation. It is then appropriate to speakof local convergence rates and of global convergenceproperties of an optimization algorithm applied to aspecific formulation. An analogous distinction existsin the field of partial differential equations. On theone hand, equations are analyzed in terms of regu-larity, well-posedness, and the existence and unique-ness of solutions. On the other, one considers numer-ous algorithms for solving differential equations. Thearea of MDO methods studies MDO formulationscombined with optimization algorithms, although attimes the distinction is blurred. It is important tonote that an optimization algorithm applied to twoformulations that are mathematically equivalent interms of solution sets may exhibit drastically differ-ent behaviors [1]. When we say "MDO methods" inthis paper, we focus more on formulations, althoughoptimization algorithms play a role as well.
A sizable and ever growing body of methods andtheir variants has been proposed for solving MDOproblems. Yet, there is much speculation, but limitedcomputational or analytical substantiation of practi-cal applicability and algorithmic properties of MDOmethods. An informative computational methodstudy—of different scope and intent—was done byShubin [2]. Haim et al. [3] compare performancesof several nonlinear programming software packageson an MDO problem, given one specific formulation.However, in general, a practitioner of MDO has littlebasis for selecting a method among those appearingin print.
Several ongoing efforts at NASA Langley areaimed at addressing this deficiency. The present pa-per will acquaint the reader with the initial results ofthe first phase of a method evaluation study initiatedlast year. The objectives of the study are as follows:
'Member AIAA. Research Scientist, MDOB, NASA Langley Research Center, Hampton, VA, n.alexandrovQlarc.nasa.gov.'Member AIAA. Chief Engineer, Engineous Software, Inc., Morrisville, NC. Research done under NASA Contract L-6317.
'The use of trademarks or names of manufacturers in this report is for accurate reporting and does not constitute anofficial endorsement, either expressed or implied, of such products or manufacturers by the National Aeronautics and SpaceAdministration.
• Accumulate computational data on the perfor-mance of promising MDO methods.
• Compare the practical performance of themethods with their proven or conjectured an-alytical properties.
• Arrive at a classification of the methods andproblems amenable to the methods, as well asa set of guidelines for using specific methods.
• Establish "standards" or guidelines for system-atic, easily reproducible, method testing proce-dures.
We consider the last objective exceedingly im-portant. Numerical results presented in publicationsabout MDO methods are rarely easily reproducibleby other researchers. This may be due to legitimatereasons, such as the complexity or unavailability of aparticular commercial or proprietary code to a spe-cific user. Nonetheless, the lack of at least a minimalset of easily reproducible results makes comparison,validation, and evaluation of methods extremely diffi-cult, if not impossible, because one can always arguethat a particular result is due strictly to an implemen-tation and not to an intrinsic property of a methodunder consideration. A remedy is, of course, to ascer-tain that the test can be replicated at least for simpleproblems, thus providing a basis for legitimate discus-sions of implementations vs. method properties.
Our objectives present a formidable task, sincecomparing methods intended for solving even the con-ventional nonlinear programming problem is notori-ously difficult, given the limitations of the problemselection, the implementation, and many other vari-ables. However, our aim is not to declare one methodsuperior to another. Instead, by accumulating com-putational data, we seek to understand under whatcircumstances the use of a specific method may beadvisable.
In this paper, we comment on testing, in general,and describe our testing procedures and their limita-tions, in particular. We give detailed results for onetest problem, followed by a summary of numerical re-sults for all problems, as well as conclusions availableto date.
A record of all tests and numerical results can befound in a forthcoming NASA contractor report [4].A complete record of all tests, codes, and descrip-tion files can be found at the NASA Langley Multi-disciplinary Optimization Branch (MDOB) methodevaluation site accessible via the MDOB homepage
(http:I Itmad-www.larc.nasa.gov/mdob/MDOB/) .§Interested readers are invited to make contributionsto this site after familiarizing themselves with thesubmission requirements.
MDO Methods
During Phase I of the study, we collected numer-ical data on the Multidisciplinary Feasible method(MDF), the Collaborative Optimization approach(CO), and the Individual Discipline Feasible method(IDF). MDF is a mathematical idealization of theconventional approach to MDO. The nomenclaturewas introduced by Cramer et al. [5]. MDF was im-plemented to serve as a baseline result in this study.Antecedents of CO [6] and IDF [5] can be traced towork on optimization of large systems, such as thatof Wismer [7] and Lasdon [8]. Both CO and IDFare aimed at solving large problems with a narrowbandwidth of coupling.
To give a brief description of the methods understudy, we state the MDO problem as a nonlinear pro-gramming problem of the following form:
minimize f ( x , u(x))subject to h(x,u(x)) = 0
g(x,u(x))>0,(1)
where, given a vector of design variables x, the statevariables u(x) are defined via a block system of equa-tions,
=0,AN(x,ui(x),...,uti(x))
where N is the number of blocks. In the context ofMDO, the blocks of the system usually represent thestate equations for the disciplinary analyses and thenecessary interdisciplinary couplings. The equationsare known as the Multidisciplinary Analysis (MDA)system.
Multidisciplinary Feasible Method
The MDF formulation is a conventional methodfor solving problem (1). It is an example of the vari-able reduction approach to nonlinear programming,where only the design variables a; are used as inde-pendent optimization variables. The statement of theproblem is unchanged from formulation (1). There-fore, theoretically, the convergence properties of anyoptimization algorithm applied to MDF are just its
§ As the paper is going to press, the NASA Langley web site is being reorganized. If the link is inoperative, please search forkeywords.
convergence properties with respect to conventionalnonlinear programming problems.
At each iteration of the optimization procedure,the design variable vector x is input into the MDAsystem. The system is then solved for the statevector u, thus reducing the dimension of the opti-mization problem. The solution of the MDA system,i.e., the multidisciplinary function evaluation, is fre-quently performed via the block Gauss-Seidel proce-dure (fixed-point iteration). The MDA system canbe, in principle, solved by any method for solvingnonlinear equations, such as the Newton's method.One should note, however, that unless a globaliza-tion strategy is used, there is no guarantee that amethod for solving nonlinear equations will solve theMDA problem from arbitrary initial points.
MDF has been in use ever since nonlinear pro-gramming techniques were first applied to engineer-ing optimization problems and, hence, is well un-derstood. It makes full use of the existing analy-sis codes and gives rise to the smallest optimizationproblems in terms of the number of design variables.At each iteration of optimization, multidisciplinaryequilibrium (or feasibility) is maintained via MDA.The term "Multidisciplinary Feasible" refers to thisproperty. Given a careful statement of a particularproblem and a good optimization algorithm appliedto the formulation, MDF can be efficient.
The main drawback of MDF is its extreme ex-pense. First, the method is costly to implement, be-cause a practitioner of MDO has to face the diffi-cult problem of analysis integration. Second, a com-plete MDA must be done not just at every iteration,but also for computing derivatives, if finite-differencederivatives are used. Moreover, optimization algo-rithms applied to MDF are sensitive to the conver-gence of MDA, in general, and the fixed-point itera-tion, in particular, which may be detrimental to therobustness of the method and its speed. Finally, themethod is not immune to failing because of attemptsto process points that cannot be analyzed.
Collaborative Optimization
In the CO approach, problem (1) is decomposedinto a number of subsystems, usually along the disci-plinary lines. The problem is then reformulated as abilevel programming problem.
A system objective / is selected for the systemlevel. The system-level constraints are the so-called"compatibility" or "coupling" constraints designed tobring the system into multidisciplinary equilibrium.The values and derivatives of the compatibility con-straints are computed by solving the lower-level, sub-
system optimization problems.The subsystem problems, generally solved in par-
allel, are nonlinear programming problems whose ob-jective is to minimize the discrepancy between theshared variables of the subsystems while satisfyingthe disciplinary constraints.
Braun et al. [9] comment on implementation andperformance features of CO. A detailed discussion ofCO's analytical and computational properties can befound in Alexandrov and Lewis [1]. The latter workcontains complete, precise notation of CO and a sim-plified notation useful in initial implementation. Weuse the simplified notation here.
Assume that for each disciplinary subsystem i,given a vector of design variables a:,- , the correspond-ing vector of responses «;(#;) is computed via theanalysis Af, and the constraints
must be satisfied, where each gi(x) is a vector. Thevectors x,- pertaining to disciplines i are not necessar-ily disjoint subsets of the complete vector of designvariables x. Then auxiliary vectors £,• and i>i are in-troduced to represent the shared components of a;,-and Ui at the system level.
The system-level optimization problem of the COformulation has the form
minimize /(£, v)subject to C(£,v) — 0, (2)
where C(£, v) is the system of N compatibility con-straints Cj(£j, Vi), each one having the form
where x* is the solution of the z'-th subsystem opti-mization problem:
The actual system compatibility constraint can bestated as a sum of the individual constraints. Otherforms of the system-level compatibility constraintsand subsystem problems exist ([6], [10], [11], [12]),but in Phase I, we focused on the CO formulation asshown above, as it is the most promising one [1] .
CO has a number of attractive features. First, itdispenses with MDA. Instead of requiring multidis-ciplinary feasibility at each iteration of optimization,the feasibility is attained in the system optimizationproblem via the introduction of compatibility con-straints. Thus, each iteration is feasible with respect
to individual analyses, but not multidisciplinary fea-sible until a solution is reached.
CO has the very appealing attribute of disci-plinary autonomy with respect to both the analysesand optimization, resulting in a relatively easy proce-dure for integration of the disciplinary analyses. Theanalyses can be processed in parallel. Another signif-icant feature is the maintenance of intradisciplinaryfeasibility at each system iteration, which is impor-tant from the application perspective. As MDF, themethod makes full use of the disciplinary analysissoftware.
The limitations of CO have to do with attain-ing multidisciplinary feasibility and with reformulat-ing the original, conventional nonlinear programmingproblem into a nonlinear bilevel programming prob-lem.
In particular, the method is intended for prob-lems where the interdisciplinary coupling has smallbandwidth. Problems with many couplings are notexpected to benefit from this formulation. The for-mulation has more optimization variables than thatarising from MDF, but given the first limitation, theincrease in the number of variables should not begreat, as the number of auxiliary variables dependson the bandwidth of the coupling.
Further, the CO formulation is aimed at solvinga narrower class of problems than do MDF and IDF;namely, it does not consider general constraints atthe system level. Technically, general constraints canbe included at the system level. Their inclusion tendsto impair the performance of the method, but is un-avoidable at times. For instance, in one of our testproblems, the hub frame design, the objective is tominimize the weight of the structure, subject to con-straints on the translational and rotational displace-ments, stresses in the frame members, and local buck-ling of the frame members. The translational androtational displacements of the frame structure areglobal responses and should not be treated as localresponses at the subsystem level.
Another limitation is that, despite maintainingdisciplinary autonomy, CO does not allow explicitoptimization with respect to disciplinary objectivesat the disciplinary level. Multiple disciplinary objec-tives have to be incorporated at the system level.
The decomposition procedure may be dictated bythe application, but it presents a difficult problem forcases where structure is not well understood.
The "small bandwidth of coupling" feature sim-ply limits the scope of problems amenable to CO. Inaddition, CO suffers from a difficulty that has to dowith reformulating a nonlinear programming probleminto a general, nonlinear, bilevel optimization prob-
lem. As such, CO is inherently difficult to solve bymeans of software intended for conventional, single-level, nonlinear programming problems. While theCO formulation is equivalent to the original nonlinearprogramming problem with respect to the solutionsets, the formulation is not equivalent to the origi-nal problem with respect to optimality conditions [1] .This is an example of two problem formulations thatare equivalent with respect to their solution sets, yetare drastically different numerically.
Based on theoretical analysis of CO [1], it is ex-pected that much fine-tuning would be required toimplement the method for a specific problem andthat convergence behavior of conventional optimiza-tion methods applied to the CO formulation mightbe erratic.
Individual Discipline Feasible Method
The IDF formulation provides another approachto "breaking" the expensive MDA iteration. Thewords "Individual Discipline Feasible" refer to main-taining disciplinary feasibility at each optimizationiteration, but not multidisciplinary feasibility until asolution is reached.
Various forms of IDF are described by Cramer etal. [5] and Lewis [13]. To state the formulation, letus assume that the system consists of two subsystemsand let us write the disciplinary analyses AI and AIin more detail as follows:
xlX.v
where Uij represent the interdisciplinary flow of in-formation from discipline j to discipline i. Then theIDF formulation introduces auxiliary variables #12and «2i , and the optimization problem can be statedin the following form:
m n m z esubject to
where, Ui(x,xi2) and ^2(2, £21) are computed bysolving the disciplinary equations
Ai(x,Ui(x,Xu),Xi2) = 0
A2(x,X21,U2(x,X2l)) = 0.
Thus, the IDF formulation is a single-level, nonlinearoptimization problem.
On the positive side, IDF is trivially equivalent tothe original nonlinear programming problem and is
thus easy to analyze. That is, the optimality condi-tions of the original problem hold for the IDF formu-lation. The convergence properties of optimizationalgorithms applied to IDF are those of the algorithmsapplied to conventional nonlinear programming prob-lems. Given a good solver for equality constrainedoptimization problems, the method is expected to beefficient.
IDF assures full use of disciplinary analysis soft-ware.
Similarly to CO, IDF is intended for problemswith small bandwidth of interdisciplinary coupling,and the problem of decomposition is similar to thatof CO. Unlike CO, IDF is not limited to problemswithout general constraints at the system level.
Also similarly to CO, formulations that arise fromIDF have more optimization variables that those aris-ing from MDF.
Importantly, although IDF maintains autonomywith respect to analyses, it lacks CO's autonomy withrespect to disciplinary optimization. -That is, whilethe analyses are performed autonomously during theanalysis stage, the coupling is restored during the op-timization step computation. This brings back thedifficulties of integration.
Another weakness of IDF is its treatment of thedisciplinary constraints. They are either ignored indescriptions of the formulation or simply relegated tothe system, despite the need to handle disciplinaryconstraints at the disciplinary level—a wish usuallyexpressed by practitioners of MDO.
Approach
We selected initial sets of ten problems, threemethods, and a set of metrics to record during datacollection. During Phase I, all methods were imple-mented in iSIGHT, which is a computational frame-work for multidisciplinary design optimization, pro-duced by Engineous Software, Inc. The data havebeen compiled and preserved for evaluation. TheiSIGHT problem description files are available to any-one who would like to duplicate the results or improveon them.
There has been much recent work in the areaof approximations for engineering optimization, andCollaborative Optimization has been reported ([15],[14]) to show improvement due to the use of responsesurface methodology. In this study, we have not at-tempted to use approximations.
Implementation and Its Limitations
In this study, we are concerned with evaluating
MDO methods, which is a notion composed of "for-mulations" and "algorithms". A formulation is astatement of the problem to be solved by an opti-mization algorithm. An algorithm is then a set ofsteps performed to solve a formulation. The two arevery much interrelated, and are even difficult to dis-tinguish for some methods. MDF, CO, and IDF areall formulations, but they each may be implementedin a variety of ways and their evaluation will be af-fected by the use of specific optimization algorithms.This makes the evaluation extremely difficult. How-ever, trends in performance can still be discerned.
The implementation of the methods and the prob-lems, as well as the software used to solve the prob-lems, have a direct bearing on the performance of themethods. Moreover, we understand completely thatto use a particular method to its full advantage, es-pecially a method for solving problems as difficult asthose that arise in MDO, one will, by necessity, per-form a significant amount of fine-tuning, both in theproblem statement and in all areas of implementa-tion.
Taken to its extreme, however, this principlemeans that, given any problem and any initialmethod, with a sufficient amount of time the methodcan be "fine-tuned" so that the problem can be solvedwith reasonable efficiency. The danger of such anextreme is that what practically amounts to a newmethod has to be developed for each problem. Thequestions are then whether an original method canstill be discerned after such an exercise and just howmuch of fine-tuning is required to produce reasonableperformance.
An important consideration in making decisionsabout fine-tuning is the available resources for solv-ing a specific problem. If an expensive problem willbe solved many times, it makes sense to fine-tune themethod and the problem to assure optimal perfor-mance. However, in some projects, it is anticipatedthat the problem under consideration will be solvedonly a few times due to its extreme expense. In suchcases, one cannot afford fine-tuning and one has to bereasonably certain of a method's robustness before itsapplication.
With this in mind, we made a decision to avoidfine-tuning as much as possible in the first phase ofthe study, in order to obtain an idea of what a reason-ably intelligent user, who is not a method developer,will face when implementing a specific method. Infurther studies, a higher degree of fine-tuning the im-plementation is planned.
The choice of optimization software for solving theformulated problems and subproblems was limited bythe set available in iSIGHT.
Within that set, the choice was somewhat subjec-tive and dependent on experience. At this stage, wehave not conducted conclusive studies on the sensi-tivity of a particular formulation to the choice of theoptimizer.
All three methods were implemented in theiSIGHT framework, using MDOL, the iSIGHT MDOLanguage. The implementation was both eased andmade more difficult because the methods were testedwithin the iSIGHT framework. On the one hand,the framework provided a unified approach to testing.On the other hand, iSIGHT is designed for handlinglarge, "production" problems, and there was signif-icant overhead connected with coding smaller prob-lems in the early stages of the study. In addition,we were restricted to using the optimization softwareavailable in iSIGHT for solving the optimization sub-problems. Overall, we feel that at this initial stage,the benefits of using iSIGHT outweighed the disad-vantages of the additional work necessary to incorpo-
rate the problems and the methods into the frame-work.
Implementation within iSIGHT also affected theperformance of the methods on some of the problems,in that stand-alone implementation has resulted in amuch faster convergence. Future method evaluationstudies are planned both in and out of iSIGHT.
Given all the limitations, however, our results areavailable for examination, as is all the software thatproduced the results. Should the authors of a partic-ular method or other researchers find that a certaintrend we discerned has to do strictly with our imple-mentation and not with the innate properties of thatmethod, we welcome suggestions on how to make thetests more informative.
Finally, promising methods evolve continually,their implementation improves with increased under-standing of their workings, and the difficulties appar-ent at one stage of their development may disappearat a more mature stage.
The ten problems selected for Phase I of the studyare listed in Table 1. Earlier references exist for mostof these problems. We indicate the sources we used.The reference for the MDOB Test Suite [16] is indi-cated for all problems directly downloaded from thatset, but the primary reference for the problem is theother reference.
At this stage, the problems are not representativeof realistic MDO applications. However, to actuallysolve a realistic MDO problem may require months oreven years, which would somewhat incapacitate theeffort. Therefore, we chose several tractable prob-lems as an initial set we could easily handle. Someof the problems, such as the Conceptual Ship DesignProblem, the Electronic Packaging Problem, and theHub Frame Problem, exhibit some of the salient fea-tures of realistic MDO problems, albeit not their size,complexity, or expense.
While a method cannot be judged applicable toa realistic problem just because it performs well ona toy problem, if it does not perform well on rela-tively simple problems, that is an indication that themethod must be further studied before attempting touse it for solving realistic problems.
During future studies, we plan to obtain numeri-cal data on solving more realistic problems with themethods that are deemed promising after this initialtesting. A summary of the problem dimensions isgiven in Table 2. In cases where the problems werefound inappropriate for being solved by a specificmethod, the table contains "—".
Note that the constraints of the system-level prob-lem for CO are just the compatibility constraints.
Metrics
We have considered a number of metrics in eval-uating the performance of the methods. Some of themetrics are objective and quantitative, the others,more subjective and qualitative. Discussion of someof the metrics follows.
Generality. This is a measure of applicability ofa method to various classes of problems. Our prob-lem pool is small at this stage, but as it grows, wewill classify the problems by their size in terms of vari-ables and constraints, smoothness, nonlinearity, com-plexity, strength and bandwidth of coupling, presenceof several objectives, and other features.
Robustness. Given that each method is designedfor solving a specific class of problems, this metricevaluates the capacity of the method to provide a so-lution or a "satisfactory" point for that class of prob-
lems for a variety of perturbed data. For the initialset of problems that we considered, we measured ro-bustness by attempting solution from several startingpoints.
Another, somewhat subjective, but importantmeasure of robustness we considered was the amountof fine-tuning required to make a method produce ananswer. For instance, we considered restructuring theproblems in a number of ways, such as the incorpora-tion of slack or squared slack variables, or changingthe tolerances on constraints and optimality.
Performance. Performance is a very complex is-sue, strongly related to implementation, problem for-mulation, and the nature of the problem. In PhaseI of the study, we quantified performance as workperformed by each method during every optimizationprocedure.
The recorded work consists of the total number ofanalyses, with each "disciplinary" analysis countedas one function evaluation. Function evaluationscomputed during finite-difference derivative compu-tations are included. In particular, for MDF, wereport the total number of analyses, including theaverage number of analyses taken during fixed-pointiterations in order to compute MDA, and those neces-sary to compute the finite-difference derivatives. ForCO, we report the number of iterations taken by thesystem-level optimization problem, as well as the sumof the function evaluations in each subsystem, includ-ing those required for finite-difference computations.For IDF, we report the total number of functionevaluations, including those taken for finite-differencecomputations, times the number of the subsystems.
Although the dimensions of the design space dif-fer for MDF, CO, and IDF, by reporting the totalnumber of analyses, we provide as complete a mea-sure of work as possible, given the serious limitationsimposed by a specific implementation of the methodsand the problem formulations.
Example: Electronic Packaging
The Electronic Packaging problem ([16], [17]) con-siders a circuit that comprises two resistors mountedon a heat sink, resulting in two coupled subsystems:electrical and thermal. Operating temperatures af-fect component resistance, while resistance, in turn,influences the temperature. The objective function iswatt density. The constraints require the operatingtemperatures of resistors to be below a threshold tem-perature and the currents through the two resistorsto be equal. The problem has eight design variables,thirteen state variables, two parameters (voltage and
where the first constraint represents the branch cur-rent equality, while the next two inequalities are thecomponent reliability constraints.
The eight design variables are as follows: xi and#2 are the heat sink width (m) and length (m); £3and #4 are the fin length (m) and width (m): z5 andx^ are the nominal resistance (fl) of components 1and 2, respectively, at temperature T°; x@ and x$ arethe temperature coefficients (°K~l) of the electricalresistance components 1 and 2.
The state variables are as follows: MI is the wattdensity (watt/m3); u^ and Us are the resistances (fi)at temperatures TI and TI; 1*4 and u$ are currents(amps) in resistors 1 and 2; UG and m representpower dissipation (watts) in resistors 1 and 2; ug,ug, and UIQ are the total circuit current, resistance,and power, respectively; MH and ui2 are the compo-nent temperatures, TI and TI (°C), of resistors 1 and2; and finally, u\z is the heat sink volume (m3).
The states are described by the following equa-tions:
where the states MH and «i2 are rather involved ex-pressions. The code for this problem can be foundat the MDO Test Suite web site accessible from theMDOB Homepage.
The reported results are for cases initiated fromthe same starting points, for all methods.
Fig. 1 depicts interdisciplinary interactions in theproblem.
The following three subsections deal with solvingthe problem by using the three methods under study.
The following conventions are used in all data ta-bles.
• "Convergence" means the attainment of a pointthat satisfies the Karush-Kuhn-Tucker (first-order necessary) conditions for a critical pointof the problem within the maximum allowablenumber of iterations (set at 10,000). However,when a particular problem was solved, knownsolutions were always found rather than justcritical points of the problem.
• "Work" is defined as the total number of disci-plinary analyses performed. For MDF, work isequal to the number of calls to the function eval-uation procedure times the average number offixed point iterations per MDA times the num-ber of "disciplines". For CO, the total numberof function evaluations at the subsystem leveland the number of system iterations is reported.For IDF, we report the number of disciplinaryfunction evaluations times the number of "dis-ciplines" .Function evaluations done during finite-difference computations are included in allwork.
• The superscript "F" added to the value of thefinal objective indicates failure to converge toa critical point within the allowable number ofiterations.
• In all tables, "-" means that the problem wasfound inappropriate for a particular method,and the tests were not run.
MDF Implementation
The MDF formulation has eight design variables,three explicit constraints, and its statement is thatof the original problem. The problem was solvedusing the method of feasible directions in iSIGHT.The derivatives were computed using finite differ-ences with the step size of 1%. The termination cri-teria included the satisfaction of the first-order neces-sary conditions for optimality, tolerances for the ab-solute and relative change in the objective functionduring several successive iterations, and the maxi-mum number of allowable iterations. The MDF re-sults are summarized in Table 3.
CO Implementation
For the CO approach, the system-level optimiza-tion problem is given by
maximize £1subject to ci < 0.0001
c-i < 0.0001.
The system-level problem has five design variables:£i> £2, £3, £11, and £i2.
The Thermal Subsystem optimization problem is:
minimize c\subject to hi = 0.0
9i = MII-85.0 < 0.09i = ui2-85.0< 0.0,
where
where the variables of the Thermal Subsystem areXj, i = 1, . . . , 4, M2, and 1/3.
The Electrical Subsystem optimization problemis:
minimize c2subject to gi = uu - 85.0 < 0.0
9i = ui2 - 85.0 < 0.0,
where
where the variables of the Electrical Subsystems are
The system-level optimization problem was solvedby a combination of the exterior penalty functionmethod and the method of feasible directions avail-able in iSIGHT. The subproblems were solved us-ing a sequential quadratic programming algorithm-DONLP-implemented in iSIGHT. The CO results aresummarized in Table 4.
IDF Implementation
The compatibility constraints of the IDF formu-lation were implemented as inequalities, since thatseemed to produce better results, given the optimiza-tion software available in iSIGHT. The optimizationproblem is:
maximize MIsubject to ci < 0.0001
c2 < 0.0001hi — M4 — MS — 0.0
9i =un -85.0< 0.09i = Mi2-85.0< 0.0,
where the twelve design variables are a;,, i = 1,. . . , 8,and four coupling variables £2, £3, £11, and £i2.
The Thermal Subsystem evaluates MI, hi, and
Cl = (t i l l -&l)2 + («12-£l2) 2 ,
while the Electrical Subsystem evaluates
The optimization problem was solved by a com-bination of the exterior penalty function method andthe method of feasible directions available in iSIGHT.The IDF results are summarized in Table 5.
The initial set of test problems was solved usingthe iSIGHT MDOL language based implementationof MDF, CO, and IDF from several starting points.The ratio of the successful runs to the number of at-tempted runs is summarized in Table 6. By a "suc-cessful run" we mean one that attains a critical pointof the problem, to a specified degree of tolerance,within the allowable maximum number of iterations.
All runs were originally planned to be attemptedfrom twelve starting points. Due to time limitations,all the data could not be generated to date. This
explains the difference in the numbers of attemptedruns among the problems and methods shown in Ta-ble 6.
During the problem formulation, we discoveredthat some of the problems did not have a suitablestructure for the CO approach or for the IDF ap-proach. We did not discard these problems, as theymay prove useful for testing other methods.
Given the small problem sample, and the limi-tations mentioned earlier, we cannot make definitivestatements about the methods under study. However,even this sample has demonstrated that the methodproperties predicted by analysis tend to hold.
Table 6: Summary: number of successful runs / number of attempted runs.
ProblemMethodMDFCOIDF
1
610156269530
2
220198728976
3
6101785382
4
812102-
5
3234837544
6
502440125932
7
8730691058-
8
245--
9
1574--
10
1353--
Table 7: Summary: representative number of analyses for convergence.
ProblemMethodMDFCOIDF
1
667130659640
2
275180056019
3
10252983406
4
772102*-
5
10626837*694
6
303540125*1071
7
6887691058*-
8
245--
9
1547--
10
1353--
Table 8: Summary: average number of analyses for convergence. Note: numbers marked by an asterisk arebased on a single data point.
In particular, the following trends were observed withrespect to convergence and the number of functionevaluations.
MDF
The method converged to known solutions for themajority of cases. However, for two of the problems,the use of the fixed-point iteration to attain MDAcaused very unstable behavior, and the optimizer wasunable to converge from many starting points.
Given the small sample, the number of functionevaluations is not conclusive. However, for the cur-rent dataset, MDF consistently requires fewer func-tion evaluations than does CO (with the exceptionof one problem). MDF did better than IDF for twoproblems, and worse than IDF for three problems.Interestingly, IDF did worse on problems that 1 and2, that exhibit more typical MDO structure than doproblems 5 and 6, on which MDF performed poorly.
CO
The method was used to solve seven of the ten
problems. The system-level compatibility constraintswere actually implemented not as equalities, but asinequalities, as this alleviates some of the numericaldifficulties associated with the equality constrainedsystem-level problem ([!]). Some adjustment of thetolerances on the system-level constraints and on thelower-level convergence criteria had to be done. Ageneral qualitative trend can be stated as follows:larger tolerances on the system-level constraints andsmaller tolerances on the lower-level convergence cri-teria lead to better chances for attaining overall con-vergence.
Compared to the other two methods, CO typi-cally required a significantly larger number of func-tion evaluations. It was more efficient than MDFonly for problem 5, and it was always less efficientthan IDF.
CO also seemed to be less robust than the othermethods, failing to find a critical point more fre-quently. On the one hand, this feature confirms theanalytical properties of CO. On the other hand, thismay be explained by our formulation of the prob-lems, and by not spending a large amount of time onfine-tuning the method, which was intentional, as we
mentioned.Again, for a large test set, with a more evident
MDO structure, the difference in performance maybe less pronounced.
IDF
Similarly to CO, the compatibility constraints forIDF were relaxed to inequalities.
IDF performed consistently better than CO onour small sample of problems. However, one mustremember that IDF is less attractive than CO withrespect to the ease of integration and its handling ofthe disciplinary constraints.
A representative number of analyses required foreach of the methods, for our test set, is summarizedin Table 7, while the average number of analyses isgiven in Table 8.
General Comments
During the testing, we also considered the lesstangible metrics, such as the ease of implementation.Here, the MDF was not typical because of the na-ture of our problems. It was the easiest to implementbecause our problems are small and relatively sim-ple. However, in real applications, the integration ofMDA can be a formidable task.
For our small test set, CO and IDF were aboutequally easy to implement. For large, more realisticapplications, we expect that CO will be easier to im-plement, because of the way it handles disciplinaryconstraints.
The current and future method studies are pro-ceeding on several fronts. Information is accumulatedon other methods, such as the multilevel methods(MAESTRO [23], [24] [25]) and concurrent subspaceoptimization methods (CSSO [26], [27]); testing pro-cedures are being fine-tuned for the methods evalu-ated in Phase I of the study; larger, more realisticproblems are being added to the problem test set.
In conclusion, while the results are by no meanscomplete, we have found this systematic study ofmethods very informative. The tests in general con-firmed the expected numerical properties; however, asignificant amount of further testing would be nec-essary to explain the numerical behavior of methodsdefinitively. We invite other researchers to contributeto accumulating systematic data, that will eventuallylead to a set of practical guidelines for the use of MDOmethods (please see the method evaluation web siteaccessible from the MDOB Homepage).
Acknowledgment s
The authors would like to thank R. Haftka for hissuggestions.
References[1] N.M. Alexandrov and R.M. Lewis. Analytical
and computational aspects of collaborative opti-mization. Engineering Optimization, 1998. Sub-mitted.
[2] G.R. Shubin. Application of alternative mul-tidisciplinary optimization formulations to amodel problem for static aeroelasticity. Techni-cal Report BCSTECH-93-022, Boeing ComputerServices, BCS Technology, December 1993.
[3] D. Haim, A.A. Giunta, M.M. Holzwarth, W.H.Mason, and L.T. Watson. Comparison of opti-mization software packages for an aircraft mul-tidisciplinary design optimization problem. De-sign Optimization, 1998. Submitted.
[4] S. Kodiyalam. Evaluation of methods for multi-disciplinary design optimization, Phase I. Tech-nical report, NASA Contractor Report, 1998.
[5] E.J. Cramer, J. E. Dennis, Jr., P.D. Frank, R.M.Lewis, and G.R. Shubin. Problem formulationfor multidisciplinary design optimization. SIAMJournal on Optimization, 4(4):754-776, Novem-ber 1994.
[6] R.D. Braun. Collaborative Optimization: An ar-chitecture for large-scale distributed design. PhDthesis, Stanford University, May 1996. Depart-ment of Aeronautics and Astronautics.
[7] D. A. Wismer, editor. Optimization Meth-ods for Large-Scale Systems with Applications.McGraw-Hill, New York, 1971.
[8] L.S. Lasdon. Optimization Theory for Large Sys-tems. Macmillan, 1970.
[9] R.D. Braun, P. Gage, I. Kroo, and I. Sobieski.Implementation and performance issues in col-laborative optimization. In Proceedings of theSixth AIAA/NASA/ISSMO Symposium on Mul-tidisciplinary Analysis and Optimization, Belle-vue, WA, Sept. 1996.
[10] H.M. Adelman, J.L. Walsh, and J.I. Pritchard.Recent advances in integrated multidisciplinaryoptimization of rotorcraft, AIAA paper AIAA-92-4777-CP. In Proceedings of the Fourth
AIAA/USAF/NASA/OAI Symposium on Mul-tidisciplinary Analysis and Optimization, pages710-721, September 1992.
[11] J.L. Walsh, K.C. Young, J.I. Pritchard, H.M.Adelman, and W.R. Mantay. Integrated aerody-namic/dynamic/structural optimization of heli-copter rotor blades using multilevel decomposi-tion. Technical Report NASA TP 3465, NASALangley Research Center, January 1995.
[12] J.L. Walsh, K.C. Young, J.I. Pritchard, H.M.Adelman, and W.R. Mantay. Multilevel de-composition approach to integrated aerody-namic/dynamic/structural optimization of heli-copter rotor blades. Technical Report NASA TM109084, NASA Langley Research Center, May1994.
[13] R.M. Lewis. Practical aspects of variable re-duction formulations and reduced basis algo-rithms in multidisciplinary design optimization.In N.M. Alexandrov and M. Y. Hussaini, editors,Multidisciplinary Design Optimization: State-of-the-Art, Philadelphia, 1997. SIAM.
[14] I.P. Sobieski and I.M. Kroo. Collaborative op-timization using response surface estimation. InProceedings of the 36th Aerospace Sciences Meet-ing and Exhibit, Reno, NV, January 1998. AIAAPaper AIAA-98-0915.
[15] S. Kodiyalam, L. Swenson, and B. Stehlin. Mul-tidisciplinary design optimization with object-oriented product modeling. In Proceedings ofthe Conference on Optimization in Industry, or-ganized by Engineering Foundation, NSF, andNASA Langley, Florida, 1997.
[16] S.L. Padula, N.M. Alexandrov, and L.L.Green. MDO Test Suite at NASA LangleyResearch Center. In Proceedings of the SixthAIAA/NASA/ISSMO Symposium on Multidis-ciplinary Analysis and Optimization, Bellevue,WA. AIAA, 1996.
[17] J.E. Renaud. An optimization strategy for mul-tidisciplinary systems design. In Proceedingsof the International Conference on EngineeringDesign, August 1993.
[18] G. Kott, G.A. Gabriele, and J. Korngold. Appli-cation of multidisciplinary design optimization
to the power stage design of a power converter.ASME Advances in Desing Automation, 2, 1993.
[19] W.-C. Li. Monotonicity and Sensitivity Analysisin Multilevel Decomposition-Based Design Opti-mization. PhD thesis, University of Mariland,1989.
[20] B. Averick, R.G. Carter, J.J. More, and G-L.Xue. The MINPACK-2 test problem collection.Technical Report ANL MCS-TM-150, ArgonneNational Laboratory, Argonne, Illinois, 1992.
[21] J.E. Dennis, D.M. Gay, and P.A. Vu. A new non-linear equations test problem. Technical Report83-16, Rice University, Houston, Texas, 1983.
[22] C.A. Floudas and P.M. Pardalos. A Collectionof Test Problems for Constrained Global Opti-mization. Lecture Notes in Computer Science,Volume 455. Springer-Verlag, Berlin; New York,1991.
[23] N.M. Alexandrov. Multilevel algorithms for non-linear equations and equality constrained opti-mization. PhD thesis, Rice University, Depart-ment of Computational and Applied Mathemat-ics, May 1993. Also available as Technical Re-port TR93-20.
[24] N.M. Alexandrov and J. E Dennis, Jr. Multi-level algorithms for nonlinear optimization. InJ. Borggaard, J. Burkardt, M. Gunzburger, andJ. Peterson, editors, Optimal Design and Con-trol, pages 1-22. Birkhauser, 1994.
[25] N.M. Alexandrov. Multilevel methods for MDO.In N.M. Alexandrov and M. Y. Hussaini, editors,Multidisciplinary Design Optimization: State ofthe Art, pages 79-89. Society for Industrial andApplied Mathematics, Philadelphia, 1997.
[26] J. Sobieszczanski-Sobieski. Optimization by de-composition: A step from hierarchic to non-hierarchic systems. Technical Report TM101494, NASA, Hampton, Virginia, September1988.
[27] J.E. Renaud. A concurrent engineering approachfor multidisciplinary design in a distributed com-puting environment. In N.M. Alexandrov andM.Y. Hussaini, editors, Multidisciplinary DesignOptimization: State of the Art. SIAM, 1997.
