Copyright ©1996, American Institute of Aeronautics and Astronautics, Inc.

AIAA Meeting Papers on Disc, 1996, pp. 190-195A9626818, AIAA Paper 96-1339

Neural network approximator with a novel learning scheme for designoptimization with variable complexity data

Ram GurumoorthyGE Corporate R & D Center, Schenectady, NY

Srinivas KodiyalamLockheed Martin Advanced Technology Center, Palo Alto, CA

IN:AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and MaterialsConference and Exhibit, 37th, Salt Lake City, UT, Apr. 15-17, 1996, Technical Papers. Pt. 1(A96-26801 06-39), Reston, VA, American Institute of Aeronautics and Astronautics, 1996,

p. 190-195

A novel learning scheme for training neural networks is proposed. The trained network is then used for functionapproximation during the numerical optimization process. The learning scheme trains the network with data of varyingcomplexity, including data that has only zeroth-order information; for data including first-order information (gradientsof responses with respect to the design parameters), it uses the combined information (response values and itsgradients) for a better approximation. The learning scheme and its function approximation capability for designoptimization are demonstrated on two realistic examples. (Author)

Page 1

96-1339

A96-26818AIAA-96-1339-CP

Neural Network Approximator with a Novel Learning Schemefor Design Optimization with Variable Complexity Data

Ram Gurumoorthy*General Electric Corporate R&D Center

Control Systems and Electronic Tech. LabSchenectady, New York 12301.

bySrinivas Kodiyalam**

Lockheed Martin Advanced Technology CenterStructures Laboratory

Palo Alto, California 94304.

AbstractA novel learning scheme for training neural networks isproposed. The trained network is then used for functionapproximation during the numerical optimizationprocess. The learning scheme trains the network withdata of varying complexity, including data that has onlyzeroth order information and when the data includes firstorder information (gradients of responses with respect tothe design parameters) it uses the combined information(response values and its gradients) for a betterapproximation. The learning scheme and its functionapproximation capability for design optimization arcdemonstrated on two realistic examples.

1.0 IntroductionThe application of neural networks as universalapproximators has been the focus of tremendous amountof activity in the past few years [1,2,3]. Neural networkshave been used in design optimization fcrapproximating and modeling responses of bothmemoryless systems and dynamical systems withmemory. These neural networks primarily use functionvalues (zeroth order information) while learning thedesign response surfaces. The authors havedemonstrated that a neural network with a modifiedbackpropagation learning scheme, using designsensitivity or gradient information, in addition tofunction values, can provide for more accurateapproximations of the response surface and quickerconvergence of the learning process [5].

Detailed design optimization of realistic structuralsystems, for example, aerospace and automotivesystems, usually involves multiple disciplines and dataof varying complexity. Not all of the physical disciplineand manufacturing process solvers provide for bothzeroth order (response values) and first order (responsegradients) information for use with the optimizationprocess. This paper addresses such variable complexitydata problems encountered in design optimization ofstructural systems.

In this paper, a novel learning scheme for trainingneural networks is proposed. This scheme trains the

network with data of varying complexity, includingdata that has only zeroth order information and whenthe data includes first order information (gradients cfresponses with respect to the design parameters) it usesthe combined information (response values and itsgradients) for a better approximation. The networktrained using the hybrid learning scheme is then usedfor function approximation during the numericaloptimization process.

The hybrid learning scheme presented here exploits theassets of the standard backpropagation learning scheme,such as, robustness of approximation and blockpresentation of input data for improving convergence,when training the network with data that represents justzeroth order information, while exploiting theadvantages of the modified backpropagation learningscheme [5] when training the net for those data that inaddition include the gradient information. The hybridlearning scheme facilitates the use of test/experimentaldata as part of network learning.

2.0 Network Architecture and Learning SchemeThis paper considers a "multi-layer feedforward"network.

hpus

Hd dan LayerFigure 1: Feedforward neural net with single hidden

layer

* Staff Engineer, Member AIAA** Senior Staff Scientist, Senior Member AIAA.

190

For data with just the zeroth order information, a multi-layer feedforward network with standardbackpropagation learning has been shown to be anefficient universal approximator [1,4], There have beenmany additions to, or adaptations of, the basicbackpropagation scheme, like, adaptive learning rate,learning with momentum [1], for improvingconvergence of the learning scheme, but this paper doesnot go into the details of these additions. In this paper,the authors restrict themselves to a single hidden layerfeedforward network, as shown in Figure 1. Therestriction to a single hidden layer is not a limitation ofthe hybrid learning scheme and the new training schemecan be used on networks with multiple hidden layersalso.

2.1 Hybrid Learning Scheme - applied to a singleneuron:In order to illustrate the philosophy of the proposedlearning scheme, a net with a single neuron isconsidered. Figure 2 shows a single tansigmoid neuronnet.

f

.Figure 2: Neural net with a single neuron

The output of the net is given by:

° = f(wu)

where,

/(*) = •l + e-2x -1

An energy function, P, is defined as follows:P = 0.5(r-o)2=0.5e2

where, T is the target value of the output. The objectiveof the training scheme is to make the difference betweenthe target output and the actual output to be zero. Thiscan be achieved by maintaining the gradient of theenergy function to be negative, as this drives P to zero,as P is always positive.

dP = (T-o)\ ——

[ / — \ T

-(\-o2\udw + \ — -(\-o2\w\du\\ / \du \ I ) J

To make dP to be negative, that is to have the trainingscheme reduce the difference between the target outputand the actual output we can use different weight updatelaws. Under the standard backpropagation learningscheme, the weight update is defined as:

= (T-o)(\-o2\u

Under the modified backpropagation learning scheme[5] - that uses the gradient of the response to the inputparameters - the weight update is redefined as:

= (T-o)(l-o2)u + \- 11-0'

dTldu_duju

The term inside the square bracket in the weight updatelaw is the modification to the standardbackpropogation. This modification has been shown tospeed up the learning process convergence and alsoimprove the accuracy of the approximation [5].

In situations with varying complexity data, let usassume that the data is categorized into two families -ul (data with zeroth and first order information) and uO(data with just zeroth order information), Tl and TOthe corresponding target outputs, and ol and oO thecorresponding neural network outputs. With the hybridbackpropagation learning scheme the weight update isdefined as:

.0

+ r'-t dTl

1-2 A,1 ..1

This update exploits the advantages of the modifiedtraining scheme when the first order information isavailable. In addition, a adaptive learning rate (lr) isimplemented in both the nonnal backpropagation termas well as the modification term.

+/r2(r'-o')(i-(o')2y+/r3dTl du1

The learning rates are adapted independently, based onthe change in the sum squared error in any epoch.

In the case of multiple neurons and multiple layers,there is an additional hybrid structure to the trainingscheme - a block and sequential treatment in thepresentation of data for weight update in each iteration.This is explained in the following section.

2.2 Hybrid Learning Scheme - Data Presentation:It has been shown in References [1,2] that in a standardbackpropagation learning scheme, a block presentationof the data during weight update is more efficient (faster

191

Block presentation of input datasets

Is change in errorwithin specified Iimits2

Sequential presentation of input datasets

Present u' datasets (one by one)

s change in errori thin specified limi

NextIteration

Figure 3: Flow of the Hybrid Neural Network Learning Scheme

Yes

Learning Complete

192

convergence) than a sequential presentation of data andupdate of weights. In the modified learning scheme, toutilize the gradient information efficiently the data hasto be presented sequentially.

In the hybrid algorithm both these features are utilized:all the data with just zeroth order information isprocessed in block while the data with zeroth and firstorder information is processed in a sequential manner.The flow chart in Figure 3 shows the method of datapresentation used while training with the hybridlearning scheme.

2.3 Neural Network Approximation ExampleRosenbrock's Valley Function:The example considered here is the approximation ofthe Rosenbrock's valley function, given as:

SlandMd NN AMTOimatlan 01 HoMflbnck Vatey FunctM

F(XI ,X2 ) = 100(*2 -

The domain of operation was chsen to be [-1.3 to 1.3]for xi and [-0.5 to 1.3] for X2- The training data setconsits of 1 65 data points evenly spaced, 83 of whichcontains just the function value, while the rest containsthe gradient information in addition. The figures belowshows a surface plot of the actual rosenbrock valleyfunction, and the neural network approximations (boththe standard network and the hybrid network). This isthe after the networks had converged to under 4% totalsum squared error (TSSE).

As can be seen from the figures, both the neuralnetworks approximate the rosenbrock valley functionaccurately. The neural network with standardbackpropagation training took 88200 epochs toconverge to under 3% TSSE, while the network withthe hybrid backpropagation training took 79200 epochsto converge to the same error.

Figure 4: Rosenbrock Valley Function

Figure 5: Standard Backpropagation Trained NetworkApproximation

Hybrid NN Approximation of RcMflbrock Valley Function

Figure 6: Hybrid Backpropagation Trained NetworkApproximation

3.0 Design Optimization MethodologyThe neural network approximtaion (NNA) basedoptimization methodology is shown in Figure 7. Thismethodology is similar to the well established TaylorSeries approximation (TSA) based optimization. Thenetwork is trained on both test data that includes justzeroth order information and analysis data that includesboth the zeroth and first order information. The neuralnetwork trained with the hybrid learning sheme thatuses the gradient information when available, is used inthe optimization for the response evaluations. This isthen compared with a neural network trained withstandard backpropagation scheme that uses just thezeroth order information in all the data.

The design optimization problem is posed asconstrained optimization problem of the following form:

To find the set of design variables, X, thatMinimizes/Maximizes: F(X)

193

( Training A>w Designs s

^s

1

/Improved Design

Finite ElementAnalysis

ResponseScreening

*-Design

SensitivityAnalysis

External Iteration

Neural NetworkTraining

InternalIteration

Figure 7: NN Approximation based Design Optimization Methodology

Subject to:gj(X) < 0; j = 1, number of inequality constraintsx't < x, < x'; bounds on design variables

The method of feasible directions, programmed in ADScode, is used to solve the constrained optimizationproblem.

3.1 Design Optimization Example - Aircraft EngineGuide Vane:

The aircraft engine composite guide vane of Reference[5] is used as the design example. In this example, bothtest and analysis data are used for training the neuralnetwork. The design problem addressed here is todetermine the composite ply orientations of the guidevane that would produce a stiffer structure. The stiffnessrequirements are specified interms of frequencyrequirements and producibility requirements of thecomposite ply layup are also an integral part of theguide vane design.

The guide vane is modeled using 990, 8-nodedisoparametric finite elements. The composite ply layupis symmetric and is given by:

3,61, 9^, 9t J, [ 92,0,, 0,,

The ply layup consists of the 2 generation sets with 4unique ply angles (61,62,83,64). Some of theproducibility requirements on the composite plyorientations, include:(i) The plies be layered as "3 ply" packs;(ii) The first and third ply orientation angle in eachpack be the same (......); and(Hi) All plies of constant thickness of approximately 5mils.

The design requirements on the guide vane frequencies,include:

Mode 1 (1st flex frequency): (OtF > l.OtfzMode 6 (3rd flex frequency): 4.0 < (837- < 6. IHzMode 7 (3rd torsion freq): 4.0 < ft)37- < 6.7/ftMode 14 (two stripe frequency): 6)2$ > 9.8/ft

The frequencies are normalized with respect to first flexfrequency corresponding to the base line layup.

A feedforward network with 10 nodes in the hiddenlayer is used. For the network with standardbackpropagation learning, all the 29 datasets (onlyzeroth order information) is used, while for the modifiedbackpropagation learning, only the 29 datasets alsoinclude the first order (gradients of frequencies withrespect to ply angles) information. The gradients areevaluated using a semi-analytical sensitivity procedure.For the hybrid backpropagation learning scheme, 7datasets only contain zeroth order information while theremaining 22 datasets include zeroth and first orderinformation. The results for the 3 different trainingschemes and the associated design optimization arepresented in Table 1.

The hybrid training presents the zeroth order data inblock, and the first order data sequentially. The resultsprovided in Table 1 for the standard and modifiedbackpropagation presents all data (both zeroth and firstorder) sequentially. This accounts for the hybridtraining requiring slightly lesser number of trainingepochs than the modified backpropagation training.

4.0 Summary:This paper investigates a hybrid backpropagationlearning algorithm for handling variable complexity

194

data commonly encountered in multidisciplinary designoptimization problems. Not all the disciplinary solverscan provide for both zeroth and first order (gradient)data. The hybrid learning algorithm can accomodate acombination of datasets, with and without first orderinformation, for training the network.

It is reasonable to conclude that the quality ofapproximations is better with the learning schemesusing gradient data. Within the schemes using thegradient data, the efficiency of approximation isimproved by the hybrid training scheme, compared tothe modified backpropagation training scheme, and, thisis primarily due to the presentation of zeroth order datain block.

5.0 References:1. Zurada, J.M., Introduction to> Artificial NeuralSystems, West Publishing Company, Minnesota,1992.

2. Kosko, B., Neural Networks for Signal Processing,Prentice Hall, 1992.

3. Hunt, K.J., Sbarbaro, D., Zbikowski, R., andGawthrop, P.J., Neural Networks for control systems:A survey," Automatica, 28, 1992, pp. 1083-1122.

4. Hajela, P., and Berke, L., NeurobiologicalComputational Models in Structural Analysis andDesign," Computers and Structures, Vol. 41, No. 4,1991, pp. 657-667.

5. Kodiyalam, S., and Gurumoorthy, R., NeuralNetworks with modified backpropagation learningapplied to Structural Optimization," To appear inAIAA Journal, March 1996. Also, AIAA ConferencePaper: 95-1370.

Data

e,e,e,e4

C»IF(0.1 F

0)JT

C02S

Baseline

0+45+90-451.05.76.09.9

No. of Datasets used in NNTraining

No.of Epochs for NN TrainingConvergence

NN Error Goal

Standard NNbased Opt.

-5+30+90-25

1.0585.525.929.9629 uu

233822

0.05

ModifiedNN based

Opt.-5

+30+90-15

1.0625.395.879.9829 u'

244658

0.05

HybridNN based

Opt.-5

+30+90-15

1.0625.395.879.9822 u1

+ 7u°200891

0.05

Table 1: Aircraft Engine Guide Vane Results (Normalized values)

195