2398 IEEE TRANSACTIONS ON MICROWAVE THEORY AND TECHNIQUES, VOL. 54, NO. 6, JUNE 2006
State-Space Dynamic Neural Network Techniquefor High-Speed IC Applications:Modeling and Stability Analysis
Yi Cao, Student Member, IEEE, Runtao Ding, and Qi-Jun Zhang, Fellow, IEEE
Abstract—We present a state-space dynamic neural net-work (SSDNN) method for modeling the transient behaviors ofhigh-speed nonlinear circuits. The SSDNN technique extends theexisting dynamic neural network (DNN) approaches into a moregeneralized and robust formulation. For the first time, stabilityanalysis methods are presented for neural modeling of nonlinearmicrowave circuits. We derive the stability criteria for both thelocal stability and global stability of SSDNN models. Stability testmatrices are formulated from SSDNN internal weight parameters.The proposed criteria can be conveniently applied to the stabilityverification of a trained SSDNN model using the eigenvalues of thetest matrices. In addition, a new constrained training algorithmis introduced by formulating the proposed stability criteria astraining constraints such that the resulting SSDNN models satisfyboth the accuracy and stability requirements. The validity ofthe proposed technique is demonstrated through the transientmodeling of high-speed interconnect driver and receiver circuitsand the stability verifications of the obtained SSDNN models.
Index Terms—Modeling, neural networks, nonlinear circuits,stability analysis, transient analysis.
I. INTRODUCTION
ARTIFICIAL neural networks (ANNs) have recentlyemerged as useful tools for modeling and design optimiza-
tion in radio-frequency (RF) and microwave computer-aideddesign (CAD) [1], [2]. ANN can be trained in a computer-drivenprocess to learn the electromagnetic (EM)/physics behaviors ofcomponents, and trained neural networks can be used in placeof original detailed EM/physics models to speed up high-levelmicrowave design [3], [4]. A recent trend in this area is toutilize ANN to enhance the CAD of nonlinear microwavedevices and circuits [5]–[10]. The early approach presented in[5] and [11] utilized a discrete-time recurrent neural networkfor learning the time-domain dynamic behavior of the originalnonlinear circuit. In [6], a dynamic neural network (DNN)technique was introduced for modeling the steady-state behav-iors of nonlinear circuits such as amplifiers and mixers. TheDNN models can be developed directly from input-output datawithout having to rely on circuit internal details. The DNNapproach maintained the ANN speed and accuracy capabilities
Manuscript received November 18, 2005; revised March 1, 2006. This workwas supported by the Natural Science and Engineering Research Council ofCanada.
Y. Cao and Q.-J. Zhang are with the Department of Electronics, CarletonUniversity, Ottawa, ON, Canada K1S 5B6.
R. Ding is with the School of Electronics and Information Engineering,Tianjin University, Tianjin 300072, China.
Digital Object Identifier 10.1109/TMTT.2006.875297
with extra flexibilities for steady-state harmonic balance simu-lations in both single-tone and multitone cases. A further studyin [7] provided the theoretical basis for the order determinationof DNN formulations. An adjoint dynamic neural networkapproach was developed in [8] and [9] paving the way fortraining neural models for nonlinear transient applications.Most recently, the DNN concept has also been considered fordeveloping high electron mobility transistor models from largesignal measurements [10].
This paper addresses a new area of DNN applications,i.e., neural-based transient modeling of nonlinear circuitssuch as nonlinear drivers and receivers in high-speed verylarge-scale integrated (VLSI) interconnect analysis. With thecontinuous increase of signal speed and frequency, signalintegrity (SI) in VLSI packages becomes important. Fast andaccurate representations of the nonlinear analog behaviors ofdriver/receiver buffers are the key to the success of SI-baseddesign of high-speed interconnects with nonlinear terminations[12]–[16]. There are two types of conventional methods inpopular use today: detailed transistor-level modeling approachsuch as SPICE models and empirical behavioral modelingapproach such as the input and output buffer informationspecification (IBIS) models [14]. The former approach offersbetter accuracy but becomes increasingly inefficient due to thegrowing size and complexity of integrated circuit (IC) designs.The IBIS model is a data-based model which is fast but usuallyhas only limited accuracy and flexibility. Therefore, developingefficient buffer models for transient applications has becomean important topic [15], [16]. In this paper, we explore theaccuracy and flexibility of neural networks and further extendthe DNN concept to the transient modeling of high-speed ICdrivers and receivers.
One of the most important aspects of nonlinear transient mod-eling is the model stability. It is a fundamental requirementfor ensuring the model reliability in circuit simulations. A non-stable model is nonpractical and can be potentially detrimentalto the overall system simulation. In general, determining the sta-bility for nonlinear systems is a challenging task. In the neuralnetwork community, global asymptotical stability (GAS) andglobal exponential stability have been studied for some specialclasses of dynamic networks, e.g., Hopfield neural networks[17], [18], recurrent neural networks [19], [20], and discrete-time state-space neural networks [21]. Recently, GAS also hasbeen studied for recurrent neural networks with time delays[22]–[24]. Various stability criteria are introduced in the litera-ture in terms of linear matrix inequalities [25]. These existing
CAO et al.: SSDNN TECHNIQUE FOR HIGH-SPEED IC APPLICATIONS 2399
methods are useful for applications such as neural optimiza-tions, signal processing, and classifications. However, for theDNN microwave behavioral models, these existing stability cri-teria cannot be directly used because of the differences in themodel structures, where the DNNs are structured for efficientlyaddressing analog microwave modeling. So far, for DNN be-havioral modeling [6], stability analysis methods are still notyet available.
In this paper, we present a nonlinear state-space dynamicneural network (SSDNN) technique, based on [26], for mod-eling nonlinear transient behaviors. The SSDNN expands theexisting DNN structure into a more general and flexible for-mulation for transient-oriented nonlinear modeling and design.Under this generalized framework, we present for the first timestability analysis of neural models for behavioral modeling ofnonlinear circuits. A set of stability criteria are derived to en-able the verifications of both local and global stabilities of theSSDNN model upon the completion of model training. Further-more, instead of using the conventional training procedure asin [26], we propose a new training algorithm that incorporatesthe proposed stability criteria into model training as constraints,such that the trained SSDNN model is guaranteed to be glob-ally asymptotically stable. The final SSDNN model can be im-plemented into general SPICE-like circuit simulators deliveringfast and accurate transient responses.
This paper is organized as follows. In Section II, we presentthe formulation of the SSDNN modeling technique. The sta-bility analysis of the SSDNN model is described in Section III,leading to a set of practically usable stability criteria. In Sec-tion IV, we derive the SSDNN sensitivities for model trainingand propose a constrained training scheme that automaticallyensures the GAS of a trained SSDNN model. In Section V,several numerical examples for modeling receiver and driverbuffers are given to validate our proposed technique.
II. SSDNN NONLINEAR MODELING STRUCTURE
A. General SSDNN Model Structure
Let be transient input signals of a nonlinear circuit,e.g., input voltages and currents, and be transient outputsignals of a nonlinear circuit, e.g., output voltages and currentswhere and are the numbers of circuit inputs and outputs,respectively. Based on combining state-space concept and con-tinuous recurrent neural network method [27], the SSDNN non-linear model is formulated as [26]
(1)
where is the vector of the state vari-ables and denotes the dimension of the state-space, or orderof the model. Here repre-sents a feed-forward multilayer perceptron (MLP) neural net-work [1], which has input neurons and output neu-rons, with weight parameters and a constant scaling param-eter . is the output matrix, which mapsstate-space into output-space.
There are several advantages of the SSDNN technique formodeling nonlinear circuit. The model accuracy can be proven
through the universal approximation property [1] of MLP neuralnetworks. Furthermore, this new format advocates first-orderdifferential equation format as opposed to the direct high-orderformat in DNN-based approaches [6], enhancing the robustnessof the model in dealing with noisy training data. In addition, theexisting DNN can be indirectly derived as a special case fromthis new SSDNN formulation. In this sense, the SSDNN is aprogressive generalization of the existing DNN.
The discrete-time counterpart of the SSDNN, namely,state-space neural networks, has been studied in, e.g., [21] and[28]–[31], and used in various applications of nonlinear systemidentification and control. If such a discrete-time structureis used for circuit modeling, certain approximations may beneeded when incorporating the model into circuit simulators.Our SSDNN provides an attractive alternative in this aspectbecause the SSDNN implementation in circuit simulation canbe achieved exactly via circuit representations [26].
The SSDNN also provides further advances in analog cir-cuit modeling over several other existing discrete-time neuralnetwork architectures, which were mostly developed fromsignal processing and control applications. An example of suchexisting methods is the Elman recurrent neural network [32],which contains a feedback connection over each hidden neuron.Another example of such architectures is to extend feedforwardMLP structure to include global feedbacks and delays fromMLP outputs to inputs [5], [33], which will be referred to asglobally recurrent neural network (GRNN) in this paper. An-other way to have discrete time recurrency is a locally recurrentneural network where the classical synapse is replaced by afinite impulse response filter or an infinite impulse responsefilter [34]. The fundamental difference between the SSDNNand the above neural networks is that the SSDNN is formulatedin continuous time while the other structures are in discretetime. To implement discrete-time models into analog equivalentcircuits for circuit simulation, certain approximations may beneeded. On the other hand, the SSDNN has an exact circuitimplementation for nonlinear time-domain simulation [26].By using recurrence over MLP inputs and outputs instead ofrecurrence over individual neurons, the SSDNN also offersadditional flexibilities in modeling nonlinear analog behaviorof the circuit.
B. SSDNN Parameters for Stability Analysis
We develop stability criteria with respect to the neural net-work structures and parameters in SSDNN. Let in (1)represent a commonly used three-layer MLP neural network.Let be the number of hidden neurons in the MLP. Letrepresent the weight of the connection between the th neuronof the 1th layer and th neuron of the th layer. To facilitatestability derivation, we split the internal weights of intothree groups, described in matrix form as
2400 IEEE TRANSACTIONS ON MICROWAVE THEORY AND TECHNIQUES, VOL. 54, NO. 6, JUNE 2006
Fig. 1. Three-layer MLP neural network used in the SSDNN. The MLP con-nection weights are grouped into three parts, as defined by WWW , WWW and WWW .The MLP input neurons include SSDNN overall input signals u and state vari-ables x . The MLP output neurons are ggg in state space equations.
which represents the connection weights between external inputsignals and all the hidden neurons
......
...(3)
which represents the connection weights between state variablesand all the hidden neurons, and
......
...(4)
which represents the connection weights between all the hiddenneurons and outputs of . Fig. 1 shows the structure ofa three-layer MLP neural network together with the corre-sponding locations of weight matrices , , and . Forthe convenience of stability analysis, we rewrite (1) in thevector form as
(5)
where is a vector of nonlinearactivation functions for hidden neurons. We follow the assump-tions that the activation functions belong to a class of functionsthat are bounded, smooth, and monotonically increasing. Ex-amples of such functions include commonly used neuron acti-vations functions such as sigmoid function, arctangent function,and hyperbolic tangent function.
III. STABILITY ANALYSIS OF THE SSDNN NONLINEAR MODEL
In this section, we perform stability analysis of the SSDNNmodel via Lyapunov’s indirect and direct methods [35]. TheLyapunov method has been widely used for deriving the stabilityconditions for recurrent neural networks with and without timedelays, e.g. [18]–[20], [22], [23]. In [18], the global exponen-tial stability of Hopfield neural networks is studied by means ofLyapunov method. Through another form of Lyapunov energyfunctions, several sufficient conditions are derived in [23] for re-current neural networks with time varying delays and Lipschitzcontinuous activation functions. Recently, this method has alsobeen used in the study of global robust stability of neural net-work with time delays [22]. These works have demonstrated theapplications of the Lyapunov method for a general category ofrecurrent neural network structures where the feedback connec-tion is applied locally at each neuron. However for our SSDNNstructure where the global feedbacks from MLP outputs to in-puts are present, these existing results can not be directly used.By following the ideas in [19], [20], here we apply the Lyapunovmethod for the stability analysis of the SSDNN and derive suf-ficient criteria for investigating both local and global stabilities.
Let the state vector be the equilibrium point of the SSDNNmodel that satisfies , i.e.,
(6)
To examine stability, we choose in (6) to be a constantexternal input. Let be the constant value for , .
By performing the variable transformation , wecan rewrite (5) in terms of as
(7)
where is a vector function defined as
(8)and
(9)
where and represent the th row of and ,respectively. Because of the monotonically increasing propertyof , we have
if
if (10)
In addition, since the activation functions are smooth switchfunctions, is non-negative and bounded,i.e.,
CAO et al.: SSDNN TECHNIQUE FOR HIGH-SPEED IC APPLICATIONS 2401
where is known as the Lipschitz constant for [20] and canbe determined by finding the maximum magnitude of the deriva-tive of . After the above transformation, the origin nowbecomes the equilibrium point of the new SSDNN representa-tion in (7). To investigate the stability of (5) at , it isequivalent to test the stability of (7) at .
A. Proposed Method for Verifying Local Stability
The most convenient way to test the local stability ofa nonlinear system is by investigating the associated lin-earized system, which is known as the Lyapunov’s indirectmethod [35]. From (7), we define a nonlinear vector function
. Linearizing (7) atyields
(12)
where is an identity matrix andis the Jacobian matrix of , computed at the equilibrium point
. For simplicity, let be the above Jacobianmatrix
(13)
where is the derivative of , . For example, ifwe use functions for , can be simplified as
(14)
The test matrix for local stability is then given by
(15)
Let be the eigenvalues of the test matrixin (15). We present the following criterion for the local
stability of a SSDNN nonlinear model through Lyapunov’sindirect method [35].
Proposed Local Stability Criterion: The equilibrium pointof the SSDNN model (7) is said to be asymptotically
(locally) stable if the real parts of all the eigenvalues forof (15) are negative, i.e., for all , .
The proposed criterion can be used to determine the asymp-totical stability of a trained SSDNN model given the constantmodel input . Since the SSDNN model is a reduced-orderrepresentation of the original nonlinear circuit, the order ofthe model is typically very small. As the size of the test ma-trix of (15) is , the proposed local stability crite-rion can be conveniently applied in practice. However, the abovecriterion is valid only if the values of the state variables are inclose neighborhood of the equilibrium point, which is not al-ways guaranteed during the usage of the SSDNN model in cir-cuit design. Next we present a complementary method that canbe used to investigate the global stability of the model.
B. Proposed Method for Verifying Global Stability
The equilibrium point of the SSDNN model (7) issaid to be globally asymptotically stable if it is locally stable inthe sense of Lyapunov and globally attractive, where the globalattractivity means that every trajectory of the original SSDNNsystem (5) would converge to a unique equilibrium point as
.The GAS, also called global stability for convenience, of
the SSDNN model can be investigated based on the conceptof Lyapunov’s direct method [35]. We assume for
in (11), which is valid for all the popular neuralnetwork activation functions, such as sigmoid function, arctan-gent function and function. In the following, we present asufficient condition for the GAS of the SSDNN model.
Proposed Condition for Global Stability: The equilibriumpoint of the SSDNN model in (7) is said to be globallyasymptotically stable if the following test matrix
(16)
is negative definite, where is an identity matrix.Derivation: We start by defining a candidate of the Lyapunov
function as
(17)
where is defined in (9). It is obvious that for
. Using property (10), each integral term in(17) is always positive for any , . Therefore is apositive definite function, satisfying the basic requirement of theLyapunov function. By differentiating along the SSDNNtrajectory defined by (7) we obtain
(18)
Based on (11) and with the assumptions of , we have
2402 IEEE TRANSACTIONS ON MICROWAVE THEORY AND TECHNIQUES, VOL. 54, NO. 6, JUNE 2006
The matrix form of (20) reads
(21)
Combining (18) and (21) gives
(22)
In general, the matrix in (22) is not symmetrical. Since, we can further derive
(23)
It is obvious that if the matrix in (16) is negative defi-nite, remains negative for any value of state variable
. Therefore is a valid Lyapunov function for the SSDNNmodel (7). Since is monotonic everywhere with ,
the integral in (17) is unbounded as. Hence the Lyapunov function is radially unbounded.
This property further ensures that the model equilibrium pointis globally attractive [35]. Thus the SSDNN model (7) is glob-ally asymptotically stable at the equilibrium point .
It is to be noted that by formulating the test matrixto be a symmetrical and real matrix in (23), the proposed GAScondition can be conveniently used in practice through the com-putation of the eigenvalues of . Let bethe eigenvalues of the test matrix in (16). We have thefollowing equivalent criterion for global stability.
Proposed Global Stability Criterion (GAS Criterion): Theequilibrium point of (5) is globally asymptotically stable ifall the eigenvalues of the test matrix of (16) are negative,i.e., for all , .
C. Discussions
The stability criteria derived by the Lyapunov method are notunique. Different formulations of Lyapunov functions can leadto different criteria. Because the Lyapunov stability conditionsare sufficient conditions, if SSDNN violates a specific set of sta-bility conditions it is still possible for SSDNN to be stable aslong as it can satisfy the stability conditions derived from an-other Lyapunov function. The present work fills a gap in devel-oping stability conditions for SSDNN. It is possible to developvarious other forms of Lyapunov functions, e.g., the multi-termformulation of Lyapunov function [22], [23], leading to a va-riety of criteria for examining the stability of SSDNN.
IV. SSDNN SENSITIVITY AND MODEL TRAINING
A. Adjoint Sensitivity for SSDNN
The SSDNN can be used to represent the nonlinear tran-sient behavior only after we train it with suitable input andoutput data. For our formulation, training is directly done intime-domain. Consequently, our training data are in the formatof sampled input and output waveforms, which can be obtainedfrom the original nonlinear circuit by simulation or measure-ment. Let denote the total number of transient waveforms
used for training. Let and represent the th inputand output transient waveform sampled in the time-interval
, respectively. Let represent the SSDNN responsedue to the th input waveform . The goal of training is toadjust parameters in and such that the objective function(overall training error)
(24)
is minimized. Here is defined as the per-waveform trainingerror for the th waveform
(25)
Since training is essentially an optimization process, fastand accurate sensitivity information is important in order toutilize gradient-based training methods. To achieve this, weneed to derive SSDNN sensitivities by modifying the adjointsensitivity technique of DNN presented in [8], [9]. Followingthe Appendix, we derive an adjoint SSDNN model as
(26)
where contains the derivatives of the hidden neuron acti-vation functions
(27)
and is a vector of state variables for the adjoint SSDNNmodel. The boundary conditions for (26) are imposed at theupper time limit as . Based on the adjoint SSDNNin (26) along with the boundary conditions, the general form oftraining error sensitivity for the th waveform can be derivedas
(28)Depending on the location of the variable , the general sen-sitivity expression of (28) can be further specified. For example,using functions for , the expression of (28) is further sim-plified as
for
for
for
(29)
To compute the sensitivity of the overall training error , weperform the sensitivity analysis using (5), (26), and (29) for each
CAO et al.: SSDNN TECHNIQUE FOR HIGH-SPEED IC APPLICATIONS 2403
Fig. 2. Flowchart for the proposed SSDNN constrained training algorithm for enforcing the GAS of a trained SSDNN model. The training error sensitivities areefficiently computed by exploiting the adjoint SSDNN model.
training waveform to obtain , .Finally, the total error sensitivity for training is calculated as
(30)
and
(31)
where and represent the th output of the SSDNN andtraining data for the th training waveform, respectively, and
is an element in matrix .
B. Constrained SSDNN Training
A basic training method for the SSDNN model is to solvethe unconstrained optimization of (24) based on a set of inputand output training waveforms, using gradient-based optimiza-tions [26]. Since the weight update is purely based on mini-mizing training error, this training method only enforces accu-racy but not stability. To overcome this problem, we present anew formulation for training which automatically ensures theGAS property of the SSDNN model once the training is ac-complished. Here we use our proposed GAS criterion as theconstraint for training. The modified SSDNN training is a con-strained optimization problem
(32)
where is the th eigenvalue of the matrix defined in(16). The constrained training is implemented in MATLAB 7.0using the function from the optimization toolbox [36].
During constrained training, the derivatives of with re-spect to model parameters are calculated analytically by ourproposed adjoint sensitivity technique following (5), (26), and(29)–(31) while the derivative of the constraint in (32) is nu-merically computed by the finite difference method. Weight pa-rameters are initialized by uniform random distribution between[ 0.1, 0.1] and are used as the starting points for the constrainedtraining. Fig. 2 is the block diagram illustrating the proposedconstrained training algorithm. Once the training is completed,the accuracy of the trained SSDNN model is further validatedwith an independent set of test waveforms.
After the successful verifications of accuracy and stability, thetrained SSDNN model can be conveniently incorporated intoa SPICE-like circuit simulator, following the exact circuit rep-resentation described in [26], for high-level circuit simulationand design. The proposed method provides a further progressbeyond the existing treatment for stability of state-space neuralnetworks [21], [28] where stability theorems were mainly usedfor verifications. The proposed constrained training of SSDNNachieves good model accuracy and at the same time enforces themodel stability.
V. NUMERICAL EXAMPLES
A. CMOS Inverter
This example is to demonstrate the validity of our proposedconstrained training algorithm using the GAS criterion asthe constraint. Two SSDNN models, namely model I andmodel II, are trained to learn transient behaviors of a CMOSinverter using the original unconstrained training method and
2404 IEEE TRANSACTIONS ON MICROWAVE THEORY AND TECHNIQUES, VOL. 54, NO. 6, JUNE 2006
TABLE ICOMPARISONS OF MATRIX EIGENVALUES USING DIFFERENT STABILITY
CRITERIA FOR EXAMPLE A. MODEL I IS OBTAINED BY ORIGINAL
UNCONSTRAINED TRAINING METHOD AND MODEL II IS OBTAINED BY OUR
PROPOSED CONSTRAINED TRAINING METHOD
the proposed constrained training method, respectively. TheCMOS inverter is a standard circuit configuration containing 2complementary MOSFETs connected with each other [37]. OurSSDNN model contains one input (input voltage) and oneoutput (output voltage). Training waveforms of the inverterare generated using level-1 HSPICE MOSFET models [38],for different rise time of the input pulse source [1 ns, 2 ns step0.2 ns]. The total training data set contains 6 waveforms. Thesame data set is also used for testing purpose. To investigate theGAS property of the model, normally distributed random noisesignals are added to the input waveforms. For this modelingcase, the three-layer MLP neural network has been applied torepresent the in the SSDNN structure of (1). Thefunction has been used as the activation function in the hiddenlayer of the MLP. The SSDNN structure with orderand number of hidden neurons was sufficient to assuremodel error within 1%. This is at similar level of accuracies forcircuit modeling as existing literature, e.g. [11], [39].
After training, we check the stability of the trained SSDNNmodels based on our local and global stability criteria and theeigenvalues of the test matrices and are summa-rized in Table I. As observed, both models can satisfy the localstability criterion and the model developed by the constrainedtraining method is globally asymptotically stable as well. Toverify the GAS by simulation, we randomly select 20 initialvalues of SSDNN state variables around the model equilibriumpoint and simulate both models up to 15 ns. Fig. 3 shows thecomparison of state trajectories for these two models under thesame initial conditions. It is noticed that for Model I, which isonly locally stable but not globally stable, the trajectories con-verge to non-unique equilibriums, meaning that the state vari-ables may have wrong solutions depending upon initial condi-tions. It is also observed that Model II which satisfies our pro-posed GAS criterion, indeed has a unique equilibrium point thatattracts all the trajectories. This example demonstrates the im-portance of stability property of the model. The example alsoconfirms that using our proposed constrained training methodthe SSDNN model can be guaranteed to preserve the GAS.
B. Receiver Buffer
This example is an expansion of that in [26]. A high-speedreceiver buffer circuit based on HSPICE level-47 BSIM3
Fig. 3. The state trajectories of the SSDNN for modeling a CMOS inverterunder 20 random initial conditions: (a) Model I without GAS property wherestate variables converge to different equilibrium points, meaning that none ofthe equilibrium points are globally asymptotically stable. (b) Model II with GASproperty where state variables converge to a unique equilibrium point.
MOSFET models [38] has been modeled through the SSDNNtechnique. The input current and voltage of the receiver arechosen as the input and output of SSDNN, respectively. Thetraining waveforms are generated by exciting the circuit withpulse signals with different rise time [0.05 ns–0.3 ns, step 0.05ns] and amplitude [ 0.5 v–2.5 v, step 0.1 v] to maximallystimulate the nonlinear dynamic behavior of the circuit indifferent working regions. The resulting training data set hasa total of 186 waveforms. In addition, a separate data set with150 test waveforms is also generated for the validation of thetrained SSDNN model. Using the SSDNN structure ofand , an accurate SSDNN model has been achievedwith average test error around 0.13%. The function wasused as the activation function for each hidden neuron. Themodel accuracy achieved here is similar to that reported in theexisting literature for modeling this type of circuit, e.g. [11],[39]. Furthermore, the SSDNN model has been verified in apractical circuit configuration where the receiver is connectedto a high-speed interconnect [26]. In [26], a set of noisy training
CAO et al.: SSDNN TECHNIQUE FOR HIGH-SPEED IC APPLICATIONS 2405
TABLE IIEIGENVALUES FOR STABILITY VERIFICATION OF THE SSDNN MODEL FOR
EXAMPLE B, AND COMPARISON BETWEEN PROPOSED CONSTRAINED TRAINING
VERSUS UNCONSTRAINED TRAINING
data where white noise is applied to the output waveforms,have been used to train both SSDNN models and existing DNNmodels. The SSDNN model has been shown to give morereliable solutions than the standard DNN models.
Here, we apply our proposed stability criteria to verify thestability of the trained SSDNN models. Table II shows the com-parison of the eigenvalues of the test matrices andusing various types of stability criteria presented in Section III.From the table, we observe that the receiver SSDNN model de-veloped by the original unconstrained training in [26] is locallystable. However, the global stability is not guaranteed. To en-force the GAS of the trained SSDNN model, we re-train a newSSDNN model with the same model structure ( and
) as the original SSDNN by applying the new con-strained training algorithm in Section IV-B. The new SSDNNmodel has been trained to have similar accuracy as the orig-inal one. After the training is completed, we verify the localstability and GAS of the model using the criteria proposed inSection III. For checking stabilities of the SSDNN model, wecompute the eigenvalues of the test matrices and ,given in Table II. It can be observed that with the new con-strained training algorithm both the local stability and GAS ofthe model have been ensured, demonstrating further advancesover [26].
C. Physics-Based Driver Buffer
This example is an expansion of that presented in [26]. Inthis example, we demonstrate the application of the SSDNNtechnique for physics-based transient modeling. We consider afour-stage CMOS driver including eight transistors connectedwith each other [26]. The driver load is a lossy transmissionline ( , , , and
) terminated with a 5 pF capacitor. Transienttraining waveforms are generated for different values of pulserise-time [0.25 ns–0.75 ns, step 0.25 ns], pulse amplitude [4.5v–5.5 v, step 0.5 v], and interconnect length [0.08 m-0.14 m, step0.02 m] using physics-based simulator MINIMOS-NT [40]. The
TABLE IIICOMPARISON OF THE MODEL TEST ERROR BETWEEN DISCRETE TIME
RECURRENT NEURAL NETWORKS, DNN AND SSDNN FOR TRANSIENT
MODELING OF PHYSICS-BASED DRIVER IN EXAMPLE C
training data set contains 36 waveforms in total. The SSDNNinputs and outputs are defined as and
, respectively. The SSDNN structure ofand has been utilized and the average test error
achieved is 0.21% using an independent data set containing 12waveforms. The function was used as the activation func-tion of the hidden neurons in MLP. The level of accuracy ofthe SSDNN model is similar to that in existing literature fornonlinear buffer modeling, e.g. [11], [39]. In [26], it has beendemonstrated that for 50 waveform evaluations, the total CPUtime was reduced from 11.6 hours using MINIMOS down to 9seconds using the SSDNN model. A set of noisy training datahas also been applied to train both SSDNN and DNN models.It was observed in [26] that the SSDNN consistently achievesbetter accuracy compared to the standard DNN when there isnoise in the training data.
Here, we perform additional comparison of the SSDNN withseveral standard recurrent neural network approaches, namely,Elman network [32], GRNN (MLP with delayed inputs andoutputs) [5], [33], and DNN [6]. As shown in Table III, all themethods except the Elman network have achieved very goodaccuracy for modeling the physics-based driver. It demonstratesthat by adopting external recurrence over the MLP structure asopposed to internal feedback over each neuron, GRNN, DNNand SSDNN have enough degree of freedom to represent thenonlinear analog behavior of the driver. To further comparethe model performance, we incorporated the four differentmodels, i.e., Elman, GRNN, DNN, and SSDNN models, intoHSPICE and verified that they can be used to perform transientcircuit simulations. Due to their discrete-time natures, Elmanand GRNN models may need certain approximations whenconverting the time-difference equations into the differentialequations in order to obtain an equivalent circuit representation.The DNN and SSDNN models, on the other hand, have exactcircuit representations in SPICE-like circuit simulators owingto their continuous-time model formats. The SSDNN techniqueprovides attractive overall quality of model accuracy, flexibilityfor circuit simulation, and reliability in the noisy environment.
Next we explore the robustness of the SSDNN models by per-forming the model stability analysis with both types of criteriaproposed in Section III. Table IV shows the comparison of theeigenvalues of the test matrices and by applyingdifferent types of stability criteria to the SSDNN models. Thetable confirms that the SSDNN buffer model trained with uncon-strained training used in [26] is locally stable. However, there isno guarantee that the model has the superior property of globalstability. To further achieve the GAS, we train another SSDNNmodel ( and ) by applying the new constrainedtraining scheme in Section IV-B. After training, the average test
2406 IEEE TRANSACTIONS ON MICROWAVE THEORY AND TECHNIQUES, VOL. 54, NO. 6, JUNE 2006
TABLE IVEIGENVALUES FOR STABILITY VERIFICATION OF THE SSDNN MODEL FOR
EXAMPLE C, AND COMPARISON BETWEEN PROPOSED CONSTRAINED TRAINING
VERSUS UNCONSTRAINED TRAINING
error is found to be 0.65%, confirming the model accuracy. Forthis new model, we again check the stability using our proposedcriteria. As shown in Table IV, all the eigenvalues of the testmatrices and are found to be negative indicatingthat both the local stability and GAS have been achieved suc-cessfully. From this study, it is demonstrated that through theproposed constrained training method, the SSDNN model canbe developed to achieve good accuracy with guaranteed GAS,showing further advances over [26].
D. 8-Bit High-Speed Driver
In this example, we consider the transient modeling of a com-mercial IC package, namely an octal driver SN74AHCT540from Texas Instruments (TI) Incorporated, as shown in Fig. 4.For this chip, both detailed transistor-level model and IBISmodel are available [41], where the former model is slow andaccurate, and the latter is faster but less accurate. Here we aimto use the SSDNN model to represent the nonlinear dynamicbehavior between the input pin and output pin. We define
and to be the inputsand outputs of the SSDNN model, respectively. Buffer input pinis excited by a pulse signal. The buffer output pin is terminatedwith a transmission line ( and ns) in serieswith a 50-pF capacitor load. We simulate the driver packageusing transistor-level model in HSPICE to gather differenttraining waveforms. Specifically, the training waveforms areobtained by varying pulse rise time [0.5 ns–2.0 ns, step0.5 ns] and the length of the transmission line load [1 cm–3cm, step 0.5 cm]. The entire training set contains 20 waveforms.The used in the SSDNN model is a three-layer MLP with
function as the activation function for the hidden layer.
Fig. 4. The schematic of a TI 8-bit high-speed driver to be considered in Ex-ample D. Training waveforms of v (t), v (t), and i (t) are generated byHSPICE simulation using detailed transistor-level models, for developing theSSDNN model.
The model structure used in this example has orderand number of hidden neurons . SSDNN model trainingis performed following the constrained training algorithm assummarized in Section IV-B.
After training, the SSDNN model accuracy is confirmed byan independent set of test waveforms with 12 waveforms thatare never applied during the training. We then incorporate thetrained SSDNN model into HSPICE and perform the transientsimulation of the model for SI analysis. Fig. 5 shows the com-parison of the voltages at the output pin using three types ofmodels, i.e., detailed transistor-level model, IBIS model and theproposed SSDNN model. It can be observed that SSDNN so-lutions can match the signal variations from the detailed tran-sistor-level model more accurately than that of the IBIS model.Therefore, the proposed SSDNN provides better accuracy thanIBIS model. We also compare the three different models in termsof relative error and CPU time in HSPICE transient simulation,as given in Table V. The SSDNN provides similar level of accu-racies as the existing literature for I/O buffer modeling, e.g. [11],[39]. The CPU time in the table is for the simulation of outputwaveforms of driver-receiver buffers with transmission line loadfor 50 variations of and . The SSDNN model is shown to be12 times faster than the detailed transistor-level model, withoutsacrificing the accuracy.
To verify the stability of the trained SSDNN model, the eigen-values of the test matrix in (16) are computed and themaximum eigenvalue is found to be 0.0059, indicating thatour SSDNN model is globally asymptotically stable. To fur-ther validate the GAS property, we perform the transient sim-ulation of the SSDNN model for randomly selected initial con-ditions around the equilibrium point and the obtained trajecto-ries of SSDNN state variables are shown in Fig. 6. It can beobserved that all the trajectories asymptotically converge to the
CAO et al.: SSDNN TECHNIQUE FOR HIGH-SPEED IC APPLICATIONS 2407
Fig. 5. Sample comparisons of the output voltage responses using detailed tran-sistor-level model (-), IBIS model (--), and proposed SSDNN model (o) for Ex-ample D. The comparisons shown here are the cases for (a) T = 1:25 ns andd = 1:25 cm, and (b) T = 1:75 ns and d = 2:75 cm. Good agreement isachieved between the responses of the proposed SSDNN models and those ofdetailed transistor-level models even though such test waveforms are never usedin training.
TABLE VCOMPARISONS OF CPU TIME AND ACCURACY FOR VARIOUS TRANSIENT
MODELS FOR HSPICE SIMULATION IN EXAMPLE D
unique equilibrium point of the model as time proceeds. Thisexample demonstrates that the SSDNN technique provides bestoverall efficiency, being more accurate than IBIS model andfaster than detailed transistor-level model. The proposed con-strained training algorithm leads to fast, accurate and robustmodeling solutions for high-speed IC design.
Fig. 6. The state-space trajectories of the SSDNN for modeling a TI driverin Example D. These trajectories are generated by transient simulation of themodel with randomly selected initial conditions, as t!1. The SSDNN modelis trained to be globally asymptotically stable through our proposed constrainedtraining algorithm. The model state variables x and x all converge to a uniqueequilibrium point as observed from the figure.
VI. CONCLUSIONS
We have presented a generalized neural-based modeling tech-nique to learn the nonlinear transient behaviors of IC driversand receivers for high-speed circuit applications. The SSDNNtechnique efficiently modifies and extends the DNN formula-tion, established mainly for frequency-domain steady state mod-eling, to cover time-domain transients. To address the robust-ness of the SSDNN model, we have derived several stabilitycriteria that facilitate convenient investigations of local stabilityand global stability based on the SSDNN model parameters. Asa further step towards enhancing model robustness, an advancedconstrained training scheme has been developed that can auto-matically produce the SSDNN models with good accuracy andguaranteed GAS. The proposed constrained training algorithmhas been demonstrated through the modeling of circuit-basedand physics-based I/O buffers. The proposed technique can beused for achieving efficient CAD of high-speed ICs includinghigh-speed interconnect and nonlinear driver/receiver buffers.
APPENDIX
DERIVATION OF ADJOINT SENSITIVITY FOR SSDNN
The adjoint sensitivity for the existing DNN [6] was intro-duced in [8], [9]. Here we derive the sensitivity for the newstate-space formulation of DNN. To derive the new SSDNN sen-sitivities, we utilize the concept of Lagrange function [42] andformulate our task as a nonlinear optimization problem of (25)with the SSDNN equations of (5) as dynamic constraints. Wedefine a Lagrange function as
2408 IEEE TRANSACTIONS ON MICROWAVE THEORY AND TECHNIQUES, VOL. 54, NO. 6, JUNE 2006
where is a vector representing time-dependent La-grange parameters. Let be the per-waveform training errordefined in (25). Subject to (5), derivatives of with respectto the weight can be expressed in terms of as
(34)
where is defined in (27). Integrating the termof (34) by parts, we obtain
(35)
Equation (35) can be rearranged as
(36)
As (36) holds true for arbitrary choice of , to eliminate theterms containing , we choose to satisfy the differ-ential equation in the form
(37)
This becomes the adjoint SSDNN equation in (26). Using theadjoint SSDNN of (26) together with its boundary conditions,(36) can be simplified as
(38)
In general, is independent of the weight , i.e.,. Equation (38) can then be simplified by elim-
inating the first term. Thus the derivation of (28) is complete.
ACKNOWLEDGMENT
The authors thank Dr. I. Erdin and H. Kwong of NortelNetworks for their valuable inputs on this paper and for pro-viding continuous technical interactions and motivations in thisresearch. Q. J. Zhang thanks his colleague Prof. M. S. Nakhla ofCarleton University for continuous interactions on high-speedVLSI packaging and interconnects. Q. J. Zhang also wishesto thank Dr. H. Maramis of Intel for technical discussions onmodeling and signal integrity.
REFERENCES
[1] Q. J. Zhang and K. C. Gupta, Neural Networks for RF and MicrowaveDesign. Norwood, MA: Artech House, 2000.
[2] P. Burrascano, S. Fiori, and M. Mongiardo, “A review of artificialneural networks applications in microwave computer-aided design,”Int. J. RF Microw. CAE, vol. 9, no. 3, pp. 158–174, May 1999.
[3] P. M. Watson and K. C. Gupta, “EM-ANN models for microstrip viasand interconnects in dataset circuits,” IEEE Trans. Microw. TheoryTech., vol. 44, no. 12, pp. 2495–2503, Dec. 1996.
[4] J. W. Bandler, M. A. Ismail, J. E. Rayas-Sánchez, and Q. J. Zhang,“Neuromodeling of microwave circuits exploiting space mappingtechnology,” IEEE Trans. Microw. Theory Tech., vol. 47, no. 12, pp.2417–2427, Dec. 1999.
[5] Y. H. Fang, M. C. E. Yagoub, F. Wang, and Q. J. Zhang, “A new macro-modeling approach for nonlinear microwave circuits based on recurrentneural network,” IEEE Trans. Microw. Theory Tech., vol. 48, no. 12, pp.2335–2344, Dec. 2000.
[6] J. J. Xu, M. C. E. Yagoub, R. T. Ding, and Q. J. Zhang, “Neural baseddynamic modeling of nonlinear microwave circuits,” IEEE Trans. Mi-crow. Theory Tech., vol. 50, no. 12, pp. 2769–2780, Dec. 2002.
[7] J. Wood, D. E. Root, and N. B. Tufillaro, “A behavioral modelingapproach to nonlinear model-order reduction for RF/microwave ICsand systems,” IEEE Trans. Microw. Theory Tech., vol. 52, no. 9, pp.2274–2284, Sep. 2004.
[8] Y. Cao, J. J. Xu, V. K. Devabhaktuni, R. T. Ding, and Q. J. Zhang,“An adjoint dynamic neural network technique for exact sensitivitiesin nonlinear transient modeling and high-speed interconnect design,”in IEEE MTT-S Int. Microw. Symp. Dig., Philadelphia, PA, June 2003,pp. 165–168.
[9] ——, “An adjoint sensitivity technique for dynamic neural networkmodeling and design of high-speed interconnect,” Int. J. RF Microw.CAE, to be published.
[10] D. Schreurs, J. Verspecht, E. Vandamme, N. Vellas, C. Gaquiere,M. Germain, and G. Borghs, “ANN model for AlGaN/GaN HEMTsconstructed from near-optimal-load large-signal measurements,” inIEEE MTT-S Int. Microw. Symp. Dig., Philadelphia, PA, Jun. 2003,pp. 447–450.
[11] I. A. Maio, I. S. Stievano, and F. G. Canavero, “NARX approach toblack-box modeling of circuit elements,” in Proc. IEEE Int. Symp. Cir-cuits Syst., Monterey, CA, Jun. 1998, pp. 411–414.
[12] S. Lum, M. Nakhla, and Q. J. Zhang, “Sensitivity analysis of lossycoupled transmission lines with nonlinear terminations,” IEEE Trans.Microw. Theory Tech., vol. 42, no. 4, pp. 607–615, Apr. 1994.
[15] I. S. Stievano, I. A. Maio, and F. G. Canavero, “Parametric macro-models of digital I/O ports,” IEEE Trans. Adv. Packag., vol. 25, pp.255–264, May 2002.
[16] B. Mutnury, M. Swaminathan, and J. Libous, “Macro-modeling of non-linear I/O drivers using spline functions and finite time-difference ap-proximation,” in Proc. Electr. Perf. Electron. Packag., Princeton, NJ,Oct. 2003, pp. 273–276.
[17] Z. H. Guan, G. R. Chen, and Y. Qin, “On equilibria, stability, and in-stability of Hopfield neural networks,” IEEE Trans. Neural Netw., vol.11, pp. 534–540, Mar. 2000.
[18] J. Cao, “Global exponential stability of Hopfield neural networks,” Int.J. Syst. Sci., vol. 32, no. 2, pp. 233–236, Feb. 2001.
[19] T. G. Chu, C. S. Zhang, and Z. D. Zhang, “Necessary and sufficient con-dition for absolute stability of normal neural networks,” Neural Netw.,vol. 16, no. 8, pp. 1223–1227, Oct. 2003.
[20] S. Q. Hu and J. Wang, “Global stability of a class of continuous-timerecurrent neural networks,” IEEE Trans. Circuits Syst., vol. 49, no. 9,pp. 1534–1547, Sep. 2002.
[21] J. M. Zamarreño and P. Vega, “State space neural network. propertiesand application,” Neural Netw., vol. 11, no. 6, pp. 1099–1112, Aug.1998.
[22] J. Cao and J. Wang, “Global asymptotic and robust stability of recurrentneural networks with time delays,” IEEE Trans. Circuits Syst., vol. 52,no. 2, pp. 417–426, Feb. 2005.
[23] ——, “Global asymptotic stability of a general class of recurrent neuralnetworks with time-varying delays,” IEEE Trans. Circuits Syst., vol. 50,no. 1, pp. 34–44, Jan. 2003.
[24] T. L. Liao and F. C. Wang, “Global stability of cellular neural networkswith time delays,” IEEE Trans. Neural Netw., vol. 11, pp. 1481–1484,Nov. 2000.
[25] S. Boyd, L. El Ghaoui, E. Feron, and V. Balakrishnan, Linear MatrixInequalities in System and Control Theory. Philadelphia, PA: SIAM,1994.
[26] Y. Cao, R. T. Ding, and Q. J. Zhang, “A new nonlinear transient mod-eling technique for high-speed integrated circuit applications basedon state-space dynamic neural network,” in IEEE MTT-S Int. Microw.Symp. Dig., Fort Worth, TX, Jun. 2004, pp. 1553–1556.
[27] L. Leistritz, M. Galicki, H. Witte, and E. Kochs, “Training trajecto-ries by continuous recurrent multilayer networks,” IEEE Trans. NeuralNetw., vol. 15, no. 2, pp. 283–291, Mar. 2002.
[28] J. M. Zamarreño, P. Vega, L. D. García, and M. Francisco, “State-space neural network for modelling, prediction and control,” Contr.Eng. Practice, vol. 8, no. 9, pp. 1063–1075, Sep. 2000.
[29] I. Rivals and L. Personnaz, “Black-box modeling with state-spaceneural networks,” in Neural Adaptive Control Technology, R.Zbikowski and K. J. Hunt, Eds. Singapore: World Scientific, 1996,pp. 237–264.
[30] P. Gil, A. Dourado, and J. O. Henriques, “State space neural networksand the unscented Kalman filter in online nonlinear system identifica-tion,” in IASTED Int. Conf. Intell. Syst. Contr., Tampa, FL, Nov. 2001,pp. 337–342.
[31] J. O. Henriques, A. Dourado, and P. Gil, “State space neural networksin nonlinear adaptive system identification and control,” in IFAC Work-shop Adv. Fuzzy/Neural Contr. 2001, Valencia, Spain, Oct. 2001, pp.99–104.
[32] J. L. Elman, “Finding structure in time,” Cogn. Sci., vol. 14, pp.179–211, 1990.
[33] K. S. Narendra and K. Parthasarathy, “Identification and control of dy-namic systems using neural networks,” IEEE Trans. Neural Netw., vol.1, no. 2, pp. 4–27, Mar. 1990.
[34] P. Campolucci, A. Uncini, F. Piazza, and B. D. Rao, “On-line learningalgorithms for locally recurrent neural networks,” IEEE Trans. NeuralNetw., vol. 10, no. 2, pp. 253–271, Mar. 1999.
[35] M. Vidyasagar, Nonlinear Systems Analysis, 2nd ed. EnglewoodCliffs, NJ: Prentice-Hall, 1993.
[36] MATLAB. ver. 7.0, The Mathworks Inc., Natick, MA, 2004.[37] A. S. Sedra and K. C. Smith, Microelectronic Circuits, 4th ed. New
York: Oxford Univ. Press, 1998.
[38] HSPICE. ver. 2001.4, Avant! Corporation, Fremont, CA, 2001.[39] J. L. Rossello and J. Segura, “Charge-based analytical model for evalu-
ation of power consumption in submicron CMOS buffer,” IEEE Trans.Computer-Aided Design, vol. 21, no. 4, pp. 433–448, Apr. 2002.
[40] MINIMOS-NT v.2.0. Inst. for Microelectronics, Technical Univ. Vi-enna, Austria.
[42] J. Vlach and K. Singhal, Computer Methods for Circuit Analysis andDesign. New York: Van Nostrand Reinhold, 1993.
Yi Cao (S’06) received the B.Eng. degree fromTianjin University, Tianjin, China, in 1999 and theM.A.Sc. degree from Carleton University, Ottawa,ON, Canada, in 2003, both in electrical engineering,where he is currently pursuing the Ph.D. degree.
His research interests include computed-aided de-sign of VLSI modules and applications of artificialneural networks for high-frequency/high-speed cir-cuit modeling and design.
Mr. Cao received the 2004 Carleton University In-dira Gandhi Memorial Fellowship.
Runtao Ding was born in Shanghai, China, in 1938.He received the diploma degree from Tianjin Univer-sity, Tianjin, China, in 1961.
Since 1961, he has been with the Department ofElectronic Engineering, School of Electronic Infor-mation Engineering, Tianjin University, where he iscurrently a Professor. From 1991 to 1996 and from1996 to 1999, he was Chairman of the Departmentof Electronic Engineering and Dean of the Schoolof Electronic Information Engineering, respectively.His research interests include nonlinear signal pro-
cessing, image processing, neural networks, and circuit design.Prof. Ding was a Cochair of the Technical Program Committee of IEEE
APCCAS’2000.
Qi-Jun Zhang (S’84–M’87–SM’95–F’06) receivedthe B.Eng. degree from East China EngineeringInstitute, Nanjing, in 1982, and the Ph.D. degree inelectrical engineering from McMaster University,Hamilton, ON, Canada, in 1987.
From 1982 to 1983, he was with the SystemEngineering Institute, Tianjin University, Tianjin,China. From 1988 to 1990, he was with OptimizationSystems Associates, Inc., Dundas, ON, Canada,where he developed advanced microwave optimiza-tion software. In 1990, he joined the Department
of Electronics, Carleton University, Ottawa, ON, where he is currently a Pro-fessor. His research interests are neural network and optimization methods forhigh-speed/high-frequency circuit design, and has more than 170 publicationsin the area. He is an author of Neural Networks for RF and Microwave Design(Artech House, 2000), a coeditor of Modeling and Simulation of High-SpeedVLSI Interconnects (Kluwer, 1994), and a contributor to Encyclopedia ofRF and Microwave Engineering, (Wiley, 2005), Fundamentals of NonlinearBehavioral Modeling for RF and Microwave Design, (Artech House, 2005),and Analog Methods for Computer-Aided Analysis and Diagnosis, (MarcelDekker, 1988). He was a Guest Coeditor for a Special Issue on High-SpeedVLSI Interconnects of the International Journal of Analog Integrated Circuitsand Signal Processing and twice a Guest Editor for the Special Issues onApplications of ANN to RF and Microwave Design for the InternationalJournal of RF and Microwave CAE. He is a member of the Editorial Boardof the International Journal of RF and Microwave CAE and the InternationalJournal of Numerical Modeling.
Prof. Zhang is a member of the Editorial Board of the IEEE TRANSACTIONS
ON MICROWAVE THEORY AND TECHNIQUES and of the Technical Committee onCAD (MTT-1) of the IEEE Microwave Theory and Techniques Society.
