Critical Reviews in Biochemistry and Molecular Biology, 35(5):359–391 (2000)
Beyond Eyeballing: Fitting Models toExperimental Data
Arthur Christopoulos and Michael J. Lew
Table of ContentsI. Introduction ............................................................................................. 361
A. “Eyeballing” ..................................................................................... 361B. Models .............................................................................................. 361
II. Empirical or Mechanistic? ..................................................................... 361
III. Types of Fitting ....................................................................................... 363A. Correlation ........................................................................................ 363
1. The Difference Between Correlation and LinearRegression ............................................................................... 363
2. The Meaning of r2 ................................................................... 3633. Assumptions of Correlation Analysis ..................................... 3644. Misuses of Correlation Analysis ............................................. 364
B. Regression ........................................................................................ 3651. Linear Regression .................................................................... 3652. Ordinary Linear Regression .................................................... 3653. Multiple Linear Regression ..................................................... 3654. Nonlinear Regression .............................................................. 3675. Assumptions of Standard Regression Analyses...................... 367
IV. How It Works .......................................................................................... 368A. Minimizing an Error Function (Merit Function) ............................. 368B. Least Squares.................................................................................... 368C. Nonleast Squares .............................................................................. 371D. Weighting.......................................................................................... 371E. Regression Algorithms ..................................................................... 372
V. When to Do It (Application of Curve Fitting Procedures)................ 374A. Calibration Curves (Standard Curves) ............................................. 374B. Parameterization of Data (Distillation) ............................................ 374
Cri
tical
Rev
iew
s in
Bio
chem
istr
y an
d M
olec
ular
Bio
logy
Dow
nloa
ded
from
info
rmah
ealth
care
.com
by
201.
216.
146.
80 o
n 02
/14/
13Fo
r pe
rson
al u
se o
nly.
360
VI. How to Do It ............................................................................................ 374A. Choosing the Right Model ............................................................... 374
1. Number of Parameters ............................................................ 3742. Shape ....................................................................................... 3753. Correlation of Parameters ....................................................... 3764. Distribution of Parameters ...................................................... 376
B. Assessing the Quality of the Fit ...................................................... 3771. Inspection................................................................................. 3772. Root Mean Square ................................................................... 3773. R2 (Coefficient of Determination)........................................... 3784. Analysis of Residuals .............................................................. 3795. The Runs Test .......................................................................... 379
C. Optimizing the Fit ............................................................................ 3801. Data Transformations .............................................................. 3802. Initial Estimates ....................................................................... 381
D. Reliability of Parameter Estimates .................................................. 3821. Number of Datapoints ............................................................. 3822. Parameter Variance Estimates from Repeated
Experiments ............................................................................. 3833. Parameter Variance Estimates from Asymptotic
Standard Errors ........................................................................ 3844. Monte Carlo Methods ............................................................. 3855. The Bootstrap .......................................................................... 3866. Grid Search Methods .............................................................. 3877. Evaluation of Joint Confidence Intervals ............................... 387
E. Hypothesis Testing ........................................................................... 3871. Assessing Changes in a Model Fit between
Experimental Treatments......................................................... 3872. Choosing Between Models ..................................................... 388
VII. Fitting Versus Smoothing ....................................................................... 388
VIII. Conclusion ................................................................................................ 389
IX. Software ................................................................................................. 389
The oldest and most commonly usedtool for examining the relationship betweenexperimental variables is the graphical dis-play. People are very good at recognizingpatterns, and can intuitively detect variousmodes of behavior far more easily from agraph than from a table of numbers. Theprocess of “eyeballing the data” thus repre-sents the experimenter’s first attempt atunderstanding their results and, in the past,has even formed the basis of formal quanti-tative conclusions. Eyeballing can some-times be assisted by judicious application ofa ruler, and often the utility of the ruler hasbeen enhanced by linearizing data transfor-mations. Nowadays it is more common touse a computer-based curve-fitting routineto obtain an “unbiased” analysis. In somecommon circumstances there is no impor-tant difference in the conclusions that wouldbe obtained by the eye and by the computer,but there are important advantages of themore modern methods in many other cir-cumstances. This chapter will discuss someof those methods, their advantages, and howto choose between them.
B. Models
The modern methods of data analysisfrequently involve the fitting of mathemati-cal models to the data. There are many rea-sons why a scientist might choose to modeland many different conceptual types ofmodels. Modeling experiments can be en-tirely constructed within a computer andused to test “what if” types of questionsregarding the underlying mathematical as-
pects of the system of interest. In one sense,scientists are constructing and dealing withmodels all the time inasmuch as they form“worldview” models; experiments are de-signed and conducted and then used in anintuitive fashion to build a mental picture ofwhat the data may be revealing about theexperimental system (see Kenakin, this vol-ume). The experimental results are then fre-quently analyzed by applying either empiri-cal or mechanistic mathematical models tothe data. It is these models that are the sub-ject of this article.
II. EMPIRICAL OR MECHANISTIC?
Empirical models are simple descrip-tors of a phenomenon that serve to approxi-mate the general shape of the relationshipbeing investigated without any theoreticalmeaning being attached to the actual pa-rameters of the model. In contrast, mecha-nistic models are primarily concerned withthe quantitative properties of the relation-ship between the model parameters and itsvariables, that is, the processes that govern(or are thought to govern) the phenomenonof interest. Common examples of mecha-nistic models are those related to mass ac-tion that are applied to binding data to ob-tain estimates of chemical dissociationconstants whereas nonmechanistic, empiri-cal models might be any model applied todrug concentration–response curves in or-der to obtain estimates of drug potency. Ingeneral, mechanistic models are often themost useful, as they consist of a quantitativeformulation of a hypothesis.1 However, theconsequences of using an inappropriatemechanistic model are worse than for em-pirical models because the parameters inmechanistic models provide informationabout the quantities and properties of realsystem components. Thus, the appropriate-
Cri
tical
Rev
iew
s in
Bio
chem
istr
y an
d M
olec
ular
Bio
logy
Dow
nloa
ded
from
info
rmah
ealth
care
.com
by
201.
216.
146.
80 o
n 02
/14/
13Fo
r pe
rson
al u
se o
nly.
362
ness of mechanistic models needs close scru-tiny.
The designation of a mathematical modelas either empirical or mechanistic is basedpredominantly on the purpose behind fittingthe model to experimental data. As such,the same model can be both empirical andmechanistic depending on its context of use.As an example, consider the following formof the Hill equation:
(1)
This equation is often used to analyzeconcentration–occupancy curves for the in-teraction of radioligands with receptors orconcentration–response curves for the func-tional interaction of agonist drugs with re-ceptors in cells or tissues. The Hill equationdescribes the observed experimental curvein terms of the concentration of drug (A), amaximal asymptote (α), a midpoint loca-tion (K), and a midpoint slope (S). In prac-tice, these types of curves are most conve-niently visualized on a semi-logarithmicscale, as shown in Figure 1.
When Hill first derived this equation,2,3
he based it on a mechanistic model for thebinding of oxygen to the enzyme, hemoglo-bin. In that context, the parameters that Hillwas interested in, K and S, were meant toreveal specific biological properties aboutthe interaction he was studying; K was ameasure of the affinity of oxygen for the
enzyme and S was the number of moleculesof oxygen bound per enzyme. Subsequentexperiments over the years have revealedthat this model was inadequate in account-ing for the true underlying molecular mecha-nism of oxygen-hemoglobin binding, butthe equation remains popular both as amechanistic model when its validity is ac-cepted, and as an empirical model where itsshape approximates that of experimentaldata. For instance, if the experimental curveis a result of the direct binding of aradioligand to a receptor, then applicationof Equation (1) to the dataset can be used todetect whether the interaction conforms tothe simplest case of one-site mass-actionbinding and, if S = 1, the parameters K andα can be used as quantitative estimates ofthe ligand-receptor dissociation constant(KD) and total density of receptors (Bmax),respectively. This is an example where theHill equation is a mechanistic equation,because the resulting parameters provideactual information about the underlyingproperties of the interaction. In contrast,concentration–response curves represent thefinal element in a series of sequential bio-chemical cascades that yield the observedresponse subsequent to the initial mass-ac-tion binding of a drug to its receptor. Thus,although the curve often retains a sigmoidalshape that is similar to the binding curve,the Hill equation is no longer valid as amechanistic equation. Hence, the Hill equa-
YA
A K
S
S S=+
α[ ][ ]
FIGURE 1. Concentration–binding (left) and concentration–response (right) curves showing theparameters of the Hill equation (α, K, and S) as mechanistic (left) or empirical (right) modeldescriptors.
Cri
tical
Rev
iew
s in
Bio
chem
istr
y an
d M
olec
ular
Bio
logy
Dow
nloa
ded
from
info
rmah
ealth
care
.com
by
201.
216.
146.
80 o
n 02
/14/
13Fo
r pe
rson
al u
se o
nly.
363
tion is useful in providing a good fit tosigmoidal concentration–response curves,but the resulting parameters are consideredempirical estimates of maximal response,midpoint slope, and midpoint location, andno mechanistic interpretation should bemade.
III. TYPES OF FITTING
The variables whose relationships thatcan be plotted on Cartesian axes do notnecessarily have the same properties. Oftenone variable is controlled by the experi-menter and the other variable is a measure-ment. Thus one variable has substantiallymore uncertainty or variability than the other,and traditionally that variable would be plot-ted on the vertical axis. In that circumstancethe Y variable can be called the “depen-dent” variable because of its dependence onthe underlying relationship and on the othervariable, which is called “independent” todenote its higher reliability. It is importantto note that not all datasets have a clearlyindependent variable. Historically, the sta-tistical determination of the relationshipbetween two or more dependent variableshas been referred to as a correlation analy-sis, whereas the determination of the rela-tionship between dependent and indepen-dent variables has come to be known as aregression analysis. Both types of analyses,however, can share a number of commonfeatures, and some are discussed below.
A. Correlation
Correlation is not strictly a regressionprocedure, but in practice it is often confusedwith linear regression. Correlation quantifiesthe degree by which two variables vary to-
gether. It is meaningful only when both vari-ables are outcomes of measurement such thatthere is no independent variable.
1. The Difference betweenCorrelation and LinearRegression
Correlation quantifies how well twodependent variables vary together; linearregression finds the line that best predicts adependent variable given one or more inde-pendent variables, that is, the “line of best-fit.” 4 Correlation calculations do not find abest-fit straight line.5
2. The Meaning of r 2
The direction and magnitude of the cor-relation between two variables can be quan-tified by the correlation coefficient, r, whosevalues can range from –1 for a perfect nega-tive correlation to 1 for a perfect positivecorrelation. A value of 0, of course, indi-cates a lack of correlation. In interpretingthe meaning of r, a difficulty can arise withvalues that are somewhere between 0 and –1 or 0 and 1. Either the variables do influ-ence each other to some extent, or they areunder the influence of an additional factoror variable that was not accounted for in theexperiment and analysis. A better “feel” forthe covariation between two variables maybe derived by squaring the value of thecorrelation coefficient to yield the coeffi-cient of determination, or r2 value. Thisnumber may be defined as the fraction ofthe variance in the two variables that isshared, or the fraction of the variance in onevariable that is explained by the other (pro-vided the following assumptions are valid).The value of r2, of course, will always bebetween 0 and 1.
Cri
tical
Rev
iew
s in
Bio
chem
istr
y an
d M
olec
ular
Bio
logy
Dow
nloa
ded
from
info
rmah
ealth
care
.com
by
201.
216.
146.
80 o
n 02
/14/
13Fo
r pe
rson
al u
se o
nly.
364
3. Assumptions of CorrelationAnalysis
1. The subjects are randomly selectedfrom a larger population. This is oftennot true in biomedical research, whererandomization is more common thansampling, but may be sufficient to as-sume that the subjects are at least rep-resentative of a larger population.
2. The samples are paired, i.e., each experi-mental unit has both X and Y values.
3. The observations are independent ofeach other. Sampling one member ofthe population should not affect theprobability of sampling another mem-ber (e.g., making measurements in thesame subject twice and treating themas separate datapoints; making mea-surements in siblings).
4. The measurements are independent. IfX is somehow involved or connectedto the determination of Y, or vice versa,then correlation is not valid. This as-sumption is very important becauseartifactual correlations can result fromits violation. A common cause of sucha problem is where the Y value is ex-pressed as either a change from the Xvalue, or as a fraction of the corre-sponding X value (Figure 2).
5. The X values were measurements, notcontrolled (e.g., concentration, etc.).The confidence interval for r2 is other-wise meaningless, and we must thenuse linear regression.
6. The X and Y values follow a Gaussiandistribution.
7. The covariation is linear.
4. Misuses of CorrelationAnalysis
Often, biomedical investigators are in-terested in comparing one method for mea-suring a biological response with another.This usually involves graphing the results asan X, Y plot, but what to do next? It is quitecommon to see a correlation analysis appliedto the two methods of measurement and thecorrelation coefficient, r, and the resulting Pvalue utilized in hypothesis testing. How-ever, Ludbrook6 has outlined some seriouscriticisms of this approach, the major onebeing that although correlation analysis willidentify the strength of the linear associationbetween X and Y, as it is intended to do, itwill give no indication of any bias betweenthe two methods of measurement. When thepurpose of the exercise is to identify andquantify fixed and proportional biases be-
FIGURE 2. An apparent correlation between two sets of unrelated random numbers (pseudo-random numbers generated with mean = 5 and standard deviation = 1) comes about where the Yvalue is expressed as a function of the X value (here each Y value is expressed as a fraction ofthe corresponding X value).
Cri
tical
Rev
iew
s in
Bio
chem
istr
y an
d M
olec
ular
Bio
logy
Dow
nloa
ded
from
info
rmah
ealth
care
.com
by
201.
216.
146.
80 o
n 02
/14/
13Fo
r pe
rson
al u
se o
nly.
365
tween two methods of measurement, thencorrelation analysis is inappropriate, and atechnique such as ordinary or weighted leastproducts regression6 should be used.
B. Regression
The actual term “regression” is derivedfrom the latin word “regredi,” and means“to go back to” or “to retreat.” Thus, theterm has come to be associated with thoseinstances where one “retreats” or “resorts”to approximating a response variable withan estimated variable based on a functionalrelationship between the estimated variableand one or more input variables. In regres-sion analysis, the input (independent) vari-ables can also be referred to as “regressor”or “predictor” variables.
1. Linear Regression
The most straightforward methods forfitting a model to experimental data are thoseof linear regression. Linear regression in-volves specification of a linear relationshipbetween the dependent variable(s) and cer-tain properties of the system under investi-gation. Surprisingly though, linear regres-sion deals with some curves (i.e., nonstraightlines) as well as straight lines, with regres-sion of straight lines being in the categoryof “ordinary linear regression” and curvesin the category of “multiple linear regres-sions” or “polynomial regressions.”
2. Ordinary Linear Regression
The simplest general model for a straightline includes a parameter that allows for
inexact fits: an “error parameter” which wewill denote as ε. Thus we have the formula:
Y = α + βX + ε (2)
The parameter, α, is a constant, oftencalled the “intercept” while b is referred toas a regression coefficient that correspondsto the “slope” of the line. The additionalparameter, ε, accounts for the type of errorthat is due to random variation caused byexperimental imprecision, or simple fluc-tuations in the state of the system from onetime point to another. This error term issometimes referred to as the stochastic com-ponent of the model, to differentiate it fromthe other, deterministic, component of themodel (Figure 3).7 When data are fitted tothe actual straight-line model, the error termdenoted by ε is usually not included in thefitting procedure so that the output of theregression forms a perfect straight line basedsolely on the deterministic component ofthe model. Nevertheless, the regression pro-cedure assumes that the scatter of thedatapoints about the best-fit straight linereflects the effects of the error term, and itis also implicitly assumed that ε follows aGaussian distribution with a mean of 0. Thisassumption is often violated, however, andthe implications are discussed elsewhere inthis article. For now, however, we will as-sume that the error is Gaussian; Figure 4illustrates the output of the linear modelwith the inclusion of the error term. Notethat the Y values of the resulting “line” arerandomly distributed above and below theideal (dashed) population line defined bythe deterministic component of the model.
3. Multiple Linear Regression
The straight line equation [Equation (2)]is the simplest form of the linear regression
Cri
tical
Rev
iew
s in
Bio
chem
istr
y an
d M
olec
ular
Bio
logy
Dow
nloa
ded
from
info
rmah
ealth
care
.com
by
201.
216.
146.
80 o
n 02
/14/
13Fo
r pe
rson
al u
se o
nly.
366
model, because it only includes one inde-pendent variable. When the relationship ofinterest can be described in terms of morethan one independent variable, the regres-sion is then defined as “multiple linear re-gression.” The general form of the linearregression model may thus be written as:
Y = α + β1X1 + β2X2 + … + βiXi + ε (3)
where Y is the dependent variable, and X1,X2 … Xi are the (multiple) independentvariables. The output of this model can de-viate from a straight line, and one may thusquestion the meaning of the word “linear”in “linear regression.” Linear regressionimplies a linear relationship between thedependent variable and the parameters, notthe independent variables of the model. ThusEquation (3) is a linear model because theparameters α, β1, β2 … βi have the (implied)exponent of unity. Multiple linear regres-sion models also encompass polynomialfunctions:
Y = α + β1X + β2X2 + … + βiXi + ε (4)
The equation for a straight line [Equa-tion (2)] is a first-order polynomial. Thequadratic equation, Y = α + β1X + β2X2, isa second-order polynomial whereas the cu-bic equation, Y = α + β1X + β2X2 + β3X3 isa third-order polynomial. Each of thesehigher order polynomial equations definescurves, not straight lines. Mathematically, alinear model can be identified by taking thefirst derivative of its deterministic compo-nent with respect to the parameters of the
model. The resulting derivatives should notinclude any of the parameters; otherwise,the model is said to be “nonlinear.” Con-sider the following second-order polyno-mial model:
Y = α + β1X + β2X2 (5)
Taking first derivatives with respect toeach of the parameters yields:
(6)
(7)
(8)
The model is linear because the firstderivatives do not include the parameters.As a consequence, taking the second (orhigher) order derivative of a linear func-tion with respect to its parameters willalways yield a value of zero.8 Thus, if theindependent variables and all but one pa-rameter are held constant, the relationshipbetween the dependent variable and theremaining parameter will always be lin-ear.
It is important to note that linear re-gression does not actually test whetherthe data sampled from the population fol-low a linear relationship. It assumes lin-earity and attempts to find the best-fitstraight line relationship based on the datasample.
FIGURE 3. The simple linear population model equation indicating the deterministic component ofthe model that is precisely determined by the parameters α and β, and the stochastic componentof the model, ε, that represents the contribution of random error to each determined value of Y.
∂∂αY = 1
∂∂β
YX
1
=
∂∂β
YX
2
2=
Cri
tical
Rev
iew
s in
Bio
chem
istr
y an
d M
olec
ular
Bio
logy
Dow
nloa
ded
from
info
rmah
ealth
care
.com
by
201.
216.
146.
80 o
n 02
/14/
13Fo
r pe
rson
al u
se o
nly.
367
4. Nonlinear Regression
Because there are so many types ofnonlinear relationships, a general model thatencompasses all their behaviors cannot bedefined in the sense used above for linearmodels, so we will define an explicit non-linear function for illustrative purposes. Inthis case, we will use the Hill equation [Equa-tion (1); Figure 1] which contains one inde-pendent variable [A], and 3 parameters, α,K, and S. Differentiating Y with respect toeach model parameter yields the following:
(9)
(10)
(11)
All derivatives involve at least two ofthe parameters, so the model is nonlinear.However, it can be seen that the partialderivative in Equation (9) does not containthe parameter, α. A linear regression of Yon [A]S/(KS + [A]S) will thus allow the es-timation of α. Because this last (linear) re-gression is conditional on knowing the val-
ues of K and S, α is referred to as a “condi-tionally linear” parameter. Nonlinear mod-els that contain conditionally linear param-eters have some advantages when it comesto actual curve fitting.7
5. Assumptions of StandardRegression Analyses 4,7
1. The subjects are randomly selectedfrom a larger population. The samecaveats apply here as with correlationanalyses.
2. The observations are independent.3. X and Y are not interchangeable. Re-
gression models used in the vast ma-jority of cases attempt to predict thedependent variable, Y, from the in-dependent variable, X and assumethat the error in X is negligible. Inspecial cases where this is not thecase, extensions of the standard re-gression techniques have been de-veloped to account for nonnegligibleerror in X.
4. The relationship between X and Y isof the correct form, i.e., the expecta-tion function (linear or nonlinearmodel) is appropriate to the data beingfitted.
5. The variability of values around theline is Gaussian.
∂∂αY A
A K
S
S S=+
[ ][ ]
∂∂
αYK
S K AK A K
S
S S= −+
( [ ])([ ] ) 2
∂∂
αYK
S K AK A K
S
S S= −+
( [ ])([ ] ) 2
FIGURE 4. A linear model that incorporates a stochastic (random error) component. The dashedline is the deterministic component, whereas the points represent the effect of random error[denoted by the symbol ε in Equation (2)].
Cri
tical
Rev
iew
s in
Bio
chem
istr
y an
d M
olec
ular
Bio
logy
Dow
nloa
ded
from
info
rmah
ealth
care
.com
by
201.
216.
146.
80 o
n 02
/14/
13Fo
r pe
rson
al u
se o
nly.
368
6. The values of Y have constant vari-ance. Assumptions 5 and 6 are oftenviolated (most particularly when thedata has variance where the standarddeviation increases with the mean) andhave to be specifically accounted forin modifications of the standard re-gression procedures.
7. There are enough datapoints to pro-vide a good sampling of the randomerror associated with the experimentalobservations. In general, the minimumnumber of independent points can beno less than the number of parametersbeing estimated, and should ideally besignificantly higher.
IV. HOW IT WORKS
A. Minimizing an Error Function(Merit Function)
The goal of both linear and nonlinearregression procedures is to derive the “bestfit” of a particular model to a set of experi-mental observations. To obtain the best-fitcurve we have to find parameter values thatminimize the difference between the ob-served experimental observations and thechosen model. This difference is assumedto be due to the error in the experimentaldetermination of the datapoints, and thus itis common to see the entire model-fittingprocess described in terms of “minimiza-tion of an error function” or minimizationof a “merit function.”9
The most common representation(“norm”) of the merit function for regres-sion models is based on the chi-square dis-tribution. This distribution and its associ-ated statistic, χ2, have long been used in thestatistical arena to assess “goodness-of-fit”with respect to identity between observed
and expected frequencies of measures. Be-cause regression analyses also involve thedetermination of the best model estimatesof the dependent variables based on theexperimentally observed dependent vari-ables, it is quite common to see the functionused to determine the best-fit of the modelparameters to the experimental data referredto as the “χ2 function,” and the procedurereferred to as “chi-square fitting.”9
B. Least Squares
The most widely used method of pa-rameter estimation from curve fitting is themethod of least squares. To explain the prin-ciple behind least squares methods, we willuse an example, in this case the simple lin-ear model. Theoretically, finding the slope,β, and intercept, α, parameters for a perfectstraight line is easy: any two X,Y pairs ofpoints can be utilized in the familiar “rise-over-run” formulation to obtain the slopeparameter, which can then be inserted intothe equation for the straight line to derivethe intercept parameter. In reality, however,experimental observations that follow lin-ear relationships almost never fall exactlyon a straight line due to random error. Thetask of finding the parameters describingthe line is thus no longer simple; in fact, itis unlikely that values for α and β definedby any pair of experimental points will de-scribe the best line through all the points.This is illustrated in Figure 5; although thedataset appears to follow a linear relation-ship, it can be seen that different straightlines, each characterized by different slopesand intercepts, are derived depending onwhich two X,Y pairs are used.
What is needed, therefore, is a “com-promise” method for obtaining an objectivebest-fit. We begin with our population model[Equation (2)]:
Cri
tical
Rev
iew
s in
Bio
chem
istr
y an
d M
olec
ular
Bio
logy
Dow
nloa
ded
from
info
rmah
ealth
care
.com
by
201.
216.
146.
80 o
n 02
/14/
13Fo
r pe
rson
al u
se o
nly.
369
Y = α + βX + ε
and derive an equation that is of the sameform:
Y = α + β
ˆ X (12)
where Y is the predicted response and α
and β ˆ are the estimates of the population
intercept and slope parameters, respectively.The difference between the response vari-able, Y, and its predictor, Y, is called the“ residual” and its magnitude is therefore ameasure of how well Y predicts Y. Thecloser the residual is to a value of zero for
each experimental point, the closer the pre-dicted line will be to that point. However,because of the error in the data (the ε termin the population model), no prediction equa-tion will fit all the datapoints exactly and,hence, no equation can make the residualsall equal zero. In the example above, eachstraight line will yield a residual of zero fortwo points, but a nonzero residual for theother two points; Figure 6 illustrates this forone of the lines.
A best-fit compromise is found by mini-mizing the sum of the squares of the residu-als, hence the name “least squares.” Math-ematically, the appropriate merit functioncan be written as:
FIGURE 5. All possible straight lines that can be drawn through a four-point dataset when only twopoints are used to define each line.
FIGURE 6. A combination of zero and nonzero residuals. The dataset is the same as in Figure 5,with only one of the lines now drawn through the points. The vertical distance of each point fromthe line (indicated by the arrows) is defined as the “residual.”
Cri
tical
Rev
iew
s in
Bio
chem
istr
y an
d M
olec
ular
Bio
logy
Dow
nloa
ded
from
info
rmah
ealth
care
.com
by
201.
216.
146.
80 o
n 02
/14/
13Fo
r pe
rson
al u
se o
nly.
370
(13)
where χ2 is the weighted sum of the squaresof the residuals (ri) and is a function of theparameters (the vector, θ), and the Ndatapoint, XiYi. The term, wi, is the statisti-cal weight (see below) of a particulardatapoint, and when used, most often re-lates to the standard error of that point. Forstandard (unweighted) least squares proce-dures such as the current example, wi equals1. The least squares fit of the dataset out-lined above is shown in Figure 7. Note thatthe best-fit straight line yields nonzero re-siduals for three of the four datapoints. Nev-ertheless, the resulting line is based on pa-rameter estimates that give the smallestsum-of-squares of those residuals.
Why do we use the sum of the square ofthe residuals and not another norm of thedeviation, such as the average of the absolutevalues of the residuals? Arguably, simplybecause of convention! Different norms ofdeviation have different relative sensitivitiesto small and large deviations and conven-tional usage suggests that sums of the squareresiduals represent a sensible compromise.4,10
The popularity of least squares estimatorsmay also be based on the fact that they arerelatively easy to determine and that they areaccurate estimators if certain assumptions are
met regarding the independence of errors anda Gaussian distribution of errors in thedata.8,9,11 Nonetheless, for extremely largedeviations due to outlier points, least squaresprocedures can fail in providing a sensible fitof the model to the data.
Although the example used above wasbased on a linear model, nonlinear leastsquares follow the same principles as linearleast squares and are based on the sameassumptions. The main difference is that thesum-of-squares merit function for linearmodels is well-behaved and can be solvedanalytically in one step, whereas for nonlin-ear models, iterative or numerical proce-dures must be used instead.
In most common applications of the leastsquares method to linear and nonlinearmodels, it is assumed that the majority ofthe error lies in the dependent variable.However, there can be circumstances whenboth X and Y values are attended by ran-dom error, and different fitting approachesare warranted. One such approach has beendescribed by Johnson,12 and is particularlyuseful for fitting data to nonlinear models.In essence, Johnson’s method utilizes a formof the standard χ2 merit function, givenabove, that has been expanded to includethe “best-fit” X value and its associatedvariance. The resulting merit function is thenminimized using an appropriate least squarescurve fitting algorithm.
χ θ22
1
2
1
= −
=
= =
∑ ∑Y f
i
N
i
Ni i
i
i
i
X
w
r
w
( , )
FIGURE 7. The minimized least squares fit of the straight line model [Equation (2)] to the datasetshown in Figures 5 and 6.
Cri
tical
Rev
iew
s in
Bio
chem
istr
y an
d M
olec
ular
Bio
logy
Dow
nloa
ded
from
info
rmah
ealth
care
.com
by
201.
216.
146.
80 o
n 02
/14/
13Fo
r pe
rson
al u
se o
nly.
371
C. Nonleast Squares
Cornish-Bowden11 has listed the mini-mal requirements for optimal behavior ofthe least squares method:
a. Correct choice of model.b. Correct data weighting is known.c. Errors in the observations are indepen-
dent of one another.d. Errors in the observations are normally
distributed.e. Errors in the observations are unbi-
ased (have zero mean).
And we can add:
f. None or the datapoints are erroneous(outliers).
Often, however, the requirements foroptimal behavior cannot be met. Other tech-niques are available for deriving parameterestimates under these circumstances, andthey are generally referred to as “robustestimation” or “robust regression” tech-niques. Because the word “robustness” hasa particular connotation, it is perhaps unfairto class all of the diverse nonleast squaresprocedures under the same umbrella. Over-all, however, the idea behind robust estima-tors is that they are more insensitive to de-viations from the assumptions that underliethe fitting procedure than least squares esti-mators.
“Maximum likelihood” calculations areone class of robust regression techniquesthat are not based on a Gaussian distributionof errors. In essence, regression proceduresattempt to find a set of model parametersthat generate a curve that best matches theobserved data. However, there is no way ofknowing which parameter set is the correctone based on the (sampled) data, and thusthere is no way of calculating a probabilityfor any set of fitted parameters being the
“correct set.” Maximum likelihood calcula-tions work in the opposite direction, that is,given a particular model with a particularset of parameters, maximum likelihood cal-culations derive a probability for the databeing obtained. This (calculated) probabil-ity of the data, given the parameters, canalso be considered to be the likelihood ofthe parameters, given the data.9 The goal isthen to fit for a set of parameters that maxi-mize this likelihood, hence the term “maxi-mum likelihood,” and the calculations at-tempt to find the regression that has themaximum likelihood of producing the ob-served dataset. It has been pointed out thatthere is no formal mathematical basis forthe maximum likelihood procedure and be-cause maximum likelihood calculations arequite involved, they are not routinely uti-lized explicitly.9 Fortunately the simpler leastsquares methods described above are equiva-lent to maximum likelihood calculationswhere the assumptions of linear and nonlin-ear regression (particularly the independenceand Gaussian distribution of the errors inthe data) are valid.8,9,11
Certain robust regression techniquesfocus on using measures of central tendencyother than the mean as the preferred statis-tical parameter estimator. For instance,Cornish-Bowden11 has described how themedian is more insensitive to outlier pointsin linear regression and certain cases ofnonlinear regression than the mean. A draw-back of this approach, however, is that itquickly becomes cumbersome when ex-tended to more complex linear problems.
D. Weighting
The simplest minimization functionsmake no distinction between different ex-perimental points, and assume that eachobservation contributes equally to the esti-
Cri
tical
Rev
iew
s in
Bio
chem
istr
y an
d M
olec
ular
Bio
logy
Dow
nloa
ded
from
info
rmah
ealth
care
.com
by
201.
216.
146.
80 o
n 02
/14/
13Fo
r pe
rson
al u
se o
nly.
372
mation of model parameters. This is appro-priate when the variance of all the observa-tions is uniform, and the error is referred toas homoscedastic. However, in reality it iscommon that different points have differentvariances associated with them with the re-sult that the points with the most variancemay have an undue influence on the param-eters obtained from an unweighted curvefit. For example, results from many biologi-cal experiments are often expressed as achange from a baseline value, with the con-sequence that the points near the baselinebecome small numbers (near zero) with alow variance. Points representing larger re-sponses will naturally have a larger vari-ance, a situation that can be described asheteroscedasticity. An unweighted curve fitthrough heteroscedastic data will allow theresulting curve to deviate from the well-defined (tight) near-zero values to improvethe fit of the larger, less well-defined val-ues. Clearly it would be better to have the fitplace more credence in the more reliablyestimated points, something that can beachieved in a weighted curve fit.
Equation (13) was used previously todefine the general, least squares, minimiza-tion function. There are a number of varia-tions available for this function that employdifferential data weighting.13 These func-tions explicitly define a value for the wi
term in Equation (13). For instance, if wi =1 or a constant, then the weighting is said tobe “uniform”; if wi = Yi, then
and the weighting is said to be “relative.”Relative weighting is also referred to as“weighting by 1/Y2” and is useful where theexperimental uncertainty is a constant frac-tion of Y. For example, counts of radioac-tive decay will have variances described bythe Poisson distribution where the variance
scales with the mean, and thus the likelyerror in each estimate is a constant percent-age of counts rather than a constant valuefor any number of counts. Thus, a curve fitallowing for relative weighting can adjustfor the resulting heteroscedastic variance.Another useful weighting value isThis yields “weighting by 1/Y” and is ap-propriate, for example, when most of theexperimental uncertainty in the dependentvariable is due to some sort of countingerror.5 Other weighting schemes utilize thenumber of replicates that are measured foreach value of Y to determine the appropri-ate weight for the datapoints.13
E. Regression Algorithms
What are the actual “mechanics” thatunderlie the χ2 minimization process be-hind least squares regression techniques?The χ2 merit function for linear models(including polynomials) is quadratic innature, and is thus amenable to an exactanalytical solution. In contrast, nonlinearproblems must be solved iteratively, andthis procedure can be summarized as fol-lows:
a. Define the merit function.b. Start with a set of initial estimates
(guesses) of the regression param-eters and determine the value of themerit function for this set of esti-mates.
c. Adjust the parameter estimates and re-calculate the merit function. If the meritfunction is improved, then keep theparameter values as new estimates.
d. Repeat step c (each repeat is an “itera-tion”). When further iterations yield anegligible improvement in the fit, stopadjusting the parameter estimates andgenerate the curve based on the last setof estimates.
χ 22
12
2
1
1=
== =
∑ ∑r
Y Yri
i ii
i
N
i
N
( )
w Yi i= .
Cri
tical
Rev
iew
s in
Bio
chem
istr
y an
d M
olec
ular
Bio
logy
Dow
nloa
ded
from
info
rmah
ealth
care
.com
by
201.
216.
146.
80 o
n 02
/14/
13Fo
r pe
rson
al u
se o
nly.
373
The rules for adjusting the parametersof the nonlinear model are based on matrixalgebra and are formulated as computer al-gorithms. The merit function can be viewedas a multidimensional surface that has allpossible sum-of-squares values as one planeand all possible values of each of the modelparameters as the other planes. This surfacemay thus vary from a smooth, symmetricalshape to one characterized by many crestsand troughs. The role of the nonlinear re-gression algorithm is to work its way downthis surface to the deepest trough that shouldthen correspond to the set of model param-eters that yield the minimum sum-of-squaresvalue.
There are a number of different algo-rithms that have been developed over theyears, and they all have their pros and cons.One of the earliest algorithms is the methodof steepest descent (or the gradient searchmethod8). This method proceeds down thesteepest part of the multidimensional meritfunction surface in fixed step lengths thattend to be rather small.9 At the end of eachiteration, a new slope is calculated and theprocedure repeated. Many iterations are re-quired before the algorithm converges on astable set of parameter values. This methodworks well in the initial iterations, but tendsto drag as it approaches a minimum value.13
The Gauss-Newton method is anotheralgorithm that relies on a linear approxima-tion of the merit function. By making thisapproximation, the merit function ap-proaches a quadratic, its surface becomes asymmetrical ellipsoid, and the iterations ofthe Gauss-Newton algorithm allow it toconverge toward a minimum much morerapidly than the method of steepest descent.The Gauss-Newton method works bestwhen it is employed close to the surfaceminimum, because at this point most meritfunctions are well approximated by linear(e.g., quadratic) functions.9 In contrast, theGauss-Newton method can work poorly in
initial iterations, where the likelihood offinding a linear approximation to the meritfunction is decreased.
A method exploiting the best features ofthe methods of steepest descent and Gauss-Newton was described by Marquardt, basedon an earlier suggestion by Levenberg,9 andthe resulting algorithm is thus often referredto as the Levenberg-Marquardt method.Marquardt realized that the size of the in-crements in an interative procedure poses asignificant scaling problem for any algo-rithm, and proceeded to refine the scalingissue and derive a series of equations thatcan approximate the steepest descent methodat early iterations and the Gauss-Newtonmethod at later stages closer to the mini-mum. The Levenberg-Marquard method(sometimes simply referred to as theMarquardt method) has become one of themost widespread algorithms used for com-puterized nonlinear regression.
Another type of algorithm that is geo-metric rather than numeric in nature is theNelder-Mead Variable Size Simplexmethod.8,14 Unlike the methods outlinedabove, this method does not require the cal-culation of any derivatives. Instead, this al-gorithm depends on the generation of anumber of starting points, called “vertices,”based on initial estimates for each param-eter of the model, as well as an initial incre-ment step. The vertices form a multidimen-sional shape called a “simplex.” Thegoodness of fit is evaluated at each vertex inthe simplex, the worst vertex is rejected anda new one is generated by combining desir-able features of the remaining vertices. Thisis repeated in an iterative fashion until thesimplex converges to a minimum. The bigadvantage of the Nelder-Mead method isthat it is very successful in converging to aminimum; its main disadvantage is that itdoes not provide any information regardingthe errors associated with the final param-eter estimates.8
Cri
tical
Rev
iew
s in
Bio
chem
istr
y an
d M
olec
ular
Bio
logy
Dow
nloa
ded
from
info
rmah
ealth
care
.com
by
201.
216.
146.
80 o
n 02
/14/
13Fo
r pe
rson
al u
se o
nly.
374
V. WHEN TO DO IT(APPLICATION OF CURVEFITTING PROCEDURES)
A. Calibration Curves (StandardCurves)
Calibration curves are most convenientwhen they are linear, but even for assayswhere a linear relationship is expected ontheoretical grounds, nonlinear curves canresult from instrumentation nonlinearitiesand other factors. The equation of a curvefitted through the calibration data will al-low convenient conversion between theraw measurement and the required value.In cases where there is no theoretical ba-sis for choosing one model over another,calibration curves can be considered to bea smoothing rather than a real fitting prob-lem and one might decide to apply a poly-nomial model to the data because of theavailability of an analytical solution. Insuch a case the order of the chosen poly-nomial would need to be low so that noisein the calibration measurements is notconverted into wobbles on the calibrationcurve.
B. Parameterization of Data(Distillation)
It is often desirable to describe data inan abbreviated way. An example of this isthe need to summarize a concentration–re-sponse curve into a potency estimate andmaximum response value. These parametersare easily obtained by eyeballing the data,but an unbiased estimate from an empiricalcurve fit is preferable and probably moreacceptable to referees!
VI. HOW TO DO IT
A. Choosing the Right Model
1. Number of Parameters
The expectation function should includethe minimum number of parameters thatadequately define the model and that allowfor a successful convergence of the fit.
If a model is overparameterized, it isconsidered to possess “redundant” param-eters (often used interchangeably with theterm “redundant variables”), and the regres-sion procedure will either fail or yield mean-ingless parameter estimates. Consider the“operational model” of Black and Leff.15
This is a model that is often used in pharma-cological analyses to describe the concen-tration–response relationship of an agonist(A) in terms of its affinity (dissociationconstant) for its receptor (KA), its “opera-tional” efficacy (τ), and the maximum re-sponse (Em) that the tissue can elicit. Onecommon form of the model is:
(14)
where E denotes the observed effect. Figure8 shows a theoretical concentration–responsecurve, plotted in semilogarithmic space, thatillustrates the relationship between the op-erational model parameters and the maxi-mal asymptote (α) and midpoint location(EC50) of the resulting sigmoidal curve. Aconcentration–response curve like the onein Figure 8 can be successfully fitted usingthe two-parameter version of the Hill equa-tion, which describes the curve in terms ofonly the EC50 and α (the slope being equalto 1):
(15)
EE A
A K Am
A
= ⋅ ⋅+ + ⋅
ττ
[ ]
([ ] ) [ ]
EA
A EC= ⋅
+α [ ]
[ ] 50
Cri
tical
Rev
iew
s in
Bio
chem
istr
y an
d M
olec
ular
Bio
logy
Dow
nloa
ded
from
info
rmah
ealth
care
.com
by
201.
216.
146.
80 o
n 02
/14/
13Fo
r pe
rson
al u
se o
nly.
375
However, it can be seen in Figure 8 thatthe midpoint and maximal asymptote of thecurve are related to the operational model ina more complicated manner; each param-eter of the sigmoidal Hill equation is com-prised of two operational model parameters.If someone were to try directly fitting Equa-tion (14) to this curve in order to deriveindividual estimates of Em, KA, and τ, theywould be unsuccessful. As it stands, theoperational model is overparameterized forfitting to a single curve; the regression algo-rithm simply will not be able to apportionmeaningful estimates between the individualoperational model parameters as it tries todefine the midpoint and maximal asymptoteof the concentration–response curve. In prac-tice, the successful application of the opera-tional model to real datasets requires addi-tional experiments to be incorporated in thecurve fitting process that allow for a betterdefinition of the individual model param-eters.16,17
2. Shape
When fitting empirical models to datathe most important feature of the modelmust be that its shape should be similar tothe data. This seems extraordinarily obvi-ous, but very little exploration of the litera-
ture is needed to find examples where thecurve and the data have disparate shapes!Empiricism allows one a great deal of free-dom in choosing models, and experiment-ers should not be overly shy of movingaway from the most common models (e.g.,the Hill equation) when their data ask for it.Even for mechanistic models it is importantto look for a clear shape match between themodel and data: a marked difference canonly mean that the model is inappropriate orthe data of poor quality.
Perhaps the only feature that practi-cally all biological responses have in com-mon is that they can be approximated bynonlinear, saturating functions. When plot-ted on a logarithmic concentration scale,responses usually lie on a sigmoid curve,as shown in Figures 71 and 8, and a num-ber of functions have been used in the pastto approximate the general shape of suchresponses. Parker and Waud,18 for instance,have highlighted that the rectangular hy-perbola, the integral of the Gaussian distri-bution curve, the arc-tangent, and the lo-gistic function have all been used by variousresearchers to empirically fit concentra-tion–response data. Some of these func-tions are more flexible than others; for in-stance, the rectangular hyperbola has a fixedslope of 1. In contrast, the logistic equationhas proven very popular in the fitting ofconcentration–response data:
FIGURE 8. The relationship between the Hill equation [Equation (15)] parameters, a and EC50, andthe operational model [Equation (14)] parameters KA, t, and Em, in the description of a concentra-tion–response curve of an agonist drug. It can be seen that each parameter of the Hill equation iscomposed of two operational model parameters.
Cri
tical
Rev
iew
s in
Bio
chem
istr
y an
d M
olec
ular
Bio
logy
Dow
nloa
ded
from
info
rmah
ealth
care
.com
by
201.
216.
146.
80 o
n 02
/14/
13Fo
r pe
rson
al u
se o
nly.
376
(16)
Part of the popularity of this equationis its flexibility and its ability to match theparameters of the Hill equation [Equation(1)] for empirical fitting purposes.
In general, the correct choice of expecta-tion function is most crucial when fittingmechanistic models. The difficulty in ascer-taining the validity of the underlying modelin these cases arises because the curve fittingprocess is undertaken with the automatic as-sumption that the model is a plausible oneprior to actually fitting the model and apply-ing some sort of diagnostics to the fit (seeAssessing the Quality of the Fit, below). Wemust always remain aware, therefore, thatwe will never really know the “true” model,but can at least employ a reasonable one thataccommodates the experimental findings and,importantly, allows for the prediction of test-able hypotheses. From a practical standpoint,this may be seen as having chosen the “right”mechanistic model.
3. Correlation of Parameters
When a model, either mechanistic orempirical, is applied to a dataset we gener-
ally consider each of the parameters to beresponsible for a single property of the curve.Thus, in the Hill equation, there is a slopeparameter, S, a parameter for the maximumasymptote (α), and a parameter for the loca-tion (K or EC50). Ideally, each of these pa-rameters would be entirely independent sothat error or variance in one does not affectthe values of the others. Such a situationwould mean that the parameters are entirelyuncorrelated. In practice it is not possible tohave uncorrelated parameters (see Figure9), but the parameters of some functions areless correlated than others. Strong correla-tions between parameters reduce the reli-ability of their estimation as well as makingany estimates from the fit of their variancesoverly optimistic.19
4. Distribution of Parameters
The operational model example can alsobe used to illustrate another practical con-sideration when entering equations for curvefitting, namely the concept of“reparameterization.”13 When fitting theoperational model or the Hill equation toconcentration–response curves, the param-eters may be entered in the equation in anumber of ways; for instance, the EC50 is
Ee X=
+ − +1
1 ( )α β
FIGURE 9. Altered estimates of the maximal asymptote, a, and the slope, S, obtained by fitting theHill equation to logistic data where the parameter K (log K) was constrained to differ from the correctvalue. The systematic relationship between the error in K and the values of the parameters S andα indicates that each is able to partially correct for error in K and thus are correlated with K.
Cri
tical
Rev
iew
s in
Bio
chem
istr
y an
d M
olec
ular
Bio
logy
Dow
nloa
ded
from
info
rmah
ealth
care
.com
by
201.
216.
146.
80 o
n 02
/14/
13Fo
r pe
rson
al u
se o
nly.
377
commonly entered as 10LogEC50. Thisreparameterization means that the regres-sion algorithm will actually provide the best-fit estimate of the logarithm of the EC50.Why reparameterize? As mentioned earlier,many of the assumptions of nonlinear re-gression rely on a Gaussian distribution ofexperimental uncertainties. Many modelparameters, including the EC50 of the Hillequation, the dissociation constant of a hy-perbolic radioligand binding equation, andthe τ parameter of the operational model,follow an approximately Gaussian distribu-tion only when transformed into loga-rithms.17 Thus, although not particularlyimportant for the estimation of the paramet-ric value, reparameterization can improvethe validity of statistical inferences madefrom nonlinear regression algorithms.13
Other examples of reparameterizations thatcan increase the statistical reliability of theestimation procedure include recasting timeparameters as reciprocals and counts of ra-dioactive decay as square roots.5
B. Assessing the Quality of theFit
The final determination of how “appro-priate” the fit of a dataset is to a model willalways depend on a number of factors, in-cluding the degree of rigor the researcheractually requires. Curve fitting for the de-termination of standard curves, for instance,will not warrant the same diagnostic criteriaone may apply to a curve fit of an experi-mental dataset that was designed to investi-gate a specific biological mechanism. In thecase of standard curves, an eyeball inspec-tion of the curve superimposed on the datais usually sufficient to indicate the reliabil-ity of the fit for that specific purpose. How-ever, when the fitting of models to experi-mental data is used to provide insight intounderlying biological mechanisms, the abil-
ity to ascribe a high degree of appropriate-ness to the resulting curve fit becomes para-mount.
1. Inspection
Although usually sufficient for empiri-cal models, an initial test for conformity ofthe data to any selected model is a simpleinspection of the curve fit superimposed onthe data. Although rudimentary, this proce-dure is quite useful in highlighting reallybad curve fits, i.e., those that are almostinvariably the consequence of having inad-vertently entered the wrong equation or set-ting certain parameter values to a constantvalue when they should have been allowedto vary as part of the fitting process. Assum-ing that visual inspection does not indicatea glaring inconsistency of the model withthe data, there are a number of statisticalprocedures that can be used to quantify thegoodness of the fit.
2. Root Mean Square
Figure 10 shows a schematic of an ex-perimental dataset consisting of 6 observa-tions (open circles labeled obs1 – obs6) andthe superimposed best-fit of a sigmoidalconcentration–response model [Equation(15)] to the data. The solid circles (exp1 –exp6) represent the expected response cor-responding to each X-value used for thedetermination of obs1 – obs6, derived fromthe model fit. The sum of the squared re-siduals, i.e., the sum of the squared differ-ences between the observed and expectedresponses has also been defined as the Er-ror Sum of Squares (SSE), and it is thisquantity that most researchers think of whendiscussing the sum-of-squares derived fromtheir curve fitting exercises [see Equation(13)]:
The SSE is sometimes used as an indexof goodness-of-fit; the smaller the value,the better the fit. However, in order to usethis quantity more effectively, an allowancemust also be made for the “degrees of free-dom” of the curve fit. For regression proce-dures, the degrees of freedom equal the to-tal number of datapoints minus the numberof model parameters that are estimated. Ingeneral, the more parameters that are addedto a model, the greater the likelihood ofobserving a very close fit of the regressioncurve to the data, and thus a smaller SSE.However, this comes at the cost of degreesof freedom. The “mean square error” (MSE)is defined as the SSE divided by the degreesof freedom (df):
(18)
Finally, the square root of MSE is equal tothe root mean square, RMS:
(19)
The RMS (sometimes referred to as Sy.x)is a measure of the standard deviation of theresiduals. It should be noted, however, thatalthough RMS is referred to as the “stan-
dard deviation” or “standard error” of themodel, this should not be confused with thestandard deviation or error associated withthe individual parameter estimates. The de-gree of uncertainty associated with anymodel parameter is derived by other meth-ods (see below).
3. R 2 (Coefficient of Determination)
Perhaps more common than the RMS,the R2 value is often used as a measure ofgoodness of fit. Like the r2 value fromlinear regression or correlation analyses,the value of R2 can range from 0 to 1; thecloser to 1 this value is, the closer themodel fits the dataset. To understand thederivation of R2, it is important to firstappreciate the other “flavors” of sums-of-squares that crop up in the mathematics ofregression procedures in addition to thewell-known SSE.
Using Figure 10 again as an example,the sum of the squared differences betweeneach observed response and the average ofall responses (obsav) is defined as the TotalSum of Squares (SST; sometimes denotedas Syy):
FIGURE 10. Relationship between a set of experimental observations (open circles; obs1 – obs6)and their corresponding least squares estimates (solid circles; exp1 – exp6). The horizontal dashedline represents the average of all the experimental observations (obsav).
The sum of the squared differences be-tween each estimated (expected) response,based on the model, and the average of allobserved responses is defined as the Re-gression Sum of Squares (SSR):
The total sum of squares, SST, is equalto the sum of SSR and SSE, and the goal ofregression procedures is to minimize SSE(and, as a consequence, SST).
Using the definitions outlined above,the value of R2 can be calculated as fol-lows:5,10
(23)
R2 is the proportion of the adjusted variancein the dependent variables that is attributedto (or explained by) the estimated regres-sion model. Although useful, the R2 value isoften overinterpreted or overutilized as themain factor in the determination of good-ness of fit. In general, the more parametersthat are added to the model, the closer R2
will approach a value of 1. It is simply anindex of how close the datapoints come tothe regression curve, not necessarily an in-dex of the correctness of the model, so whileR2 may be used as a starting point in theassessment of goodness of fit, it should beused in conjunction with other criteria.
4. Analysis of Residuals
Because the goal of least squares re-gression procedures is to minimize the sum
of the squares of the residuals, it is notsurprising that methods are available foranalyzing the final residuals in order to as-sess the conformity of the chosen model tothe dataset. The most common analysis ofresiduals relies on the construction of a scat-ter diagram of the residuals.13,20 Residualsare usually plotted as a function of the val-ues of the independent variable. If the modelis adequate in describing the behavior of thedata, then the residuals plot should show arandom scatter of positive and negative re-siduals about the regression line. If, how-ever, there is a systematic deviation of thedata from the model, then the residuals plotwill show nonrandom clustering of positiveand negative residuals. Figure 11 illustratesthis with an example of a radioligand com-petition binding experiment. When the dataare fitted to a model of binding to a singlesite, a systematic deviation of the pointsfrom the regression curve is manifested asclustering in the residuals plot. In contrast,when the same dataset is fitted to a model ofbinding to two sites, a random scatter of theresiduals about the regression line indicatesa better fit of the second model. This type ofresidual analysis is made more quantitativewhen used in conjunction with the “runstest” (see below).
There are many other methods of per-forming detailed analyses of residuals inaddition to the common method describedabove. These methods include cumulativeprobability distributions of residuals, χ2 tests,and a variety of tests for serial correla-tion.7,10,11,20
5. The Runs Test
The runs test is used for quantifyingtrends in residuals, and thus is an additionalmeasure of systematic deviations of themodel from the data. A “run” is a consecu-
RSSR
SST
SSE
SST2 1= = −
Cri
tical
Rev
iew
s in
Bio
chem
istr
y an
d M
olec
ular
Bio
logy
Dow
nloa
ded
from
info
rmah
ealth
care
.com
by
201.
216.
146.
80 o
n 02
/14/
13Fo
r pe
rson
al u
se o
nly.
380
tive series of residuals of the same sign(positive or negative). The runs test involvesa calculation of the expected number ofruns, given the total number of residualsand expected variance.20 The test uses thefollowing two formulae:
(24)
(25)
where Np and Nn denote the total number ofpositive and negative residuals, respectively.The results are used in the determination ofa P value.5,13 A low P value indicates a sys-tematic deviation of the model from the data.In the example shown in Figure 11, the one-site model fit was associated with a P valueof less than 0.01 (11 runs expected, 4 ob-served), whereas the two-site model gave a Pvalue of 0.4 (10 runs expected, 9 observed).
C. Optimizing the Fit
With the ubiquitous availability of pow-erful computers on most desktops, the im-pressive convergence speed of modern curvefitting programs can often lead to a falsesense of security regarding the reliability ofthe resulting fit. Assuming that the appro-priate model has been chosen, there are stilla number of matters the biomedical investi-gator must take into account in order toensure that the curve fitting procedure willbe optimal for their dataset.
1. Data Transformations
Most standard regression techniques as-sume a Gaussian distribution of experimen-tal uncertainties and also assume that anyerrors in Y and X are independent. As men-tioned earlier, however, these assumptionsare not always valid. In particular, the vari-ance in the experimental dataset can be
FIGURE 11. An example of residuals plots. The top panel represents a curve fit based on a onebinding site model to a data set obtained from a radioligand competition binding assay (left) andits corresponding residuals plot (right). Note the clustering of positive and negative residuals. Thebottom panel represents a curve fit based on a two binding site model to the same dataset (left)and its corresponding residuals plot (right). Note the random scatter of positive and negativeresiduals in this case.
Expected Runs=+
+2
1N N
N Np n
p n
Expected Variance=− −
+ + −2 2
12
N N N N N N
N N N Np n p n p n
p n p n
( )
( ) ( )
Cri
tical
Rev
iew
s in
Bio
chem
istr
y an
d M
olec
ular
Bio
logy
Dow
nloa
ded
from
info
rmah
ealth
care
.com
by
201.
216.
146.
80 o
n 02
/14/
13Fo
r pe
rson
al u
se o
nly.
381
heteroscedastic, that is, it changes in a sys-tematic fashion with the variables. Onemethod for optimizing the curve fitting pro-cess to adjust for heteroscedastic errors is toweight the data, as discussed earlier, whileanother approach is to transform the data toa form where the errors become morehomoscedastic prior to the application of theregression technique. Transformations suchas the square root or logarithm of the depen-dent or independent variables do not neces-sarily cause any problems of their own, pro-vided they reduce rather than increase anyheteroscedasticity in the data. In contrast,classical “linearising” transformations, wherea new variable is derived from both the origi-nal dependent and independent variables, arequite dangerous and it is unfortunate thatthey are still common practice in some labo-ratories. Indiscriminate data transforms ofthe latter kind are troublesome because theyhave the potential of distorting homoscedasticerrors in experimental uncertainties and thusviolating the assumptions of any subsequentregression procedure. Transforms are appro-priate if they have a normalizing effect onheteroscedastic errors; they are not valid oth-erwise. In addition, some data transforms,embodied in reciprocal plots (e.g.,Lineweaver-Burk) or the Scatchard transfor-mation, violate the assumption of indepen-dence between X and Y variables and areequally inappropriate. In contrast, transfor-mation of model parameters (as describedearlier) may often have an optimising effecton the fitting procedure.
2. Initial Estimates
All curve fitting algorithms require thespecification of initial estimates of the pa-rameters that are then optimized to yield thebest fit. No regression algorithm is perfect,and failure to specify reasonable parameterestimates may result in a failure of the algo-
rithm to converge or, more insidiously, aconvergence of the curve fit on a “localminimum.” If we recall our earlier discus-sion of the surface of the merit function thatthe various algorithms travel down, it ispossible to envisage a multiparameter modelthat results in a series of troughs such thatthe algorithm may settle in one as if it hasconverged on the best fit when, in fact, adeeper trough is available elsewhere on themerit function surface. This is an exampleof the program converging on a local mini-mum (Figure 12), where the curve fit is notoptimal although the user may think that thebest fit has been obtained. The best safe-guard against this problem is to perform theregression analysis a number of times usingdifferent initial estimates. A well-behavedmodel should converge on essentially thesame final estimates each time.
Some commercial programs make theprocess of finding initial parameter estimatesrelatively painless by incorporating approxi-mate rules that find initial estimates for theuser. Although this is expedient, there is nosubstitute for the researcher personally ad-dressing the issue of initial parameter esti-mates. This forces one to focus on the un-derlying model and the meaning of the modelparameters, and it is then not too difficult tocome up with a best guess. If further assis-tance is required, or if there are some pa-rameters that the user does not have a par-ticular “feel” for, then a simplex algorithmor a Monte Carlo-based algorithm (see be-low) may be utilized to derive estimates thatcan subsequently be improved upon by themore standard derivative-based algorithms.
D. Reliability of ParameterEstimates
The determination of the reliability ofthe estimated parameters derived from acurve fit is as important as the actual esti-
Cri
tical
Rev
iew
s in
Bio
chem
istr
y an
d M
olec
ular
Bio
logy
Dow
nloa
ded
from
info
rmah
ealth
care
.com
by
201.
216.
146.
80 o
n 02
/14/
13Fo
r pe
rson
al u
se o
nly.
382
mation of the parametric values themselves.All known methods for the calculation ofstandard errors and confidence intervalsfrom regression algorithms are based onthe mathematics of linear models. Sincenonlinear models are more common in bi-ology than linear models, it is perhaps dis-heartening to have to accept that there areno exact theories for the evaluation of para-metric errors in nonlinear regression. How-ever, there are a number of proceduresavailable for approximating these errorssuch that, in most practical applications, areasonable measure of parameter error isobtained.
1. Number of Datapoints
The number of experimental datapointscollected and analyzed will play a crucialrole in the curve fitting process in one (orboth) of two ways:
a. Determination of the appropriatenessof the model.
b. Determination of the accuracy of theparameter estimates.
Different measures for goodness-of-fithave already been covered, but some dis-cussion on the influence of datapoint num-
ber is also warranted at this point, since it
can form an important component of choos-ing the right model, that adequately accountsfor the data. Figure 13 illustrates the effectof datapoint number on one of the mostcommon statistical procedures utilized indiscriminating between variants of the samemodel, i.e., the “F-test” (or “extra-sum-of-squares” test). The actual test is describedin greater detail in the next section. Fornow, it is sufficient to point out that the F-test relies heavily on the degrees of freedomassociated with the fit to any model, whichare in turn dependent on the number ofdatapoints minus the number of parametersestimated. Although all the points in each ofthe panels in Figure 13 are taken from thesame simulated dataset, the “correct” model(a two binding site model) can only be sta-tistically resolved when the datapoints wereincreased from 6 (panel A) or 10 (panel B),to 20 (panel C).
Assuming that the researcher has apriori reasons for deciding that a particu-lar model is most appropriate under theircircumstances, the number of datapointswill still be crucial in determining theaccuracy of the parameters based on thatmodel. Table 1 lists the parameter esti-mates and corresponding 95% confidenceintervals of a two binding site model (i.e.,the correct model) applied to the datasets
FIGURE 12. Multiple minima in parameter space. The best fit is obtained at that set of parametervalues yielding the smallest possible sum of squares. Depending on the initial estimates, however,the fitting algorithm may converge on parameter sets which, although yielding a reduced sum ofsquares, do not correspond to the minimum possible sum of squares. The regression is then saidto have converged on a “local minimum.”
Cri
tical
Rev
iew
s in
Bio
chem
istr
y an
d M
olec
ular
Bio
logy
Dow
nloa
ded
from
info
rmah
ealth
care
.com
by
201.
216.
146.
80 o
n 02
/14/
13Fo
r pe
rson
al u
se o
nly.
383
of Panel A and Panel C, respectively, ofFigure 13. Although the final parameterestimates appear comparable in each in-stance, the fit based on the small numberof datapoints is associated with unaccept-ably large confidence intervals. There aresimply not enough points to accuratelydefine all the parameters of the model. Incontrast, increasing the number ofdatapoints to 20 allowed for reasonableestimates of the error associated with eachparameter estimate. The confidence inter-vals reported in the table were calculatedfrom the asymptotic standard errors de-
rived by the computer program from thefitting algorithm and are most likely un-derestimates of the true error (see below),thus rendering our (already shaken) con-fidence in the accuracy of minimal-data-point parameter esti-mates virtually nonexistent. There havebeen some methods presented in the lit-erature for maximizing the reliability ofparameter estimates under conditions ofminimal datapoint number (e.g., Refer-ences 21 and 22), but there really is nosubstitute for a good sampling of experi-mental datapoints.
FIGURE 13. Influence of data point number on choice of model. The radioligand competitionbinding curves above were simulated (with random error) according to a model for binding to two-sites. The sampling of points in each of the panels is from exactly the same simulated dataset. Thecurves in each panel are the least squares fit of the data to either a one- or two-site binding model,as determined by an F-test (see Section VI. E). Panels A (6 points) and B (10 points) were notstatistically significant from a one-site model. Only in panel C (20 points) were the data able to bestatistically resolved into the (correct) two-site model fit.
The most straightforward and conser-vative approach to building up an errorprofile of a given parameter is to simplyrepeat the same experiment many times,obtain single parameter estimates from eachindividual curve fit, and then derive themean and standard deviation (and error) ofthe parameters using standard textbookmethods. Assuming that each curve fit isperformed under optimal conditions, e.g.,appropriate number of datapoints, appro-priate transformation and weighting, etc.,biomedical research is still fraught withsmall overall sample sizes; it is not uncom-mon to see n = 3 – 6 given in many publi-cations as the number of times an experi-ment is repeated. As such, the conservative,albeit straightforward approach to param-eter error estimation just described maynot have the power to resolve small differ-
ences between experimental treatments, asit is based on small sample sizes and, fur-thermore, does not utilize all the availabledatapoints. The remaining methods forparameter error estimation utilize all thedatapoints in some form or other.
3. Parameter VarianceEstimates from AsymptoticStandard Errors
The standard errors reported by practi-cally all commercially available least squaresregression programs fall under this category.Asymptotic standard errors arecomputationally the easiest to determine and,perhaps not surprisingly, the least accurate.In most instances, these standard errors willunderestimate the true error that is likely tobe associated with the parameter of interest.
The calculation of the asymptotic stan-dard error and associated confidence inter-
Table 1Parameter Estimates and Associated Confidence Intervals from Fitting a Two-Site Modelof Radioligand Competition Binding to Different Data Point Numbers Taken from theSame Dataset (Panels A and C; Figure 13)
Parameter Estimate 95% Confidence Interval
Datapoints = 6Maximum Asymptotea –96.7 45.6 to 148.3Minimum Asymptoteb 1.3 –84.8 to 87.56Log IC50Highc –6.7 –9.93 to –3.53Log IC50Lowd –5.1 –16.1 to 5.9Fraction Highe 0.74 –1.4 to 2.9
Datapoints = 20Maximum Asymptote 99.9 95.4 to 104.4Minimum Asymptote 0.9 –5.5 to 7.4Log IC50High –6.8 –7.3 to –6.5Log IC50Low –5.6 –6.4 to –4.6Fraction High 0.64 0.4 to 0.8
a Y-axis value in the absence of competing drug.b Y-axis value in the presence of saturating concentrations of competing drug.c Potency estimate for competition at the high affinity binding site.d Potency estimate for competition at the low affinity binding site.e Fraction of high affinity binding sites.
Cri
tical
Rev
iew
s in
Bio
chem
istr
y an
d M
olec
ular
Bio
logy
Dow
nloa
ded
from
info
rmah
ealth
care
.com
by
201.
216.
146.
80 o
n 02
/14/
13Fo
r pe
rson
al u
se o
nly.
385
vals involves matrix algebra, but may besummarized as follows:23
1. Determine the Hessian (or “informa-tion”) matrix. This is the matrix con-taining the second derivatives of theparameters with respect to the mini-mized χ2 merit function.
2. Evaluate the variance-covariance ma-trix by multiplying the inverse of theHessian matrix by the variance of theresiduals of the curve fit.
3. The diagonal elements of the resultingvariance-covariance matrix are thesquares of the asymptotic standard er-rors; the off-diagonal elements of thematrix are the covariances of the pa-rameters, and are a measure of theextent to which the parameters in themodel are correlated with one another.
The computer program then reports theresulting standard errors. For these errors toactually be a good measure of the accuracyof the parameter estimates, the followingassumptions must hold:23
a. The fitting equation is linear.b. The number of datapoints is very large.c. The covariance terms in the variance-
covariance matrix are negligible.
For nonlinear models, the first assump-tion is invalid, however, the impact of fail-ure to conform to this assumption may belessened for models that are well behaved,e.g., contain conditionally linear parametersor can be approximated by linear functions.The second assumption can also be reason-able provided the experimenter is able toensure an adequate sampling of datapoints.Unfortunately, the third assumption is al-most never realized. As described earlier,most parameters in nonlinear models showsome degree of correlation with one an-other; indeed, high correlations are indica-tive of parameter redundancies in the model.
As such, ignoring the covariances from thevariance-covariance matrix in the reportingof parameter errors will underestimate thetrue error.
Nevertheless, asymptotic standard er-rors may serve a useful diagnostic role. Sincethey will invariably be underestimates ofthe true error, very large standard errors orconfidence intervals reported after a curvefit are indicative of a very poor fit of theassociated parameter (see Table 1). Thismay occur, for instance, because the param-eter is ill defined by the available data.
4. Monte Carlo Methods
The most reliable method for the deter-mination and validation of model parameterconfidence intervals is also the most com-puter-intensive. Monte Carlo simulations in-volve the generation of multiple (hundreds tothousands) of pseudodatasets, based on achosen model, and the subsequent analysisof the simulated datasets with the same modelused to generate them followed by construc-tion of a frequency histogram showing thedistribution of parameter estimates.17,24 Fig-ure 14 shows a flowchart summarizing thegeneral approach to Monte Carlo simulation.
The crucial factor in the implementationof the Monte Carlo approach is the ability toadd random “error” to the pseudodatasetpoints that accurately reflects the distributionof experimental uncertainties associated withthe determination of “real” datasets. The bestdeterminant of this error is the variance ofthe fit of the chosen model to real experimen-tal data, provided that the standard assump-tions underlying least squares regressionanalyses are valid. In addition to the appro-priate choice of variance for the simulations,other key features in this approach are thechoice and the number of independent vari-ables, which again should match those deter-mined in a typical experiment.
Cri
tical
Rev
iew
s in
Bio
chem
istr
y an
d M
olec
ular
Bio
logy
Dow
nloa
ded
from
info
rmah
ealth
care
.com
by
201.
216.
146.
80 o
n 02
/14/
13Fo
r pe
rson
al u
se o
nly.
386
The beauty of the Monte Carlo approachis that the level of accuracy with regard tothe confidence interval profiles is very muchin the hands of the researcher; the greaterthe number of simulated datasets, the greaterthe resolution of the confidence intervals.However, this comes at the expense of com-puter time; a Monte Carlo simulation of1000 datasets may take 1000 times longerthan a least squares fit of the actual experi-mental dataset used to pattern the simula-tions. Coupled with the fact that many com-mercially available curve fitting packagesdo not contain Monte Carlo-compatible pro-gramming features, the time factor involvedin generating parameter confidence inter-vals from the Monte Carlo approach dis-suades many researchers from routinelyusing this method. Nonetheless, great in-sight can be gained from Monte Carlo ap-proaches. For instance, in addition to pro-viding the greatest degree of accuracy in
parameter error estimation, Monte Carlomethods can also guide the experimentertoward the most appropriate modelreparameterizations in order to optimize theactual curve fitting procedure.17
One potential problem with the stan-dard Monte Carlo approach is that it is nec-essary to define the population distributionsfor the errors applied to the datapoints. Anormal distribution is most commonly used,but it is not always clear that it is appropri-ate. The bootstrap, described below, explic-itly overcomes that problem.
5. The Bootstrap
“Bootstrapping” is an oddly-named pro-cess that allows an approximate reconstructionof the parameters of the population from whichthe data have been (at least conceptually)
FIGURE 14. A general approach to Monte Carlo simulation.
Cri
tical
Rev
iew
s in
Bio
chem
istr
y an
d M
olec
ular
Bio
logy
Dow
nloa
ded
from
info
rmah
ealth
care
.com
by
201.
216.
146.
80 o
n 02
/14/
13Fo
r pe
rson
al u
se o
nly.
387
sampled.25 Bootstrapping differs from standardMonte Carlo methods in that it makes no as-sumption about the form of the population, andinstead assumes that the best estimate of theproperties of the population is the experimen-tally determined dataset. The population is re-constructed by repeated resampling of thedatapoints to give a large number (hundreds oreven thousands) of new pseudodatasets. Theresampling is done “with replacement,” whichis to say that any particular real datapoint canappear in each pseudodataset more than onetime. The result is a population of pseudodatasetsthat represents a pseudo-population that hasapproximately the same properties as the origi-nal population.
Bootstrapping can be used in severalways relevant to model fitting. First, it canprovide a pseudopopulation of any param-eter calculable from each pseudodataset.Thus it can be used to give confidence inter-vals for fitted parameters obtained frommethods that do not directly provide esti-mates of parameter variance, such as thesimplex method. Similarly, it has been usedto estimate the reliability of the varianceestimates obtained from other methods thatrely on the covariance matrix.19
Bootstrapping is not without poten-tial problems. One arises from the factthat the real dataset is unlikely to includeany samples from the extreme tails of theoverall population of possible datapoints.This means that bootstrapped populationsgenerally have less area under the ex-treme tails than the real population fromwhich the data were sampled. There arecorrections that can be applied,25 butbootstrapping is not universally acceptedby statisticians.
6. Grid Search Methods
Another computer-intensive approach toerror determination involves the construc-
tion of multidimensional grids based onmodel parameter values and then “search-ing” for those parameter value combina-tions where the variance of the overall fitincreases significantly. The confidence in-tervals are then defined as those regions ofthe grid (which resemble a multidimensionalellipsoid) that surround the minimum overwhich the variance does not change signifi-cantly.8,23
7. Evaluation of JointConfidence Intervals
As discussed earlier, the parametersin most models tend to show some corre-lation with one another. The evaluation ofjoint confidence intervals is a procedurethat is designed to include the covarianceof the parameters in the determination ofparameter error estimates.8,13 The equa-tions underlying this approach, however,assume that the fitting equation is linearin order to derive a symmetrical ellipti-cally-shaped confidence interval profileof parameters. Unfortunately, this methodyields asymmetric confidence regions forthose nonlinear models that cannot ap-proximate to a linear model, and is thusnot as reliable as Monte Carlo or Gridsearch methods.
E. Hypothesis Testing
Often, the desire to ascertain the stan-dard error or confidence interval associ-ated with model parameters is a prelude tothe statistical testing of the parameters ac-cording to a particular hypothesis. There-fore, some objective statistical test is re-quired in order to allow for comparisonsbetween parameters or comparisons be-tween models.
Cri
tical
Rev
iew
s in
Bio
chem
istr
y an
d M
olec
ular
Bio
logy
Dow
nloa
ded
from
info
rmah
ealth
care
.com
by
201.
216.
146.
80 o
n 02
/14/
13Fo
r pe
rson
al u
se o
nly.
388
1. Assessing Changes in aModel Fit between ExperimentalTreatments
There are three broad approaches toperforming statistical comparisons be-tween the same model parameters beforeand after an experimental treatment. Thefirst relies on the use of standard paramet-ric tests, such as the Student’s t-test. Thesecond approach relies on more computer-intensive, but preferable comparisons be-tween parameters based on permutationtests. The third approach differs from theother two in that it uses all the experimen-tal data generated before and after a par-ticular treatment in a comparison of glo-bal changes in goodness of fit. The lastprocedure may be summarized as fol-lows.5,26
1. Analyze each dataset separately.2. Sum the SSE resulting from each fit to
give a new “total” sum-of-squaresvalue (SSA). Similarly, sum the twodegrees of freedom values from eachfit to give a “total” degrees of freedom(dfA).
3. Pool the two sets of data into one largeset.
4. Analyze this new “global” dataset toobtain a new sum-of-squares value(SSB) and degrees of freedom (dfB).
5. Calculate the following F ratio:
(26)
The F value is used to obtain a P value,with the numerator having (dfB – dfA) de-grees of freedom and the denominator hav-ing dfA degrees of freedom. A small P value(i.e., large F value) indicates that the indi-vidual fits are better than the global, pooledfit, i.e., the experimental treatment resulted
in a significant difference in the model pa-rameters between the two datasets.
2. Choosing between Models
The F ratio can also be used to comparethe fit of a single dataset to two differentversions of the same model:
(27)
In this instance, SS1 and df1 are definedas the SSE and degrees of freedom, respec-tively, of the model with fewer parameters,whereas SS2 and df2 are defined as the SSEand degrees of freedom, respectively, of themodel with the greater number of param-eters. The addition of more parameters to amodel will result in an improvement of thegoodness of fit and a reduction in SSE, butat the cost of degrees of freedom. The F test[Equation (27)] attempts to quantify whetherthe loss of degrees of freedom on goingfrom a simpler to a more complicated modelis worth the gain in goodness of fit. A lowP value is indicative of the more compli-cated model being the statistically bettermodel. It should be noted, however, that theF test can only be applied to two differentversions of the same model, e.g., a one bind-ing-site versus a two binding-site curve fit.In addition, the F test is particularly harshsince it relies so heavily on degrees of free-dom and, hence, datapoints and number ofparameters. As a consequence, the test maybe too conservative and reject the morecomplicated model for the simpler one, evenwhen this is not the case. Thus, results fromthe test should be regarded with caution ifthe number of datapoints is limited and othermeasures of goodness of fit appear to indi-cate that the simpler model is not a reason-able fit to the data. When in doubt, repeat
FSS SS df df
SS dfB A B A
A A
= − −( ) / ( )
/
FSS SS df df
SS df= − −( ) / ( )
/
1 2 1 2
2 2
Cri
tical
Rev
iew
s in
Bio
chem
istr
y an
d M
olec
ular
Bio
logy
Dow
nloa
ded
from
info
rmah
ealth
care
.com
by
201.
216.
146.
80 o
n 02
/14/
13Fo
r pe
rson
al u
se o
nly.
389
the experiment with greater numbers ofdatapoints.
VII. FITTING VERSUS SMOOTHING
Throughout this article, the process offitting empirical or mechanistic models toexperimental data has generally been encom-passed within the umbrella term “curve fit-ting.” However, some distinctions can bemade. Simulation refers to the processwhereby the properties of the model are ex-amined in order to determine the theoreticalconsequences of imposing specified condi-tions on the parameters and variables. Theterm fitting refers to the process whereby themodel parameters are altered to discoverwhich set of parameter values best approxi-mate a set of experimental observations de-rived from the actual system of interest. Aspecial case of the fitting process is the pro-cedure known as smoothing, whereby a modelis chosen to generate a fit that simply passesnear or through all the experimental datapointsin order to act as a guide for the eye.
If the purpose of the curve fitting proce-dure is simply to smooth or to generate a
standard curve for extrapolation, then thenature of the underlying model and accom-panying regression technique is not crucial.If, however, the purpose of the curve fittingprocedure is to obtain insight into the fea-tures of the model that describe an aspect ofthe biological system of interest, then thechoice of model is paramount. Althoughlinear models can give curved lines, (e.g.,the polynomial equations described earlier),most biological experiments that yield datadescribed by a curve are probably best ana-lyzed using nonlinear regression. This isbecause it is much more common to find anonlinear model that can be related in ameaningful and realistic fashion to the sys-tem under study than a general linear model.
VIII. CONCLUSION
Computerized curve fitting has becomenearly ubiquitous in the analysis of bio-medical research. The ease of use and speedof the modern curve fitting programs en-courage researchers to use them routinelyfor obtaining unbiased parameter estimateswhere in the not very distant past, they might
Table 2Selected List of Commercially-Available Curve Fitting Programs and Their AssociatedLeast Squares Algorithms. Distributors are listed in parentheses
Program Algorithm
Enzfitter (Biosoft) Levenberg-Marquardt; SimplexExcel (Microsoft) SimplexFig. P (Biosoft) Levenberg-MarquardtKaleidagraph (Synergy) Levenberg-MarquardtKELL (Biosoft) Levenberg-MarquardtOrigin (Microcal) Levenberg-MarquardtPrism (GraphPad) Levenberg-MarquardtProFit (QuantumSoft) Levenberg-Marquardt; Robust; Monte Carlo (Simplex)Scientist (Micromath) Levenberg-Marquardt; SimplexSigmaPlot (SPSS) Levenberg-Marquardt
Cri
tical
Rev
iew
s in
Bio
chem
istr
y an
d M
olec
ular
Bio
logy
Dow
nloa
ded
from
info
rmah
ealth
care
.com
by
201.
216.
146.
80 o
n 02
/14/
13Fo
r pe
rson
al u
se o
nly.
390
have used eyeballing or linearization pro-cesses that would have contained substan-tial subjective elements and systematic dis-tortions. Nevertheless, indiscriminate use ofcurve fitting without regard to the underly-ing features of the model and data is a haz-ardous approach. We hope that the contentof this chapter is useful in illustrating bothstrengths and some pitfalls of computer-based curve fitting, and some ways to opti-mize the quality and utility of the param-eters so obtained.
IX. SOFTWARE
Table 2 contains a limited sampling ofcommercially available curve fitting pro-grams. Some of them (e.g., EnzFitter andKELL) are more specialized in their appli-cations than others, but all are commonlyapplied to curve fitting of biological modelsto data. Also shown in the table are theassociated regression algorithms utilized byeach program.
REFERENCES
1. Wells, J.W., Analysis and interpreta-tion of binding at equilibrium, in Re-ceptor-Ligand Interactions: A Practi-cal Approach, E.C. Hulme, Ed., OxfordUniversity Press, Oxford, 1992, 289–395.
2. Hil l , A.V., The possible effects ofthe aggregation of the moleculesof haemoglobin on i ts dissociat ioncurves , J . Phys io l. , 40 , i v–v i i ,1910.
3. Hill, A.V., The combinations of hae-moglobin with oxygen and with carbonmonoxide. I, Biochem. J., 7, 471–80,1913.
4. Motulsky, H.J., Intuitive Biostatistics,Oxford University Press, New York, 1995.
5. Motulsky, H.J., Analyzing Data withGraphPad Prism, GraphPad Software Inc.,San Diego, CA, 1999.
7. Bates, D.M. and Watts, D.G., NonlinearRegression Analysis and Its Applications,Wiley and Sons, New York, 1988.
8. Johnson, M.L. and Faunt, L.M., Param-eter estimation by least-squares methods,Methods Enzymol., 210, 1–37, 1992.
9. Press, W.H., Teukolsky, S.A., Vetterling,W.T., and Flannery, B.P., Numerical Reci-pes in C. The Art of Scientific Computing,Cambridge University Press, Cambridge,MA, 1992.
10. Gunst, R.F. and Mason, R.L., RegressionAnalysis and Its Applications: A Data Ori-ented Approach, Marcel Dekker, New York,1980.
11. Cornish-Bowden, A., Analysis of EnzymeKinetic Data, Oxford University Press,New York, 1995.
12. Johnson, M.L., Analysis of ligand-bind-ing data with experimental uncertaintiesin independent variables, MethodsEnzymol., 210, 106–17, 1992.
13. Motulsky, H.J. and Ransnas, L.A., Fittingcurves to data using nonlinear regression:a practical and nonmathematical review,FASEB J., 1, 365–74, 1987.
14. Jurs, P.C., Curve fitting, in Computer Soft-ware Applications in Chemistry, Wiley andSons Inc., New York, 1996, 25–51.
15. Black, J.W. and Leff, P., Operationalmodels of pharmacological agonism, Proc.Roy. Soc. (Lond.) B., 220, 141–62, 1983.
16. Leff, P., Prentice, D.J., Giles, H., Martin,G.R., and Wood, J., Estimation of agonistaffinity and efficacy by direct, operational
Cri
tical
Rev
iew
s in
Bio
chem
istr
y an
d M
olec
ular
Bio
logy
Dow
nloa
ded
from
info
rmah
ealth
care
.com
by
201.
216.
146.
80 o
n 02
/14/
13Fo
r pe
rson
al u
se o
nly.
391
model-fitting, J. Pharm. Methods, 23, 225–37, 1990.
17. Christopoulos, A., Assessing the distribu-tion of parameters in models of ligand-receptor interaction: to log or not to log,Trends Pharmacol. Sci., 19, 351–7, 1998.
18. Parker, R.B. and Waud, D.R., Pharmaco-logical estimation of drug-receptor disso-ciation constants. Statistical evaluation. I.Agonists, J. Pharmacol. Exp. Ther., 177,1–12, 1971.
