Mount Sinai School of MedicineSystems Biology—Biomedical Modeling

Development of Models II:Model Fitting and Error Estimation

Kevin D. Costawith

Steven H. Kleinstein and Uri Hershberg

1

Biomathematical Model

• A system of mathematical equations or computer simulations that provides a quantitative picture of how a complex biological system functions under healthy and diseased conditions

• Computational models use numerical methods to examine mathematical equations or systems of equations too complex for analytical solution

2

Advantages of the Modeling Approach

• Concise summary of present knowledge of operation of a particular system

• Predict outcomes of modes of operation not easily studied experimentally in a living system

• Provide diagnostic tools to test theories about the site of suspected pathology or effect of drug treatment

• Clarify or simplify complex experimental data

• Suggest new experiments to advance understanding of a system

3

Limitations of the Modeling Approach

• Models often require many simplifying assumptions– beware of garbage in, garbage out

• Validation of model predictions is essential– examination of behavior under known limiting

conditions– experimental validation– limits of model performance can point out

what we don’t understand about a system

4

Forward Model

– Parameter values often obtained from published literature– Ex: cardiac electromechanical coupling, cell signaling networks

• Used for simulating realistic experimental data under precisely defined conditions to test hypotheses in silico

• Can help design better experiments and reduce animal use

• Generally too complicated for fitting to experimental data

• Allows generation of synthetic data sets with prescribed noise characteristics (Monte Carlo simulation) for evaluating parameters obtained by inverse modeling

• A detailed mathematical model designed to incorporate a desired level of anatomic, physical, or physiologic features– Can have arbitrary complexity as desired Trayanova and Tice, 2009

5

Inverse Model

• A mathematical model designed to fit experimental data so as to explicitly quantify physical or physiological parameters of interest

• Values of model elements are obtained using parameter estimation techniques aimed at providing a “best fit” to the data

• Generally involves an iterative process to minimize the average difference between the model and the data

• Evaluating the quality of an inverse model involves a combination of established mathematical techniques as well as intuition and creative insight

6

Characteristics of a Good Inverse Model

• Fit is good—model should be able to adequately describe a relatively noise-free data set (of course a poor fit provides some insight also)

• Model parameters are unique– Theoretically identifiable for noise-free data– Well-determined model parameters in presence of

measurement noise

• Values of parameter estimates are consistent with hypothesized physical or physiologic meanings and change appropriately in response to alterations in the actual system

7

Forward-Inverse Modeling

• A process of combined data simulation and model fitting used for evaluating the robustness, uniqueness, and sensitivity of parameters obtained from an inverse model of interest

• A powerful tool for improving data analysis and understanding the limitations on model parameters used for system characterization and distinguishing normal from abnormal populations

8

Six Steps for Inverse-Modeling of Data

1. Select an appropriate mathematical model• Polynomial or other functional form• Based on underlying theoretical equations

2. Define a “figure of merit” function• Measures agreement between data & model for given parameters

3. Adjust model parameters to get a “best fit”• Typically involves minimizing the figure of merit function

4. Examine “goodness of fit” to data• Never perfect due to measurement noise

5. Determine whether a much better fit is possible• Tricky due to possible local minima vs. global minimum• F-test for comparing models of different complexity

6. Evaluate accuracy of best-fit parameter values• Provide confidence limits and determine uniqueness• Assess physical reasonability of estimated parameter values

9

Selecting the Model• “Trend lines”

– Polynomial, exponential, and other standard functions are often used when a data set seems to follow a mathematical trend but the governing formula is not known

• Physically-based equations– Given knowledge of a governing physical process,

the desired model equations are derived from underlying theoretical principles

– Resulting model parameters have a specific physical interpretation

10

Least-Squares Error Minimization

y

x

data (xi,yi)

x

xx

xx

xx x x

x model (xi,ŷi)

11

• Goal is to fit N data points (xi, yi) i=1..N

ˆ y i ˆ y (x i,a1..aM )• The model is a function with M adjustable parameters ak, k=1..M used to generate N model points (xi, ŷi)

y i ˆ y (x i,a1..aM )• The residual measures the difference

between a data point and the corresponding model estimate

[y i ˆ y (x i,a1..aM )]i1

N

• Since residuals can be positive or negative,

a sum of residuals is not a good measure of overall error in the fit

E [y i ˆ y (x i,a1..aM )]2

i1

N

• A better measure is the sum of squared

residuals, E, which is only zero if each and every residual is zero

Maximum Likelihood Estimation

• Not meaningful to ask “What is the probability that my set of model parameters is correct?”– Only one correct parameter set Mother Nature!

• Better to ask “Given my set of model parameters, what is the probability that this data set could be obtained?”– What is the likelihood of the parameters given the

data?

• Inverse modeling is also known as “maximum likelihood estimation”.

12

The Chi-Square Error Measure and Maximum Likelihood Estimation

• For Gaussian distribution of measurement noise with varying standard deviation, i, the probability, P, of the data set coming from the model parameters is given by

• Maximizing this probability involves maximizing ln(P) or minimizing –ln(P), yielding the chi-square function of weighted residuals, 2

– The “weight” is the inverse of the variance of each measurement (wi = i

-2)– Other functions may be useful for non-

Gaussian measurement noise, yielding so-called “robust estimation” methods

• If variance is assumed to be uniform, then let = constant = 1, and chi-square function yields the sum of squared residuals function defined earlier

P exp [y i ˆ y (x i)]

2

2 i2

i1

N

ln(P) [y i ˆ y (x i)]

2

i2

i1

N

2

2 |1 [y i ˆ y (x i)]2

i1

N

E

13

Minimizing Chi-Square• Since the error in the model fit depends on the model

parameters, ak, minimizing the chi-square function requires finding where the derivatives are zero

• This yields a general set of M (nonlinear) equations for the M unknowns ak

• The model derivatives dŷ/dak are often known exactly, or may be approximated numerically using finite differences

2 [y i ˆ y (x i)]

2

i2

i1

N

( 2)

ak

2[y i ˆ y (x i)]

i2

ˆ y (x i,a1..aM )

ak

i1

N

0 ; k 1..M

14

Linear Regression Analysis• Consider a set of measurements of

photodetector voltage (dependent variable) as a function of incident laser intensity (independent variable).

• Define the least-squares error norm quantifying the “goodness” of the linear fit. Then adjust model parameters a and b to minimize this error.

E(a,b) y i (a bx i) 2

i1

N

• We propose to examine a linear relationship between voltage (y) and intensity (x).

ŷ = a + bx

x = [x1, x2, …, xN]

y = [y1, y2, …, yN]

ŷ = [ŷ1, ŷ2, …, ŷN]

• We need to find the best set of values of a and b to fit the data.

data

data

model

ŷ = a +bx

15

syx N 1

N 2(sy

2 b2sx2)

• Standard error of the estimate, Standard error of the estimate, ssy•xy•x approximates standard deviation of approximates standard deviation of population about the line of means in population about the line of means in terms of regression slope, terms of regression slope, bb, and , and xx and and yy standard deviations, standard deviations, ssxx and and ssyy

sa sx,y

1

N

x 2

(N 1)sx2

sb 1

N 1

sx,y

sx

• Standard error of slope and intercept, Standard error of slope and intercept, ssaa and and ssbb, used for , used for tt-test of -test of aa,,b b = 0 or = 0 or to compute confidence intervals to compute confidence intervals

Computing Model Parameters for Linear Regression

E(a,b) y i (a bx i) 2

i1

N

b x iy i N x y

x i2 N(x )2 a y b x

E(a,b)

a0

E(a,b)

b0

16

• We can determine the best values of We can determine the best values of aa and and bb by calculating the partial by calculating the partial derivatives of derivatives of EE with respect to with respect to aa and and bb, and setting these to zero. This , and setting these to zero. This yields 2 equations in terms of the yields 2 equations in terms of the mean values and to be solved for mean values and to be solved for the 2 unknowns the 2 unknowns aa and and bb, yielding:, yielding:

x

y

Regression versus Correlation• Correlation coefficient describes Correlation coefficient describes

the strength of the association the strength of the association between the two variablesbetween the two variablesrr +1 if they increase together +1 if they increase togetherrr -1 if one decreases as other increases -1 if one decreases as other increasesrr 0 if they do not relate to one another 0 if they do not relate to one another

• The correlation coefficient can be The correlation coefficient can be related to results of the regressionrelated to results of the regression

• Unlike the regression parameters, Unlike the regression parameters, aa and and bb, the correlation , the correlation coefficient, coefficient, rr, is symmetric in , is symmetric in xx and and yy and therefore does not and therefore does not require choosing of independent require choosing of independent and dependent variables and dependent variables

r (x i x )(y i y )

(x i x )2(y i y )2

1 (N 2)

(N 1)

sx,y2

sy2

17

Linearization of Nonlinear Models• Many nonlinear equations can be “linearized”

by selecting a suitable change of variables– (A) Nonlinear Michaelis-Menten model of enzymatic

reaction rate, V, in terms of substrate concentration [S], and kinetic constants, Vmax and Km

– (B) Linearized double reciprocal Lineweaver-Burk representation of Michaelis-Menten equation

adapted from Lobemeier, 2000

V Vmax

[S]

Km [S]

1

V

Km

Vmax

1

[S]

1

Vmax

18

• Historically this has been a common approach in analysis of scientific data, mainly due to ease of implementation

• However, “linearization” often distorts the error structure, violates key assumptions, and affects resulting model parameter values, which may lead to incorrect conclusions

• In our modern era of computers it is usually wisest to perform nonlinear least squares analysis when dealing with nonlinear data sets

General Model Fitting

• It is important to understand where these regression It is important to understand where these regression equations come from, but this is rarely done by hand.equations come from, but this is rarely done by hand.

• Spreadsheet programs typically have several trend-Spreadsheet programs typically have several trend-line functions built in, including nonlinear models line functions built in, including nonlinear models which follow the same idea but cannot readily be which follow the same idea but cannot readily be solved analytically.solved analytically.

• Often in biomedical experiments, a data set is Often in biomedical experiments, a data set is governed by a system of equations determined by governed by a system of equations determined by underlying physical principles rather than just the underlying physical principles rather than just the apparent shape of the curve. apparent shape of the curve.

19

Nonlinear Model Fitting• The selected model ŷ is a nonlinear

function of model parameters ak, k=1..M comprising the vector a

• The 2 merit function is still given by

• The gradient with respect to model parameters ak must approach zero at minimum 2

• However, because the gradients are nonlinear functions of a, minimization must proceed by iteratively updating a until 2 stops decreasing.

• In the steepest descent method, the constant, , must be small enough to not exhaust the downhill direction.

ˆ y i ˆ y (x i,a)

2(a) [y i ˆ y (x i,a)]2

i2

i1

N

( 2)

ak

2[y i ˆ y (x i,a)]

i2

ˆ y (x i,a)

ak

i1

N

anext a current 2(a current )

• Alternative numerical methods include the inverse-Hessian method, the popular hybrid Levenberg-Marquardt method, and the robust but inefficient full Newton-type methods.

20

2

Goodness of Fit and theResiduals Plot• The correlation coefficient (r2) is

often used to characterize the goodness of fit between model and data

• However, a high correlation can exist even for a model that systematically differs from the data (all 3 examples have r2 > 0.99)

adapted from Lobemeier, 200021

model fitsA

B

C

residualsA

B

C

• One must also examine the distribution of residuals—a good model fit should yield residuals equally distributed along x and normally distributed around zero with no systematic trends, as in A rather than B or C

Global Error Minimization• The error function depends on

model parameters ak, and can be thought of as an M-dimensional “surface” of which we seek the minimum

• Depending on the complexity of the model (i.e. the number of model degrees of freedom) the error surface may be quite “bumpy”

• A challenge is to ensure that a given set of “optimal” model parameters represents the true global minimum of the error surface, and not a local minimum

• This can be tested by varying the initial guesses and comparing the resulting model parameters and fitting error

from Numerical Recipes online

22

Implementation in MATLAB% basic model fitting routineglobal known;filename = input('Enter the name of file: ','s'); % read x and y data from filedata = dlmread(filename);x_data = data(:,1);y_data = data(:,2);known = 5; % assign known model parametersguess = [.1 .1 1 1]; % guess initial values of unknown model parametersLB = [-1000, -1000, -1000, -1000]; % LB and UB can be used to enforce lower and upperUB = [1000, 1000, 1000, 1000]; % bounds on parameter values if desired

[optimum,resnorm,residual] = lsqcurvefit(@model,guess,x_data,y_data,LB,UB) % optimum contains the fitted model parameters % resnorm is the sum of squared residuals fitting error % residual is a vector of residuals for further inspection

y_model=model(optimum,x_data); % generate vector of simulated data for plotting plot(x_data,y_data,'bx',x_data,y_model,'r-');xlabel('Independent Variable (units)');ylabel('Dependent Variable (units)');

file model.mfunction y = model(a,x)global known;y = a(1)+a(2)*x.^2+a(3).*sin(a(4).*x) – known; % model may depend on known variablesreturn

fit the datafit the data

23

MATLAB Example (continued)y=a(1)+a(2)*x.^2+a(3).*sin(a(4).*x) - known

x data- initial guess- optimal fit

X values

Y v

alue

s

guess= 0.8000 1.5000 2.0000 2.0000

optimum = 0.0288 2.2895 0.9220 1.7929

resnorm = 12.2049

true= known= 1.0000 2.0000 -1.0000 6.0000 5.0000

24

local minimumx data- initial guess- optimal fit

X values

Y v

alue

s

guess= 0.8000 1.5000 -2.0000 5.0000

optimum = 1.0000 2.0000 -1.0000 6.0000

resnorm = 8.0159e-10

global minimum

Comparing Two Model Fits• The number of data points, N, must exceed

the number of model parameters, M, yielding the degrees of freedom (DOF = N-M)

• Increasing M using a more complex model will generally improve the quality of fit and reduce 2

M N 1

y

x

model 1 model 2

25

F

(simple2 complex

2 )

(DOFsimple DOFcomplex)

complex2

DOFcomplex

• An F-statistic can be computed to compare the results of two model fits – F ~ 1, the simpler model is adequate– F > 1, the more complex model is better, or

random error led to a better fit with the complex model

– P-value defines the probability of such a “false positive” result (lookup in F table)

MSE 2

N M

2

DOF

• The mean squared error, MSE, can be used to compare two models fit to a given data set

• Increasing MSE with decreasing 2 can reveal an over-parameterized model

df1 DOFsimple DOFcomplex Mcomplex Msimple

df2 DOFcomplex N Mcomplex

26

D(o)

D(1)

D(2)

D(3)

Accuracy of Estimated Model Parameters

• The underlying true set of system parameters, atrue, is known to Mother Nature but unknown to the experimenter

• Fitting D(0) using 2 minimization yields the estimated model parameters a(0)

• Other experiments could have resulted in data sets D(1), D(2), etc. which would have yielded model parameters a(1), a(2), etc.

• We wish to estimate the probability distribution of a(j) - atrue without knowing atrue and without a limitless number of hypothetical data sets. Hmmmm…

• True parameters are statistically realized, along with measurement errors, as the measured data set D(0)

from Numerical Recipes online

27

DS(1)

DS(2)

DS(3)

DS(4)

D(o)

Monte Carlo Simulation of Synthetic Data Sets

from Numerical Recipes online

28

2 [y i ˆ y (x i,a(0))]

2

i1

N

N M

• Note: if experimental i2 are not known, can

estimate after fit and use randn function in MATLAB

• Assume that if a(0) is a reasonable estimate of atrue, then the distributionof a(j)-a(0) should be similar to that of a(j)-atrue

• With the assumed a(0), and some understanding of the characteristics of the measurement noise, we can generate “synthetic data sets” DS

(1), DS(2),… at

the same xi values as the actual data set, D(0), that have the same relationship to a(0) as D(0) has to atrue

• For each DS(j), perform a model fit to obtain

corresponding aS(j), yielding one point aS

(j)- a(0) for simulating the desired M-dimensional probability distribution. This is a very powerful technique!!

2-parameter probability distribution for 1,000 Monte Carlo simulations

The Bootstrap Method• If you don’t know enough about the measurement errors

(i.e. cannot even say they are normally distributed) then Monte Carlo simulation cannot be used.

• Bootstrap Method uses actual data set D(0), with its N data points, to generate synthetic data sets DS

(1), DS(2),… also

with N data points.

• Randomly select N data points from D(0) with replacement, which makes DS

(j) differ from D(0) with a fraction of the original points replaced by duplicated original points.

• The 2 merit function does not depend on the order of (xi,yi), so fitting the DS

(j) data yields model parameter sets aS(j) as

with Monte Carlo, except using actual measurement noise.29

Confidence Intervals and Accuracy of Model Parameters

• The probability distribution is a function defined on M-dimen-sional space of parameters a

• A confidence interval is a region that contains a high percentage of the total distribution relative to model parameters of interest

• You choose the confidence level (e.g. 68.3%, 90%, etc.) and the region shape– e.g. lines, ellipses, ellipsoids

• You want a region that is compact and reasonably centered on a(0)

In MATLAB: y=prctile(x,[5 95])

from Numerical Recipes online

30

Validating Physical Interpretation of Model Parameters• Physical sensibleness

– Chemical rate constant cannot be negative– Poisson’s ratio cannot exceed 0.5– Can enforce lower and upper bounds on parameters, but

should examine closely if these end up “optimal”

• Independent measurements of key physical quantities– Comparison with published values or limiting behavior– Measure steady state modulus of viscoelastic material

• Experimentally alter specific parameters, collect data, and examine results of model fit– May involve building a physical model for testing

• Compare model fitting results using data from normal and abnormal populations– In asthma patients, airway resistance should be higher than normal

31

Problem Set

Harnett, Science, 2006

B lymphocytes in the immune response

www.EnCognitive.com

32

The movie is available athttp://www.youtube.com/watch?v=iDYL4x1Q6uU

Problem Set• Ordinary differential equation (ODE) model of BrdU

labeling to estimate proliferation and death rates of B cells

U – number of unlabeled B cellsL – number of BrdU labeled B cells

p – rate of proliferation (per hour)d – rate of death (per hour)s – rate of B cell inflow from source (cells/hr)

• Given experimental data on fraction of total B cells labeled with BrdU versus time, develop a model to fit the data, estimate values of p, s, and d, and evaluate the model performance

Steven Kleinstein and Uri Hershberg33

Resources• Numerical Recipes online

http://www.nrbook.com/a/bookfpdf.php

• MATLAB online helphttp://www.mathworks.com/access/helpdesk/help/techdoc/

• GraphPad PRISM online guide to curve fittinghttp://www.graphpad.com/manuals/prism4/regressionbook.pdf

• ReferencesAnderson SM, Khalil A, Uduman M, Hershberg U, Louzoun Y, Haberman AM,

Kleinstein SH, Shlomchik MJ. Taking advantage: high-affinity B cells in the germinal center have lower death rates, but similar rates of division, compared to low-affinity cells, J Immunol, 183:7314-7325, 2009.

Glantz SA. Primer of Biostatistics, 6th Ed., McGraw-Hill, 2005.

Lobemeier ML. Linearization plots: time for progress in regression, BioMedNet, issue 73, March 3, 2000.

Lutchen KL and Costa KD. Physiological interpretations based on lumped element models fit to respiratory impedance data: use of forward-inverse modeling, IEEE Trans Biomed Eng, 37:1076-1086, 1990.

34

www.sciencesignaling.org

Slides from a lecture in the course Systems Biology—Biomedical Modeling

Citation: K. D. Costa, S. H. Kleinstein, U. Hershberg, Biomedical model fitting and error analysis, Sci. Signal. 4, tr9 (2011).