Understanding Protein Dynamics withL1-Regularized Reversible Hidden Markov Models

Robert T. McGibbon RMCGIBBO@STANFORD.EDU

Department of Chemistry, Stanford University, Stanford CA 94305, USA

Bharath Ramsundar RBHARATH@STANFORD.EDU

Department of Computer Science, Stanford University, Stanford CA 94305, USA

Mohammad M. Sultan MSULTAN@STANFORD.EDU

Department of Chemistry, Stanford University, Stanford CA 94305, USA

Gert Kiss GKISS@STANFORD.EDU

Department of Chemistry, Stanford University, Stanford CA 94305, USA

Vijay S. Pande PANDE@STANFORD.EDU

Department of Chemistry, Stanford University, Stanford CA 94305, USA

Abstract

We present a machine learning framework formodeling protein dynamics. Our approach usesL1-regularized, reversible hidden Markov mod-els to understand large protein datasets generatedvia molecular dynamics simulations. Our modelis motivated by three design principles: (1) therequirement of massive scalability; (2) the needto adhere to relevant physical law; and (3) thenecessity of providing accessible interpretations,critical for both cellular biology and rational drugdesign. We present an EM algorithm for learningand introduce a model selection criteria based onthe physical notion of convergence in relaxationtimescales. We contrast our model with stan-dard methods in biophysics and demonstrate im-proved robustness. We implement our algorithmon GPUs and apply the method to two large pro-tein simulation datasets generated respectivelyon the NCSA Bluewaters supercomputer and theFolding@Home distributed computing network.Our analysis identifies the conformational dy-namics of the ubiquitin protein critical to cellularsignaling, and elucidates the stepwise activationmechanism of the c-Src kinase protein.

Proceedings of the 31 st International Conference on MachineLearning, Beijing, China, 2014. JMLR: W&CP volume 32. Copy-right 2014 by the author(s).

1. IntroductionProtein folding and conformational change are grand chal-lenge problems, relevant to a multitude of human diseases,including Alzheimer’s disease, Huntington’s disease andcancer. These problems entail the characterization of theprocess and pathways by which proteins fold to their ener-getically optimal configuration and the dynamics betweenmultiple long-lived, or “metastable,” configurations on thepotential energy surface. Proteins are biology’s molec-ular machines; a solution to the folding and conforma-tional change problem would deepen our understanding ofthe mechanism by which microscopic information in thegenome is manifested in the macroscopic phenotype of or-ganisms. Furthermore, an understanding of the structureand dynamics of proteins is increasingly important for therational design of targeted drugs (Wong & McCammon,2003).

Molecular dynamics (MD) simulations provide a computa-tional microscope by which protein dynamics can be stud-ied with atomic resolution (Dill et al., 1995). These simula-tions entail the forward integration of Newton’s equationsof motion on a classical potential energy surface. The po-tential energy functions in use, called forcefields, are semi-emprical approximations to the true quantum mechanicalBorn-Oppenheimer surface, designed to reproduce experi-mental observables (Beauchamp et al., 2012). For moder-ately sized proteins, this computation can involve the prop-agation of more than a million physical degrees of freedom.Furthermore, while folding events can take milliseconds

arX

iv:1

405.

1444

v1 [

q-bi

o.B

M]

6 M

ay 2

014

Understanding Protein Dynamics with L1-Regularized Reversible Hidden Markov Models

(10−3 s) or longer, the simulations must be integrated withfemtosecond (10−15 s) timesteps, requiring the collectionof datasets containing trillions of data points.

While the computational burden of performing MD sim-ulations has been a central challenge in the field, signif-icant progress has been achieved recently with the de-velopment of three independent technologies: ANTON, aspecial-purpose supercomputer using a custom ASIC to ac-celerate MD (Shaw, 2007); Folding@Home, a distributedcomputing network harnessing the desktop computers ofmore than 240,000 volunteers (Shirts & Pande, 2000); andGoogle Exacycle, an initiative utilizing the spare cycles onGoogle’s production infrastructure for science (Kohlhoffet al., 2014).

The analysis of these massive simulation datasets now rep-resents a major difficulty: how do we turn data into knowl-edge (Lane et al., 2013)? In contrast to some other machinelearning problems, the central goal here is not merely pre-diction. Instead, we view analysis – often in the form ofprobabilistic models generated from MD datasets – as atool for generating scientific insight about protein dynam-ics.

Useful probabilistic models must embody the appropriatephysics. The guiding physical paradigm by which chem-ical dynamics are understood is one of states and rates.States correspond to metastable regions in the configurationspace of the protein and can often be visualized as wellson the potential energy surface. Fluctuations within eachmetastable state are rapid; the dominant, long time-scaledynamics can be understood as a jump process movingwith various rates between the states. This paradigm moti-vates probabilistic models based on a discrete-state Markovchain. A priori, the location of the metastable states areunknown. As a result, each metastable state should corre-spond to a latent variable in the model. Hidden Markovmodels (HMMs) thus provide the natural framework.

Classical mechanics at thermal equilibrium satisfy a sym-metry with respect to time: a microscopic process and itstime-reversed version obey the same laws of motion. Thestochastic analogue of this property is reversibility (alsocalled detailed balance): the equilibrium flux between anytwo states X and Y is equal in both directions. Probabilis-tic models which fail to capture this essential property willassign positive probability to systems that violate the sec-ond law of thermodynamics (Prinz et al., 2011). Hence, weenforce detailed balance in our HMMs.

In addition to the constraints motivated by adherence tophysical laws, suitable probabilistic models should, inbroad strokes, incorporate knowledge from prior experi-mental and theoretical studies of proteins. Numerous stud-ies indicate that only a subset of the degrees of freedom

are essential for describing the protein’s dominant longtime-scale dynamics (see Cho et al. (2006) and referencestherein). Furthermore, substantial prior work indicates thatprotein folding occurs via a sequence of localized shifts(Maity et al., 2005). Together, these pieces of evidencemotivate the imposition of L1-fusion regularization (Tib-shirani et al., 2005). The L1 term penalizes deviationsamongst states along uninformative degrees of freedom,thereby suppressing their effect on the model. Furthermore,the pairwise structure of the fusion penalty minimizes thenumber of transitions which involve global changes: manypairs of states will only differ along a reduced subset of thedimensions.

The main results of this paper are the formulation of theL1-regularized reversible HMM and the introduction of asimple and scalable learning algorithm to fit the model. Wecontrast our approach against standard frameworks for theanalysis of MD data and demonstrate improved robustnessand physical interpretability.

This paper is organized as follows. Section 2 describesprior work. Section 3 introduces the model and associatedlearning algorithm. Section 4 applies the model to threesystems: a toy double well potential; ubiquitin, a humansignaling protein; and c-Src kinase, a critical regulatoryprotein involved in cancer genesis. Section 5 provides dis-cussion and indicates future directions.

2. Prior WorkEarlier studies have applied machine learning techniquesto investigate protein structure prediction – the problemof discovering a protein’s energetically optimal configura-tion – using CRFs, belief propagation, deep learning, andother general ML methods (Sontag et al., 2012; Di Lenaet al., 2012; Chu et al., 2006; Baldi & Pollastri, 2003). Butproteins are fundamentally dynamic systems, and none ofthese approaches offer insight into kinetics; rather, they areconcerned with extracting static information about proteinstructure.

The dominant computational tool for studying protein dy-namics is MD. Traditional analyses of MD datasets are pri-marily visual and non-quantitative. Standard approachesinclude watching movies of a protein’s structural dynamicsalong simulation trajectories, and inspecting the time evo-lution of a small number of pre-specified degrees of free-dom (Humphrey et al., 1996; Karplus & Kuriyan, 2005).While these methods have been successfully applied tosmaller proteins, they struggle to characterize the dynamicsof the large and complex biomolecules critical to biologi-cal function. Quantitative methods like PCA can elucidateimportant (high variance) degrees of freedom, but fail tocapture the rich temporal structure in MD datasets.

Understanding Protein Dynamics with L1-Regularized Reversible Hidden Markov Models

Markov state models (MSMs) are a simple class of proba-bilistic models, recently introduced to capture the temporaldynamics of the folding process. In an MSM, protein dy-namics are modeled by the evolution of a Markov chain ona discrete state space. The finite set of states is generatedby clustering the set of configurations in the MD trajec-tories (Beauchamp et al., 2011). MSMs can be viewed asfully observable HMMs. More recently, HMMs with multi-nomial emission distributions have been employed on thisdiscrete state space (Noe et al., 2013).

Although MSMs have had a number of notable successes(Voelz et al., 2012; Sadiq et al., 2012), they are brittle andcomplex. Traditional MSMs lack complete data likelihoodfunctions, and learning cannot be easily characterized bya single optimization problem. For these reasons, MSMlearning requires significant manual tuning. For example,because clustering is purely a preprocessing step, the like-lihood function contains no guidance on the choice of themetastable states. Moreover, the lack of uncertainty in theobservation model necessitates the introduction of a verylarge number of states, typically more than ten thousand, inorder to cover the protein’s phase space at sufficient reso-lution. This abundance of states is statistically inefficient,as millions of pairwise transition parameters must be esti-mated in typically-sized models, and renders interpretationof learned MSMs challenging.

3. Fusion L1-Regularized Reversible HMMWe introduce the L1-regularized reversible HMM withGaussian emissions, a generative probabilistic model overmultivariate discrete-time continuous-space time series. Asdiscussed in Section 1, we integrate necessary physicalconstraints on top of the core hidden Markov model (Ra-biner, 1989).

Let {Yt} be the observed time series in RD of length T(i.e., the input simulation data), and let {Xt} be the cor-responding latent time series in {1, . . . ,K}, where K isa hyperparameter indicating the number of hidden statesin the model. Each hidden variable xt corresponds to ametastable state of the physical system. The emission dis-tribution givenXt = k is a multivariate normal distributionparameterized by mean µk ∈ RD and diagonal covariancematrix Diag(σ2

k) ∈ RD·D (where σ2k ∈ RD is the vector of

diagonal covariance elements). We use the notation (µk)jto indicate the jth element of the vector µk.

Controlling the means {µk} is critical for achieving phys-ically interpretable models. As discussed in Section 1, wewish to minimize the differences between µk and µk′ tothe extent possible. Consequently, we place a fusion L1

penalty (Tibshirani et al., 2005) on our log likelihood func-

tion, which adds the following pairwise cost:

λ∑k,k′

∑j

τ(j)k,k′ |(µk)j − (µk′)j | .

Here, λ governs the overall strength of the penalty, whilethe adaptive fusion weights, {τ (j)k,k′}, control the contribu-tion from each pair of states (Guo et al., 2010). Duringlearning, the adaptive fusion weights are computed as

τ(j)k,k′ = |(µk)j − (µk′)j |−1,

where the {µk} are the learned metastable state means inthe absence of the penalty. The intuition motivating theadaptive strength of the penalty is that if degree of freedomj is informative for separating states k and k′, the corre-sponding fusion penalty should be applied lightly.

The reversible time evolution of the model is parameter-ized by an irreducible, aperiodic, row-normalized K byK stochastic matrix T, which satisfies detailed balance.Mathematically, the detailed balance constraint is

∀k, k′, πkTk,k′ = πk′Tk′,k,

where row vector π is the stationary distribution of T.The stationary distribution π also parameterizes the ini-tial distribution over the metastable states. By the Perron–Frobenius theorem, π is the dominant left eigenvector of Twith eigenvalue 1 and is not an independent parameter inthis model.

The initial distributions and evolution of {Xt, Yt} satisfythe following equations:

X0 ∼K∑k=1

πkδk,

Xt+1 ∼K∑k=1

TXt,k δk,

Yt ∼ N (µXt, σ2Xt

).

The complete data likelihood {xt, yt} is

L({xt}, {yt}|T, µ, σ) =

πx0

T−1∏t=1

Txt−1,xt

T−1∏t=0

N (yt;µxt, σ2xt

).

The hyperparameter ∆ controls the discretization intervalat which a protein’s coordinates are sampled to obtain {yt}.In the absence of downsampling by ∆, subsequent samplesyt, yt+1 would be highly correlated. On the other hand,subsequent samples from an HMM are conditionally inde-pendent given the hidden state. Choice of ∆ large enoughrecovers this conditional independence (vide infra).

Understanding Protein Dynamics with L1-Regularized Reversible Hidden Markov Models

3.1. Learning

The model is fit using expectation-maximization. The E-step is standard, while the M-step requires modification toenforce the detailed balance constraint on T and the adap-tive fusion penalty on the {µk}.

3.1.1. E-STEP

Inference is identical to that for the standard HMM, usingthe forward-backward algorithm (Rabiner, 1989) to com-pute the following quantities:

γi(t) = P(Xt = i|{yt}),ξij(t) = P(Xt = i,Xt+1 = j | {yt}).

3.1.2. M-STEP

Both the penalty on {µk} and the reversibility constraintaffect only the M-step. The M-step update to the means inthe t-th iteration of EM consists of maximizing the penal-ized log-likelihood function

µ(t+1)k = argmin

µk

N∑i

K∑k

γk(i)(xi − µk)2

2(σ2k)(t)

+ λ∑k,k′

∑j

τ(j)k,k′ |µk,j − µk′,j | .

The {µk} update is a quadratic program, which can besolved by a variety of methods. We compute {µk} by it-erated ridge regression. Following Guo et al. (2010) andFan & Li (2001), we use the local quadratic approximation∣∣∣ µ(t,s+1)

k,j − µ(t,s+1)k′,j

∣∣∣ ≈(µ(t,s+1)k,j − µ(t,s+1)

k′,j

)22∣∣∣µ(t,s)

k,j − µ(t,s)k′,j

∣∣∣ +1

2

∣∣∣µ(t,s)k,j − µ

(t,s)k′,j

∣∣∣ .where s is the iteration index for this procedure within thet-th M-step. This approximation is based on the identity

|x− y| = (x− y)2

2 |x− y|+

1

2|x− y|.

Under the approximation, we obtain a generalized ridge re-gression problem which can be solved in closed form dur-ing each iteration s. Note that this approximation is onlyvalid when |µ(t,s)

k,j −µ(t,s)k′,j | > 0. For numerical stability, we

threshold |µ(t,s)k,j − µ

(t,s)k′,j | to zero at values less than 10−10.

The variance update is standard:

σ2k =

∑t γk(t)(yt − µk)T (yt − µk)∑

t γk(t).

The transition matrix update is

T = arg maxT

∑ij

log(Tij)∑t

ξij(t).

Because the Gaussian emission distributions have infinitesupport, T is irreducible and aperiodic by construction.However, we must explicitly constrain T to satisfy detailedbalance.Lemma 1. T satisfies detailed balance if and only if Tij =

Wij∑k Wik

, where W = WT .

Proof. If T satisfies detailed balance, then let Wij =πiTij = πjTji = Wji. Then note

Wij∑kWik

=πiTij∑k πiTik

=Tij∑kTik

= Tij

To prove the converse, assume Tij =Wij∑k Wik

, with W =

WT . Let πi =∑kWik. Then πiTij = Wij = Wji =

πjTji.

Substituting the results of Lemma 1, we rewrite the transi-tion matrix update as

W = arg maxW

([∑ij

log(Wij)− log πi

]∑t

ξij(t)).

We compute the derivative of the inner term with respect tologWij and optimize with L-BFGS (Nocedal & Wright,2006).

3.2. Model Selection

There are two free model parameters: K and ∆. The num-ber of metastable states, K, is expected to be small – atmost a few dozen. To chooseK, we can use the AIC or BICselection criteria, or alternatively enumerate a few smallvalues.

The choice of ∆ is more difficult than the choice of K, aschanging the discretization interval alters the support of thelikelihood function. Recall that choosing ∆ too small re-sults in subsequent samples yt, yt+1 becoming highly cor-related, while the model satisfies the conditional indepen-dence assumption Yt |= Yt+1 |Xt. Moreover, small ∆ in-creases data-storage requirements, while ∆ too large willneedlessly discard data. Thus a balance between these twoconflicting directives is necessary.

We use the physical criterion of convergence in the relax-ation timescales to evaluate when ∆ is large enough. Thepropagation of the dynamics from an initial distributionover the hidden states, Xt, can be described by

P (yt+n |xt) =

K−1∑k=0

N (yt+n;µk, σ2k)(xTt Tn

)k.

Understanding Protein Dynamics with L1-Regularized Reversible Hidden Markov Models

Diagonalize T in terms of its left eigenvectors φi, righteigenvectors ψi, and eigenvalues λi (such a diagonalizationis always possible since T is a stochastic matrix).

P (yt+n |xt) =

K∑k=1

[N (yt+n;µk, σ

2k)

(K∑i=1

λni 〈xt, ψi〉φi

)k

]Since π is the stationary left eigenvector of T, and the re-maining eigenvalues lie in the interval −1 < λi < 1, thecollective dynamics can be interpreted as a sum of expo-nential relaxation processes.

P (yt+n |xt) =

K∑k=1

[fk(yt+n)

(π +

K∑i=2

e−n/τi 〈xt, ψi〉φi

)k

]In the equation, we define fk(y) = N (y;µk, σ

2k). Each

eigenvector of T (except the first) describes a dynamicalmode with characteristic relaxation timescale

τi = − 1

lnλi.

The longest timescales, τi, are of central interest froma molecular modeling perspective because they describedynamical modes visible in time-resolved protein experi-ments (Zhuang et al., 2011) and are robust against perturba-tions (Weber & Pande, 2012). We choose ∆ large enoughto converge the τi: for adequately large ∆, we expect τi(∆)to asymptotically converge to the true relaxation timescaleτ∗i . For simple systems, we may evaluate τ∗i explicitly,while for larger systems, we choose ∆ large enough so thatτi(∆) no longer changes with further increase in the dis-cretization interval.

3.3. Implementation

We implement learning for both multithreaded CPU andNVIDIA GPU platforms. In the CPU implementation,we parallelize across trajectories during the E-step usingOpenMP. The largest portion of the run time is spent inLOG-SUM-EXP operations, which we manually vectorizewith SSE2 intrinsics for SIMD architectures. Parallelismon the GPU is more fine grained. The E-step populates twoT ×K ×K arrays with forward and backwards sweeps re-spectively. To fully utilize the GPU’s massive parallelism,each trajectory has a team of threads which cooperate onupdating the K ×K matrix at each time step. SpecializedCUDA kernels were written for K = 4, 8, 16 and 32 alongwith a generic kernel for K > 32.

Even in log space, for long trajectories, the forward-backward algorithm can suffer from an accumulation offloating point errors which lead to catastrophic cancelationduring the computation of γi(t). This risk requires that theforward-backward matrices be accumulated in double pre-cision, whereas the rest of the calculation is safe in singleprecision.

Figure 1. Simulations of Brownian dynamics on a double well po-tential (A) illustrate the advantages of the HMM over the MSM.When the dynamics are discretized at a time interval of > 500steps, the 2-state HMM, unlike the 2-state MSM achieves a quan-titatively accurate prediction of the first relaxation timescale (B).The MSM (C) features hard cutoffs between the states wheres theHMM (D) each have infinite support.

The speedup using our GPU implementation is 15× com-pared to our optimized CPU implementation and 75× withrespect to a standard numpy implementation using K = 16states on a NVIDIA GTX TITAN GPU / Intel Core i7 4core Sandy Bridge CPU platform. Further scaling of theimplementation could be achieved by splitting the compu-tation over multiple GPUs with MPI.

4. Experiments4.1. Double Well Potential

We first consider a one-dimensional diffusion process ytgoverned by Brownian dynamics. The process is describedby the stochastic differential equation

dytdt

= −∇V (yt) +√

2DR(t)

where V is the reduced potential energy, D is the diffusionconstant, and R(t) is a zero-mean delta-correlated station-ary Gaussian process. For simplicity, we set D = 1 andconsider the double well potential

V (y) = 1 + cos(2y)

with reflecting boundary conditions at y = −π and y = π.Using the Euler-Maruyama method and a time step of∆t = 10−3, we produced ten simulation trajectories oflength 5 × 105 steps each. The histogrammed trajectoriesare shown in Fig. 1(A). The exact value of the first re-laxation timescale was computed by a finite element dis-cretization of the corresponding Fokker-Planck equation(Higham, 2001).

Understanding Protein Dynamics with L1-Regularized Reversible Hidden Markov Models

We applied both a two-state MSM and two-state HMM,with fusion L1 regularization parameter λ = 0, to the sim-ulation trajectories. The MSM states were fixed, with a di-viding surface at y = 0, as shown in Fig. 1(C). The HMMstates were learned, as shown in Fig. 1(D). Both the MSMand the HMM display some sensitivity with respect to thediscretization interval, with more accurate predictions ofthe relaxation timescale at longer lag times.

The two-state MSM is unable to accurately learn thelongest timescale, τ1, even with large lag times, while thetwo-state HMM succeeds in identifying τ1 with ∆ ≥ 500Fig. 1(B).

4.2. Ubiquitin

Figure 2. Dynamics of Ub. (A) The HMM identifies the twometastable states of Ub, varying primarily in the loop and helixregions (axes in yellow and red respectively). (B) The MSM failsto cleanly separate the two underlying physical states. Three post-processed macrostates from the MSM are shown (in blue, green,and yellow). (C) A structural rendering of the conformationalstates of the Ub system. S0, shown in grey, binds to the UCHfamily of proteins, and S1 (with characteristic structural differ-ences to S0 in red and yellow) binds to the USP family.

Ubiquitin (Ub) is a regulatory hub protein at the intersec-tion of many signaling pathways in the human body (Her-shko & Ciechanover, 1998). Among its many tasks are theregulation of inflammation, repair of DNA, and the break-down and recycling of waste proteins. Ubiquitin interactswith close to 5000 human signaling proteins. Understand-ing the link between structure and function in ubiquitinwould elucidate the underlying framework of the human

signaling network.

We obtained a dataset of MD simulations of human Ubconsisting of 3.5 million data points. The protein, shownin Fig. 2, is composed of 75 amino acids. The simulationswere performed on the NCSA Blue Waters supercomputer.The resulting structures were featurized by extracting thedistance from each amino acid’s central carbon atom to itsposition in the simulations’ starting configurations. HMMswere constructed with 2 to 6 states. We chose ∆ by moni-toring the convergence of the relaxation timescales as dis-cussed in Sec. 3.2, and set the L1 fusion penalty heuristi-cally to a default value of λ = 0.01. In agreement with ex-isting biophysical data (Zhang et al., 2012), the HMMs cor-rectly determined that Ub was best modeled with 2 states(Fig. 2A). For ease of representation, the learned HMM isshown projected onto two critical degrees of freedom (dis-cussed below).

For comparison, we generated MSM models with 500 mi-crostates (Fig. 2B) and projected upon the same criticaldegrees of freedom. We used a standard kinetic lump-ing post-processing step to identify 3 macrostates (shownin green, blue, and yellow respectively); the lumping al-gorithm collapsed when asked to identify 2 macrostates(Bowman, 2012). Contrast the simple, clean output of the2 state HMM in Fig. 2(A) with the standard MSM ofFig. 2(B). Note how significant post-processing and man-ual tuning would be required to piece together the true two-state structural dynamics of Ub from the MSM output.

We display a structural rendering of the Ub system in Fig.2(C). The imposed L1 penalty of the HMM suppressesdifferences among the uninformative degrees of freedomdepicted in grey. The remaining portions of the protein(shown in color) reveal the two critical axes of motion ofthe Ub system: the hinge dynamics of the loop region dis-played in yellow and a kink in the lower helix shown in red.We use these axes in the simplified representations shownin Figs. 2(A,B).

The states S0 and S1 identified by the HMM have direct bi-ological interpretations. Comparison to earlier experimen-tal work reveals that configuration S0 binds to the UCHfamily of proteins, while configuration S1 binds to the USPfamily instead (Komander et al., 2009). The families playdiffering roles in the vital task of regenerating active Ub forthe cell-signaling cycle.

Together, MD and the HMM analysis provide atomic in-sight into the effect of protein structure on ubiquitin’s rolein the signaling network. Our analysis approach may havesignificant value for protein biology and for the furtherstudy of cellular signaling networks. Although experimen-tal studies of protein signaling provide the gold standard forhard data, they struggle to provide structural explanations

Understanding Protein Dynamics with L1-Regularized Reversible Hidden Markov Models

— knowing why a certain protein is more suited for cer-tain signaling functions is challenging at best. In contrast,the MD/HMM approach can provide a direct link betweenstructure and function and give a causal basis for observedprotein activity.

4.3. c-Src Tyrosine Kinase

Protein kinases are a family of enzymes that are criticalfor regulating cellular growth whose aberrant activation canlead to uncontrolled cellular proliferation. Because of theircentral role in cell proliferation, kinases are a critical targetfor anti-cancer therapeutics. The c-Src tyrosine kinase is aprominent member of this family that has been implicatedin numerous human malignancies (Aleshin & Finn, 2010).

Due to the protein’s size and complexity, performing MDsimulations of the c-Src kinase is a formidable task. Theprotein, shown in Fig. 3A, consists of 262 amino acids;when surrounding water molecules – necessary for accu-rate simulation – are taken into account, the system hasover 40,000 atoms. Furthermore, transition between theactive and inactive states takes hundred of microseconds.Adequate sampling of these processes therefore requireshundreds of billions of MD integrator steps. Simulationsof the c-Src kinase were performed on the Folding@Homedistributed computing network, collecting a dataset of 4.7million configurations from 550 µs of sampling, for a totalof 108 GB of data (Shukla et al., 2014).

In order to understand the molecular activation mechanismof the c-Src kinase, we analyzed this dataset using the L1

regularized reversible HMM. We featurized the configu-rations by extracting the distance from each amino acid’scentral carbon atom to its position in an experimentally de-termined inactive configuration. We built HMMs with 2 to6 states, and singled out the 3 state model for achieving abalance of complexity and interpretability. As with Ub, wechose ∆ by monitoring the convergence of the relaxationtimescales, same default L1 fusion penalty of λ = 0.01.

The L1-regularized reversible HMM elucidates the c-Srckinase activation pathway, revealing a stepwise mechanismof the dynamics. A projection of the learned HMM statesonto two key degrees of freedom is shown in Fig. 3B.Fig. 3C shows a structural representation of the means ofthe three states, highlighting a sequential activation mech-anism. The transformation from the inactive to the inter-mediate state occurs first by the unfolding of the A-loop(the subsection of the protein highlighted in red). Acti-vation is completed by the inward rotation of the C-helix(highlighted in orange) and rupture of a critical side chaininteraction between two amino acids on the C-helix and theA-loop respectively.

Although the protein structure is complex, the activation

Inac%ve( Intermediate( Ac%ve(

A( B(

C(

Figure 3. Activation of the c-Src Kinase. (A) Structure of the pro-tein system. (B) The 3 state HMM, projected onto two degrees offreedom representing the positions of the A-loop (shown in red)and C-helix (shown in orange) respectively. (C) Structural render-ings of the means of the hidden states showing atomistic detailsof the activation pathway.

process takes place only in a small portion of the overallprotein; the random fluctuations of the remaining degreesof freedom are largely uncoupled from the activation pro-cess. As with Ub, theL1 penalty suppresses the signal fromunimportant degrees of freedom shown in grey. In contrastto the simplicity of HMM approach, a recent MSM analysisof this dataset found similar results, but required 2,000 mi-crostastates and significant post-processing of the modelsto generate physical insight into the activation mechanism(Shukla et al., 2014).

The identification of the intermediate state along the activa-tion pathway has substantial implications in the field of ra-tional drug design. Chemotherapy drugs often have harm-ful side effects because they target portions of proteins thatare common across entire families, interfering with boththe uncontrolled behavior of tumor proteins as well as thecritical cellular function of healthy proteins. Intermediatestates, such as the one identified by the HMM, are morelikely to be unique to each kinase protein; future therapeu-tics that target these intermediate states could have signifi-cantly fewer deleterious side effects (Fang et al., 2013).

5. Discussion and ConclusionCurrently, MSMs are a dominant framework for analyz-ing protein dynamics datasets. We propose replacing this

Understanding Protein Dynamics with L1-Regularized Reversible Hidden Markov Models

methodology with L1-regularized reversible HMMs. Weshow that HMMs have significant advantages over MSMs:whereas the MSM state decomposition is a prepreprocess-ing procedure without guidance from a complete-data like-lihood function, the HMM couples the identification ofmetastable states with the estimation of transition probabil-ities. As such, accurate models require fewer states, aidinginterpretability from a physical perspective.

The switch is not without tradeoffs. MSMs are backed bya significant body of theoretical work: the MSM is a di-rect discretization of an integral operator which formallycontrols the long timescale dynamics known as the trans-fer operator. This connection enables the quantification ofapproximation error in the MSM framework (Prinz et al.,2011). No such theoretical guarantees yet exist for the L1-regularized reversible HMM because the evolution of Yt isno longer unconditionally Markovian. However, becausethe HMM can be viewed as a generalized hidden MSM,there is reason to believe that analogues of MSM theoreti-cal guarantees extend to the HMM framework.

While the L1-regularized reversible hidden Markov modelrepresents an improvement over previous methods for an-alyzing MD datasets, future work will likely confront anumber of remaining challenges. For example, the cur-rent model does not learn the featurization and treats∆ as a hyperparameter. Bringing these two aspects ofthe model into the optimization framework would reducethe required amount of manual tuning. Adapting tech-niques from Bayesian nonparametrics, unsupervised fea-ture learning and linear dynamical systems may facilitatethe achievement of these goals.

Our results show that structured statistical analysis of mas-sive protein datasets is now possible. We reduce com-plex dynamical systems with thousands of physical degreesof freedom to simple statistical models characterized by asmall number of metastable states and transition rates. TheHMM framework is a tool for turning raw molecular dy-namics data into scientific knowledge about protein struc-ture, dynamics and function. Our experiments on the ubiq-uitin and c-Src kinase proteins extract insight that may fur-ther the state of the art in cellular biology and rational drugdesign.

AcknowledgmentsWe thank Diwakar Shukla for kindly providing the c-Srckinase MD trajectories. B.R. was supported by the Fan-nie and John Hertz Foundation. G.K and V.S.P acknowl-edge support from the Simbios NIH Center for BiomedicalComputation (NIH U54 Roadmap GM072970). V.S.P. ac-knowledges NIH R01-GM62868 and NSF MCB-0954714.

ReferencesAleshin, A. and Finn, R. S. SRC: a century of science

brought to the clinic. Neoplasia, 12(8):599–607, 2010.

Baldi, Pierre and Pollastri, Gianluca. The principled designof large-scale recursive neural network architectures–DAG-RNNs and the protein structure prediction prob-lem. J. Mach, Learn. Res., 4:575–602, 2003.

Beauchamp, Kyle A., Bowman, Gregory R., Lane,Thomas J., Maibaum, Lutz, Haque, Imran S., and Pande,Vijay S. MSMBuilder2: Modeling conformational dy-namics on the picosecond to millisecond scale. J. Chem.Theory Comput., 7(10):3412–3419, 2011.

Beauchamp, Kyle A., Lin, Yu-Shan, Das, Rhiju, and Pande,Vijay S. Are protein force fields getting better? A sys-tematic benchmark on 524 diverse NMR measurements.J. Chem. Theory Comput., 8(4):1409–1414, 2012.

Bowman, Gregory R. Improved coarse-graining of markovstate models via explicit consideration of statistical un-certainty. J. Chem. Phys., 137(13):134111, 2012.

Cho, Samuel S., Levy, Yaakov, and Wolynes, Peter G. Pversus Q: Structural reaction coordinates capture proteinfolding on smooth landscapes. Proc. Natl. Acad. Sci.U.S.A., 103(3):586–591, 2006.

Chu, Wei, Ghahramani, Zoubin, Podtelezhnikov, Alexei,and Wild, David L. Bayesian segmental models withmultiple sequence alignment profiles for protein sec-ondary structure and contact map prediction. IEEE/ACMTrans. Comput. Biol. Bioinf., 3(2):98–113, 2006.

Di Lena, Pietro, Baldi, Pierre, and Nagata, Ken. Deepspatio-temporal architectures and learning for proteinstructure prediction. In Adv. Neural Inf. Process. Syst.25, NIPS ’12, pp. 521–529, 2012.

Dill, Ken A, Bromberg, Sarina, Yue, Kaizhi, Chan,Hue Sun, Ftebig, Klaus M, Yee, David P, and Thomas,Paul D. Principles of protein folding – a perspective fromsimple exact models. Protein Sci., 4(4):561–602, 1995.

Fan, Jianqing and Li, Runze. Variable selection via non-concave penalized likelihood and its oracle properties. J.Am. Stat. Assoc., 96(456):1348–1360, 2001.

Fang, Zhizhou, Grtter, Christian, and Rauh, Daniel. Strate-gies for the selective regulation of kinases with al-losteric modulators: Exploiting exclusive structural fea-tures. ACS Chem. Biol., 8(1):58–70, 2013.

Guo, Jian, Levina, Elizaveta, Michailidis, George, andZhu, Ji. Pairwise variable selection for high-dimensionalmodel-based clustering. Biometrics, 66(3):793–804,2010.

Understanding Protein Dynamics with L1-Regularized Reversible Hidden Markov Models

Hershko, Avram and Ciechanover, Aaron. The ubiquitinsystem. Annu. Rev. Biochem., 67(1):425–479, 1998.

Higham, Desmond J. An algorithmic introduction to nu-merical simulation of stochastic differential equations.SIAM Rev., 43(3):525–546, 2001.

Humphrey, William, Dalke, Andrew, and Schulten, Klaus.VMD: Visual molecular dynamics. J. Mol. Graphics, 14(1):33 – 38, 1996.

Karplus, M. and Kuriyan, J. Molecular dynamics and pro-tein function. Proc. Natl. Acad. Sci. U.S.A., 102(19):6679–6685, 2005.

Kohlhoff, Kai J, Shukla, Diwakar, Lawrenz, Morgan, Bow-man, Gregory R, Konerding, David E, Belov, Dan, Alt-man, Russ B, and Pande, Vijay S. Cloud-based simu-lations on Google Exacycle reveal ligand modulation ofgpcr activation pathways. Nat. Chem., 6(1):15–21, 2014.

Komander, David, Clague, Michael J, and Urbe, Sylvie.Breaking the chains: structure and function of the deu-biquitinases. Nat. Rev. Mol. Cell Biol., 10(8):550–563,2009.

Lane, Thomas J, Shukla, Diwakar, Beauchamp, Kyle A,and Pande, Vijay S. To milliseconds and beyond: chal-lenges in the simulation of protein folding. Curr. Opin.Struct. Biol., 23(1):58 – 65, 2013.

Maity, Haripada, Maity, Mita, Krishna, Mallela M. G.,Mayne, Leland, and Englander, S. Walter. Protein fold-ing: The stepwise assembly of foldon units. Proc. Natl.Acad. Sci. U.S.A., 102(13):4741–4746, 2005.

Nocedal, J. and Wright, S. Numerical Optimization.Springer Series in Operations Research and FinancialEngineering. Springer, New York, 2nd edition, 2006.

Noe, Frank, Wu, Hao, Prinz, Jan-Hendrik, and Plattner,Nuria. Projected and hidden markov models for calculat-ing kinetics and metastable states of complex molecules.J. Chem. Phys., 139(18):184114, 2013.

Prinz, Jan-Hendrik, Wu, Hao, Sarich, Marco, Keller, Bet-tina, Senne, Martin, Held, Martin, Chodera, John D.,Schutte, Christof, and Noe, Frank. Markov models ofmolecular kinetics: Generation and validation. J. Chem.Phys., 134(17):174105, 2011.

Rabiner, Lawrence R. A tutorial on hidden markov modelsand selected applications in speech recognition. Proc.IEEE, 77(2):257–286, 1989.

Sadiq, S. Kashif, Noe, Frank, and De Fabritiis, Gianni. Ki-netic characterization of the critical step in HIV-1 pro-tease maturation. Proc. Natl. Acad. Sci. U.S.A., 109(50):20449–20454, 2012.

Shaw, David E. et al. Anton, a special-purpose machine formolecular dynamics simulation. In ACM Comp. Ar. 34,ISCA ’07, pp. 1–12, 2007.

Shirts, Michael and Pande, Vijay S. Screen savers of theworld unite! Science, 290(5498):1903–1904, 2000.

Shukla, Diwakar, Meng, Yilin, Roux, Benoıt, and Pande,Vijay S. Activation pathway of Src kinase reveals in-termediate states as novel targets for drug design. Nat.Commun., 5, 2014.

Sontag, David, Meltzer, Talya, Globerson, Amir, Jaakkola,Tommi S, and Weiss, Yair. Tightening LP relax-ations for MAP using message passing. arXiv preprintarXiv:1206.3288, 2012.

Tibshirani, Robert, Saunders, Michael, Rosset, Saharon,Zhu, Ji, and Knight, Keith. Sparsity and smoothness viathe fused lasso. J. R. Statistic. Soc. B, 67(1):91–108,2005.

Voelz, Vincent A., Jger, Marcus, Yao, Shuhuai, Chen, Yu-jie, Zhu, Li, Waldauer, Steven A., Bowman, Gregory R.,Friedrichs, Mark, Bakajin, Olgica, Lapidus, Lisa J.,Weiss, Shimon, and Pande, Vijay S. Slow unfolded-state structuring in Acyl-CoA binding protein folding re-vealed by simulation and experiment. J. Am. Chem. Soc.,134(30):12565–12577, 2012.

Weber, Jeffrey K and Pande, Vijay S. Protein folding ismechanistically robust. Biophys. J., 102(4):859–867,2012.

Wong, Chung F. and McCammon, J. Andrew. Protein flexi-bility and computer-aided drug design. Annu. Rev. Phar-macol. Toxicol., 43(1):31–45, 2003.

Zhang, Yingnan, Zhou, Lijuan, Rouge, Lionel, Phillips,Aaron H, Lam, Cynthia, Liu, Peter, Sandoval, Wendy,Helgason, Elizabeth, Murray, Jeremy M, Wertz, In-grid E, et al. Conformational stabilization of ubiqui-tin yields potent and selective inhibitors of USP7. Nat.Chem. Biol., 9(1):51–58, 2012.

Zhuang, Wei, Cui, Raymond Z., Silva, Daniel-Adriano,and Huang, Xuhui. Simulating the T-jump-triggeredunfolding dynamics of trpzip2 peptide and its time-resolved IR and two-dimensional IR signals using themarkov state model approach. J. Phys. Chem. B, 115(18):5415–5424, 2011.