Novel Tools and Methods Computational Phenotyping in Psychiatry: A Worked Example Philipp Schwartenbeck, 1,2,3,4 and Karl Friston 1 DOI:http://dx.doi.org/10.1523/ENEURO.0049-16.2016 1 The Wellcome Trust Centre for Neuroimaging, UCL, London WC1N 3BG, UK, 2 Centre for Cognitive Neuroscience, University of Salzburg, 5020 Salzburg, Austria, 3 Neuroscience Institute, Christian-Doppler-Klinik, Paracelsus Medical University Salzburg, A-5020 Salzburg, Austria, and 4 Max Planck UCL Centre for Computational Psychiatry and Ageing Research, London WC1B 5EH, UK Abstract Computational psychiatry is a rapidly emerging field that uses model-based quantities to infer the behavioral and neuronal abnormalities that underlie psychopathology. If successful, this approach promises key insights into (path- ological) brain function as well as a more mechanistic and quantitative approach to psychiatric nosology—structuring therapeutic interventions and predicting response and relapse. The basic procedure in computational psychiatry is to build a computational model that formalizes a behavioral or neuronal process. Measured behavioral (or neuronal) responses are then used to infer the model parameters of a single subject or a group of subjects. Here, we provide an illustrative overview over this process, starting from the modeling of choice behavior in a specific task, simulating data, and then inverting that model to estimate group effects. Finally, we illustrate cross-validation to assess whether between-subject variables (e.g., diagnosis) can be recovered successfully. Our worked example uses a simple two-step maze task and a model of choice behavior based on (active) inference and Markov decision processes. The procedural steps and routines we illustrate are not restricted to a specific field of research or particular computational model but can, in principle, be applied in many domains of computational psychiatry. Key words: active inference; computational psychiatry; generative model; Markov decision process; model inversion Introduction Recent advances in computational neuroscience—and the lack of a mechanistic classification system in mental disorders (Stephan et al., 2015)— have motivated the ap- plication of computational models in clinical research. This has led to the emergence of a field of research called computational psychiatry, which has attracted much re- cent interest (Friston et al., 2014b; Stephan and Mathys, 2014; Wang and Krystal, 2014; Huys et al., 2015). Its aim is to use computational models of behavior or neuronal Received March 8, 2016; accepted June 9, 2016; First published July 18, 2016. The authors declare no competing financial interests. Author contributions: P.S. and K.F. performed the simulations and wrote the paper. Significance Statement We provide an overview over the process of using formal models to understand psychiatric conditions, which is central in the emerging research field of “computational psychiatry.” This approach promises key insights into both healthy and pathological brain function as well as a more mechanistic understanding of psychiatric nosology, which may have important consequences for therapeutic interventions or predicting response and relapse. In a worked example, we discuss the generic aspects of using a computational model to formalize a task, simulating data and estimating parameters, as well as inferring group effects between patients and healthy control subjects. We also provide routines that can be used for these steps and are freely available in the academic software SPM. Methods/New Tools July/August 2016, 3(4) e0049-16.2016 1–18
Transcript
Novel Tools and Methods
Computational Phenotyping in Psychiatry: AWorked ExamplePhilipp Schwartenbeck,1,2,3,4 and Karl Friston1
DOI:http://dx.doi.org/10.1523/ENEURO.0049-16.2016
1The Wellcome Trust Centre for Neuroimaging, UCL, London WC1N 3BG, UK, 2Centre for Cognitive Neuroscience,University of Salzburg, 5020 Salzburg, Austria, 3Neuroscience Institute, Christian-Doppler-Klinik, Paracelsus MedicalUniversity Salzburg, A-5020 Salzburg, Austria, and 4Max Planck UCL Centre for Computational Psychiatry and AgeingResearch, London WC1B 5EH, UK
AbstractComputational psychiatry is a rapidly emerging field that uses model-based quantities to infer the behavioral andneuronal abnormalities that underlie psychopathology. If successful, this approach promises key insights into (path-ological) brain function as well as a more mechanistic and quantitative approach to psychiatric nosology—structuringtherapeutic interventions and predicting response and relapse. The basic procedure in computational psychiatry is tobuild a computational model that formalizes a behavioral or neuronal process. Measured behavioral (or neuronal)responses are then used to infer the model parameters of a single subject or a group of subjects. Here, we provide anillustrative overview over this process, starting from the modeling of choice behavior in a specific task, simulating data,and then inverting that model to estimate group effects. Finally, we illustrate cross-validation to assess whetherbetween-subject variables (e.g., diagnosis) can be recovered successfully. Our worked example uses a simpletwo-step maze task and a model of choice behavior based on (active) inference and Markov decision processes. Theprocedural steps and routines we illustrate are not restricted to a specific field of research or particular computationalmodel but can, in principle, be applied in many domains of computational psychiatry.
IntroductionRecent advances in computational neuroscience—and
the lack of a mechanistic classification system in mentaldisorders (Stephan et al., 2015)—have motivated the ap-plication of computational models in clinical research.
This has led to the emergence of a field of research calledcomputational psychiatry, which has attracted much re-cent interest (Friston et al., 2014b; Stephan and Mathys,2014; Wang and Krystal, 2014; Huys et al., 2015). Its aimis to use computational models of behavior or neuronal
Received March 8, 2016; accepted June 9, 2016; First published July 18, 2016.The authors declare no competing financial interests.
Author contributions: P.S. and K.F. performed the simulations and wrote thepaper.
Significance Statement
We provide an overview over the process of using formal models to understand psychiatric conditions,which is central in the emerging research field of “computational psychiatry.” This approach promises keyinsights into both healthy and pathological brain function as well as a more mechanistic understanding ofpsychiatric nosology, which may have important consequences for therapeutic interventions or predictingresponse and relapse. In a worked example, we discuss the generic aspects of using a computationalmodel to formalize a task, simulating data and estimating parameters, as well as inferring group effectsbetween patients and healthy control subjects. We also provide routines that can be used for these stepsand are freely available in the academic software SPM.
function to infer the hidden causes of measurable quantities,such as symptoms, signs, and neuroimaging or psycho-physical responses. In consequence, this approach prom-ises new insights into the computational mechanisms ofcertain pathologies, which would otherwise be hidden, whenassessing the observations alone.
This tutorial addresses a particular but important aspect ofcomputational psychiatry; namely, how to characterize indi-viduals in terms of their computational phenotypes. In otherwords, it describes how to quantify the beliefs and prefer-ences of an individual—within a formal framework—by fit-ting their choice behavior to a computational model. Ourfocus is on a particular sequence of analyses that is sup-ported by routines in the SPM software. These routines havebeen written in a way that they should be applicable to anychoice or decision tasks that can be modeled in terms of(partially observable) Markov decision processes (Toussaintet al., 2006; Alagoz et al., 2010; Rao, 2010; FitzGerald et al.,2015; see below). The purpose of this note is to describe theoverall structure of the analyses and the functionality of afew key (Matlab) routines that can be used to analyze thebehavior of subjects in a relatively straightforward and effi-cient fashion. An overview of these routines is provided inFigure 1 and the software notes below.
P.S. is a recipient of DOC Fellowship 501100001822 from the AustrianAcademy of Sciences atthe Centre for Cognitive Neuroscience, University ofSalzburg. The Wellcome Trust Centre for Neuroimaging is supported by corefunding from Wellcome Trust Grant 091593/Z/10/Z.
Correspondence should be addressed to Philipp Schwartenbeck, Centre forCognitive Neuroscience, University of Salzburg, Hellbrunnerstrasse 34, 5020Salzburg, Austria. E-mail: [email protected].
Figure 1. A schematic overview of the analysis stream underlying the treatment of computational psychiatry in this article. The basicprocedure involves specifying a model of behavior cast in terms of a Markov decision process (MDP). Under the assumption thatchoices are made in an approximately Bayes optimal fashion using active (Bayesian) inference, this model is sufficient to predictbehavior. If we supplement the model specification (MDP) with empirical choice behavior (Data), we can estimate the prior beliefsresponsible for those choices. If this is repeated for a series of subjects, the ensuing priors can then be analyzed in a random-effects(PEB) model to make inferences about group effects or to perform cross-validation. Furthermore, physiological and behavioralpredictions can be used as expansion variables for fMRI or other neuroimaging time series (bottom left). The routines in the boxes referto MATLAB routines that are available in the academic software SPM. These routines are sufficient to both simulate behavioralresponses and analyze empirical or observed choice behaviour, at both the within-subject and between-subject levels. The finalroutine also enables cross-validation and predictions about a new subject’s prior beliefs using a leave-one-out scheme that may beuseful for establishing the predictive validity of any models that are considered.
Characterizing choice behavior in terms of a formalmodel represents a subtle challenge, especially underactive inference models of behavior (Friston et al., 2011,2015a). This is because it is generally assumed that sub-jects make decisions based on a generative model of thetask at hand and behave in an approximately Bayesianway by making bounded rational choices (Berkes et al.,2011). Generative models provide a probabilistic mappingfrom hidden states or parameters to observations. Inother words, a generative model specifies how conse-quences (i.e., observed outcomes) are generated fromtheir causes (i.e., unobserved or hidden states and pa-rameters). When modeling behavior under a generativemodel, the (objective) model includes the (subjective)model we assume is used by each subject (we provide aworked example of this in the following sections). Thismeans that fitting choice behavior becomes a meta-Bayesian problem, in which we are trying to infer thebeliefs adopted by a subject based on our beliefs aboutthis (active) inference process (Daunizeau et al., 2010).Therefore, under the perspective of (active) Bayesian in-ference, the only things one can infer about a subject aretheir prior beliefs (Houlsby et al., 2013). In other schemes,for example normative economic models or reinforcementlearning, prior expectations would correspond to keymodel parameters, such as temporal discounting or thesensitivity to rewards. In turn, this means that the differ-ence between one subject and another has to be cast interms of their generative models, which can always beformulated as prior beliefs or, when their priors pertain tothe parameters of probability distributions, hyperpriors.Crucially, understanding individual behavior in terms ofindividual (subjective) generative models of a task adds anadditional hypothesis space for investigating individualand group differences, and speaks to the idea of under-standing pathological behavior in terms of pathologicalmodels of (i.e., abnormal prior beliefs about) the world(Beck et al., 2012; Dayan, 2014; Schwartenbeck et al.,2015c). In what follows, we will illustrate this generalprinciple using a particular example (and simulated data).The end point of this analysis will be a characterization ofsingle subjects (and group differences) in terms of (hyper)priors encoding uncertainty or confidence about choicebehavior. However, the same procedure can be applied toany prior belief that shapes an individual’s response totheir changing world.
The formal approachFormally, this approach rests on (both subjective and objec-tive) generative models. For the subjective generative model,these observations are experimental outcomes or cues ob-served by a subject, while for the objective model, theoutcomes would be the subject’s responses or choices. Instatistical terms, generative models provide a likelihoodfunction, P�y��, m�, of data y given a set of parameters �and the model structure m, as well as a prior over parame-ters, P���m�. Crucially, one can invert this model usingBayes rule to infer the most likely parameter values (hiddenstates) causing observed data, as follows:
P(y��, m), P(��m)
P(y�m)� P(��y, m).
Here, P�y�m� refers to the evidence or marginal likeli-hood of the model, which can be obtained by integratingor marginalizing out � in the numerator, as follows:
P(y�m) � � P(y��, m)· P(��m)d�.
Usually, this integral cannot be solved analytically (ex-actly) but has to be approximated, for example throughheuristics like the Akaike/Bayesian information criterion ormore refined but computationally more expensive solu-tions, such as sampling or variational methods (Attias,2000; Beal, 2003; Bishop, 2006).
In the following, we will illustrate the use of generativemodels in computational phenotyping and focus on thecrucial steps in (1) specifying the model, (2) using themodel to simulate or generate data, (3) model inversion orfitting to estimate subject-specific parameters, and (4)subsequent inference about between-subject or groupeffects using hierarchical or empirical Bayes: for example,comparing a group of healthy control subjects to a patientgroup. We will use a simple decision-making task toillustrate these processes and use a recently proposedcomputational model of behavior, which casts decision-making as a Markov decision process based on (active)Bayesian inference (Friston et al., 2013; Friston et al.,2015). The details of the task and computational modelare not of central importance and are discussed else-where (Friston et al., 2015a). Here, the model serves toillustrate the typical procedures in computational ap-proaches to psychiatry. When appropriate, we will referexplicitly to Matlab routines that implement each step.These routines use (variational) procedures for Bayesianmodel inversion and comparison. They have been devel-oped over decades as part of the SPM software, and havebeen applied extensively in the modeling of neuroimagingand other data. The key routines called on in this articleare described in the software notes below.
Model specificationFigure 1 provides an overview of the procedures we willbe illustrating. Usually, one starts by developing and op-timizing the task paradigm. Clearly, to describe a sub-ject’s response formally, it is necessary to specify asubjective generative model that can predict a subject’sresponses. For experimental paradigms, it is often con-venient to use discrete state–space models (i.e., partiallyobservable Markov decision processes), in which variouscues and choices can be labeled as discrete outcomes.We will illustrate this sort of model using a relativelysimple two-step maze task.
An active inference model of epistemic foragingIn the following, we introduce the ingredients for casting adecision-making or planning task as an active Bayesianinference. Note that this treatment serves as an illustrationfor the general steps in computational phenotyping, which
do not depend on the use of a particular computational ornormative model. In practice, one often wants to comparedifferent variants of computational models of choice be-havior, such as (Bayesian) inference and (reinforcement)learning models (Sutton and Barto, 1998), which we dis-cuss below. In fact, the active inference routine we pres-ent here allows one to perform these formal comparisons,which are the subject of current research (Friston et al.,2009; Mathys et al., 2011; Schwartenbeck et al., 2015b).
For our worked example, we will use a task that requiresboth exploratory (epistemic) and exploitative behavior(Friston et al., 2015a). In brief, in this task a subject has tochoose whether to sample the left or right arm of a
T-shaped maze to obtain a reward or a cue. The rewardsare in the upper arms, while the cue is in the lower arm.This cue indicates the location of the reward with highvalidity (Fig. 2A). Crucially, the left and the right arm of themaze are absorbing states, which means that the agenthas to stick with its choice. Therefore, in the absence ofany prior knowledge, the optimal policy (i.e., a sequenceof actions) involves first sampling the cue and then se-lecting the reward location indicated by the cue. While thisseems like a very simple task, it captures interestingaspects of behavior such as planning and a trade-offbetween exploration (i.e., sampling the cue) and exploita-tion (i.e., moving to the arm that is likely to contain a
Figure 2. A, Task: we used a simple two-step maze task for our simulations, where a subject starts in the middle of a T-shaped mazeand has to decide whether to sample the left or right arm, knowing that one of the two arms will contain a reward but it can sampleonly one of them (i.e., the arms are absorbing states). Alternatively, the subject could sample a cue at the bottom of the maze, whichwill tell her which arm to sample. B, State space: Here, the subject has four different control states or actions available: she can moveto the middle location, the left or the right arm or the cue location. Based on these control states, we can specify the hidden states,which are all possible states that a subject can visit in a task and often are only partially observable. In this task, the hidden statecomprises the location � the context (reward left or right), resulting in 4 � 2 � 8 different hidden states. Finally, we have to specifythe possible outcomes or observations that an agent can make. Here, the subject can find itself in the middle location, in the left orright arm with or without obtaining a reward or at the cue location.
reward). As such, it can be easily extended to model morecomplex types of choice problems.
The first step is to specify the (subjective) generativemodel of this task. These sorts of problems can be mod-eled efficiently as partially observable Markov decisionprocesses, where the transition probabilities are deter-mined by the current action and state, but not by thehistory of previous states (see below). In the context ofactive inference, the specification of the generative modelfor a Markov decision process is based on three matrices,called A, B, and C. These describe the mapping fromhidden states to outcomes, the transition probabilitiesand the preferences (expectations) over outcomes, re-spectively. The specification of these matrices dependson the state space of the task (Fig. 2B). In our example,the agent can be in one of eight possible hidden statesthat are determined by the agent’s location (middle, bot-tom, left arm, or right arm) and context (cue indicates theleft or right arm).
As the name implies, hidden states are usually not fully(but only partially) observable and have to be inferredbased on observations. This motivates the A-matrix,which maps from hidden states to observations (outcomestates; i.e., states that the subject can observe). Here, wecan differentiate seven different observable states (middleposition, left arm rewarded or unrewarded, right arm re-warded or unrewarded, cue location indicating to go leftor right; Fig. 2B); thus, the A-matrix is a 7 � 8 matrix (Fig.3A). The A-matrix accounts for any partially observableaspect of a Markov decision process, where this mappingbecomes an identity matrix if all states are fully observable(i.e., if there is no uncertainty about which hidden statecaused an observation).
Second, the B-matrix encodes the transition probabili-ties in a decision process (i.e. the probability of the nexthidden state contingent on the current hidden state andthe action taken by the agent; Fig. 3B, illustration of thetransition probabilities in our task). These transition prob-abilities are a particular feature of Markov decision pro-cesses, because they depend only on the current state(and action), not on previous states. This is called theMarkov or “memory-less” property. Finally, one has tospecify the agent’s preferences over outcomes states(observations), which are encoded in the C-vector. Pref-erences over outcomes are (prior) expectations, whichcan be based on task instructions or, in the context ofeconomic decision-making or reinforcement learning, util-ity or reward (or any combination of these).
From the perspective of active inference, the preferred(desired) states are the states that the subject expects tofind itself in. Note that casting an agent’s preferences interms of prior expectations does not imply that the con-cept of reward is meaningless; rather, the notion of areward is absorbed into an agent’s expectations, whichguide its inferences about policies. Therefore, a Bayesoptimal subject will infer or select policies that bring aboutexpected or preferred states. This inference is modeledby Bayesian updates that accumulate evidence to opti-mize posterior beliefs about hidden states of the worldand the current policy being enacted. Mathematically, this
can be expressed as a minimization of variational freeenergy (as an upper bound on surprise; see below). Be-cause variational free energy is an approximation tonegative Bayesian model evidence, belief updating tominimize free energy for surprise is exactly the same asmaximizing model evidence. This Bayes optimal inferenceensures that subjects obtain the outcomes they expect(i.e., desire). Equivalently, they will avoid unexpected (i.e.,undesired) outcomes because they are surprising. Thekey quantities that endow behavior with a purposeful,goal-directed aspect are the prior preferences that nu-ance the selection of policies. These preferences aregenerally treated as free parameters that can be esti-mated through model inversion, as we will see below.
For a full discussion of the task described above andthe specifics of casting choice behavior as an activeinferential Markov decision process, please see (Fristonet al., 2013; Friston et al., 2015a). Having said this, theparticular details of this paradigm are not terribly impor-tant. The important thing to note is that, in principle, nearlyevery experimental paradigm can be specified with twosets of matrices (A and B), while every subject can becharacterized in terms of their preferences (C). Practicallyspeaking, the key challenge is not to specify these matri-ces; but, the greatest challenge is to understand anddefine the hidden state space implicit in the paradigm (inother words, the states in which these matrices operate).
Simulating dataHaving specified the subjective generative model for thistask, we can now use the model to simulate or generatechoice behavior. While eventually one will use real data,simulating data is useful to assess whether the generativemodel of a task produces sensible behavior. Once anycounterintuitive behavior has been resolved, one can thenuse simulations to optimize the design parameters (orstate space) that will be used empirically as the subjectivemodel.
To simulate choices, we need to specify specific valuesfor the priors and hyperpriors of the generative model,which we want to recover when working with real data(see Model inversion, below). In our case, we need tospecify the following two parameters: the preferences(expectations) over outcomes and a hyperprior on theconfidence or precision of beliefs about policies. Thepreferences over outcomes simply determine the desir-ability of each observable (outcome) state. For our simu-lations, we have assigned a high value to outcomes inwhich the agent obtains a reward (�4), a low value foroutcome states in which the agent does not obtain areward and is trapped in an absorbing state (�4), and amedium value for the remaining outcomes, after which it isstill possible to obtain a reward later (0; Fig. 3C). Becausethese expectations are defined in log-space, this can beunderstood as the subject’s (prior) belief that obtaining areward is exp�4� � 55 times more likely than ending up ina “neutral” state �exp�0� � 1�. In addition, we can specifya hyperprior on precision. Precision ���
reflects an agent’s stochasticity or goal directedness inchoice behavior but, crucially, itself has a Bayes-optimal
Figure 3. Generative model. A, The A-matrix maps from hidden states to observable outcome states (resulting in a 7 � 8 matrix).There is a deterministic mapping when the subject is either in the middle position (simply observing that she is in the middle) or atthe cue location (simply observing that she is at the cue location where the cue indicates either left or right). However, when thesubject is in the left or right arm, there is a probabilistic mapping to a rewarded and an unrewarded outcome. For example, if thesubject is in the left arm and the cue indicated is in the left arm (third column), there is a high probability, p, of a reward, whereas thereis a low probability q � 1 � p of no reward. B, The B-matrix encodes the transition probabilities (i.e. the mapping from the currenthidden state to the next hidden state contingent on the action taken by the agent). Thus, we need as many B-matrices as there areactions available (four in this example). Illustrated here is the B-matrix for a move to the left arm. We see that the action never changesthe context, but (deterministically) does change the location, by always bringing it to the left arm, except when starting from anabsorbing state (right arm). C, Finally, we have to specify the preferences over outcome states in a C-vector. Here, the subject stronglyprefers ending up in a reward state and strongly dislikes ending up in a left or right arm with no reward, whereas it is somewhatindifferent about the “intermediate” states. Note that these preferences are (prior) beliefs or expectations; for example, the agentbeliefs that a rewarding state is exp �4� � 55 times more likely than an “intermediate” state [exp(0) � 1].
solution that can be inferred on a trial-by-trial basis. Theimportance of precision in decision processes and itsputative neuronal implementation are discussed in detailelsewhere (Friston et al., 2014a; Schwartenbeck et al.,2015a). In brief, the values of policies are scored asvariational free energies, which follow a Gibbs distribu-tion. Precision is the inverse temperature of this distribu-tion and itself is parameterized by a � distribution with ascale ��� and rate ��� hyperparameter. Thus, precisionplays the same role as an inverse temperature in classicsoftmax choice rules, with the difference that it is contin-uously optimized. For our simulations, we have set � and� to a value of 2, resulting in an expected value for(inverse) precision of � � 1. Hyperpriors on precision areof great importance when recovering the parametersbased on observed behavior and may play a central role inpsychiatry, as discussed below.
Finally, we need to specify initial states for each trial inthe experiment (i.e., the state from which the subjectstarts navigating through the maze). In our simulations,the initial (hidden) state of every trial is set to the middlelocation and (randomly) to one of the two contexts (rewardon the left or right), where the entire experiment com-prises 128 trials. Details of the model specification andsimulation (and the steps below) can be found in the DEMtoolbox of the academic software SPM12 (WellcomeTrust Centre for Neuroimaging, London, UK, http://www-.fil.ion.ucl.ac.uk/spm) under the option “behavioral mod-eling.”
Having specified the generative model, we can now usethe function spm_MDP_VB to simulate behavior. Thisroutine provides solutions of behavior based on activeinference, such that agents believe they will minimizeexpected free energy. Expected free energy can be de-composed into the Kullback–Leibler (KL) divergencebetween predicted and preferred outcomes plus the ex-pected surprise (uncertainty) about future outcomes.Therefore, minimizing expected free energy implicitlymaximizes expected utility or preferred outcomes in arisk-sensitive fashion (compare with KL control), whileresolving ambiguity. This active inference scheme isbased on the following three variational update equations:agents are assumed to perform within-trial inference oncurrent states, actions, and (expected) precision. In addi-tion to these updates, the parameters of the generativemodel are updated between trials (i.e., learned). Details ofthe model and variational Bayesian updates can be foundin the study by Friston et al. (2015a), and the output ofthese simulations can be found in Figure 4A for a singletrial and in Figure 4B for the entire experiment.
These simulated responses (and other electrophysio-logical responses not considered in this article) can nowbe used to verify the efficiency of the paradigm and itsbasic behavior. One can also assess the sensitivity ofsimulated behavior to variations in preferences and priorprecision. The sensitivity determines the efficiency withwhich a subject’s preferences and hyperpriors can beestimated. We now turn to this estimation in terms ofmodel inversion.
Model inversionWe have described the first two steps of computationalmodeling; namely, translating a particular paradigm into agenerative model and simulating data by exploiting the abil-ity of generative models to simulate or generate data. Inthese simulations, we used a specific model that castsdecision-making as a Markov decision process based onactive inference. While recent work has highlighted the roleof deficient decision processes in psychiatry (Montagueet al., 2012; Hauser et al., 2014; Huys et al., 2015), thecentral prerequisite of defining a generative model for a taskgeneralizes to all applications in computational psychiatry.
We can now turn to the inversion of the generative modelto recover its parameters, based on observed (or in our casesimulated) behavior. This is an important step in empiricalresearch, because the aim of computational psychiatry is tocharacterize psychopathology in a quantitative and compu-tationally meaningful fashion. In other words, we want toexplain people’s behavior in terms of a specific parameter-ization of a (subjective) generative model.
A common approach is to compute maximum a poste-riori (MAP) estimates of parameters obtained by invertingan objective generative model. As described above, agenerative model is a mapping from hidden parameters toobserved data. Thus, by inverting the model one can mapfrom observations to hidden parameters, resulting in aposterior distribution over the most likely (MAP) parame-ters contingent on the model and observed data. To doso, one has to decide on how to approximate the modelevidence or marginal likelihood. Here, we will use (nega-tive) variational free energy as a proxy for the log-evidence of a model. The log-model evidence can beexpressed as follows:
In P(y | m) � DKL[q(�) | | ]P(��y, m)] � F(q(�), y) ,
where the first term is the KL divergence between the trueposterior and an approximate posterior, which has a lowerbound of zero, and thus makes the (negative) variational freeenergy in the second term a lower bound of the (negative)log-model evidence (which is at most zero). It is thus suffi-cient to minimize free energy to maximize the log-evidence ofthe model itself. When fitting subject-specific choices, thedata at the observed choices and the active inferencescheme cited above provide a likelihood model. The likeli-hood is the probability of obtaining a particular sequence ofchoices, given a subject’s preferences and hyperpriors.Model inversion corresponds to estimating the preferencesand hyperpriors, given an observed sequence of choices.
To do this model inversion, one can use the routinespm_dcm_mdp, which inverts an objective generativemodel given the subjective model, observed states, andresponses of a subject. This is made particularly easybecause the subjective model provides the probability ofvarious choices or actions from which the subject selectsbehavior. We can now simply integrate or solve the activeinference scheme using the actual outcomes observedempirically and evaluate the probability of the ensuingchoices. To complete the objective generative model, weonly need to specify priors over the unknown model pa-
rameters. In what follows, we use fairly uninformativeshrinkage priors. Shrinkage generally refers to any regu-larization method in statistics that prevents overfitting (butsee Bishop, 2006, page 10). In particular, shrinkage isinherent in Bayesian inference due to the use of priors,which “shrink” the parameter estimates toward the priormean, and thus preclude overfitting. In our case, we usedpriors with a mean of 0 and a variance of 1/16, thus inducingshrinkage toward 0. These priors can be changed in thespm_dcm_mdp.m script, to specify any prior constraints on,or knowledge about, the parameters that are estimated(e.g., time constants that fall in natural ranges). We willsee later that priors can themselves be optimized, usingBayesian model comparison (BMC). This follows because
any model is defined in terms of its (shrinkage) priors.Technically, model inversion uses a standard (Newtonmethod) gradient ascent on variational free energy (inwhich the curvature or Hessian is estimated numerically).Practically, this involves specifying the (MDP) model usedto explain the subject’s behavior, the observed outcomes,and their associated choices or actions (and objectivepriors on the unknown subjective model parameters).These quantities are specified as fields in a Matlab struc-ture usually called DCM (for dynamic causal model).
Figure 5 shows the output of this routine when appliedto our simulated data. This provides estimates of thepreferences over outcomes and the hyperprior � on pre-cision. Here, the estimation converges at the 13th itera-
Figure 4. Data simulation using the routine spm_MDP_VB. A, A simulated example trial, where the left top panel shows the hiddenstates, the right top panel shows the inferred actions, the middle panels show the inference on policies (i.e., the possible sequencesof actions), the bottom left panel shows the preferences over outcome states (c-vector), and the bottom right panel shows theexpected precision, which could be encoded by dopamine (Friston et al., n.d.). In this trial, the subject starts in the middle positionwhere the reward is (most likely) on the right arm. She then makes a selection to sample the cue and, finally, moves to the right arm,as indicated by the cue (darker colors reflect higher posterior probabilities). B, Overview of a simulated experiment comprising 128trials. The first panel shows the inferred policies (black regions) and initial states (shown as colored circles: red circles, reward islocated at the right arm; blue circles, reward is located at the left arm) at every given trial. The second panel shows estimated reactiontimes (cyan dots), outcome states (colored circles), and the value of those outcomes (black bars). Note that the value of outcomesis expressed in terms of an agent’s (expected) utility, which is defined as the logarithm of an agent’s prior expectations. Thus, theutility of an outcome is at most 0 [� log(1)]. Reaction times reflect the choice conflict at any given trial and are simulated by using thetime it takes Matlab to simulate inference and subsequent choice in any given trial (using the tic-toc function in Matlab). The third andfourth panels show simulated event-related potentials for hidden state estimation and expected precision, respectively. The specificsof these simulations are discussed in detail elsewhere (Friston et al., 2016). Finally, panels five and six illustrate learning and habitformation. Our simulations did not include any learning or habitual responses.
tion, and provides the trajectory of the two parameters aswell as their conditional expectations and posterior devi-ations. The inversion of simulated data can also be helpfulto ensure that the subjective model can be inverted priorto testing subjects or patients. For example, one cansimulate how many trials are necessary to recover param-eters with sufficient confidence. An example of this isshown in Figure 6A for the hyperprior on precision.
Importantly, there is usually more than one free param-eter in a subject’s generative model, and these parame-ters could have similar effects on behavior. One can useintuitions about the effects of two parameters on re-sponse variables and use simulations to test for anysimilarity and any ensuing conditional dependencies. Anefficient parameterization (or experimental design) would
usually suppress conditional dependencies and thereforemake the parameter estimation more efficient. In our exam-ple, we treated the hyperprior on precision and the prefer-ences over outcomes as free parameters, where the formeraccounts for an agent’s stochasticity in behavior and thelatter controls the agent’s preferences for different states.Thus, these two parameters control distinct aspects of ob-servable behavior and can be estimated relatively efficiently.Any conditional dependencies among parameters can beassessed (post hoc) in terms of their shared variance; forexample, by assessing their posterior covariance.
Inferring group effectsFinally, having described the specification of the genera-tive model and model inversion, we can now turn to
Figure 5. Model inversion, as implemented by the routine spm_dcm_mdp, is based on simulated behavior. In this routine, (negative)variational free energy as a lower bound of log-model evidence is maximized and converges after the 13th iteration (top right). Thetrajectory of two estimated parameters in parameter space is provided (top left) as well as their final conditional estimates (bottom left)and their posterior deviation from the prior value (bottom right). The black bars on the bottom right show the true values, while thegray bars show the conditional estimates, illustrating a characteristic shrinkage toward the prior mean.
inference about between-subject or group effects. As out-lined above, when casting decision-making as a Markovdecision process based on active inference, a key role isplayed by precision, which determines the confidencethat subjects place in their beliefs about choices. Further-more, it has been previously suggested that precisionmight be encoded by neuromodulators, in particular do-pamine (Friston et al., 2012; Friston et al., 2014a; FitzGer-ald et al., 2015; Schwartenbeck et al., 2015a), but alsoacetylcholine (Moran et al., 2013). Therefore, precisionmight be central for understanding certain pathologies,such as psychosis, obsessive–compulsive disorder, oraddiction (Adams et al., 2013; Schwartenbeck et al.,2015c; Hauser et al., 2016).
We simulated a group difference in precision by repeatingthe model specification, data simulation, and model inver-sion steps described above for two groups of eight subjectseach; where, crucially, we introduced a difference in thehyperprior on precision (of one-quarter) between the twogroups and an intersubject variability with a log precision offour (SD, 0.135). The result of the model inversion for eachsubject is illustrated in Figure 6B in terms of real and esti-mated subject-specific hyperpriors. These results immedi-ately indicate of a group effect in this parameter that isfurther evidenced by a significant difference between theestimates in the two groups �t � 6.627, p 0.001�. Noticethat inferences about group effects call on a model ofbetween-subject differences. In Bayesian terms, these mod-els are called hierarchical models or empirical Bayesianmodels and are the Bayesian equivalent of (random-effects)ANOVAs, with between-subject and within-subject effects.A full random-effects analysis can be implemented usingparametric empirical Bayes (PEB) implemented for nonlinear
models of the sort we are dealing with here (for details, seeEfron and Morris, 1973; Friston et al., 2015b; Friston et al.,2015c).
The Matlab routine to directly assess the group effect isspm_dcm_peb. In brief, this routine uses hierarchical em-pirical Bayes for second-level group inversion based on adesign matrix modeling group effects. In this case, thebetween-subject model (or design matrix) contained twoexplanatory variables modeling a group mean and a groupdifference, respectively. This is the between-subject modelX in Figure 1 (for details, see Friston et al., 2015b,c; for areproducibility study using this approach, see Litvak et al.,2015).
Figure 7A shows the output of Bayesian model com-parison, which can be understood as the posterior evi-dence that there is a group difference in the full model(i.e., a group mean and difference) or in a reduced model(i.e. only one or no effects). In Figure 7A (top right), we findthat the models with a group mean and difference andmodels with just a group difference have the highestposterior evidence (with a posterior probability that isslightly � 0.5 and �0.5, respectively). Figure 7B showsthe corresponding parameter estimates (the group meanresults are shown on the left, and the group differencesare shown on the right). Here, Figure 7B (top right) showsthat the group difference of one-quarter is recoveredaccurately. Note that while these results correspond tothe result obtained by a simple t test, using this routine forBayesian model comparison offers more flexibility andprovides more information than standard parametric ap-proaches. One obvious difference, in relation to classicinference, is the possibility of assessing the posteriorevidence for the null hypothesis (i.e., evidence that there
Figure 6. A, Conditional estimate and confidence interval for the hyperprior on precision (�) as a function of the number of trials ina simulated experiment. B, True and estimated subject-specific parameters, following model inversion for 16 subjects with a groupeffect in the hyperprior (�). The two groups can be seen as two clusters along the diagonal.
is no difference between groups) and the posterior esti-mate of the effect size for the group difference. For ex-ample, if we repeat the above simulation with a groupdifference of zero, we obtain a high posterior probabilityfor the model with no mean and group differences � �0.7� and the models that assume a group difference scor-ing a posterior probability of �0.1, whereas a nonsignifi-cant t test would only provide inconclusive results.
While this empirical Bayesian procedure is very usefulfor inferring group differences within a model, in practice,researchers are also interested in comparing differentmodels of the same (choice response) data. For example,although we have illustrated computational phenotypingbased on active inference, other models that are used incomputational neuroscience (and psychiatry) are basedon biophysical neural network models or reinforcementlearning (for recent review, see Huys et al., 2016). Com-paring different models of a task, such as inference and
(reinforcement) learning models, is necessary to identifywhich computational framework best explains observedbehavior. This usually rests on some form of Bayesianmodel comparison (Stephan et al., 2009). In the presentsetting, competing models are compared in terms of theirmodel evidence, as scored by variational free energycomputed during model inversion (at the within-subjectlevel or at the between-subject level using spm_dcm_peb). This has been shown to outperform alternative ap-proximations to model evidence, such as the Akaike andBayesian information criteria (Penny, 2012). It is possibleto perform such model comparisons within the activeinference toolbox; for example, by comparing active in-ference (surprise minimization) to classic expected utilitytheory (Schwartenbeck et al., 2015a,b).
Furthermore, one might also be interested in formalmodel comparisons that entertain different hidden statespaces underlying the (subjective) generative models
Figure 7. Hierarchical empirical Bayesian inference on group effects using the function spm_dcm_peb. A, Results of Bayesian modelcomparison (reduction) to infer whether the full model (with both group mean and group differences) or a reduced (nested) model(bottom left) provides a better explanation for the data. These results indicate high posterior evidence for a model with a groupdifference, with slightly less evidence for the full model, which also includes a group mean effect (i.e., a deviation from the group priormean; top panels). Middle panels show the maximum a posteriori estimates of the mean and group effects for the full and reducedmodels. B, Estimated (gray bars) group mean (left) and difference (right) in �. These estimates are about one-quarter (top right), whichcorresponds to the group effect that was introduced in the simulations (black bars). The small bars correspond to 90% Bayesianconfidence intervals. A reduced parameter estimate corresponds to the Bayesian model average over all possible models (full andreduced) following Bayesian model reduction.
used by subjects. Crucially, patient groups and healthycontrol subjects might differ in both the hyperparametersof their subjective models (e.g., the hyperprior on preci-sion) as well as the state space underlying these models.For example, we have illustrated a subjective model inwhich precision is a hidden state that has to be inferred.However, it is possible that some subjects or patientgroups do not perform inference on precision, such thattheir best model would be one in which precision is notinferred. Likewise, patient groups and healthy control subjectsmight differ in how they represent prior expectations; for exam-ple, by encoding losses and wins separately. This can betested by using formal model comparisons as describedabove, where models that include precision as a hiddenstate are compared to models where no inference on pre-cision is made, or to models that have a different parame-terization for losses and wins. This has been the focus ofprevious work (Schwartenbeck et al., 2015b) and, impor-tantly, also speaks to the issue of Bayesian model compar-ison and averaging (FitzGerald et al., 2014), as well asstructure learning (Tervo et al., 2016), when performing atask. More generally, any assumption that is implicit in theform of hyperparameterization of a model can be testedempirically through Bayesian model comparison. In thissense, the particular model used to illustrate the approach inthis article should not be taken too seriously; every aspectcan be optimized in relation to empirical choice behavioruntil an optimal description of the paradigm is identified.
Finally, we can use a cross-validation scheme imple-mented by the routine spm_dcm_loo to test whethergroup membership can be accurately recovered, basedon a leave-one-out scheme. This scheme uses the pre-dictive posterior density of a predictive variable (Fristonet al., 2015c), which is � in our example. In brief, thiscross-validation or predictive scheme switches the rolesof the between-subject explanatory variables (e.g., diag-nosis) and the subject-specific parameter estimates theyare trying to explain. This allows one to classify a partic-ular subject based on that subject’s parameter estimates,and to classify group effects based on independent data.Figure 8 shows the results of this cross-validation scheme,which speaks to a high accuracy in recovering the groupmembership, as indicated by a high correlation betweenrecovered and actual group membership (Fig. 8, top right)and a conclusive posterior probability for group member-ship for each subject (Fig. 8, bottom). This facility may beuseful for classifying new subjects based on their com-putational phenotyping. Note that the inferred differencein the latent variable precision (Fig. 7) and the inferredgroup membership based on this latent variable (Fig. 8)corresponds closely with the actual (simulated) groupdifference in an observed variable; namely, the averagereward received by each group in this task (Fig. 9). Im-portantly, this shows that the inference scheme discussedabove can infer the latent variable underlying the ob-served variables. This is important because the implicitinverse problem is usually ill posed and can have manysolutions.
A common approach to evaluate the goodness of dif-ferent models relies on cross-validation. In other words,
model parameters are optimized using a (training) subsetof the data and tested on another (test) subset. Modelperformance can then be assessed purely in terms ofaccuracy, without having to worry about complexity. Thisis because the use of independent training and test dataprecludes overfitting. Therefore, cross-validation accu-racy can be regarded as a proxy for model evidence andused, in a straightforward way, to compare different mod-els. The approaches described in this article eschew cross-validation, because the model evidence is assessed directlythrough its variational free-energy approximation. There arepros and cons of a variational assessment of model evi-dence, in relation to cross-validation accuracy. In brief,the variational approach is universally more efficient (byNeyman–Pearson Lemma) than cross-validation. This canbe seen heuristically by noting that the model inversionand parameter estimation in cross-validation uses incom-plete (training) data (e.g., a leave one scheme). On theother hand, cross-validation accuracy is robust to modelassumptions, and does not rely on variational approxima-tions to model evidence. This means that it can be usefulin assessing the robustness of variational schemes of thesort described in this article.
ConclusionWe have tried to provide an overview of the key stepsentailed by phenotyping in computational psychiatry,using a worked example based on a (Markov) decisionprocess. The first, and probably most important, step isthe specification of the (subjective) generative model for atask or paradigm. This model encodes a mapping fromhidden states or parameters to outcomes and can beused to simulate (generate) data. More importantly, itforms the basis of an objective generative model forempirical choice behavior; enabling one to map fromchoices to (subject-specific) model (hyper) parameters.This allows one to estimate the prior preferences andhyperpriors used by the subject to select their behavior.
Our simulations were based on a particular computationalapproach called active inference, which casts (choice) be-havior and planning as pure Bayesian inference with respectto an agent’s prior expectations. This approach can beparticularly useful when we want to cast a decision processas inference [i.e., assuming a stable (subjective) generativemodel that is used to infer hidden states or policies]. Fur-thermore, this allows one to compare a (Bayesian) inferencemodel to a (Bayesian or reinforcement) learning model, inwhich the parameterization of the (subjective) generativemodel is continuously updated (FitzGerald et al., 2014). Fu-ture work will implement the aspect of learning within theactive inference scheme, such that the parameterization ofthe generative model can be updated and simultaneouslyused to infer hidden states. A limitation of this computationaltoolbox is that it provides solutions only for discrete state-space problems, which significantly simplifies a given deci-sion or planning problem at the expense of biological realismabout the inferred neuronal mechanisms underlying the de-cision process.
An important aim of computational psychiatry is tocharacterize the generative processes that underlie path-
ological choice behavior. A first step to achieve this is toperform single-subject model inversion to estimate theirprior beliefs and then compare these estimates at thebetween-subject level. Hierarchical or empirical Bayes
and Bayesian cross-validation can then be used to testhypotheses about group differences such as diagnosis or re-sponse to treatment. While we have used a specific example ofMarkov decision processes based on active Bayesian infer-
Figure 8. Cross-validation based on a leave-one-out scheme. Using the function spm_dcm_loo, we find that group membership isaccurately recovered based on the parameter estimate of the hyperprior on each subject. This is evidenced by a high correlationbetween inferred and true group membership in the top right panel. These reflect out-of-sample estimates of effect sizes, which werelarge (by design) in this example. The top right panel provides the estimate of the group indicator variable (which is �1 for the firstgroup and �1 for the second group). The bottom panel provides the posterior probability that each subject belongs to the first group.
ence, the procedures we have described are generic, and mayprovide an exciting, straightforward, and principled approachto personalized treatment in psychiatry.
Software notesHere we describe the key routines called on in the text. Theseroutines are called in a demo script that can be edited andexecuted to change various parameters. The demo script isdescribed first, followed by the key routines it calls.
DEM_demo_MDP_fitThis routine uses a Markov decision process formulationof active inference (with variational Bayes) to model for-aging for information in a three-arm maze. This demoillustrates the inversion of single-subject and group datato make inferences about subject-specific parameters,such as their prior beliefs about precision and utility. Wefirst generate some synthetic data for a single subject andillustrate the recovery of key parameters using variationalLaplace. We then consider the inversion of multiple trialsfrom a group of subjects to illustrate the use of empiricalBayes in making inferences at the between-subject level.Finally, we demonstrate the use of Bayesian cross-validation to retrieve out-of-sample estimates (and theclassification of new subjects).
In this example, the agent starts at the center of athree-way maze that is baited with a reward in one of thetwo upper arms. However, the rewarded arm changes
from trial to trial. Crucially, the agent can identify wherethe reward (US) is located by accessing a cue (CS) in thelower arm. This tells the agent whether the reward is onthe left or the right upper arm. This means the optimalpolicy would first involve maximizing information gain orepistemic value by moving to the lower arm and thenclaiming the reward thus signified. Here, there are eighthidden states (four locations times right or left reward),four control states (that take the agent to the four loca-tions), and seven outcomes (three locations times twocues plus the center). The central location has an ambig-uous or uninformative outcome, and the upper arms arerewarded probabilistically.
spm_MDP_VB% active inference and learning using variational Bayes
% FORMAT [MDP] � spm_MDP_VB(MDP,OPTIONS)%% MDP.S(N,1)- true initial state% MDP.V(T - 1,P)- P allowable policies (control se-
quences)%% MDP.A(O,N)- likelihood of O outcomes given N hid-
den states% MDP.B{M}(N,N)- transition probabilities among hid-
den states (priors)% MDP.C(N,1)- prior preferences (prior over future out-
comes)
Figure 9. Simulated group difference between control subjects and patients (with a group difference in precision of one-quarter) inthe average reward received. Note that this difference in an observable variable was successfully traced back to a difference in thehyperprior on precision (a latent variable) by our inference scheme, which is important because such inverse problems are usually illposed and hard to solve.
’Expected Utility’};% OPTIONS.habit- switch to suppress habit learning:
(default: [1])%%% produces:%% MDP.P(M,T)- probability of emitting action 1,. . .,M at
time 1,. . .,T% MDP.Q(N,T)- an array of conditional (posterior) ex-
pectations over%N hidden states and time 1,. . .,T% MDP.X- and Bayesian model averages over policies% MDP.R- conditional expectations over policies%% MDP.un- simulated neuronal encoding of hidden
states% MDP.xn- simulated neuronal encoding of policies% MDP.wn- simulated neuronal encoding of precision
(tonic)% MDP.dn- simulated dopamine responses (phasic)% MDP.rt- simulated reaction timesThis routine provides solutions of an active inference
scheme (minimization of variational free energy) using agenerative model based on a Markov decision process. Thismodel and inference scheme is formulated in discrete spaceand time. This means that the generative model and processare finite-state machines or hidden Markov models, whosedynamics are given by transition probabilities among states,and the likelihood corresponds to the probability of an out-come given hidden states. For simplicity, this routine as-sumes that action (the world) and hidden control states (inthe model) are isomorphic.
This implementation equips agents with the prior beliefsthat they will maximize expected free energy: expectedfree energy is the free energy of future outcomes underthe posterior predictive distribution. This can be inter-preted in several ways)—most intuitively as minimizing theKL divergence between predicted and preferred out-
comes (specified as prior beliefs)—while simultaneouslyminimizing the (predicted) entropy of outcomes condi-tioned on hidden states. Expected free energy thereforecombines KL optimality based on preferences or utilityfunctions with epistemic value or information gain.
This particular scheme is designed for any allowable pol-icies or control sequences specified in MDP.V. Constraintson allowable policies can limit the numerics or combinato-rics considerably. For example, situations in which one ac-tion can be selected at one time can be reduced to T polices,with one (shift) control being emitted at all possible timepoints. This specification of polices simplifies the generativemodel, allowing a fairly exhaustive model of potential out-comes, eschewing a mean field approximation over succes-sive control states. In brief, the agent encodes beliefs abouthidden states in the past and in the future conditioned oneach policy (and a nonsequential state–state policy called ahabit). These conditional expectations are used to evaluatethe (path integral) of free energy that then determines theprior over policies. This prior is used to create a predictivedistribution over outcomes, which specifies the next action.
In addition to state estimation and policy selection, thescheme also updates model parameters, including the statetransition matrices, mapping to outcomes, and the initialstate. This is useful for learning the context. In addition, byobserving its own behavior, the agent will automatically learnhabits. Finally, by observing policies chosen over trials, theagent develops prior expectations or beliefs about what itwill do. If these priors (over policies, which include the habit)render some policies unlikely (using an Ockham’s window),they will not be evaluated.
spm_dcm_mdp% MDP inversion using Variational Bayes
% FORMAT [DCM] � spm_dcm_mdp(DCM)%% Expects:%———————————————————————% DCM.MDP% MDP structure specifying a generative
model% DCM.field% parameter (field) names to optimize% DCM.U% cell array of outcomes (stimuli)% DCM.Y% cell array of responses (action)%% Returns:%———————————————————————% DCM.M% generative model (DCM)% DCM.Ep% Conditional means (structure)% DCM.Cp% Conditional covariances% DCM.F% (negative) Free-energy bound on log evi-
denceThis routine inverts (cell arrays of) trials specified in
terms of the stimuli or outcomes and subsequent choicesor responses. It first computes the prior expectations (andcovariances) of the free parameters specified by DCM-.field. These parameters are log-scaling parameters thatare applied to the fields of DCM.MDP.
If there is no learning implicit in multitrial games, onlyunique trials (as specified by the stimuli) are used togenerate (subjective) posteriors over choice or action.
Otherwise, all trials are used in the order specified. Theensuing posterior probabilities over choices are used withthe specified choices or actions to evaluate their logprobability. This is used to optimize the MDP (hyper)parameters in DCM.field using variational Laplace (withnumerical evaluation of the curvature).
spm_dcm_peb% Hierarchical (PEB) inversion of DCMs using BMR andVL
% FORMAT [PEB,DCM] � spm_dcm_peb(DCM,M,field)% FORMAT [PEB,DCM] � spm_dcm_peb(DCM,X,field)%% DCM - {N [x M]} structure array of DCMs from N
subjects%
————————————————————————-% DCM{i}.M.pE- prior expectation of parameters% DCM{i}.M.pC- prior covariances of parameters% DCM{i}.Ep- posterior expectations% DCM{i}.Cp- posterior covariance% DCM{i}.F- free energy%% M.X- second-level design matrix, where X(:,1) �
ones(N,1) [default]% M.pC- second-level prior covariances of parameters% M.hE- second-level prior expectation of log preci-
sions% M.hC- second-level prior covariances of log preci-
sions% M.bE- third-level prior expectation of parameters% M.bC- third-level prior covariances of parameters%% M.Q- covariance components:
{’single’,’fields’,’all’,’none’}% M.beta- within:between precision ratio: [default � 16]%% field- parameter fields in DCM{i}.Ep to optimize [de-
fault: {’A’,’B’}]%’All’ will invoke all fields. This argument effectively
allows%one to specify the parameters that constitute random
effects.%% PEB- hierarchical dynamic model%
————————————————————————-% PEB.Snames- string array of first-level model names% PEB.Pnames- string array of parameters of interest% PEB.Pind- indices of parameters in spm_vec
trix% PEB.M.Q- precision [components] of second-level
random effects% PEB.M.pE- prior expectation of second-level param-
eters
% PEB.M.pC- prior covariance of second-level param-eters
% PEB.M.hE- prior expectation of second-level log-precisions
% PEB.M.hC- prior covariance of second-level log-precisions
% PEB.Ep- posterior expectation of second-level pa-rameters
% PEB.Eh- posterior expectation of second-level log-precisions
% PEB.Cp- posterior covariance of second-level pa-rameters
% PEB.Ch- posterior covariance of second-level log-precisions
% PEB.Ce- expected covariance of second-level ran-dom effects
% PEB.F- free energy of second-level model%% DCM- first-level (reduced) DCM structures with em-
pirical priors%% If DCM is an (N x M} array, hierarchical inversion will be% applied to each model (i.e., each row) - and PEB will be a% {1 x M} cell array.This routine inverts a hierarchical DCM using variational
Laplace and Bayesian model reduction. In essence, it opti-mizes the empirical priors over the parameters of a set offirst-level DCMs, using second-level or between-subject con-straints specified in the design matrix X. This scheme is efficientin the sense that it does not require inversion of the first-levelDCMs—it just requires the prior and posterior densities fromeach first-level DCM to compute empirical priors under theimplicit hierarchical model. The output of this scheme (PEB)can be re-entered recursively to invert deep hierarchical mod-els. Furthermore, BMC can be specified in terms of the empir-ical priors to perform BMC at the group level. Alternatively,subject-specific (first-level) posterior expectations can be usedfor classic inference in the usual way. Note that these (summarystatistics) are optimal in the sense that they have been esti-mated under empirical (hierarchical) priors.
If called with a single DCM, there are no between-subject effects, and the design matrix is assumed tomodel mixtures of parameters at the first level. If calledwith a cell array, each column is assumed to containfirst-level DCMs inverted under the same model.
spm_dcm_loo% Leave-one-out cross-validation for empirical Bayes andDCM
% FORMAT [qE,qC,Q] � spm_dcm_loo(DCM,M,field)%% DCM - {N [x M]} structure DCM array of (M) DCMs
from (N) subjects% ——————————————————————% DCM{i}.M.pE- prior expectation of parameters% DCM{i}.M.pC- prior covariances of parameters% DCM{i}.Ep- posterior expectations% DCM{i}.Cp- posterior covariance%
% M.X- second-level design matrix, where X(:,1) �ones(N,1) [default]
% field- parameter fields in DCM{i}.Ep to optimize [de-fault: {’A’,’B’}]
%’All’ will invoke all fields%% qE- posterior predictive expectation (group effect)% qC- posterior predictive covariances (group effect)% Q- posterior probability over unique levels of X(:,2)This routine uses the posterior predictive density over
the coefficients of between-subject effects encoded bya design matrix X. It is assumed that the second columnof X contains classification or predictor variables. Across-validation scheme is used to estimate the mixtureof parameters at the first (within-subject) level that areconserved over subjects in terms of a constant (firstcolumn of X) and differences (second column of X).Using a leave-one-out scheme, the predictive posteriordensity of the predictive variable is used to assesscross-validation accuracy. For multiple models, thisprocedure is repeated for each model in the columns ofthe DCM array.
ReferencesAdams RA, Stephan KE, Brown HR, Frith CD, Friston KJ (2013) The
computational anatomy of psychosis. Front Psychiatry 4:47.CrossRef Medline
Alagoz O, Hsu H, Schaefer AJ, Roberts MS (2010) Markov decisionprocesses: a tool for sequential decision making under uncer-tainty. Med Decis Mak 30:474–483.
Attias H (2000) A variational Bayesian framework for graphical mod-els. In: Neural information processing systems 12 (Solla SA, LeenTK, Müller K, eds). La Jolla, CA: Neural Information ProcessingSystems Foundation, Inc.
Beal MJ (2003) Variational algorithms for approximate Bayesianinference. Masters thesis, University of Cambridge, UK.
Beck JM, Ma WJ, Pitkow X, Latham PE, Pouget A (2012) Not noisy,just wrong: the role of suboptimal inference in behavioral variabil-ity. Neuron 74:30–39. CrossRef Medline
Berkes P, Orbán G, Lengyel M, Fiser J (2011) Spontaneous corticalactivity reveals hallmarks of an optimal internal model of theenvironment. Science 331:83–87.
Bishop C (2006) Pattern recognition and machine learning. NewYork: Springer.
Daunizeau J, den Ouden HEM, Pessiglione M, Kiebel SJ, StephanKE, Friston KJ (2010) Observing the observer (I): meta-Bayesianmodels of learning and decision-making. Sporns O, ed. PLoS One5:e15554. CrossRef Medline
Dayan P (2014) Rationalizable irrationalities of choice. Top Cogn Sci6:204–228.
Efron B, Morris C (1973) Stein’s estimation rule and its competi-tors—an empirical Bayes approach. J Am Stat Assoc 68:117–130.
FitzGerald THB, Dolan RJ, Friston KJ (2014) Model averaging, opti-mal inference, and habit formation. Front Hum Neurosci 8:457.CrossRef Medline
FitzGerald THB, Schwartenbeck P, Moutoussis M, Dolan RJ, FristonK (2015) Active inference, evidence accumulation, and the urntask. Neural Comput 27:306–328.
Friston KJ, Shiner T, FitzGerald T, Galea JM, Adams R, Brown H,Dolan RJ, Moran R, Stephan KE, Bestmann S (2012) Dopamine,affordance and active inference. PLoS Comput Biol 8:e1002327.CrossRef Medline
Friston K, Schwartenbeck P, FitzGerald T, Moutoussis M, Behrens T,Dolan RJ (2013) The anatomy of choice: active inference andagency. Front Hum Neurosci 7:598 CrossRef Medline
Friston K, Schwartenbeck P, FitzGerald T, Moutoussis M, Behrens T,Dolan RJ (2014a) The anatomy of choice: dopamine and decision-making. Philos Trans R Soc Lond B Biol Sci 369:). CrossRef Med-line
Friston K, Stephan KE, Montague R, Dolan RJ (2014b) Computa-tional psychiatry: the brain as a phantastic organ. Lancet Psychi-atry 1:148–158. CrossRef Medline
Friston K, Rigoli F, Ognibene D, Mathys C, FitzGerald T, Pezzulo G(2015a) Active inference and epistemic value. Cogn Neurosci 6:187–214.
Friston K, Zeidman P, Litvak V (2015b) Empirical Bayes for DCM: agroup inversion scheme. Front Syst Neurosci 9:164.
Friston KJ, Litvak V, Oswal A, Razi A, Stephan KE, van Wijk BCM,Ziegler G, Zeidman P (2015c) Bayesian model reduction and em-pirical Bayes for group (DCM) studies. Neuroimage 128:413–431.
Friston, K., FitzGerald, T., Rigoli, F., Schwartenbeck, P., O’Doherty,J., & Pezzulo, G. (2016). Active inference and learning. Neurosci-ence & Biobehavioral Reviews. In press. CrossRef
Hauser TU, Iannaccone R, Ball J, Mathys C, Brandeis D, Walitza S,Brem S (2014) Role of the medial prefrontal cortex in impaireddecision making in juvenile attention-deficit/hyperactivity disorder.JAMA psychiatry 71:1165–1173.
Hauser TU, Fiore VG, Moutoussis M, Dolan RJ (2016) Computationalpsychiatry of ADHD: neural gain impairments across Marrian levelsof analysis. Trends Neurosci 39:63–73.
Huys QJM, Guitart-Masip M, Dolan RJ, Dayan P (2015) Decision-theoretic psychiatry. Clin Psychol Sci 3:400–421.
Huys QJM, Maia TV, Frank MJ (2016) Computational psychiatry as abridge from neuroscience to clinical applications. Nat Neurosci19:404–413.
Litvak V, Garrido M, Zeidman P, Friston K (2015) Empirical Bayes forgroup (DCM) studies: a reproducibility study. Front Hum Neurosci9:670.
Mathys C, Daunizeau J, Friston KJ, Stephan KE (2011) A Bayesianfoundation for individual learning under uncertainty. Front HumNeurosci 5:39. CrossRef Medline
Montague PR, Dolan RJ, Friston KJ, Dayan P (2012) Computationalpsychiatry. Trends Cogn Sci 16:72–80. CrossRef Medline
Moran RJ, Campo P, Symmonds M, Stephan KE, Dolan RJ, FristonK (2013) Free energy, precision and learning: the role of cholinergicneuromodulation. J Neurosci 33:8227–8236. CrossRef Medline
Rao RPN (2010) Decision making under uncertainty: a neural modelbased on partially observable markov decision processes. FrontComput Neurosci 4:146.
Schwartenbeck P, FitzGerald THB, Mathys C, Dolan R, Friston K(2015a) The dopaminergic midbrain encodes the expected cer-tainty about desired outcomes. Cereb Cortex 25:3434–3445.
Schwartenbeck P, FitzGerald THB, Mathys C, Dolan R, KronbichlerM, Friston K (2015b) Evidence for surprise minimization over valuemaximization in choice behavior. Sci Rep 5:16575.
Schwartenbeck P, FitzGerald THB, Mathys C, Dolan R, Wurst F,Kronbichler M, Friston K (2015c) Optimal inference with subopti-mal models: addiction and active Bayesian inference. Med Hy-potheses 84:109–117.
Stephan KE, Mathys C (2014) Computational approaches to psychi-atry. Curr Opin Neurobiol 25:85–92.
Stephan KE, Penny WD, Daunizeau J, Moran RJ, Friston KJ (2009)Bayesian model selection for group studies. Neuroimage 46:1004–1017. CrossRef Medline
Stephan KE, Iglesias S, Heinzle J, Diaconescu AO (2015) Translationalperspectives for computational neuroimaging. Neuron 87:716–732.
Sutton R, Barto A (1998) Reinforcement learning: an introduction.Cambridge, MA: MIT.
Tervo DGR, Tenenbaum JB, Gershman S (2016) Toward the neural
implementation of structure learning. Curr Opin Neurobiol 37:99–105. CrossRef Medline
Toussaint M, Harmeling S, Storkey A (2006) Probabilistic inferencefor solving (PO) MDPs. Edinburgh, UK: Institute for Adaptive andNeural Computation, School of Informatics, University of Edin-burgh, Informatics Research Report 0934.
Wang XJ, Krystal JH (2014) Computational Psychiatry. Neuron 84:638–654.
