Markov Models in Medical Decision Making

322

Markov Models in Medical Decision Making:A Practical Guide

FRANK A. SONNENBERG, MD, J. ROBERT BECK, MD

Markov models are useful when a decision problem involves risk that is continuous overtime, when the timing of events is important, and when important events may happen morethan once. Representing such clinical settings with conventional decision trees is difficultand may require unrealistic simplifying assumptions. Markov models assume that a patientis always in one of a finite number of discrete health states, called Markov states. All eventsare represented as transitions from one state to another. A Markov model may be evaluatedby matrix algebra, as a cohort simulation, or as a Monte Carlo simulation. A newer repre-sentation of Markov models, the Markov-cycle tree, uses a tree representation of clinicalevents and may be evaluated either as a cohort simulation or as a Monte Carlo simulation.The ability of the Markov model to represent repetitive events and the time dependence ofboth probabilities and utilities allows for more accurate representation of clinical settings thatinvolve these issues. Key words: Markov models; Markov-cycle decision tree; decision mak-ing. (Med Decis Making 1993;13:322-338)

A decision tree models the prognosis of a patient sub-sequent to the choice of a management strategy. For

example, a strategy involving surgery may model theevents of surgical death, surgical complications, andvarious outcomes of the surgical treatment itself. Forpractical reasons, the analysis must be restricted to afinite time frame, often referred to as the time horizonof the analysis. This means that, aside from death, theoutcomes chosen to be represented by terminal nodesof the tree may not be final outcomes, but may simplyrepresent convenient stopping points for the scope ofthe analysis. Thus, every tree contains terminal nodesthat represent &dquo;subsequent prognosis&dquo; for a particularcombination of patient characteristics and events.There are various ways in which a decision analyst

can assign values to these terminal nodes of the de-cision tree. In some cases the outcome measure is acrude life expectancy; in others it is a quality-adjustedlife expectancy.’ One method for estimating life ex-pectancy is the declining exponential approximationof life expectancy (DEALE),2 which calculates a patient-specific mortality rate for a given combination of pa-tient characteristics and comorbid diseases. Life ex-

pectancies may also be obtained from Gompertz models

of survival’ or from standard life tables.’ This paperexplores another method for estimating life expec-tancy, the Markov model.

In 1983, Beck and Pauker described the use of Mar-kov models for determining prognosis in medical ap-plications.’ Since that introduction, Markov modelshave been applied with increasing frequency in pub-lished decision analyses.’-9 Microcomputer softwarehas been developed to permit constructing and eval-uating Markov models more easily. For these reasons,a revisit of the Markov model is timely. This paperserves both as a review of the theory behind the Mar-kov model of prognosis and as a practical guide forthe construction of Markov models using microcom-puter decision-analytic software.Markov models are particularly useful when a de-

cision problem involves a risk that is ongoing over time.Some clinical examples are the risk of hemorrhagewhile on anticoagulant therapy, the risk of rupture ofan abdominal aortic aneurysm, and the risk of mor-

tality in any person, whether sick or healthy. Thereare two important consequences of events that haveongoing risk. First, the times at which the events willoccur are uncertain. This has important implicationsbecause the utility of an outcome often depends onwhen it occurs. For example, a stroke that occurs im-mediately may have a different impact on the patientthan one that occurs ten years later. For economic

analyses, both costs and utilities are discounted&dquo;,&dquo;such that later events have less impact than earlierones. The second consequence is that a given eventmay occur more than once. As the following exampleshows, representing events that are repetitive or thatoccur with uncertain timing is difficult using a simpletree model.

Received February 23, 1993, from the Division of General InternalMedicine, Department of Medicine, UMDNJ Robert Wood JohnsonMedical School, New Brunswick, New Jersey (FAS) and the Infor-mation Technology Program, Baylor College of Medicine, Houston,Texas (JRB). Supported in part by Grant LM05266 from the NationalLibrary of Medicine and Grant HS06396 from the Agency for HealthCare Policy and Research.

Address correspondence and reprint requests to Dr. Sonnenberg:Division of General Internal Medicine, UMDNJ Robert Wood John-son Medical School, 97 Paterson Street, New Brunswick, NJ 08903.

at National Institutes of Health Library on January 21, 2009 http://mdm.sagepub.comDownloaded from

http://mdm.sagepub.com

323

A Specific ExampleConsider a patient who has a prosthetic heart valve

and is receiving anticoagulant therapy. Such a patientmay have an embolic or hemorrhagic event at any time.Either kind of event causes morbidity (short-term and/or chronic) and may result in the patient’s death. Thedecision tree fragment in figure 1 shows one way ofrepresenting the prognosis for such a patient. The firstchance node, labelled ANTICOAG, has three branches,labelled BLEED, EMBOLUS, and NO EVENT. Both BLEED and

EMBOLUS may be either FATAL or NON-FATAL. If NO EVENT

occurs, the patient remains WELL.There are several shortcomings with this model. First,

the model does not specify when events occur. Sec-ond, the structure implies that either hemorrhage orembolus may occur only once. In fact, either may oc-cur more than once. Finally, at the terminal nodeslabelled POST EMBOLUS, POST BLEED, and WELL, the analyststill is faced with the problem of assigning utilities, atask equivalent to specifying the prognosis for each ofthese non-fatal outcomes.

The first problem, specifying when events occur,may be addressed by using the tree structure in figure1 and making the assumption that either BLEED orEMBOLUS occurs at the average time consistent withthe known rate of each complication. For example, ifthe rate of hemorrhage is a constant 0.05 per personper year, then the average time before the occurrence

of a hemorrhage is 1/0.05 or 20 years. Thus, the eventof having a fatal hemorrhage will be associated witha utility of 20 years of normal-quality survival. However,the patient’s normal life expectancy may be less than20 years. Thus, the occurrence of a stroke would havethe paradoxical effect of improving the patient’s lifeexpectancy. Other approaches, such as assuming thatthe stroke occurs halfway through the patient’s nor-mal life expectancy, are arbitrary and may lessen thefidelity of the analysis.Both the timing of events and the representation of

events that may occur more than once can be ad-

dressed by using a recursive decision tree.12 In a re-cursive tree, some nodes have branches that have ap-

peared previously in the tree. Each repetition of thetree structure represents a convenient length of timeand any event may be considered repeatedly. A re-cursive tree that models the anticoagulation problemis depicted in figure 2.

Here, the nodes representing the previous terminalnodes POST-BLEED, POST-EMBOLUS, and No EVENT are re-

placed by the chance node ANTICOAG, which appearedpreviously at the root of the tree. Each occurrence ofBLEED or EMBOLUS represents a distinct time period, sothe recursive model can represent when events occur.However, despite this relatively simple model and car-rying out the recursion for only two time periods, thetree in figure 2 is &dquo;bushy,&dquo; with 17 terminal branches.If each level of recursion represents one year, then

FIGURE 1. Simple tree fragment modeling complications of antico-agulant therapy.

carrying out this analysis for even five years wouldresult in a tree with hundreds of terminal branches.

Thus, a recursive model is tractable only for a veryshort time horizon.

The Markov Model

The Markov model provides a far more convenientway of modelling prognosis for clinical problems withongoing risk. The model assumes that the patient isalways in one of a finite number of states of healthreferred to as Markov states. All events of interest aremodelled as transitions from one state to another. Eachstate is assigned a utility, and the contribution of thisutility to the overall prognosis depends on the lengthof time spent in the state. In our example of a patientwith a prosthetic heart valve, these states are WELL,DISABLED, and DEAD. For the sake of simplicity in thisexample, we assume that either a bleed or a non-fatalembolus will result in the same state (DISABLED) andthat the disability is permanent.The time horizon of the analysis is divided into equal

increments of time, referred to as Markov cycles. Dur-ing each cycle, the patient may make a transition fromone state to another. Figure 3 shows a commonly usedrepresentation of Markov processes, called a state-transition diagram, in which each state is representedby a circle. Arrows connecting two different states in-dicate allowed transitions. Arrows leading from a stateto itself indicate that the patient may remain in thatstate in consecutive cycles. Only certain transitionsare allowed. For example, a person in the WELL statemay make a transition to the DISABLED state, but atransition from DISABLED to WELL is not allowed. A per-son in either the WELL state or the DISABLED state maydie and thus make a transition to the DEAD state. How-ever, a person who is in the DEAD state, obviously,cannot make a transition to any other state. Therefore,a single arrow emanates from the DEAD state, leadingback to itself. It is assumed that a patient in a givenstate can make only a single state transition during acycle.



324

FIGURE 2. Recursive tree mod-

eling complications of antico-agulant therapy.

The length of the cycle is chosen to represent a

clinically meaningful time interval. For a model thatspans the entire life history of a patient and relativelyrare events the cycle length can be one year. On theother hand, if the time frame is shorter and modelsevents that may occur much more frequently, the cycletime must be shorter, for example monthly or evenweekly. The cycle time also must be shorter if a ratechanges rapidly over time. An example is the risk ofperioperative myocardial infarction (MI) following pre-vious MI that declines to a stable value over six months.&dquo;The rapidity of this change in risk dictates a monthlycycle time. Often the choice of a cycle time will be

determined by the available probability data. For ex-ample, if only yearly probabilities are available, thereis little advantage to using a monthly cycle length.

INCREMENTAL UTILITY

Evaluation of a Markov process yields the averagenumber of cycles (or analogously, the average amountof time) spent in each state. Seen another way, the

patient is &dquo;given credit&dquo; for the time spent in eachstate. If the only attribute of interest is duration of

survival, then one need only add together the average



325

FIGURE 3. Markov-state diagram. Each circle represents a Markovstate. Arrows indicate allowed transitions.

times spent in the individual states to arrive at an

expected survival for the process.

n

Expected utility = ~ tss=l i

where ts is the time spent in state s.

Usually, however, the quality of survival is consid-ered important. Each state is associated with a qualityfactor representing the quality of life in that state rel-ative to perfect health. The utility that is associatedwith spending one cycle in a particular state is referredto as the incremental utility. Consider the Markov pro-cess depicted in figure 3. If the incremental utility ofthe DISABLED state is 0.7, then spending the cycle inthe DISABLED state contributes 0.7 quality-adjusted cyclesto the expected utility. Utility accrued for the entireMarkov process is the total number of cycles spent ineach state, each multiplied by the incremental utilityfor that state.

n

Expected utility = ~ ts X Uss=l i

Let us assume that the DEAD state has an incremen-

tal utility of zero,* and that the WELL state has an in-cremental utility of 1.0. This means that for every cyclespent in the WELL state the patient is credited with aquantity of utility equal to the duration of a singleMarkov cycle. If the patient spends, on average, 2.5cycles in the WELL state and 1.25 cycles in the DISABLEDstate before entering the DEAD state, the utility assignedwould be (2.5 X 1) + (1.25 X 0.7), or 3.9 quality-ad-justed cycles. This number is the quality-adjusted lifeexpectancy of the patient.

* For medical examples, the incremental utility of the absorbingDEAD state must be zero, because the patient will spend an infiniteamount of time in the DEAD state and if the incremental utility werenon-zero, the net utility for the Markov process would be infinite.

When performing cost-effectiveness analyses, aseparate incremental utility may be specified for eachstate, representing the financial cost of being in thatstate for one cycle. The model is evaluated separatelyfor cost and survival. Cost-effectiveness ratios are cal-culated as for a standard decision tree.10,11

TYPES OF MARKOV PROCESSES

Markov processes are categorized according towhether the state-transition probabilities are constantover time or not. In the most general type of Markovprocess, the transition probabilities may change overtime. For example, the transition probability for thetransition from WELL to DEAD consists of two compo-nents. The first component is the probability of dyingfrom unrelated causes. In general, this probabilitychanges over time because, as the patient gets older,the probability of dying from unrelated causes willincrease continuously. The second component is theprobability of suffering a fatal hemorrhage or embolusduring the cycle. This may or may not be constantover time.

A special type of Markov process in which the tran-sition probabilities are constant over time is called aMarkov chain. If it has an absorbing state, its behaviorover time can be determined as an exact solution bysimple matrix algebra, as discussed below. The DEALEcan be used to derive the constant mortality ratesneeded to implement a Markov chain. However, theavailability of specialized software to evaluate Markovprocesses and the greater accuracy afforded by age-specific mortality rates have resulted in greater reli-ance on Markov processes with time-variant proba-bilities.

The net probability of making a transition from onestate to another during a single cycle is called a tran-sition probability. The Markov process is completelydefined by the probability distribution among thestarting states and the probabilities for the individualallowed transitions. For a Markov model of n states,there will be n2 transition probabilities. When theseprobabilities are constant with respect to time, theycan be represented by an n x n matrix, as shown intable 1. Probabilities representing disallowed transi-tions will, of course, be zero. This matrix, called the P

matrix, forms the basis for the fundamental matrixsolution of Markov chains described in detail by Beckand Pauker.’

TaMe 1 . P Matrix



326

FIGURE 4. Markov-state diagram. The shaded circle labeled &dquo;STROKE&dquo;represents a temporary state.

THE MARKOV PROPERTY

The model illustrated in figure 3 is compatible witha number of different models collectively referred toas finite stochastic processes. In order for this modelto represent a Markov process, one additional restric-tion applies. This restriction, sometimes referred to asthe Markovian assumption’ or the Markov property) 14specifies that the behavior of the process subsequentto any cycle depends only on its description in thatcycle. That is, the process has no memory for earliercycles. Thus, in our example, if someone is in the

DISABLED state after cycle n, we know the probabilitythat he or she will end up in the DEAD state after cyclen + 1. It does not matter how much time the person

spent in the WELL state before becoming DISABLED. Putanother way, all patients in the DISABLED state have thesame prognosis regardless of their previous histories.For this reason, a separate state must be created foreach subset of the cohort that has a distinct utility orprognosis. If we want to assign someone disabled froma bleed a different utility or risk of death than someonedisabled from an embolus, we must create two dis-abled states. The Markovian assumption is not fol-lowed strictly in medical problems. However, the as-sumption is necessary in order to model prognosiswith a finite number of states.

MARKOV STATES

In order for a Markov process to terminate, it musthave at least one state that the patient cannot leave.Such states are called absorbing states because, aftera sufficient number of cycles have passed, the entirecohort will have been absorbed by those states. Inmedical examples the absorbing states must representdeath because it is the only state a patient cannotleave. There is usually no need for more than one DEAD

state, because the incremental utility for the DEAD stateis zero. However, if one wishes to keep track of thecauses of death, then more than one DEAD state maybe used.

Temporary states are required whenever there is anevent that has only short-term effects. Such states aredefined by having transitions only to other states andnot to themselves. This guarantees that the patientcan spend, at most, one cycle in that state. Figure 4illustrates a Markov process that is the same as that

shown in figure 3 except that a temporary state hasbeen added, labeled STROKE. An arrow leads to STROKE

only from the WELL state, and there is no arrow fromthe STROKE back to itself. This ensures that a patientmay spend no more than a single cycle in the STROKEstate. Temporary states have two uses. The first use isto apply a utility or cost adjustment specific to thetemporary state for a single cycle. The second use isto assign temporarily different transition probabilities.For example, the probability of death may be higherin the STROKE state than in either the WELL state or theDISABLED state.

A special arrangement of temporary states consistsof an array of temporary states arranged so that eachhas a transition only to the next. These states are calledtunnel states because they can be visited only in a fixedsequence, analogous to passing through a tunnel. Thepurpose of an array of tunnel states is to apply toincremental utility or to transition probabilities a tem-porary adjustment that lasts more than one cycle.An example of tunnel states is depicted in figure 5.

The three tunnel states, shaded and labelled POST Mil

through POST M13, represent the first three months fol-lowing an MI. The POST Mil state is associated with thehighest risk of perioperative death. POST MI2 and POSTM13 are associated with successively lower risks of per-ioperative death. If a patient passes through all threetunnel states without having surgery, he or she entersthe POST Mi state, in which the risk of perioperativedeath is constant.

Because of the Markovian assumption, it is not pos-sible for the prognosis of a patient in a given state todepend on events prior to arriving in that state. Often,however, patients in a given state, for example WELL,may actually have different prognoses depending onprevious events. For example, consider a patient whois WELL but has a history of gallstones. Each cycle, thepatient has a certain probability of developing com-plications from the gallstones. Following a cholecys-tectomy, the patient will again be WELL but no longerhas the same probability of developing biliary com-plications. Thus, the state WELL actually contains twodistinct populations of people, those with gallstonesand those who have had a cholecystectomy. In orderfor the model to reflect the different prognoses forthese two classes of well patients, it must contain twodistinct well states, one representing WELL WITH GALL-STONES and the other representing WELL, STATUS-POST



327

FIGURE 5. Tunnel states: the three shaded circles represent tem-

porary states that can be visited only in a fixed sequence.

CHOLECYSTECTOMY. In general, if prognosis depends inany way on past history, it requires that there be onedistinct state to represent the different histories.

USE OF THE MARKOV PROCESS IN

DECISION ANALYSIS

The Markov process models prognosis for a givenpatient and thus is analogous to a utility in an ordinaiydecision tree. For example, if we are trying to choosebetween surgeiy and medical therapy, we may con-struct a decision tree like that shown in figure 6A. Inthis case, events of interest, such as operative deathand cure, are modelled by tree structure &dquo;outside&dquo; theMarkov process. The Markov process is being usedsimply to calculate survival for a terminal node of thetree. This structure is inefficient, because it requiresthat an entire Markov process be run for each terminal

node, of which there may be dozens or even hundreds.A far more efficient structure is shown in figure 6B. Inthis case, the Markov process incorporates all eventsof interest and the decision analysis is reduced simplyto comparing the values of two Markov processes. Theuse of the cycle tree representation (discussed in detailbelow) permits representing all relevant events withinthe Markov process.

1#nufllati’is of Markov Models

THE FUNDAMENTAL MATRIX SOLUTION

When the Markov process has constant transition

probabilities (and constant incremental utilities) forall states, the expected utility may be calculated bymatrix algebra to yield the fundamental matrix, whichshows, for each starting state, the expected length oftime spent in the state. The matrix solution is fast and

provides an &dquo;exact&dquo; solution that is not affected by thecycle length. There are three main disadvantages ofthe matrix formation. The first is the difficulty in per-forming matrix inversion. However, this is less of a

problem than when Beck and Pauker’ described thetechnique, because many commonly available micro-computer spreadsheet programs now perform matrixalgebra. The second disadvantage is the restriction toconstant transition probabilities. The third disadvan-tage is the need to represent all the possible ways ofmaking a transition from one state to another as asingle transition probability. At least for medical ap-plications, the matrix algebra solution has been largelyrelegated to the history books. For more details of thematrix algebra solution the reader is referred to Beckand Pauker.’

FIGURE 6. Use of Markov processes in a decision model. In panel A(top), the Markov process is used only as a utility. In panel B (bottom),the Markov process is used to represent all events.



328

FIGURE 7. Markov cohort simulation. PanelA (top), shows the initialdistribution with all patients in the WELL state. Panel B (middle)shows the distribution partway through the simulation. Panel C(bottom) shows the final distribution, with the entire cohort in the

DEAD state.

MARKOV COHORT SIMULATION

The Markov cohort simulation is the most intuitive

representation of a Markov process. The differencebetween a cohort simulation and the matrix formu-

lation may be thought of as analogous to the differencebetween determining the area under a curve by divid-ing it into blocks and summing their areas versus cal-

culating the area by solving the integral of the functiondescribing the curve. The simulation considers a hy-pothetical cohort of patients beginning the processwith some distribution among the starting states. Con-sider again the prognosis of a patient who has a pros-thetic heart valve, represented by the Markov-state dia-gram in figure 3. Figure 7A illustrates the cohort at thebeginning of the simulation. In this example, all pa-tients are in the WELL state. However, it is not necessaryto have all patients in the same state at the beginningof the simulation. For example, if the strategy repre-sents surgery, a fraction of the cohort may begin the

simulation in the DEAD state as a result of operativemortality.The simulation is &dquo;run&dquo; as follows. For each cycle,

the fraction of the cohort initially in each state is par-titioned among all states according to the transitionprobabilities specified by the P matrix. This results ina new distribution of the cohort among the variousstates for the subsequent cycle. The utility accrued forthe cycle is referred to as the cycle sum and is cal-culated by the formula:

where n is the number of states, fs is the fraction ofthe cohort in state s, and U, is the incremental utilityof state s. The cycle sum is added to a running totalthat is referred to as the cumulative utility. Figure 7Bshows the distribution of the cohort after a few cycles.Fifty percent of the cohort remains in the WELL state.Thirty percent of the cohort is in the SICK state and20% in the DEAD state. The simulation is run for enoughcycles so that the entire cohort is in the DEAD state(fig. 7C).The cohort simulation can be represented in tabular

form, as shown in table 2. This method may be im-

plemented easily using a microcomputer spreadsheetprogram. The first row of the table represents the start-

ing distribution. A hypothetical cohort of 10,000 pa-tients begins in the WELL state. The second row showsthe distribution at the end of the first cycle. In ac-cordance with the transition probabilities specified inthe P-matrix (table 1), 2,000 patients (20% of the originalcohort) have moved to the DISABLED state and another

2,000 patients to the DEAD state. This leaves 6,000 (60%)remaining in the WELL state. This process is repeatedin subsequent cycles. The fifth column in table 2 showsthe calculation of the cycle sum, which is the sum ofthe number of cohort members in each state multi-

plied by the incremental utility for that state. For ex-ample, because the incremental utility of the DISABLEDstate is 0.7, the cycle sum during cycle 1 is equal to(6,000 X 1) + (2,000 X 0.7) = 7,400. The DEAD state

does not contribute to the cycle sum because its in-

Table 2 . Markov Cohort Simulation



329

cremental utility is zero. The sixth column shows the ‘

cumulative utility following each cycle.Because the probabilities of leaving the WELL and

DEAD states are finite and the probability of leaving theDEAD sate is zero, more and more of the cohort ends

up in the DEAD state. The fraction of the cohort in theDEAD state actually is always less than 100% because,during each cycle, there is a finite probability of apatient’s remaining alive. For this reason, the simu-lation is stopped when the cycle sum falls below somearbitrarily small threshold (e.g., 1 person-cycle) or whenthe fraction of the cohort remaining alive falls belowa certain amount. In this case, the cycle sum fallsbelow 1 after 24 cycles. The expected utility for thisMarkov cohort simulation is equal to the cumulativeutility when the cohort has been completely absorbeddivided by the original size of the cohort. In this case,the expected utility is 23,752/10,000, or 2.3752 quality-adjusted cycles. The unadjusted life expectancy maybe found by summing the entries in the columns forthe WELL and DISABLED states and dividing by the cohortsize. Notice that the cohort memberships at the startdo not contribute to these sums. Thus, the cohortmembers will spend, on average, 1.5 cycles in the WELLstate and 1.25 cycles in the DISABLED state, for a netunadjusted life expectancy of 2.75 cycles.

THE HALF-CYCLE CORRECTION

The Markov model assumes that during a singlecycle, each patient undergoes no more than one statetransition. One way to visualize the Markov process is

to imagine that a clock makes one &dquo;tick&dquo; for each cyclelength. At each tick, the distribution of states is ad-justed to reflect the transitions made during the pre-ceding cycle. The Markov cohort simulation requiresexplicit bookkeeping (as illustrated in table 2) duringeach cycle to give credit according to the fraction ofthe cohort in each state. In the example illustrated intable 2, the bookkeeping was performed at the end ofeach cycle.

In reality, transitions occur not only at the clockticks, but continuously throughout each cycle. There-fore, counting the membership only at the beginningor at the end of the cycle will lead to errors. Theprocess of carrying out a Markov simulation is anal-ogous to calculating expected survival that is equal tothe area under a survival curve. Figure 8 shows a sur-vival curve for members of a state. The smoothness of

the curve reflects the continuous nature of state tran-sitions. Each rectangle under the curve represents theaccounting of the cohort membership during one cyclewhen the count is performed at the end of each cycle.The area of the rectangles consistently underestimatesthe area under the curve. Counting at the beginningof each cycle, as in figure 9, consistently overestimatessurvival. To more accurately reflect the continuousnature of the state transitions, we make the assump-

FIGURE 8. Counting cohort membership at the end of each cycle.

FIGURE 9. Counting cohort membership at the beginning of eachcycle.

FIGURE 10. Illustration of the half-cycle correction. at National Institutes of Health Library on January 21, 2009 http://mdm.sagepub.comDownloaded from


330

tion that state transitions occur, on average, haywaythrough each cycle. There is no way to determine thestate membership in the middle of the cycle. However,if we consider the count at the end of each cycle tobe in the middle of a cycle that begins halfway throughthe previous cycle and ends halfway through the sub-sequent cycle, as in figure 10, then the under- andoverestimations will be balanced. This is equivalent toshifting all cycles one half cycle to the right. We mustthen add a half cycle for the starting membership atthe beginning to compensate for this shift to the right.Adding a half cycle for the example in table 2 resultsin an expected utility of 2.875 quality-adjusted cyclesand a life expectancy of 3.25 cycles.The shift to the right makes no difference at the end

of the simulation if the cohort is completely absorbedbecause the state membership at that time is infini-tesimal. However, if the simulation is terminated priorto the absorption of the cohort, the shift to the rightwill result in overestimation of the expected survival.Therefore, for simulations that terminate prior to ab-

sorption, an additional correction must be made bysubtracting a half cycle for members of the state whoare still alive at the end of the simulation. The im-

portance of the half cycle correction depends on cyclelength. If the cycle length is veiy short relative to av-erage survival, the difference between actual survivaland simulated survival (as shown in figure 8) will besmall. If the cycle time is larger relative to suivival, thedifference will be more significant. The interested readershould note that the fundamental matrix represen-tation is equivalent to counting state membership atthe beginning of each cycle. Therefore) the correctionthat should be applied to the result of a matrix solutionis subtraction of one half cycle from the membershipof each starting state.

FIGURE 11. Probability tree corresponding to the WELL state.

THE MARKOV-CYCLE TREE

In the preceding discussion, transition probabilitieswere provided as if they were elemental data suppliedwith a problem. However, for actual clinical settings,transition probabilities may be quite complicated tocalculate because transitions from one state to another

may happen in a variety of ways. For example, a patientin the WELL state may make a transition to the DEAD

state by having a fatal stroke, by having an accident,or by dying of complications of a coexisting disease.Each transition probability must take into account allof these transition paths. Hollenberg15 devised an el-egant representation of Markov processes in which thepossible events taking place during each cycle are rep-resented by a probability tree.The probability tree corresponding to the WELL state

is illustrated in figure 11. It contains a chance node

modeling the occurrence of death from age, sex, andrace (ASR)-specific mortality, the branch labelled DIEASR. If the patient does not die from natural causes,the branch labelled SURVIVE leads to a chance node

modelling whether the patient has a BLEED or an EM-BOLUS, either of which may be fatal. If neither BLEED

nor EMBOLUS occurs (the branch NO EVENT), the patientremains WELL. Each terminal node in the probabilitytree is labelled with the name of the state in which a

patient reaching that terminal node will begin the nextcycle. Thus, a patient reaching any terminal node la-belled DEAD will begin the next cycle in the DEAD state.A patient surviving either an embolus or a bleed willbegin the next cycle in the DISABLED state. The prob-ability tree for patients beginning in the DISABLED stateis identical to that for the WELL state, except that pa-tients having NO EVENT will still be DISABLED. The prob-ability tree for patients beginning in the DEAD stateconsists only of the terminal node labelled with thename of the DEAD state since no event is possible, anda patient in the DEAD state will always remain in thatstate.

The subtrees are attached to a special type of nodedesignated a Markov node as depicted in figure 12.There is one branch of the Markov node for each Mar-

kov state. Each probability from the Markov node toone of its branches is equal to the probability that thepatient will start in the corresponding state. The Mar-kov node together with its attached subtrees is referredto as a Markov-cycle tree15 and, along with the incre-mental utilities and the probabilities of the branchesof chance nodes, is a complete representation of aMarkov process. Starting at any state branch, the sumof the probabilities of all paths leading to terminalnodes labelled with the name of a particular endingstate is equal to the transition probability from thebeginning state to the ending state. For example, thehighlighted paths in figure 12 show all transitions fromWELL to DISABLED.



331

EVALUATING CYCLE TREES

A cycle tree may be evaluated as a Markov cohortsimulation. First, the starting composition of the co-hort is determined by partitioning the cohort amongthe states according to the probabilities leading fromthe Markov node to the individual branches. Each sub-

tree is then traced from its root to its termini (&dquo;foldingforward&dquo;), partitioning the subcohort for the corre-sponding state according to the probability tree. Theresult is a new distribution of the cohort among the

states, which reflects how the cohort appears after a

single cycle. The fraction of the cohort currently ineach state is then credited with the appropriate in-cremental utility to form the cycle sum, which is addedto the cumulative utility. The new distribution of thecohort is then used as the starting distribution for thenext cycle. The process is repeated until some pre-determined criterion is reached, usually when thequantity of utility accumulating for each state dropsbelow some specified small quantity. This occurs whenthe fraction of the cohort in the DEAD state approachesone.

ADVANTAGES OF THE CYCLE TREE REPRESENTATION

Cycle trees have many of the same advantages thatdecision trees have for modelling complex clinical sit-uations. They allow the analyst to break up a largeproblem into smaller, more manageable ones. Thisclarifies issues for the analyst and for others trying tounderstand the results. The use of subtrees promotesappropriate symmetiy among the various states, thusenhancing the fidelity of the model. The model pro-vides a great deal of flexibility when changing or re-fining a Markov model. If a single component proba-bility or a detail of a subtree needs to be changed, thiscan be done without recalculating the aggregate tran-sition probabilities. Finally, the disaggregation of tran-sition probabilities permits sensitivity analysis to beperformed on any component probability. Because ofits advantages, the cycle tree representation has beenused most often in recently published Markov deci-sion analyses.’-’

MONTE CARLO SIMULATION

As an alternative to simulating the prognosis of ahypothetical cohort of patients, the Monte Carlo sim-ulation determines the prognoses of a large numberof individual patients. This is illustrated in figure 13.Each patient begins in the starting state (i.e., the WELLstate), and at the end of each cycle, a random-numbergenerator is used together with the transition proba-bilities to determine in which state the patient willbegin the next cycle. Just as for the cohort simulation,the patient is given credit for each cycle spent in a

FIGURE 12. Complete Markov-cycle tree corresponding to the an-ticoagulation problem.

non-DEAD state and each state may be adjusted forquality of life. When the patient enters the DEAD state,the simulation is stopped. For the example in figure13, the patient spends two cycles in the WELL state andthree cycles in the DISABLED state before being &dquo;ab-

sorbed,&dquo; resulting in a utility of (2 X 1) + (3 X 0.7) or

4.1 quality-adjusted cycles. The process is repeated avery large number (on the order of 104) of times. Eachtrial generates a quality-adjusted survival time. After alarge number of trials, these constitute a distributionof survival values. The mean value of this distribution

will be similar to the expected utility obtained by acohort simulation. However, in addition to the mean

survival, statistical measures such as variance and

standard deviation of the expected utility may be de-termined from this distribution. It should be noted

that a Markov cycle tree may be evaluated as a MonteCarlo simulation.



332

FIGURE 13. Monte Carlo simulation. The figure shows the state tran-sitions of a single person until death occurs during cycle 6.

COMPARING THE DIFFERENT REPRESENTATIONS

Each representation has specific advantages anddisadvantages for particular purposes. The simula-tions (Markov cohort, Monte Carlo, and cycle tree) per-mit the analyst to specify transition probabilities andincremental utilities that vary with time. Such variation

is necessary to model certain clinical realities, such asthe increase in baseline mortality rate with age. A dis-advantage common to all simulations (cohort, cycletree, and Monte Carlo) is the necessity for repetitiveand time-consuming calculations. However, the avail-ability of specialized microcomputer software to per-form these simulations has made this much less of an

issue. The fundamental matrix solution is very fastbecause it involves only matrix algebra and providesan &dquo;exact&dquo; solution that is not sensitive to cycle time(as in the cohort simulation) or number of trials (as inthe Monte Carlo simulation). The major disadvantages

of the matrix formulation are the restriction to prob-lems with constant transition probabilities, the needto express each composite transition probability as asingle number, and the difficulty of performing matrixalgebra. The matrix manipulations required for a Mar-kov process with a large number of states may requirespecial computational resources. The Monte Carlomethod and the matrix solutions provide measures ofvariability, if these are desired. Such measures are notpossible with a cohort simulation. The features of thethree representations are summarized in table 3.

Tpmsiflon Probabilides

CONVERSION OF RATES TO PROBABILITIES

The tendency of a patient to make a transition fromone state to another is described by the rate of tran-sition. The rate describes the number of occurrencesof an event (such as death) for a given number ofpatients per unit of time and is analogous to an in-stantaneous velocity. Rates range from zero to infinity.A probability, on the other hand, describes the like-lihood that an event will occur in a given length oftime. Probabilities range from zero to 1. Rates may beconverted to probabilities if their proper relationshipis considered.

The probability of an event that occurs at a constantrate (r) in a specified time (t) is given by the equation:

This equation can be easily understood by examiningthe survival curve for a process defined by a constantrate. The equation describing this survival curve is:

where f is the fraction surviving at time t and r is theconstant transition rate. At any given time, the fraction

Table 3 9 Characteristics of Markov Approaches*

... --- -. - -

* Adapted from Beck et al.5



333

that has experienced the event is equal to 1 - f. Thus,the curve describing the probability that the event willoccur in time t is simply 1 - f, or 1 - e - rt as shown

in equation 1. The probability of transition in time tis always less than the corresponding rate per time tbecause as the cohort members die, fewer are at riskfor the transition later in the time period. When therate is small or t is short, the rate and probability are

very similar. Often, data supplied for an analysis pro-vide rates of complications. For use in a Markov anal-ysis, these rates must be converted to the correspond-ing transition probabilities by substituting the Markov-cycle length for t in equation 1.

PRECAUTIONS IN CHANGING THE CYCLE LENGTH

When changing the Markov-cycle duration from yearlyto monthly, one cannot simply divide the calculatedtransition probabilities by 12 to arrive at the appro-priate transition probabilities for the shorter cycle. Ifthe original rate is a yearly rate, then the monthlyprobability is p = 1 - e - r/12. If one has only the yearlytransition probability and not the rate, the transitionprobability can be converted to a rate by solving equa-tion 2 for r:

Then, the calculated rate is used, as above, to recal-culate the transition probability.

TIME DEPENDENCE OF PROBABILITIES

In the most general case, the transition probabilitiesin a Markov model vary with time. An obvious exampleis the probability of death, which increases as the co-hort ages. If the time horizon for the analysis is a longone, the mortality rate will increase significantly dur-ing later cycles. There are two ways of handling suchchanging probabilities. One is with a continuous func-tion, such as the Gompertz function.3 For each clockcycle, the appropriate mortality rate is calculated froma formula and converted to a transition probability.Some rates are not easily described as a simple func-

tion. One example is the actual mortality rate over alifetime, which initially is high during early childhood,falls to a minimum during late childhood, and thengradually increases during adulthood. Another ex-ample is the risk of acquiring a disease (such as Hodg-kins’ disease) that has a bimodal age distribution. Insuch cases, the necessary rates (or correspondingprobabilities) may be stored in a table, indexed by cyclenumber, and retrieved as the Markov model is eval-uated. Some computer software used for evaluatingMarkov processes provides facilities for constructingand using such tables.

DISCOUNTING: TIME DEPENDENCE OF UTILITIES

Incremental utilities, like transition probabilities, mayvary with time. One important application of this timedependence is the discounting used in cost-effec-tiveness analyses.’° This is based on the fact that costsor benefits occurring immediately are valued morehighly than those occurring in the future. The dis-counting formula is:

where Ut is the increment utility at time t, Uo is theinitial incremental utility, and d is the discount rate.10Because of the time variance, discounting cannot beused when the fundamental matrix solution is used.

A Med ExampleThe following example is a Markov implementation

of a decision analysis that has been published in detailelsewhere as an ordinary decision tree.l6,’; This anal-ysis was performed for an actual patient at the NewEngland Medical Center. The implementation of themodel is a Markov-cycle tree as used by two specificdecision analysis microcomputer programs DecisionMaker1s and SMLTREE.’9

Case history. A 42-year-old man had had a cadaverickidney transplant 18 months previously and had donewell except for an early rejection episode, which hadbeen treated successfully. He had maintained normalkidney function. While he was receiving standardtreatment with azathioprine and prednisone, how-ever, two synchronous malignant melanomas ap-peared and required wide resection. Continuation ofimmunosuppressive therapy increases the chance ofanother, possibly lethal melanoma. Cessation of thistherapy ensures that the patient’s kidney will be re-jected and will require his return to dialysis, a ther-apeutic modality he prefers to avoid.The key assumptions in the construction of this

model are:

1. If therapy is stopped, the patient will reject thekidney immediately.

2. If therapy is continued, the patient still may rejectthe kidney, but with a lower probability.

3. If the patient rejects the kidney despite contin-uation of therapy, the therapy will be stopped atthe time the rejection occurs.

4. A second transplant will not be considered.

5. Quality of life is lower on dialysis than with afunctioning transplant. Based on the original util-ity assessment from the patient, the utility of lifeon dialysis was 0.7 and that of life with a func-tioning transplant 1.0&dquo;6



334

FIGURE 14. Simple decision tree for the kid-ney transplant, melanoma case.

6. No adjustment is made to quality of life for havingrecurrent melanoma.

7. The patient’s life expectancy is reduced becauseof having had a renal transplant and a previousmelanoma. It will be further reduced if the patientgoes on dialysis or if melanoma recurs.

The simple tree modeling this problem is shown infigure 14. There are two branches of the decision node,representing CONTINUE and STOP therapy, respectively.In the case of CONTINUE) a chance node models the

occurrence of REJECT or NO REJECT. The developmentof a new melanoma is modelled by the chance nodewith the branches MELANOMA and No MELANOMA. Ter-

minal nodes represent the six possible combinationsof therapy, renal status, and occurrence of a new mel-anoma. The two combinations representing sTOY Trrrr~-APY and NO REJECT are assumed not to occur. Proba-

bilities in this model must be assigned to reflect thedifferent risks of developing a new melanoma de-pending on whether or not therapy has been contin-ued. The lowest probability is for patients whose ther-apy is stopped immediately. The highest probabilityis for those whose therapy is continued indefinitely.Because therapy will be stopped, patients who ex-perience rejection after an initial period of continuingtherapy will have an intermediate risk of melanomarecurrence.

Because it was a simple tree, the original modelrequired several simplifying assumptions. The first wasthat recurrent melanoma occurred at a fixed time in

the future (one year)16 although, in reality, it may occurat any time. The second was that transplant rejectionoccurred at a fixed time, the midpoint of the patient’slife expectancy. Therefore, the utility of continuingtherapy, then experiencing transplant rejection wasassigned the average of the utilities for transplant anddialysis. If the patient values time on dialysis differ-ently now compared with later, this is an oversimpli-fication. The third assumption was that the probability

of melanoma recurrence in this intermediate scenario

was the average of the high and low probabilities. Again,this is an oversimplification because the patient ac-tually has a high probability while on the therapy anda low probability while off it. The Markov model canaddress all of these issues.

’I’he Markov decision model is shown in figure 15and figure 16. ’I’he root of the tree in figure 15 is a

decision node with two branches representing the twochoices CONTINUE and sTOP. The Markov-cycle tree de-picted in figure 16 consists of a Markov node with onebranch for each Markov state. If we assume that the

utility of a state depends only on whether the patientis on dialysis or not and that the probability of mel-anoma depends only on whether or not the patientis receiving immunosuppressive therapy, then onlyfive states are required to represent the scenario. Theseare (from top to bottom) TRANSWEL (transplant well),TRANSMEL (transplant with melanoma), DIALWELL (di-

alysis, no melanoma), DIALMEL (dialysis and mela-noma), and DEAD. Separate states are not needed basedon treatment because it is assumed that patients inthe transplant states are on immunosuppressive ther-apy and those in the dialysis states are not.

FIGURE 15. Root of the tree representing the Markov model of thekidney transplant, melanoma case.



335

INITIAL PROBABILITIES

The first task is to assign probabilities to the branchesof the Markov node. Recall that these probabilities rep-resent the probabilities of starting in the individualstates. For the CONTINUE strategy, all patients begin inthe TRANSWEL state, so the probability of that state shouldbe 1. Similarly, for the STOP strategy, all patients beginin the oraLwELL state. We can implement these as-sumptions by assigning the probabilities of the

TRANSWEL and DIALWELL branches as variables. A bind-

ing set between the strategy branch and the Markovnode can set the appropriate variable to 1. Thus, thesame Markov-cycle tree can be used as a subtree&dquo; torepresent the prognosis for both strategies.

SUBSEQUENT PROGNOSIS

Each branch of the Markov node is attached to a

subtree that models the possible events for each Mar-kov state. The most complex is for the TRANSWEL state,shown at the top of figure 16. The first event modelledis the chance of dying from all causes (the branch Die).Die leads to a terminal node. In this case the utility isDEAD, because a patient who dies during one cycle willbegin the next cycle in the DEAD state. For patientswho do not die (the branch Survive), the next chancenode models the chance of transplant rejection (thebranches Reject and NoReject). Subsequent to each ofthese branches is a chance node modelling the riskof recurrent melanoma (the branches Recur andNoRecur and Recur2 and NoRecur2). Each of these

branches leads to a terminal node. For Recur followingReject, the appropriate state is DIALMEL, for Recur2 fol-lowing NoReject the appropriate sate is TRANSMEL. ForNoRecur following Reject and NoRecur2 followingNoReject, the appropriate states are DIALWELL and

TRANSWEL, respectively. Only the latter branch repre-sents a return to the starting state.The event tree for the TRANSMEL state is also shown

in figure 16. It is simpler than that for TRANSWEL be-cause the risk of melanoma recurrence need not be

modeled. Assignment of terminal states is similar tothat for the TRANSWEL state except that patients maynot make a transition to the TRANSWEL or DIALWELL state.

Similarly, the probability tree for the DIALWELL statemodels only the risks of death and of melanoma re-currence and that for the DIALMEL state models onlythe risk of death. The event tree for the DEAD state is

simply a terminal node assigned to the state DEAD,since no event is possible and all patients return tothe DEAD state in the subsequent cycle.

CHOICE OF CYCLE LENGTH

Before the probabilities can be assigned, the analystmust decide on the cycle length. The cycle lengthshould be short enough so that events that changeover time can be represented by changes in successive

FIGURE 16. Markov-cycle tree representing the kidney transplant,melanoma case.

cycles. For example, if the risk of allograft rejectionwere markedly different in month 3 than in month 1,then a monthly cycle should be used. Another con-sideration is that the cohort simulation is an approx-imation and will more closely approximate the &dquo;exact&dquo;

&dquo;

solution when the cycle length is short. In practice,however, it makes little difference whether the cyclelength is one year or one month, if the appropriatehalf-cycle corrections are made.20 A final considerationis evaluation time. A monthly cycle length will resultin a 12-fold increase in evaluation time over a yearlycycle length. For this example, since there is no im-portant change during a year, a yearly cycle length isused.

ASSIGNMENT OF PROBABILITIES

The next task is to assign probabilities to the eventsin each event tree. Each state has a chance node rep-resenting the occurrence of death during a given cycle.



336

DEFERRED EVALUATION

There are two ways to introduce the expression de-fining the probability of death into the Markov decisionmodel. At first glance, it may seem that the expressionabove can be placed in a binding proximal to the Mar-kov node, since it is shared by all branches. However,the values of mEXCESS are different for the individualstates. Moreover, the value of the expression for pDIEshould change for each cycle of the simulation as mASRincreases. A simple binding would be evaluated onlyonce and thus would not allow the value of the expres-sion to change. One solution is to place a binding withthe above expression on each branch of the Markovnode. However, this is cumbersome, because it re-

quires entering the expression four times (it isn’t neededfor the DEAD state), and slows evaluation, because theentire expression must be placed on the binding stackduring the evaluation of each state during each cycle.An ideal solution is provided by deferring the eval-

uation of the expression until the value of pDIE isneeded, thus ensuring that the evaluation will use thecurrent value of m.CYCLE and the appropriate localvalue of mEXCESS. This is accomplished using thePASSBIND function in Decision Maker&dquo; and SMLTREE.19This function tells the computer program to place theentire expression on the binding stack instead of eval-uating the expression first. Thus, when the value ofpDIE is needed at any time, anywhere in the tree, theexpression will be evaluated with the prevailing valuesof m.CYCLE and mEXCESS. The expression thus canbe entered in a binding proximal to the Markov node.The required binding expression is:

Each branch of the Markov node then needs a bindingfor the appropriate value of mEXCESS. The values ofthe probabilities of pReject and pRecur depend onwhether the patient is on immunosuppressive therapyand therefore must be specified for each state. Thevalues of mEXCESS, pReject, and pRecur for each stateare shown in table 4.

ASSIGNING UTILITIES

As described above, utilities in a Markov cohort sim-ulation are associated with a state, rather than withterminal nodes of the tree. Therefore, each state mustbe assigned an incremental utility that reflects thevalue of being in that state for one cycle. In DecisionMaker18 and SML TREE) 19 this is accomplished by settingthe values of three special variables. The variablem.uINCR represents the incremental utility of a statefor one cycle. The variable m.uINIT is a one-time ad-justment to the incremental utility that is made at thebeginning of the Markov simulation. It is used to im-

This probability is based on the mortality rate for eachstate, which consists of two components, the baseline

mortality rate and the excess mortality due to anycomorbid diseases. The baseline mortality rate (mASR)depends on the patient’s age, sex, and race. We canassign a different mASR for each cycle to reflect theincreasing mortality rate as patients get older. Themost convenient way to implement this is to use a

table of age-specific mortality rates and look up theappropriate value for each cycle. With a yearly cyclelength, the patient’s age at the end of cycle n is:

There are three refinements to the age used to cal-culate the cycle-specific age. First, because we assumethat transitions occur, on average, halfway through acycle, the age should be reduced by 0.5 cycle. Second,published mortality rates for patients of a given age(e.g., 50 years old) are derived from all patients betweenthat age and the next (50 and 51 years). Thus, thepublished rate for age 50 actually applies to a groupwith an average age of 50.5 years and the cycle-specificage should be reduced by an additional 0.5 year toretrieve the appropriate rate. Finally, deaths are slightlymore likely to occur among the older members of aheterogeneous cohort and toward the end of a year(when all members are older), so that the observeddeath rate applies to patients who are slightly olderthan the average. Empirically, reducing the age by anadditional 0.1 to 0.2 years corrects for these effects.

Thus, the starting age should be corrected accordingto the formula:

where cyclen is the length of the Markov cycle in years.For a detailed discussion of these corrections, the in-terested reader is referred to Sonnenberg and Wong.’OThe mortality rate may be retrieved from the table

(MTABLE) by the following expression, where StartAgeis corrected as above and m.CYCLE is the Markov-cyclenumber:

For this example, the initial value of mASR is 0.0036/year (for a 43-year-old male). The excess componentof mortality due to the patient’s coexisting diseases isadded to the baseline mortality rate to produce a totalcompound mortality rate.2 The total mortality rate maythen be used to calculate the probability of death dur-ing any cycle:



337

TO* 4 o Mortality Rates and Probabilities

plement the half-cycle correction and therefore its valueis usually set to 0.5 X m.uINCR. The variable m.uTAILis used when the Markov cohort simulation is termi-

nated before the entire cohort is in the absorbing state.Its value is added to the incremental utility for a stateat the end of the simulation. The tail utility has twouses. One is to represent the prognosis beyond thestopping point in the Markov simulation. For example,if a Markov process is used only to represent the eventstaking place during the first six months following anoperation, then m.uTAIL will represent the life ex-

pectancy of the patient beyond the first six months.The second use is to apply the half-cycle correctionto a simulation that is stopped prior to absorption ofthe cohort, even if the subsequent prognosis is of nointerest. In this case, the tail utility must be set to - 0.5X m.uINCR.

The values of these special variables are set withbindings on each branch of the Markov node. For theTRANSWEL and TRANSMEL states the bindings are:

The value of m.uINCR is 1 because there is no utilitydecrement for the TRANSPLANT states. m.u.TAIL is 0 be-cause we are planning to run the simulation until thecohort is completely absorbed.

For the DwLwELL and DIALMEL states, the bindingsare:

because the DIALYSIS states are associated with a utilityof only 0.7 relative to perfect health.16

For the DEAD state, the bindings are:., ,.- .

because no utility accrues for membership in the DEADstate. In practice, these bindings may be omitted forthe DEAD state because if their values are not specified,they will be assumed to be zero.

MARKOV-STATE BINDINGS: A REFINEMENT

Examination of figure 16 reveals that a chance nodewith branches Reject and NoReject appears in two

places in the tree. Similarly, a chance node withbranches Recur and NoRecur appears in three placesin the tree. We would like to use a common subtree

to represent these events in all portions of the tree.The problem is that several of the branches are ter-minal nodes and their utilities apply only to one spe-cific context. The solution to this problem is the useof Markov-state bindings. When Markov-state namesare used on the right side of a binding expression, theprogram substitutes the state on the right side for thevariable on the left side wherever it appears. This per-mits representing the prognoses of all four non-deadstates with a single subtree, as in figure 17. With thestate bindings shown, this Markov-cycle tree will befunctionally identical to the one in figure 16.

EVALUATION

When the Markov model is evaluated as a cohort

simulation, the expected utilities are:

CONTINUE THERAPY 7.4

STOP THERAPY 5.2

Thus, the analysis favors continuing therapy by a largemargin, more than two quality-adjusted life years. Ifthe quality adjustment is removed from the analysis(by setting quality of life on dialysis to unity), then theresults are:

Conelusion

Markov models consider a patient to be in one of afinite number of discrete states of health. All clinicallyimportant events are modelled as transitions from onestate to another. Markov processes may be repre-sented by a cohort simulation (one trial, multiple sub-jects), by a Monte Carlo simulation (many trials, a sin-gle subject for each), or by a matrix algebra solution.The matrix algebra solution requires the least com-putation, but can be used only when transition prob-abilities are constant, a special case of the Markovprocess called a Markov chain. The Markov-cycle treeis a formalism that combines the modelling power ofthe Markov process with the clarity and convenienceof a decision-tree representation. Specialized com-puter software18,19 has been developed to implementMarkov-cycle trees.The assignment of quality adjustments to incre-

mental utility permits Markov analyses to yield quality-adjusted life expectancy. Discounting may be appliedto incremental utilities in cost-effectiveness analyses.The Markov model provides a means of modelling clin-ical problems in which risk is continuous over time,in which events may occur more than once, and whenthe utility of an outcome depends on when it occurs.



338

FIGURE 17. Markov-cycle tree using subtrees and state bindings.

Most analytic problems involve at least one of theseconsiderations. Modelling such problems with con-ventional decision trees may require unrealistic or un-justified simplifying assumptions and may be com-putationally intractable. Thus, the use of Markov modelshas the potential to permit the development of deci-sion models that more faithfully represent clinicalproblems.

Referenced1. Mehrez A, Gafni A. Quality-adjusted life years, utility theory, and

healthy-years equivalents. Med Decis Making. 1989;9:142-9.2. Beck JR, Kassirer JP, Pauker SG. A convenient approximation of

life expectancy (the "DEALE"). I. Validation of the model. Am JMed. 1982;73:883-8.

3. Gompertz B. On the nature of the function expressive of the lawof human mortality. Philos Trans R Soc London. 1825;115:513-85.

4. National Center for Health Statistics. Vital Statistics of the United

States, 1988. Vol II, Mortality, Part A, Section 6. Washington,Public Health Service, 1991.

5. Beck JR, Pauker SG. The Markov process in medical prognosis.Med Decis Making. 1983;3:419-58.

6. Eckman MH, Beshansky JR, Durand-Zaleski I, Levine HJ, PaukerSG. Anticoagulation for noncardiac procedures in patients withprosthetic heart valves. Does low risk mean high cost? JAMA,1990;263:1513-21.

7. Wong JB, Sonnenberg FA, Salem D, Pauker SG. Myocardial re-vascularization for chronic stable angina: an analysis of the roleof percutaneous transluminal coronary angioplasty based ondata available in 1989. Ann Intern Med. 1990;113:852-71.

8. Hillner BE, Smith TJ, Desch CE. Efficacy and cost-effectivenessof autologous bone marrow transplantation in metastatic breastcancer. Estimates using decision analysis while awaiting clinicaltrial results. JAMA. 1992;267:2055-61.

9. Birkmeyer JD, Marrin CA, O’Connor G-T. Should patients withBjork-Shiley valves undergo prophylactic replacement? Lancet.1992;340:520-3.

10. Weinstein MC, Stason WB. Foundations of cost-effectiveness

analysis for health and medical practices. N Engl J Med.

1977;296:716-21.

11. Detsky AS, Naglie IG. A clinician’s guide to cost-effectivenessanalysis. Ann Intern Med. 1990;113:147-54.

12. Lau J, Kassirer JP, Pauker SG. DECISION MAKER 3.0: improved de-cision analysis by personal computer. Med Decis Making.1983;3:39-43.

13. Goldman L. Cardiac risks and complications of noncardiac sur-gery. Ann Intern Med. 1983;98:504-13.

14. Kemeny JB, Snell JL. Finite Markov Chains. New York: Springer-Verlag, 1976.

15. Hollenberg JP. Markov cycle trees: a new representation for com-plex Markov processes (abstr). Med Decis Making. 1984;4:529.

16. Cucharal GJ, Levey AS, Pauker SG. Kidney Failure or Cancer:Should Immunosuppression Be Continued in a Transplant Pa-tient with Malignant Melanoma? Med Decis Making. 1984;4:82-107.

17. Kassirer JP, Sonnenberg FA. Decision analysis. In Kelley WN, ed.Textbook of Internal Medicine. Philadelphia: J. B. Lippincott,1988, 1991.

18. Sonnenberg FA, Pauker SG. Decision Maker: an advanced per-sonal computer tool for clinical decision analysis. Proceedingsof the Eleventh Annual Symposium on Computer Applicationsin Medical Care, Washington, D.C.: IEEE Computer Society,1987.

19. Hollenberg J. SMLTREE: The All Purpose Decision Tree Builder.Boston: Pratt Medical Group; 1985.

20. Sonnenberg FA, Wong JB. Fine-tuning Markov models for life-expectancy calculations. Med Decis Making. 1993;13:170-2.



Date post:	17-Jan-2022
Category:	Documents
Upload:	others
View:	4 times
Download:	0 times

Markov Models in Medical Decision Making

Documents