Discrete Time Markov Chains

2

Discrete Time Markov Chain (DTMC)• Consider a system where transitions take place

at discrete time instants, and where the state of the system at time k is denoted as Xk

• Markov property states that

– Future only depends on the present…

– The pij‘s are the state transitions probabilities

• System evolution is fully specified by its state transition probability matrix [P]ij = Pij

ij

kkkkkk

P

iXjXPiXiXiXjXP

)/(},,,/( 100111

3

State Transition Probability Matrix

• P is a stochastic matrix– All entries are positive and sum of entries in a row

add up to 1

• n-step transition probability matrix– Pij

n is the probability of being in state j after n steps when starting in state i

• Limiting probabilities: the odds of being in a given state after a very large (infinite) number of transitions

4

Chapman-Kolmogorov Equations• Track state evolution based on transition

probabilities between all states– is the probability of being in state j after n

transitions when starting in state i

• This provides a simple recursion to compute higher order transition probabilities

)(nijp

nmppppp

ppp

pp

mnkj

k

mik

nkj

kik

nij

kkjikij

ijij

0 ,)()()1()(

)2(

)1(

(C-K)

5

More on C-K Equations

• In matrix form, where P is the one-step transition probability matrix

• The Chapman-Kolmogorov equations state–Pn

= Pm Pn-m, 0 m n

ijii

j

j

ppp

ppp

ppp

10

11110

00100

P

6

Limiting Distribution• From P we know that pij is the probability

of being in state j after step 1 given that we start in state i

• In general, let be the initial sate probability vector– Then the state probability vector 1 after the

first step is given by 1 = 0P

– And in general after n steps, we have n = 0Pn

• Does n converges to a limit as n ? (limiting distribution)

,, 01

00

0 def

7

A Simple 3-State Example

0 1 21-pp2

p(1-p)

p(1-p)1-p

p p2

1-p

)1(1

10

)1(1

P2

2

pppp

pp

pppp

Note that the relation n = 0Pn implies that conditioned on

starting in state i, the value of n is simply the ith row of Pn

nnn

nnn

nnn

i

nn

ijii

j

j

ppp

ppp

ppp

10

11110

00100

10

,0,1,0,0,,

8

A Simple 3-State Example

0.18560.03840.776

0.160.040.8

0.18560.03840.776

P2

0.1845760.0384640.77696

0.18560.03840.776

0.1845760.0384640.77696

P3

0.1846170.0384610.776922

0.1845760.0384640.77696

0.1846170.0384610.776922

P4

0.1846150.0384620.776923

0.1846170.0384610.776922

0.1846150.0384620.776923

P5

16.004.08.0

8.002.0

16.004.08.0

P have we2.0for p

Appears independent ofthe starting state i

9

Stationary Distribution

• A Markov chain with M states admits a stationary probability distribution = [0, 1,…, M-1] if

P = and {i=0 to M-1}i =1

• In other words

{i=0 to M-1}i pij = j , j and {i=0 to M-1}i =1

10

Stationary vs. Limiting Distributions

• Assume that the limiting distribution defined by = [0, 1,…, M-1], where j = limnPij

n > 0,

exists and is a distribution, i.e., {i=0 to M-1}i =1

• Then is also a stationary distribution, i.e., = P and no other distribution exists

• We can find the limiting distribution of a DTMC either by solving the stationary equations or by raising the matrix P to some large power

11

Summary: Computing DTMC Probabilities

• Use the stationary equations: = P and ii = 1

• Numerically compute successive values of Pn

(and stop when all rows are approximately equal)– The limit converges to a matrix with identical rows,

where each row is equal to

• Guess a solution to the recurrence relation ,1,0 ,

0

jpk

kjkj

12

Back to Our Simple 3-State Example

0 1 21-pp2

p(1-p)

p(1-p)1-p

p p2

1-p

)1(1

10

)1(1

P2

2

pppp

pp

pppp

2102

22

02

1

2100

)1()1()1(

)1()1(

P

ppppp

pp

ppp

184615.038462,0.0.776923,0

gives this2.0for

p

1

)1(

1

1

1

2

2

2

2

2

1

2

3

0

p

ppp

pp

pp

13

The Umbrella Example

0 2 11

(1-p)p

1-p p

01

10

100

P

pp

pp

p

pp

p

202

211

20

1

1

1

P

3

1

3

1

3

1

2

1

0

p

p

p

p

• For p=0.6, this gives

= [0.16667, 0.41667, 0.41667]

• Probability Pwet that professor gets wet = Prob[zero umbrella and rain]

- They are independent, so that Pwet = 0p = 0.16667 0.6 = 0.1

• Average number of umbrellas at a location: E[U] = 1 + 22 = 1.25

14

Infinite DTMC

• Handling chains with an infinite state space–We cannot anymore use matrix multiplication

• But the result that if the limiting distribution exists it is the only stationary distribution still holds– So we can still use the stationary equations,

provided they have “some structure” we can exploit

15

A Simple (Birth-Death) Example

• The (infinite) transition probability matrix and

correspondingly the stationary equations are of the form

0 1 2 3 41-r

r

s

1-r-s

r r r r

s s s s

1-r-s1-r-s 1-r-s

srs

rsrs

rsrs

rr

100

10

01

001

P

3210

4323

3212

2101

100

1

111

1

ssrrssrrssrr

sr

16

A Simple (Birth-Death) Example

• The stationary equations can be rewritten as follows

– 0 = 0(1-r)+1s 1 = (r/s)0

– 1 = 0r+1(1-r-s)+2s 2 = r/s1 = (r/s)20

– 2 = 1r+2(1-r-s)+3s 3 = r/s2 = (r/s)30

– We can then show by induction that i = (r/s)i0, i 0

• The normalization condition {i=0 to ∞}i =1 gives

0 = (1- r/s), and hence i = (r/s)i(1- r/s)

0 1 2 3 41-r

r

s

1-r-s

r r r r

s s s s

1-r-s1-r-s 1-r-s

17

A Simple (Birth-Death) Example

• Defining = r/s (a natural definition of “utilization”

consistent with Little’s Law), the stationary

probabilities are of the form

0 = (1- ) and i = i(1- )

• From those, we readily obtain E[N]=Σiii = /(1- )

0 1 2 3 41-r

r

s

1-r-s

r r r r

s s s s

1-r-s1-r-s 1-r-s

18

Putting Things on a Firmer Footing

• When does the limiting distribution exist?

• How does the limiting distribution compare to time averages (fraction of time spent in state j)?– The limiting distribution is an ensemble average

• How does the average time between successive visits to sate j compare to j (the probability of being in state j)?

19

Some Definitions & Properties• Period (of state j): GCD of integers n s.t. Pjj

n>0

– State is aperiodic if period is 1

– A Markov chain is aperiodic if all states are aperiodic

• A Markov chain is irreducible if all states communicate

– For all i and j, Pijn>0 for some n

• State j is recurrent (transient) if the probability fj of starting at j and ever

returning to j is =1 (<1)

– Number of visits to a recurrent (transient) state is infinite (finite) with probability 1

– If state j is recurrent (transient), then ΣnPjjn = (< )

– In an irreducible Markov chain, states are either all transient or all recurrent

• A transient Markov chain does not have a limiting distribution

• Positive recurrence and null recurrence: A Markov chain is positive recurrent

(null recurrent) if the mean time between returning to a state is finite (infinite)

• An ergodic Markov chain is aperiodic, irreducible and positive recurrent

20

Existence of the Limiting Distribution

An irreducible, aperiodic DTMC (finite or infinite) is in either

of the following two classes:

1. States are either all transient or all null recurrent, in which

case j = limnPijn= 0 for all j, and the stationary distribution

does NOT exists

2. States are all positive recurrent, the limiting distribution

exists and is equal to the stationary distribution, with a

positive probability for each state. In addition, j = 1/mjj,

where mjj is the average time between successive visits to

state j

21

Existence of Limiting Distribution• Existence of limiting

distribution depends on the structure of the underlying Markov chain

• Stationary state probabilities exist for ergodic Markov chains– Many practical systems of

interest give rise to ergodic Markov chains

• Stationary probabilities are of the form

0

8

7

6

5

94

321

1011

Absorbing

Periodic

Transient

Positive recurrent, irreducible, aperiodic ergodic

)(lim nij

nj P

independent of 0

22

Time Averages

• For a positive recurrent, irreducible Markov chain, with probability 1

pj = limtNj(t)/t = 1/mjj > 0

– Where Nj(t) is the number of transitions to state j by time t, and mjj is the average time between visits to state j

– pj is the time-average fraction of time is state j

• For an ergodic DTMC, with probability 1,

pj = j = 1/mjj – Time-average and stationary probabilities are equal

23

Probabilities and Rates

• For an ergodic DTMC j is the limiting probability of

being in state j as well as the long-run fraction of time the chain is in state j

– i Pij can also be interpreted as the rate of transitions from

state i to state j

• The stationary equation for state i gives us

– i = Σjj Pji but we also know i = i ΣjPij = Σji Pij

– So that we have Σjj Pji = Σji Pij or in other words, the total

rate leaving state i equals the total rate entering state i

= 1

24

Balance Equations

• For an ergodic Markov chain, the rate leaving any set of states equals the rate of entering that set of states

• Application to our earlier example

• We immediately get r1 = s2

0 1 2 3 41-r

r

s

1-r-s

r r r r

s s s s

1-r-s1-r-s 1-r-s

25

Conservation Law – Single State

j

k4

k3

k2

k8

k7

k6

k1k5 j

k4

k3

k2

k8

k7

k6

k1k5pjk1

pjk2

pjk3

pjk4

pjk5

pjk6

pjk7

pjk8

pk1j

pk2j

pk3j

pk4j

pk5j

pk6j

pk7j

pk8j

jk

kjkjk

jkj pp

26

Conservation Law – Set of States

k4

k3

k2

k8

k7

k6

k1k5

pj1k1

pj1k2

pj1k3

pj3k4

pj3k5

pj2k6 pj2k7

pj2k8

j3

j2

j1

S

S SS S j k

kjkj k

jkj pp

k4

k3

k2

k8

k7

k6

k1k5

pk1j1

pk2j1

pk3j1

pk4j3

pk5j3

pk6j2 pk7j2

pk8j2

j3

j2

j1

S

27

Back to our Simple Queue

• Applying the machinery we have just developed to the “right” set of

states, we get

where as before = p(1-q) and β=q(1-p)

• Basically, it directly identifies the right recursive expression for us

1 ,0 ,

0

1

1

01

np

n

pn

n

nn

0 1 21-pp p(1-q)

q(1-p) q(1-p)

(1-p)(1-q)+qp (1-p)(1-q)+qp

np(1-q)

q(1-p)

(1-p)(1-q)+qp

n+1p(1-q)

q(1-p)

(1-p)(1-q)+qp

p(1-q)

q(1-p)

S

28

Extending To Simple Chains• Birth-death process: One-dimensional Markov chain with transitions only

between neighbors

– Main difference is that we now allow arbitrary transition probabilities

• The balance equation for S gives us

– npn(n+1) = n+1p(n+1)n , for n 0

– You could actually derive this directly by solving the balance equations progressively

from the “left,” i.e., other terms would eventually cancel out, but after quite a bit of

work…

0 1 2p00

p01

p10

p11

n n+1

p22 pnn p(n+1)(n+1)

p21

p12 p(n-1)n pn(n+1)

p(n+1)npn(n-1) p(n+2)(n+1)

p(n+1)(n+2)

S

29

Solving Birth-Death Chains

• By induction on npn(n+1) = n+1p(n+1)n we get

• The unknown 0 can be determined from ii = 1, so that we finally obtain

1

1 )1(

)1(0

)1(

)1(1

n

i ii

ii

nn

nnnn p

p

p

p

0,1

0

1

1 )1(

)1(

1

1 )1(

)1(

1

n

p

pp

p

m

m

i ii

ii

n

i ii

ii

n

30

Time Reversibility

• Another option for computing state probabilities exists (though

not always) for aperiodic, irreducible Markov chains, namely,

• If x1, x2, x3,… exists s.t. for all i and j

Σixi = 1 and xiPij = xj Pji

• Then i = xi (the xi’s are the limiting probabilities) and the

Markov chain is called time-reversible

• To compute the i‘s we first assume time-reversibility, and

check if we can find xi’s that work. If yes, we are done. If no,

we fall back on the stationary and/or balance equations

31

Periodic Chains

• In an irreducible, positive recurrent periodic chain, the limiting distribution does not exist, but the stationary distribution does

P = and ii =1

and represents the time-average fraction of time spent in each state

• Conversely, if the stationary distribution of an irreducible periodic DTMC exists, then the chain is positive recurrent