Page 1
Chapter 2: Markov Chains
(part 1)
1 Stochastic Process
Definition: A stochastic Process {Xt, t ∈ T} is a collection of random variables where t is the
time index ( or space index as in spatial analysis).
T is the index set
The set of possible values of Xt is state space, denoted by S.
If Xt = i ∈ S, then we say that the process is in state i at time t.
X0 is usually called the initial state.
For convenience, we write X(t) for Xt.
Example Toss a coin n times. Let Xi be the outcome of the ith toss. Then {Xt : t =
0, 1, 2, · · ·n} is a stochastic Process with
T = {0, 1, 2, · · · , n} S = {H,T}.
Example (Gambler’s ruin) A gambler starts with 3$, win 1$ with probability 1/3; losses 1$
with probability 2/3. He must leave when he goes broke or he wins N$. Let Xt be the money
he has after the t′th game. Then {Xt : t = 0, 1, 2, · · ·} is a stochastic Process with
T = {0, 1, 2, · · · , } S = {0, 1, · · · , N}.
Example (Poisson Process) Xt counts the number of times that a specified event occurs
during the time period from 0 to t. Then {Xt : t ∈ [0,∞]} is a stochastic process with
T = [0,∞], S = {0, 1, 2, · · · , }.
1
Page 2
Example (Stock market index) Xt is the S&P 500 index in the t′th day of this year. Then
{Xt : t = 1, 2, · · · , 365} is a stochastic process with
T = {1, 2, · · · , 365}, S = (0,∞).
Classification of a stochastic process :
If T is a countable set — {Xt : t ∈ T} a discrete time stochastic process .
If T is a continuum — {Xt : t ∈ T} a continuous-time stochastic process .
If S is a countable set — {Xt : t ∈ T} a discrete-state stochastic process .
If S is a continuum — {Xt : t ∈ T} a real-state stochastic process .
Example Examples above:
A stochastic process is defined by T , S and the joint distribution at any finite
number of time points
2 Definition of Markov Chain
Definition: Markov Chain1 Let {Xt : t ∈ T} be a stochastic process with discrete-state
space S and discrete-time space T satisfying
P (Xn+1 = j|Xn = i,Xn−1 = in−1, · · · , X0 = i0) = P (Xn+1 = j|Xn = i)
for any set of state i0, i1, · · · , in−1, i, j in S and n ≥ 0 is called a Markov Chain (MC).
A process with the property stated in the above definition is said to have the Markovian
property, i.e. the conditional distribution of any future state Xn+1 depends on the present
state and is independent of the past states.
Example (Gambler’s ruin (continued) with N = 5)
P (Xt+1 = j|Xt = i)1Markov chain was named after A.A. Markov who developed the theoretical functions for the finite state
Markov chains in the 1900s. An interesting example from the 19th century is called the Galton-Watson processwhich attempts to answer the question of when and with what probability would a given family name become
extinct.
2
Page 3
=
1/3, j = i + 1 and 5 > i > 0;2/3, j = i− 1 and 5 > i > 0;1, if j = i = 0;1, if j = i = 5;0, otherwise
0 1 2 3 4 5 6 7 8 9 100
1
2
3
4
5
Sample path
Definition Let Pn,n+1ij = P (Xn+1 = j|Xn = i), called one-step transition probability.
It is easy to see that the one-step transition probability dependents on n and i, j.
Definition: (Stationary transition probability2 ) A Markov chain {Xn : n = 0, 1, 2, · · ·}with state space S is said to have stationary transition probability if
P (Xn+1 = j|Xn = i)
= P (Xn = j|Xn−1 = i)
= · · ·
= P (X1 = j|X0 = i)
for each i, j ∈ S, i.e. probability of the one-step transition does not change as n in creases.
A counterexample of stationary Markov Chain: for people’s promotion, time space is T :
people’s age; State space {junior, senior}, then
...
> P (X50 = senior|X49 = junior)
> ...
> P (X40 = senior|X39 = junior)
> P (X39 = senior|X38 = junior)
...2In this module, we consider Markov chains with stationary transition probability only.
3
Page 4
One-step transition probability matrix We can use a matrix to denote the transition prob-
ability.
P = (pij) =
012...
0 1 2 · · ·∥∥∥∥∥∥∥∥
p00 p01 p02 · · ·p10 p11 p12 · · ·p20 p21 p22 · · ·· · ·
∥∥∥∥∥∥∥∥
Example Toss a coin (probability of Head 0.6) independently n times. Xi is outcome of i’th
toss. Then the one-step transition matrix is
P = (pij) =
H TH ‖
‖ 0.6 0.4 ‖‖
T ‖‖ 0.6 0.4 ‖
‖
Example (Gambler’s ruin (continued) with N = 5)
P (Xt+1 = j|Xt = i)
=
1/3, j = i + 1 and 5 > i > 0;2/3, j = i− 1 and 5 > i > 0;1, if j = i = 0;1, if j = i = 5;0, otherwise
Then the one-step transition matrix is
P = (pij) =
012345
0 1 2 3 4 5∥∥∥∥∥∥∥∥∥∥∥∥
1 0 0 0 0 02/3 0 1/3 0 0 00 2/3 0 1/3 0 00 0 2/3 0 1/3 00 0 0 2/3 0 1/30 0 0 0 0 1
∥∥∥∥∥∥∥∥∥∥∥∥
Properties of P
(1) pij ≥ 0, for i, j ∈ S
(2)∞∑
j=0
pij = 1, for i ∈ S
—–summation of each row in P is 1.
Proof: It follows immediately from the definition of probability and law of total probability.
4
Page 5
State-diagram associated with transition probabilities A diagram for the Markov chain
that represents the movement and the probability for the movement.
Example Example of tossing a coin
H T 0.4
0.4
0.6
0.6
Example Draw a state diagram associated with the following transition probability matrix.
P =
1234
1 2 3 4∥∥∥∥∥∥∥∥
1.0 0 0 00 0.3 0.7 00 0.5 0.5 0
0.2 0 0.1 0.7
∥∥∥∥∥∥∥∥
1 2
3 4
1.0 0.3
0.5 0.7
0.7 0.5
0.1
0.2
5
Page 6
Example (Gambler’s ruin (continued)) Draw a state diagram associated with the following
transition probability matrix.
P =
012345
0 1 2 3 4 5∥∥∥∥∥∥∥∥∥∥∥∥
1 0 0 0 0 02/3 0 1/3 0 0 00 2/3 0 1/3 0 00 0 2/3 0 1/3 00 0 0 2/3 0 1/30 0 0 0 0 1
∥∥∥∥∥∥∥∥∥∥∥∥
States 0 and 5 are called absorbing states since once these states are reached, the process
can never leave.
6
Page 7
Example A salesman lives in Town A and is responsible for the sales consisting of Towns
A, B and C. Each week he is required to visit a different town. When he is in his own town,
it makes no difference which town he visits next so he slips a coin and if it is heads he goes to
B and if tails he goes to C. However, after spending a week away from home, he has a slight
preference for going home so when he is in either towns B or C, he has a slight preference for
going home so when he is in either towns B or C, he flips two coins. If two heads occur, then
he goes to the other town; otherwise he goes to A.
The successive towns that the salesman visits form a Markov chain with state space {A,B, C}where random variable Xn equals A, B or C according to his location during week n. The
transition probability matrix of this problem is given by
P =
∥∥∥∥∥∥
0 0.50 0.500.75 0 0.250.75 0.25 0
∥∥∥∥∥∥
Given a Markov chain, what do we want to know next?
• After t steps, what’s the probability that the MC is in state i?
• As t →∞, what’s the probability that the MC is in state i?
• Given that we’ve taken t steps, what’s the probability we’ve ever been in state i?
• What’s the expected number of steps before we reach state i for the first time?
7
Page 8
3 Chapman-Kolmogorov Equations
Denote the n-step transition probability of the Markov chain {Xn : n = 0, 1, 2, · · ·} with
state space S by
p(n)ij = P (Xm+n = j|Xm = i),
∀m, n = 0, 1, 2, · · · , ∀i, j ∈ S.
Similarly, we define the n-step transition probability matrix as
P(n) = (p(n)ij ) =
∥∥∥∥∥∥∥∥∥∥∥∥∥
p(n)00 p
(n)01 p
(n)02 · · ·
p(n)10 p
(n)11 p
(n)12 · · ·
...p(n)i0 p
(n)i1 p
(n)i2 · · ·
...
∥∥∥∥∥∥∥∥∥∥∥∥∥
.
Example Consider the salesman example where the salesman starts from Town B. The
transition probability matrix indicates that the probability of being in Town A after one step
(in one week) is 0.75. But what is the probability that he will be in Town A after two steps (or
even more steps)?
Notice that the 2-step transition probability from Town B to Town A is given by
P (X2 = A|X0 = B)
= P (X1 = A|X0 = B)P (X2 = A|X1 = A)
+P (X1 = B|X0 = B)P (X2 = A|X1 = B)
+P (X1 = C|X0 = B)P (X2 = A|X1 = C)
= 0.75× 0 + 0× 0.75 + 0.25× 0.75
= 0.1875
Consider that
[P×P]BA
=
∥∥∥∥∥∥
0 0.50 0.500.75 0 0.250.75 0.25 0
∥∥∥∥∥∥·∥∥∥∥∥∥
0 0.50 0.500.75 0 0.250.75 0.25 0
∥∥∥∥∥∥BA
8
Page 9
=
∥∥∥∥∥∥
0.75 0.125 0.1250.1875 0.4375 0.3750.1875 0.3750 0.4375
∥∥∥∥∥∥= 0.1875.
The n-step transition probabilities can be computed using the Chapman-Kolmogorov equa-
tions.
Theorem For Markov chain {Xn : n = 0, 1, 2, · · ·}, the n-step transition probability from
state i to state j satisfies the Chapman-Kolmogorov equation
p(n+m)ij
=∞∑
k=0
P (Xn = k|X0 = i)P (Xm+n = j|Xn = k)
=∞∑
k=0
p(n)ik p
(m)kj
or
P(m+n) = P(m)P(n).
P(n) = Pn.
Proof:
p(m+n)ij
= P (Xm+n = j|X0 = i)
=∞∑
k=0
P (Xm+n = j, Xn = k|X0 = i)
=∞∑
k=0
P (Xn = k|X0 = i)P (Xm+n = j|Xn = k, X0 = i)
=∞∑
k=0
P (Xn = k|X0 = i)P (Xm+n = j|Xn = k)
=∞∑
k=0
P(n)ik p
(m)kj .
2
Example (salesman example continued) The 5-step transition probability matrix of the
salesman example is given by
P (5) = P 5
9
Page 10
=
∥∥∥∥∥∥
0.2930 0.3535 0.35350.5303 0.2344 0.2350.5303 0.2354 0.2344
∥∥∥∥∥∥
we have P (X5 = A|X0 = B) = .5303.
P (∞) = limn→∞Pn
=
∥∥∥∥∥∥
0.4286 0.2857 0.28570.4286 0.2857 0.28570.4286 0.2857 0.2857
∥∥∥∥∥∥
10
Page 11
CHAPTER 2: Markov Chains
(part 2)
Basic questions
First step analysis
1 Some Markov Chain Models
1.1 An inventory Model
Let Xn denote the number consumable items somebody has at the end of a week. In week n,
ξn items will be consumed with
P (ξn = k) = ak, k = 0, 1, 2, ...
on the weekend, if Xn > s, no buying; if Xn < s, S −Xn items will be bought. Then
Xn+1 ={
Xn − ξn+1, if s < Xn ≤ SS − ξn+1, if Xn ≤ s.
1. It is clear that {Xn : n = 0, 1, 2, ...} is a MC (why?)
2. Let s = 0, S = 2 with P (ξn = 0) = 0.5, P (ξn = 1) = 0.4 and P (ξn = 2) = 0.1, what is the
transition probability matrix ?
P =
-1 0 1 2-1 0 0.1 0.4 0.50 0 0.1 0.4 0.51 0.1 0.4 0.5 02 0 0.1 0.4 0.5
(‘-’ means ‘borrowing’)
3. (unsolved problem) how frequently you need to purchase?
P∞ =
-1 0 1 2-1 0.0444 0.2333 0.4444 0.27780 0.0444 0.2333 0.4444 0.27781 0.0444 0.2333 0.4444 0.27782 0.0444 0.2333 0.4444 0.2778
we will discuss it later. Note that Pn converges as n →∞
1
Page 12
1.2 The Ehrenfest Urn Models
suppose there are 2a balls. amongst them, k balls are in container (urn) A, and 2a − k are in
container B. A balls is selected at random (all selections are equally likely) from the totally 2a
balls and moved to the other container. Let Yn be the number of balls in container A at the nth
stage. Then Yn is a Markov Chain with
S = {0, 1, 2, · · · , 2a}.
and
P (Yn+1 = i + 1|Yn = i) =2a− i
2a, if 0 ≤ i ≤ 2a
P (Yn+1 = i− 1|Yn = i) =i
2a,
P (Yn+1 = j|Yn = i) = 0, if j 6= i− 1, i + 1.
For a = 2, the transition probability matrix is
P =
0 1 2 3 40 0 1 0 0 01 1/4 0 3/4 0 02 0 1/2 0 1/2 03 0 0 3/4 0 1/44 0 0 0 1 0
We have
P2k =
0 1 2 3 40 0.125 0 0.75 0 0.1251 0 0.5 0 0.5 02 0.125 0 0.75 0 0.1253 0 0.5 0 0.5 04 0.125 0 0.75 0 0.125
and
P2k+1 =
0 1 2 3 40 0 0.5 0 0.5 01 0.125 0 0.75 0 0.1252 0 0.5 0 0.5 03 0.125 0 0.75 0 0.1254 0 0.5 0 0.5 0
Note that Pn does not converge.
1.3 A discrete queueing Markov Chain
Customers arrive for service and take their place in a waiting line1. Suppose that
P (k customer arrive in a service period )1A more advanced topic
2
Page 13
= P (ξn = k)
= ak.
where ξn has the same distribution as ξ. In each service period, only one customer is served.
Let Xn be the customers waiting for service. Then
Xn+1 = max(Xn − 1, 0) + ξn.
Based on this, the transition probability matrix is
P =
0 1 2 3 40 a0 a1 a2 a3 ...1 a0 a1 a2 a3 ...2 0 a0 a1 a2 ...3 0 0 a0 a1 ......
......
...... ...
Unsolved questions
1. what is Pn?
2. Intuitively,
(a) If Eξ > 1, then the number of customers waiting for service will increase infinitely.
(b) If Eξ < 1, what is the probability that there will be k customers waiting for service. [if
you are the only hairdresser in a barbershop, how many chairs you need to provide?]
2 First Step Analysis
2.1 Motivating Example
In the Gambler’s Ruin example (with N=4 and X0 = 3),
1. what is probability that the gambler eventually goes broke (or win N$)?
2. On average, how many games he can play before he goes broke?
3. how many times he can have 0 < k ≤ N dollars before the game ends?
Recall (for N = 4) that
P =
0 1 2 3 40 1 0 0 0 01 2/3 0 1/3 0 02 0 2/3 0 1/3 03 0 0 2/3 0 1/34 0 0 0 0 1
3
Page 14
States 0 and 4 are absorbing states.
Definition If Pii = 1, then state i is an absorbing state.
Two important variables
T = minn{n : Xn = 0 or Xn = N}—the length of the game
{XT = 0}—{the game ends in state 0}(or {XT = N}—{the game ends in state 0}.)
By this notation, we go back to the questions
* Question 1 is to calculate u3 = P (XT = 0|X0 = 3) More generally,
ui = P (XT = 0|X0 = i),
where i = 1, 2, 3
* Question 2 is to calculate v3 = E(T |X0 = 3) More generally,
vi = E(T |X0 = i), i = 1, 2, 3
* Question 3 is to calculate w3k = E(∑T−1
n=0 I(Xn = k)|X0 = 3) More generally,
wik = E(T−1∑
n=0
I(Xn = k)|X0 = i), i = 1, 2, 3
We have the following important relations
P (XT = 0|X0 = i) = P (XT = 0|X1 = i),
E(T |X1 = i) = 1 + E(T |X0 = i)
E(T−1∑
n=0
I(Xn = k)|X1 = i) =
1 + E(∑T−1
n=0 I(Xn = k)|X0 = i) if i = k
0 + E(∑T−1
n=0 I(Xn = k)|X0 = i) if i 6= k
Calculation of ui in the gambler’s example
Let ui = P (XT = 0|X0 = i), i = 0, 1, 2, 3, 4. It is easy to see from the example (N=4)
u0 = P (XT = 0|X0 = 0) = 1, u4 = P (XT = 4|X0 = 0) = 0
4
Page 15
0 1 2 3 4 5 6 7 8 9 100
1
2
3
4
5
0 1 2 3 4 5 6 7 8 9 100
1
2
3
4
5
0 1 2 3 4 5 6 7 8 9 100
1
2
3
4
5
By the important relations, we have
u3 = P (XT = 0|X0 = 3)
=4∑
i=0
P (XT = 0|X0 = 3, X1 = i)P (X1 = i|X0 = 3)
=4∑
i=0
P (XT = 0|X1 = i)P (X1 = i|X0 = 3)
=4∑
i=0
uiP (X1 = i|X0 = 3)
=4∑
i=0
uip3i
More generally,
uj =4∑
i=0
uipji, j = 1, 2, 3
i.e.
u0 = 1,
5
Page 16
u1 = p10u0 + p11u1 + p12u2 + p13u3 + p14u4
u2 = p20u0 + p21u1 + p22u2 + p23u3 + p24u4
u3 = p30u0 + p31u1 + p32u2 + p33u3 + p34u4
u4 = 0
or
u0 = 1
u1 =23u0 + 0u1 +
13u2 + 0u3 + 0u4
u2 = 0u0 +23u1 + 0u2 +
13u3 + 0u4
u3 = 0u0 + 0u1 +23u2 + 0u3 +
13u4
u4 = 0
we have
u1 =1415
, u2 =45, u3 =
815
Interestingly,
Pn →
∥∥∥∥∥∥∥∥∥∥
1 0 0 0 014/15 0 0 0 1/154/5 0 0 0 1/58/15 0 0 0 7/15
0 0 0 0 1
∥∥∥∥∥∥∥∥∥∥
Calculation of vi = E(T |X0 = i) in the gambler’s example
Note that
v0 = E(T |X0 = 0) = 0, v4 = E(T |X0 = 4) = 0
By the important relations
v3 = E(T |X0 = 3)
=4∑
i=0
E(T |X0 = 3, X1 = i)E(X1 = i|X0 = 3)
=4∑
i=0
E(T |X1 = i)× P (X1 = i|X0 = 3)
=4∑
i=0
(1 + vi)× P (X1 = i|X0 = 3)
=4∑
i=0
p3i +4∑
i=0
vip3i
6
Page 17
Generally
vj = E(T |X0 = j) = 1 +4∑
i=0
vipji, j = 1, 2, 3.
or
v0 = 0
v1 = 1 +23v0 + 0v1 +
13v2 + 0v3 + 0v4
v2 = 1 + 0v0 +23v1 + 0v2 +
13v3 + 0v4
v3 = 1 + 0v0 + 0v1 +23v2 + 0v3 +
13v4
v4 = 0
we have
v1 =115
, u2 =185
, v3 =175
Calculation of wi3 = E(∑T−1
n=0 I(Xn = 3)|X0 = i) in the gambler’s example
Following the same idea
w33 = 1 + p30w03 + p31w13 + p32w23 + p33w33 + p34w43
and for i 6= 3
wi3 = pi0w03 + pi1w13 + pi2w23 + pi3w33 + pi4w43
special cases
w03 = 0, w43 = 0
Combining these, we have
w03 = 0
w13 =23w03 + 0w13 +
13w23 + 0w33 + 0w43
w23 = 0w03 +23w13 + 0w23 +
13w33 + 0w43
w33 = 1 + 0w03 + 0w13 +23w23 + 0w33 +
13w43
w43 = 0
we have
w13 = 0.2, w23 = 0.6, w33 = 1.4.
7
Page 18
Example (Random walk with 3 states) A person walks randomly between positions 0, 1
and 2. Let Xn be the position: 0, 1, 2, at n time point. Then, the transition probability matrix
is
P =
0 1 20 1 0 01 α β γ2 a b c
Suppose he starts with state 2,
1. what is probability that he will be strapped in 0?
2. what is the average time that he first reaches 0?
3. what is the expected number of times he will visit state 1 before absorption.
Define
T = minn{n : Xn = 0}—the length of the process
{XT = 0}—{the walk strapped in state 0}.
Let
ui = P (XT = 0|X0 = i),
vi = E(T |X0 = i),
wk = P (T−1∑
n=0
I(Xn = k)|X0 = 2)
where i, k = 0, 1, 2.
We have u0 = 1 and
uj =2∑
i=0
uipji, j = 1, 2
i.e.
u0 = 1,
u1 = p10u0 + p11u1 + p12u2
u2 = p20u0 + p21u1 + p22u2
or
u0 = 1
u1 = αu0 + βu1 + γu2
u2 = au0 + bu1 + cu2
8
Page 19
we have(
1− β −γ−b 1− c
)(u1
u2
)=
(αa
).
Then
u1 =(1− c)α + γa
(1− β)(1− c)− bγ,
u2 =bα + (1− β)a
(1− β)(1− c)− bγ
The answer to question 1 is u2 = {bα + (1− β)a}/{(1− β)(1− c)− bγ}Note that v0 = 0 and
vj = 1 +2∑
i=0
vipji, j = 1, 2.
or
v0 = 0
v1 = 1 + βv1 + γv2
v2 = 1 + bv1 + cv2
we have
v1 =(1− c) + γ
(1− β)(1− c)− bγ,
v2 =b + (1− β)
(1− β)(1− c)− bγ
The answer to the second question is v2 = {b + (1− β)}/{(1− β)(1− c)− bγ}Similarly, w0 = 0,
w1 = 1 +2∑
i=0
wip1i
w2 =2∑
i=0
wip2i
i.e.
w0 = 0
w1 = 1 + βw1 + γw2
w2 = bw1 + cw2
9
Page 20
we have
w1 =1− c
(1− β)(1− c)− bγ,
w2 =b
(1− β)(1− c)− bγ
The answer to the second question is w2 = b/{(1− β)(1− c)− bγ}
10
Page 21
CHAPTER 2: Markov Chains
(part 3)
Example A Maze A white rat is put into the maze
1 2 3
4 5 6
7 8 9 shock
shock
food
In the absence of learning, one might hypothesize that the rat would move through the maze
at random, i.e. if there are k ways to leave a compartment, then the rat would choose each
of them with equal probability 1/k. Assume that the rat makes one changes to some adjacent
compartment at each unit of time. Let Xn be the compartment occupied at stage n. Suppose
that compartment 9 contains food and 3 and 7 contain electrical shocking mechanisms.
1. what is the transition probability matrix
P =
1 2 3 4 5 6 7 8 91 0 1/2 0 1/2 0 0 0 0 02 1/3 0 1/3 0 1/3 0 0 0 03 0 0 1 0 0 0 0 0 04 1/3 0 0 0 1/3 0 1/3 0 05 0 1/4 0 1/4 0 1/4 0 1/4 06 0 0 1/3 0 1/3 0 0 0 1/37 0 0 0 0 0 0 1 0 08 0 0 0 0 1/3 0 1/3 0 1/39 0 0 0 0 0 0 0 0 1
2. If the rat starts in compartment 1, what is the probability that the rate encounters food
before being shocked.
1
Page 22
Let T = min{n : Xn = 3 or Xn = 7 or Xn = 9}. Let ui = E(XT = 9|X0 = i), i,e. the
probability that the rat absorbed by food. Note that there are 3 absorbing states 3, 7 and
9. It is easy to see that
u3 = 0, u7 = 0, u9 = 1
we have, for the others i = 1, 2, 4, 5, 6, 8,
ui = pi1u1 + pi2u2 + pi3u3 + pi4u4 + pi5u5 + pi6u6 + pi7u7 + pi8u8 + pi9u9
in details
u1 = u0 + u1 + u2 + u3 + u4 + u5 + u6 + u7 + u8
u2 = u0 + u1 + u2 + u3 + u4 + u5 + u6 + u7 + u8
u4 = u0 + u1 + u2 + u3 + u4 + u5 + u6 + u7 + u8
u5 = u0 + u1 + u2 + u3 + u4 + u5 + u6 + u7 + u8
u6 = u0 + u1 + u2 + u3 + u4 + u5 + u6 + u7 + u8
u8 = u0 + u1 + u2 + u3 + u4 + u5 + u6 + u7 + u8
[PLEASE FILL IN THE COEFFICIENTS] Finally, we have
u1 = 0.1429, u2 = 0.1429, u2 = 0, u4 = 0.1429,
u5 = 0.2857, u6 = 0.4286, u7 = 0, u8 = 0.4286, u9 = 1
Example A model of Fecundity Changes in sociological patterns such as increase in age at
marriage, more remarriages after widowhood, and increased divorce rates have profound effects
on overall population growth rates. Here we attempt to model the life span of a female in a
population in order to provide a framework for analyzing the effect of social changes on average
fecundity.
For a typical woman, we may categorize her in one of the follows states
E0 : Prepuberty E1 : Single E2 : Married E3 : Divorced,
E4 : Widowed E5 : Died or emigration from the population
Suppose the transition probability matrix is
P =
E0 E1 E2 E3 E4 E5
E0 0 0.9 0 0 0 0.1E1 0 0.5 0.4 0 0 0.1E2 0 0 0.6 0.2 0.1 0.1E3 0 0 0.4 0.5 0 0.1E4 0 0 0.4 0 0.5 0.1E5 0 0 0 0 0 1
2
Page 23
We are interested in the mean duration spent in state E2, Married, since this corresponds to
the state of maximum fecundity.
Let wi2 be the mean duration in state E2 given the initial state is Ei.
From the first step analysis, we have
w22 = 1 + p21w12 + p22w22 + p23w32 + p24w42 + p22w52.
For absorbing state 5, we have
w52 = 0
If i is not a state 2 and absorbing state 5,
wi2 = pi1w12 + pi2w22 + pi3w32 + pi4w42 + pi2w52
All together, we have
w02 = 0w02 + 0.9w12 + 0w22 + 0w32 + 0.1w42 + 0.1w52
w12 = w02 + w12 + w22 + w32 + w42 + 0w52
w22 = 1 + w02 + w12 + w22 + w32 + w42 + 0w52
w32 = w02 + w12 + w22 + w32 + w42 + 0w52
w42 = w02 + w12 + w22 + w32 + w42 + 0w52
w52 = 0
The solution is
w02 = 4.5, w12 = 5, w22 = 6.25, w32 = 5, w42 = 5, w52 = 0.
Each female, on the average, spend w02 = 4.5 periods in the childbearing state E2 during her
lifetime.
Example [A process with short-term memory, e.g. the weather depends on the past m-days]
We constrain the weather to two states s: sunny, s: cloudy
- - - s c s s c s - -Xn−1 Xn Xn+1
Suppose that given the weathers in the previous two days, we can predict the weather in the
following day as
sunny (yesterday) + sunny (today) =⇒ sunny (tomorrow) with probability 0.8;
3
Page 24
cloudy (tomorrow) with probability 0.2;
cloudy (yesterday)+sunny (today) =⇒ sunny (tomorrow) with probability 0.6;
cloudy (tomorrow) with probability 0.4;
sunny (yesterday)+cloudy (today) =⇒ sunny (tomorrow) with probability 0.4;
cloudy (tomorrow) with probability 0.6;
cloudy (yesterday)+ cloudy (today) =⇒ sunny (tomorrow) with probability 0.1;
cloudy (tomorrow) with probability 0.9;
Let Xn be the weather of the n’th day. Then the state space is
S = {s, c}
We have
P (xn+1 = s|xn−1 = s, xn = s) = 0.8
P (xn+1 = s|xn−1 = c, xn = s) = 0.6
Therefore {Xt} is not a MC .
Let Yn = (Xn−1, Xn) be the weather of the day and the previous day. Then the state space
for {Yn} is
S = {(s, s), (c, s), (s, c), (c, c)}Then {Yt} is a MC with transition probability matrix
P =
(s,s) (s,c) (c,s) (c,c)(s,s) 0.8 0.2 0 0(s,c) 0 0 0.4 0.6(c,s) 0.6 0.4 0 0(c,c) 0 0 0.1 0.9
Suppose that in the past two days the weathers are (c, c). In how many days on average can
we expect to have two successive sunny days?
To solve the question, we define a new MC by recording the weathers in successive days.
If there are two successive sunny days, we stop. Denote the process by {Zn}. The transition
probability matrix is then
P =
(s,s) (s,c) (c,s) (c,c)(s,s) 1 0 0 0(s,c) 0 0 0.4 0.6(c,s) 0.6 0.4 0 0(c,c) 0 0 0.1 0.9
4
Page 25
Denote the states (s,s), (s, c), (c,s) and (c,c) by 1,2,3 and 4 respectively. Let vi denote the
expected number of days we first have two sunny days?
By the first step analysis, we have the following equations
v1 = 0v2 = 1 + p21v1 + p22v2 + p23v3 + p24v4
v3 = 1 + p31v1 + p32v2 + p33v3 + p34v4
v4 = 1 + p41v1 + p42v2 + p43v3 + p44v4
i,e.
v1 = 0v2 = 1 + 0.4v3 + 0.6v4
v3 = 1 + 0.6v1 + 0.4v2
v4 = 1 + 0.1v3 + 0.9v4
we have
v2 = 13.3333, v3 = 6.3333, v4 = 16.3333
5
Page 26
CHAPTER 2: Markov Chains
(part 4)
September 21, 2005
Basic questions
1. Regular MC; 2 How to calculate the limit distribution; 3. How to interpret the distribution.
1 Regular transition probability matrix
Question:
1. limn→∞P(n)ij =?
2. If the above limit exists, does it dependent on i?
Definition A MC (with transition probability matrix P) is called regular if there exists
an integer k > 0 such that all the elements of Pk are strictly positive. We call the corresponding
transition probability matrix the regular transition probability matrix .
Example
P =
∥∥∥∥∥∥∥∥
0.8 0.2 0 00.8 0 0.2 00.8 0 0 0.20.8 0 0 0.2
∥∥∥∥∥∥∥∥
We have
P2 =
∥∥∥∥∥∥∥∥
0.8 0.16 0.04 00.8 0.16 0 0.040.8 0.16 0 0.040.8 0.16 0 0.04
∥∥∥∥∥∥∥∥
P3 =
∥∥∥∥∥∥∥∥
0.8 0.16 0.032 0.0080.8 0.16 0.032 0.0080.8 0.16 0.032 0.0080.8 0.16 0.032 0.008
∥∥∥∥∥∥∥∥
1
Page 27
Thus, P is a regular transition probability matrix. We further have
Pn →
∥∥∥∥∥∥∥∥
0.8 0.16 0.032 0.0080.8 0.16 0.032 0.0080.8 0.16 0.032 0.0080.8 0.16 0.032 0.008
∥∥∥∥∥∥∥∥
Example (a NON-regular MC )
P =
∥∥∥∥∥∥
1 0 00.2 0.3 0.50 0 1
∥∥∥∥∥∥
we have
P2 =
∥∥∥∥∥∥
1 0 00.26 0.09 0.650 0 1
∥∥∥∥∥∥,
P3 =
∥∥∥∥∥∥
1 0 00.278 0.027 0.695
0 0 1
∥∥∥∥∥∥,
P100 =
∥∥∥∥∥∥
1 0 00.2857 0.000 0.7134
0 0 1
∥∥∥∥∥∥,
...
P∞ =
∥∥∥∥∥∥
1 0 00.2857 0 0.7134
0 0 1
∥∥∥∥∥∥.
Some methods to check whether a MC is regular.
1. (sufficient and necessary) if a MC is regular, then there is a k such that all elements in
the Pn is greater than 0 (for all n > k). [instead of checking Pk: k = 1,2,3, we may check
P2k : k = 1, 2,.]
2. (sufficient) for every pair of states i, j there is a path k1, k2 · · · kr such that Pik1Pk1k2 · · ·Pkrj >
0 and there is at least one state i for which Pii > 0
3. (necessary) if a MC has absorbing states, then it is not a regular MC.
For a regular MC, we have the following observation:
1. the limit of
limn→∞Pn
exists.
2
Page 28
2. the starting state does not affect the long-run behaviors of the MC
Theorem Suppose that P is a regular transition probability matrix with states S =
{0, 1, 2, · · · , N}. Then
(1) 1 limn→∞ P(n)ij exists;
(2) 1 The limit does not depend on starting state i, denote by πj = limn→∞ P(n)ij ;
(3)∑N
k=0 πk = 1. (we call (π0, π1, · · · , πN ) a limiting distribution.)
(4) the limits π = (π0, π1, · · · , πN )T are the solution of the
πj =∑N
k=0 πkPkj , j = 0, 1, 2, · · · , N,
∑Nk=0 πk = 1.
(1.1)
or in matrix
πP = π,
N∑
k=0
πk = 1.
(5) the limiting distribution is unique
[Proof of (3): Note thatN∑
j=1
P (Xn = j|X0 = i) = 1;
Let n tend to ∞, we have
limn→∞P (Xn = j|X0 = i) = πj
Thus, (3) follows.
proof of (4): By the Chapman-Kolmogorov equations, we have
P(n)ij =
N∑
k=0
P(n−1)ik Pkj , j = 0, 1, · · · , N
let n →∞. It follows that
πj =N∑
k=0
πkPkj
1the proofs are beyond the scope of the module
3
Page 29
proof of (5): We need to show that if x0, x1, · · · , xN is a solution of (1.1), i.e.
xj =N∑
k=0
xkPkj , (1.2)
j = 0, 1, 2, · · · , N,N∑
k=0
xk = 1, (1.3)
then
xj = πj . (1.4)
—– Multiplying (1.2) by Pjl, we have
xjPjl =N∑
k=0
πkPkjPjl, j = 1, 2, · · ·N
The sum of the above equations isN∑
j=0
xjPjl =N∑
j=0
N∑
k=0
πkPkjPjl
=N∑
k=0
πk
N∑
j=0
PkjPjl
=N∑
k=0
πkP(2)kj (why?)
Note that by (1.2), the LHS above is
N∑
j=0
xjPjl = xj ;
Thus
xj =N∑
k=0
πkP(2)jl , j = 1, 2, · · ·N
By induction
xj =N∑
k=0
xkP(n)kj for all n
Let n →∞, we have
xj =N∑
k=0
xkπj = πj
N∑
k=0
xk = πj for all j
Thus,
xj = πj for all j
Interpretation of πj : j = 0, 1, 2, · · · , N .
4
Page 30
1. πj is the (unconditional) probability that the MC is in state j;
2. the limiting distribution
πj = limn→∞P (Xn = j|X0 = i)
(note that it is independent of the initial state)
3. Long run mean fraction of visits to state j.
[Define indicator function
I{Xk = j} ={
1, if Xk = j0, if Xk 6= j.
By the definition, we have
(a) E(I{Xk = j}|X0 = j) = P (Xk = j|X0 = j);
(b) I{Xk = j} = 1 means the MC is visiting state j at time k.
Then the fraction of visit is 1
E
(1m
m−1∑
k=0
I{Xk = j}|X0 = i
)
=1m
m−1∑
k=0
E (I{Xk = j}|X0 = i)
=1m
m−1∑
k=0
E (I{Xk = j}|X0 = i)
=1m
m−1∑
k=0
P (Xk = j|X0 = i)
→ πj .
]
4. the limit of Pn:0 1 2 ... N
0 π0 π1 π2 ... πN
1 π0 π1 π2 ... πN
P∞ = 2 π0 π1 π2 ... πN...N π0 π1 π2 ... πN
1A mathematic fact is used here: if an → a, then
n−1n∑
k=1
an → a.
5
Page 31
Calculation of the limiting distribution
1. (if you have software) approximate it by definition: calculate Pn for sufficiently large n.
If all the column are the same, the πi is the value in column i.
2. solution of the equations (note the there are N+2 equations with N+1 parameters). Delete
any one in the first N+1 equations. Then solve the equation.
Example For one-step transition probability matrix
1.
0 1 2 30 0.1 0.5 0 0.4
P= 1 0 0 1 02 0 0 0 13 1 0 0 0
[Let (π0, π1, π2, π3) be the limiting distribution. Then
0.1π0 + π3 = π0
0.5π0 = π1
π1 = π2
0.4π0 + π2 = π3
π0 + π1 + π2 + π3 = 1
The solutions are
(π0, π1, π2, π3) = (0.3448, 0.1724, 0.1724, 0.3103)
]
2. IF the matrix is not regular, then we cannot use the above method. The Ehrenfest Urn
Model with a = 1
P =
0 1 20 0 1 01 1/2 0 1/22 0 1 0
Example [A process with short-term memory, e.g. the weather depends on the past m-days]
We constrain the weather to two states s: sunny, s: cloudy
- - - s c s s c s - -Xn−1 Xn Xn+1
Let Yn = (Xn−1, Xn) be the weather of the day and the previous day. Then the state space
for {Yn} is
S = {(s, s), (c, s), (s, c), (c, c)}
6
Page 32
Then {Yt} is a MC with transition probability matrix
P =
(s,s) (s,c) (c,s) (c,c)(s,s) 0.8 0.2 0 0(s,c) 0 0 0.4 0.6(c,s) 0.6 0.4 0 0(c,c) 0 0 0.1 0.9
To calculate the limiting distribution
π = (π0, π1, π2, π3).
Approach 1: (by definition)
P 10 =
∥∥∥∥∥∥∥∥
0.3228 0.1029 0.0893 0.48500.2679 0.0898 0.0911 0.55120.3087 0.0995 0.0898 0.50190.2425 0.0837 0.0919 0.5820
∥∥∥∥∥∥∥∥
P 100 =
∥∥∥∥∥∥∥∥
0.2727 0.0909 0.0909 0.54550.2727 0.0909 0.0909 0.54550.2727 0.0909 0.0909 0.54550.2727 0.0909 0.0909 0.5455
∥∥∥∥∥∥∥∥
we have
π = (0.2727, 0.0909, 0.0909, 0.5455)T
Approach 2 (by theorem 4.1):
0.8π0 + 0.6π2 = π0
0.2π0 + 0.4π2 = π1
0.4π1 + 0.1π3 = π2
0.6π1 + 0.9π3 = π3
π0 + π1 + π2 + π3 = 1
Deleting one of the first 4 equations and solving the rest, we have
π0 = 3/11, π1 = 1/11, π2 = 1/11, π3 = 6/11.
Moreover,
P (s) = P ((s, s)) + P ((c, s)) = 4/11;
P (c) = P ((s, c)) + P ((c, c)) = 6/11;
7
Page 33
CHAPTER 2: Markov Chains
(part 5)
September 26, 2005
1 Classification of States
Definition For a Markov Chain {Xn : n = 0, 1, 2, , · · ·} with transition probability matrix P,
state j is said to be accessible from state i, denoted by i → j, if P(n)ij > 0 for some n ≥ 0.
Furthermore, two states i and j which are accessible to each other are said to communicate
and we write i ↔ j.
Example Consider the following transition probability matrices
P =
∥∥∥∥∥∥
0 0.50 0.500.75 0 0.250.75 0.25 0
∥∥∥∥∥∥
1 ↔ 21 ↔ 32 ↔ 3
P =
∥∥∥∥∥∥∥∥
1.0 0 0 00 0.3 0.7 00 0.5 0.5 0
0.2 0 0.1 0.7
∥∥∥∥∥∥∥∥
2 ↔ 34 → 1, 4 → 3
P =
∥∥∥∥∥∥
0.55 0.36 0.090.40 0.60 00.75 0 0.25
∥∥∥∥∥∥
1 ↔ 21 ↔ 32 ↔ 3
P =
∥∥∥∥∥∥
0.3 0.4 0.31.0 0.0 0.00.0 0.3 0.7
∥∥∥∥∥∥
1 ↔ 21 ↔ 32 ↔ 3
Theorem Communication is an equivalence relation, i.e.
(1). i ↔ i (reflexivity)
(2). i ↔ j ⇒ j ↔ i (symmetry)
(3). i ↔ j and j ↔ k ⇒ i ↔ k (transitivity)
1
Page 34
Proof: (1) and (2) are straightforward. Next, we show (3). By the conditions, there exists
m > 0 and n > 0 such that
P(m)ij > 0, P
(n)jk > 0
By the Chapman-Kolmogorov equations,
p(m+n)ik =
∞∑
l=0
p(m)il p
(n)lk ≥ p
(m)ij p
(n)jk > 0
i.e. i → k. Similarly, k → i and hence i and k communicate.
Thus, we can write
i ←→ j, j ←→ k and i ←→ k
as
i ←→ j ←→ k
Definition A Markov Chain is irreducible if all the states communicate with one another.
Example which of the following Markov Chain is irreducible?
P =
∥∥∥∥∥∥
0 0.50 0.500.75 0 0.250.75 0.25 0
∥∥∥∥∥∥
P =
∥∥∥∥∥∥∥∥
1.0 0 0 00 0.3 0.7 00 0.5 0.5 0
0.2 0 0.1 0.7
∥∥∥∥∥∥∥∥
P =
∥∥∥∥∥∥∥∥
0.5 0 0 0.50 0.3 0.7 00 0.1 0.9 0
0.6 0 0 0.4
∥∥∥∥∥∥∥∥
A regular Markov Chain is a irreducible Markov Chain.
2 Periodicity of a Markov Chain
Definition For a state i, let d(i) be the greatest common divisor of {n ≥ 1 : P(n)ii > 0}. If
{n ≥ 1 : P(n)ii > 0} = ∅ then define d(i) = 0. We call d(i) the period of state i.
2
Page 35
If d(i) = 1, then state i is aperiodic.
Example
P =
∥∥∥∥∥∥
0 1 00 0 11 0 0
∥∥∥∥∥∥
It is easy to check that P00 = 0, P(2)00 = 0, P
(3)00 = 1, P
(4)00 = 0,, P
(5)00 = 0, P
(6)00 = 1, ... and
{n ≥ 1 : P(n)ii > 0} = {3, 6, 9, · · ·}
The common divisor is d(0) = 3. Thus the period for state 0 is d(0) = 3.
what about state 1 and 2?
3
Page 36
Example
P =
0 1 2 30 0 1 0 01 0 0 1 02 0 0 0 13 0.5 0 0.5 0
we have
d(0) = 2 because {n ≥ 1 : P(n)00 > 0} = {4, 6, 8, ...}
d(3) = 2 because {n ≥ 1 : P(n)33 > 0} = {2, 4, 6, 8, ...};
what about state 1 and 2?
Example
P =
0 1 2 30 0 0 0.3 0.71 0 0 0.4 0.62 0.5 0.5 0 03 0.4 0.6 0 0
we have d(i) = 2, i = 0, 1, 2, 3.
Theorem1
1. i ↔ j =⇒ d(i) = d(j).
2. P(Nd(i))ii > 0 =⇒ P
(nd(i))ii > 0, for n ≥ N
3. P(m)ji > 0 =⇒ P
(m+nd(i))ji > 0, for n ≥ N
1The proof is beyond the scope of the module
4
Page 37
3 Recurrent and Transient States
For any state i, define the probability that starting from state i, the first return to i is at the
nth transition
f(n)ii = P (X1 6= i, X2 6= i, · · · , Xn−1 6= i,Xn = i|X0 = i).
[We define f(0)ii = 0]
Theorem
P(n)ii = P (Xn = i|X0 = i) =
n∑
k=0
f(k)ii P
(n−k)ii .
Proof: Define
Ck = (first return to i is at kth transition, 0 < k ≤ n)
We have
P (Xn = i|X0 = i)
= P (n⋃
k=1
Ck, Xn = i|X0 = i)
=n∑
k=1
P (Ck, Xn = i|X0 = i)
=n∑
k=1
P (Ck|X0 = i)P (Xn = i|Xk = i)
=n∑
k=1
f(k)ii pn−k
ii
=n∑
k=0
f(k)ii pn−k
ii .
The proof is now complete.
Let
fii =∞∑
n=0
f(n)ii = lim
N→∞
N∑
n=0
f(n)ii
Then, fii is the probability that starting from state i, a Markov Chain returns to the state i (at
some time).
Definition If fii = 1, then i is recurrent; if fii < 1, then i is transient.
5
Page 38
If i is a transient state, let Ni be the number of times that the Markov Chain visits state i.
Theorem If i is transient, then
E(Ni|X0 = i) =fii
1− fii.
[Proof: 2 We have
fii = P (Ni ≥ 1|X0 = i)
=∞∑
j=1
P (X1 6= i, · · · , Xj−1 6= i,Xj = i|X0 = i).
Let’s calculate
fii = P (Ni ≥ 2|X0 = i)
=∞∑
j=1
P (Ni ≥ 2|X0 = i, X1 6= i, · · · , Xj−1 6= i, Xj = i)
×P (X1 6= i, · · · , Xj−1 6= i,Xj = i|X0 = i)
=∞∑
j=1
P (Ni ≥ 2|Xj = i)
×P (X1 6= i, · · · , Xj−1 6= i,Xj = i|X0 = i)
=∞∑
j=1
P (Ni ≥ 1|X0 = i)
×P (X1 6= i, · · · , Xj−1 6= i,Xj = i|X0 = i)
= P (Ni ≥ 1|X0 = i)∞∑
j=1
P (X1 6= i, · · · , Xj−1 6= i,Xj = i|X0 = i)
= P (Ni ≥ 1|X0 = i)P (Ni ≥ 1|X0 = i) = f2ii.
In general
P (Ni ≥ k|X0 = i) = fkii.
Therefore
E(Ni|X0 = i) =∞∑
k=0
E(Ni ≥ k|X0 = i)
= fii + f2ii + · · ·
=fii
1− fii.
2the proof might be difficult for you
6
Page 39
Example For a Markov Chain with transition probability matrix
P =
∥∥∥∥∥∥∥∥
1.0 0 0 00 0.3 0.7 00 0.6 0.4 0
0.2 0 0.1 0.7
∥∥∥∥∥∥∥∥
we have
1. f11 = 1 because
f(1)11 = 1, f
(2)11 = 0, f
(3)11 = 0, · · ·
2. f22 = 1 because
f(1)22 = 0.3,
f(2)22 = 0.7× 0.6,
f(3)22 = 0.7× 0.4× 0.6,
f(4)22 = 0.7× 0.42 × 0.6,
...
f22 =∞∑
k=1
f(k)22 = 0.3
+0.7× (1 + 0.4 + 0.42 + ...)× 0.6
= 1.
3. f33 = 1 (why?)
4. f44 < 1 because
f(1)44 = 0.7, f
(2)44 = 0, f
(3)44 = 0, · · ·
The expected number of visits are
E(N1|X0 = 1) = ∞E(N2|X0 = 2) = ∞E(N3|X0 = 3) = ∞E(N4|X0 = 4) < ∞.
7
Page 40
Theorem State i is recurrent if and only if
∞∑
n=1
p(n)ii = ∞,
State i is transient if and only if
∞∑
n=1
p(n)ii < ∞.
[Proof: We write Ni in terms of indicator random variables
Ni =∞∑
n=1
I(Xn = 1)
where
I(Xn = i) ={
1, if Xn = i,0, if Xn 6= i
We have
E(Ni|X0 = i)
= E
[ ∞∑
n=0
I(Xn = i)|X0 = i
]
=∞∑
n=0
E[I(Xn = i)|X0 = i]
=∞∑
n=0
P (Xn = i|X0 = i)
=∞∑
n=0
p(n)ii .
By the previous theorem, the theorem follows. ]
Theorem If state i is recurrent (transient) and state i communicates with state j, then state
j is recurrent (transient)
[proof:
i ↔ j
=⇒ there exist m,n ≥ 0 such that
P(m)ij > 0 and P
(n)ji > 0
8
Page 41
Let v > 0. It is easy to see that (why)
P(m+n+v)jj ≥ P
(m)ij P
(v)ii P
(n)ij
Thus∞∑
v=0
P(m+n+v)jj ≥
∞∑
v=0
P(m)ij P
(v)ii P
(n)ij
= P(m)ij P
(n)ij
∞∑
v=0
P(v)ii = ∞
]
Example (Continued)
P =
∥∥∥∥∥∥∥∥
1.0 0 0 00 0.3 0.7 00 0.5 0.5 0
0.2 0 0.1 0.7
∥∥∥∥∥∥∥∥
Because 2 is recurrent, and 2 ←→ 3 therefore, 3 is recurrent.
Example Consider a Markov Chain with state space S = {a, b, c, d, e, f} and one-step tran-
sition probability matrix given by
P =
∥∥∥∥∥∥∥∥∥∥∥∥
0.3 0.2 0.2 0.2 0.1 00 0.5 0 0 0 0.50 0 0.4 0.6 00 0 0.3 0.2 0.5 00 0 1 0 0 00 0.8 0 0 0 0.2
∥∥∥∥∥∥∥∥∥∥∥∥
9
Page 42
CHAPTER 2: Markov Chains
(part 6)
October 3, 2005
1 The Basic Limit theorem of MC
Consider a recurrent state i. recall that
f(n)ii = P (X1 6= i,X2 6= i, · · · , Xn−1 6= i,Xn = i|X0 = i).
Define the first return time
Ri = min{n ≥ 1, Xn = i}
Then
f(n)ii = P (Ri = n|X0 = i) for n = 1, 2, · · ·
Since i is recurrent, fii =∑∞
n=1 f(n)ii = 1, i.e. f
(n)ii , n = 1, 2, · · · are mass probability function.
The mean duration between visits is
mi = E(Ri|X0 = i) =∞∑
n=1
nf(n)ii
Theorem1 For a recurrent irreducible aperiodic MC.
1.
limn→∞P
(n)ii =
1∑∞
n=1 nf(n)ii
=1
mi.
2.
limn→∞P
(n)ji = lim
n→∞P(n)ii for all states j.
3.
limn→∞P
(n)ii = πi = 1/mi.
1the proof is beyond the scope of the module
1
Page 43
Example Consider a MC whose transition probability matrix is given by
P =
0 1 2 30 0 1 0 01 0.1 0.5 0.2 0.22 0.2 0.2 0.5 0.13 0.3 0.3 0.4 0
Find the mean time for the MC to go from state 0 to state 0
[solution: Let π = (π0, π1, π2, π3) be the limit distribution. Then{
π = πPπ1 + π2 + π3 + π4 = 1
The solution is
π0 = 0.1383, π1 = 0.4609, π2 = 0.2806, π3 = 0.1202
Thus, the mean time is
m0 = 1/π0 = 7.2319
2 A brief Review of Markov chain
1. Definition
(a) Definition: Markov Chain, (stationary Markov Chain)
Example[Weather forecasting] Let Xn be the weather, sunny (s) and cloudy (c), in
the nth day. Whether {Xn, n > 0} is a MC depending the rule that the weather
changes. For example
i. If the weather in any day in determined by the weather in the previous day, then
Xn is a MC.
ii. If the weather in any day in determined by the weather in the past 2 days, then
Xn is not a MC.
Example[Gambler’s Ruin] A fair coin is tossed repeatedly. At each toss a gambler
wins 1$ if a head shows and loses 1$ if tails. He continues playing until his capital
reaches m or he goes broke.
(b) One-step transition probability matrix. For a MC, we are interested in
pij = P (Xn+1 = j|Xn = i)
2
Page 44
We can write this in form of a matrix.
Example[Weather forecasting (continued)] If we know
P (s|c) = 0.2, P (c|c) = 0.8
P (s|s) = 0.6, P (c|s) = 0.4.
Then the transition probability matrix is
s cP = s 0.6 0.4
c 0.2 0.8
(c) n-step transition probability matrix and Chapman-Kolmogorov equations. For a MC,
we are now interested in
p(n)ij = P (Xn = j|X0 = i)
Example[Weather forecasting (continued)]
s cP(3) = P3 = s 0.376 0.624
c 0.312 0.688
n-step transition probability matrix if “sufficient” to analyze short-term properties of a
MC
2. Long term behavior for a MC: (a) with absorbing states and (b) without absorbing state.
(a) Long term behavior for a MC with absorbing states: first-step analysis. Derive the
set of equations for any given transition probability matrix .
Example[Gambler’s Ruin (continued)] Find pi, the probability that he goes broke if
his initial capital is i$. How many games he can play before the game is over? How
may times he can have j$ before the game is over?
(b) Long term behavior for a MC without absorbing states:limiting distribution (or
stationary distribution). How to calculate the limiting distribution and how to
explain the limiting distribution.
Example[Weather forecasting (continued)] what is the proportion of sunny days?
3
Page 45
3. Classification of states in a MC: recurrent and transient.
fii =∞∑
n=1
f(n)ii .
state i is recurrent ⇐⇒ fii = 1
⇐⇒∞∑
n=1
P(n)ii = ∞
state i is transient ⇐⇒ fii < 1
⇐⇒∞∑
n=1
P(n)ii < ∞
4. The basic limit theorem of MC: the mean duration between visits to a recurrent aperiodic
state i is
mi = E(Ri|X0 = i) = 1/ limn→∞Pn
ii .
3 More examples
Example 3.1 (A discrete queueing Markov Chain) Suppose that
P (k customer arrive in a service period )
= P (ξn = k)
= ak.
In each service period, only one customer is served. Let Xn be the customers waiting for service.
Then
Xn+1 = (Xn − 1)+ + ξn.
Based on this, the transition probability matrix is
P =
0 1 2 3 ...0 a0 a1 a2 a3 ...1 a0 a1 a2 a3 ...2 0 a0 a1 a2 ...3 0 0 a0 a1 ...... 0 0 0 0 ...
1. If Eξ > 1, then the number of customers waiting for service will increase infinitely.
2. If Eξ < 1, what is the probability that there will be k customers waiting for service. [if
you are the only hairdresser in a barbershop, how many chairs you need to provide?]
4
Page 46
Example 3.2 (Independent random variable ) suppose ξ is a random variable such that
P (ξ = i) = ai ≥ 0, i = 0, 1, · · · ,
ξ0, ξ1, · · · , ξn · · · are independent random samples from ξ. Define
{Xn = ξn : n = 0, 1, 2, · · ·}
It is a MC because
Pij = P (Xn+1 = j|X1 = i1, · · · , Xn−1 = in−1, Xn = i)
= P (ξn+1 = j|ξ1 = i1, · · · , ξn−1 = in−1, ξn = i)
= P (ξn+1 = j)
The transition probability matrix is
P =
0 1 2 3 ...0 a0 a1 a2 a3 ...1 a0 a1 a2 a3 ...2 a0 a1 a2 a3 ...3 a0 a1 a2 a3 ......
......
...... ...
Example 3.3 (Successive Maxima) suppose ξ is a random variable such that
P (ξ = i) = ai ≥ 0, i = 0, 1, · · · ,
ξ0, ξ1, · · · , ξn · · · are independent random samples from ξ. Define
Xn = max{ξ1, ξ2, · · · , ξn}
Xn = max{Xn−1, ξn}.
Then {Xn} is a MC with
pij = P (Xn+1 = j|X1 = i1, X2 = i2, · · · , Xn−1 = in−1, Xn = i)
= P (max{Xn, ξn+1} = j|X1 = i1, X = i2, · · · , Xn−1 = in−1, Xn = i)
= P (max{Xn, ξn+1} = j|Xn = i)
= P (max{i, ξn+1} = j) = P (max{i, ξ} = j).
transition probability matrix
5
Page 47
P =
0 1 2 3 ...0 a0 a1 a2 a3 ...1 0 a0 + a1 a2 a3 ...2 0 0 a0 + a1 + a2 a3 ...3 0 0 0 a0 + a1 + a2 + a3 ......
......
...... ...
Example 3.4 (partial sums)
X0 = 0, Xn = ξ1 + · · ·+ ξn, n = 1, 2, · · ·
Then {Xn} is a MC with
P =
0 1 2 3 ...0 a0 a1 a2 a3 ...1 0 a0 a1 a2 ...2 0 0 a0 a1 ...3 0 0 0 a0 ......
......
...... ...
Example 3.5 (successive trials) Consider success trials with outcomes (F: failure) and (S:
Success)
- - - F S S S F S - -Xn−1 Xn Xn+1
Let Xn be the maximum success trials before trail n (including trial n). In the above example,
Xn−1 = 1, Xn = 2, Xn+1 = 3, Xn+2 = 0.
Suppose Pr(S) = α and P (F ) = β (with α + β = 1 ).
{Xn, n = 1, 2, · · ·} is a MC with
S = {0, 1, 2, · · ·}
and
Xn+1 ={
Xn + 1, if the n+1 trial is a success0, if the n+1 trial is a failure
with transition probability matrix
P =
∥∥∥∥∥∥∥∥∥
β α 0 0 · · ·β 0 α 0 · · ·β 0 0 α · · ·...
......
......
∥∥∥∥∥∥∥∥∥
6
Page 48
Example 3.6 (current age in a renewal process) The lifetime of a light bulb is ξ with
P (ξ = k) = ak > 0, k = 1, 2, 3,
Let each bulb be replaced by a new one when it burns out. Let Xn be the age of the bulb
at time n. (set X0 = 0). Then {Xn : n = 0, 1, 2, · · ·} is a MC because
Xn+1 ={
Xn + 1, if the bulb works at n+10, if the bulb burns out at time n+1
We have
P (Xn+1 = 0|Xn = i) = P (ξ ≤ i|ξ > i) = ak+1/(ak+1 + ak+2 + · · ·).
Example 3.7 (Gambler’s ruin) Player A has i$; Player B (e.g. a Casino) has N − i$. A
wins 1$ with probability p and losses 1$ with probability q (p + q = 1). The game is over is
either A or B goes broke.
Let Xn be A’s fortune after game n. Then {Xn : n = 0, 1, 2, · · ·} is a MC with state space
S = {0, 1, 2, · · · , N}.
the transition probability matrix
P =
∥∥∥∥∥∥∥∥∥∥∥
1 0 0 · · · 0 0q 0 p · · · 0 00 q 0 · · · 0 0...
......
......
...0 0 0 · · · 0 1
∥∥∥∥∥∥∥∥∥∥∥
Define
T = minn{n : Xn = 0 or Xn = N},µi = P (Xn = 0|X0 = i),
wi = P (Xn = N |X0 = i).
Questions:
1. what is the prob that A will lose all his money?
2. how many games they can expect to play?
Let Xn be the money player A has after the n′th game. Define
T = min{n ≥ 0 : Xn = 0 or Xn = N}
then the questions is equivalent to calculate
7
Page 49
1. uk = P (XT = 0|X0 = k)
By the first step analysis, we have
uk = puk+1 + quk−1, for k = 1, 2, , N − 1,
u0 = 1,
uN = 0.
Note that p + q = 1. Thus
uk = puk+1 + quk−1
=⇒uk(p + q) = puk+1 + quk−1
=⇒0 = p(uk+1 − uk)− q(uk − uk−1)
Let xk = uk − uk−1
0 = p(uk+1 − uk)− q(uk − uk−1)
=⇒0 = pxk+1 − qxk
or
xk = (q/p)xk−1
=⇒xk = (q/p)k−1x1.
Note also that
uk − u0 = (uk − uk−1) + ... + (u1 − u0) = [1 + (q/p) + ... + (q/p)(k−1)]x1
Especially,
uN − u0 = (uk − uk−1) + ... + (u1 − u0) = [1 + (q/p) + ... + (q/p)(N−1)]x1
we have
x1 =−1
1 + (q/p) + ... + (q/p)N−1
8
Page 50
Thus
uk = 1− 1 + (q/p) + ... + (q/p)(k−1)
1 + (q/p) + ... + (q/p)N−1
Note that
1 + (q/p) + ... + (q/p)(k−1) =
{k if p = q = 1/21−(q/p)k
1−(q/p) if q 6= q
whence
uk =
{1− k/N if p = q = 1/21− 1−(q/p)k
1−(q/p)N if p 6= q
The prob that A wins all the N$ with prob.
αk = 1− uk =
{k/N if p = q = 1/21−(q/p)k
1−(q/p)N if p 6= q
Let N →∞
αk = 1− uk →{
0 p ≤ q1 if p > q
[Even the gambling is fair, A will eventually go broke]
Example 3.8 (A continuous sampling Plan) Consider a production line where each item
has probability p of being defective. Assume that the condition of a particular item (defective
od nondefective) does not depend on the conditions of the other items. Consider the following
continuous sampling Plan. Initially, every item is sampled as it is produced; this procedure
continues until i consecutive nondefective items are found. Then the sampling the sampling
plan calls for sampling only one out of every r items at randome until a defective one is found.
When this happens the plan calls for reverting to 100% sampling until i consecutive nondefective
items are found. The process continues in the same way.
State Ek (k = 0, 1, ..., i-1) denotes that k consecutive nondefective items have been found
in the 100 percent sampling portion of the plan, while state Ei denotes that the plan is in the
second stage (sampling one out of r ). Time m is considered to follow the mth item, whether
sampled or not.
Pjk = P (in state Ek after m + 1 items | in state Ej after m items)
=
p for k = 0, 0 ≤ j < i1− p for k = j + 1 ≤ ipr for k = 0, j = i1− p
r for k = j = i0 otherwise
9
Page 51
The transition probability matrix is
P =
∥∥∥∥∥∥∥∥∥
p 1− p 0 ... 0 0p 0 1− p ... 0 0...
p/r 0 0 .... 0 1− p/r
∥∥∥∥∥∥∥∥∥
Let πk be the limiting distribution that the system is in state Ek for k = 0, 1, 2, ..., i. The
equations determining these limiting probability are
(0) pπ0 + pπ1 + ... + pπi−1 + (p/r)πi = π0
(1) (1− p)π0 + 0 + ... + 0 + 0 = π1
(2) 0 + (1− p)π1 + ... + 0+ 0 = π2...
(i) 0 + 0 ... + (1− p)πi−1 + (1− p/r)πi = πi
and
π0 + π1 + · · ·+ πi = 1
From (1) to (i) we have
πk = (1− p)πk−1 and πi = (r/p)(1− p)πi−1.
it follows
πk = (1− p)kπ0, k = 0, · · · , i− 1
together with the final equation, we have
{[1 + ... + (1− p)i−1] +r
p(1− p)i}π0 = 1.
Hence
πk =p(1− p)k
1 + (r − 1)(1− p)ik = 0, 1, · · · , i− 1
and
πi =r(1− p)i
1 + (r − 1)(1− p)i
The Average fraction inspected, the long run fraction of items that are inspected, is
AFI = (π0 + ... + πi−1) + (1/r)πi =1
1 + (r − 1)(1− p)i,
because each item is inspected while in states E0, · · · , Ei−1 but only one out of r is inspected in
state Ei. The average fraction not inspected is
1−AFI =(r − 1)(1− p)i
1 + (r − 1)(1− p)i.
10
Page 52
Let us assume taht each item founded to be defective is replaced by an item known to be good.
Thus the the average outgoing quality (AOQ) is only in the uninspected items, with defective
rate p. thus
AOD =(r − 1)(1− p)ip
1 + (r − 1)(1− p)i
Example 3.9 (A Maze) A white rat is put into the maze
1 2 3
4 5 6
7 8 9
shock
In the absence of learning, one might hypothesize that the rat would move through the maze
at random, i.e. if there are k ways to leave a compartment, then the rat would try each of them
with equal probability 1/k. Suppose that doors between 4 and 7, 6 and 9 are not accessible from
each other and the door between 5 and 8 is only accessible from 5 to 8, the door will be closed.
Assume that the rat makes one try to some adjacent compartment at each unit of time. Let
Xn be the compartment occupied at stage n. Suppose that compartment 3 contains electrical
shocking mechanisms.
The transition probability matrix
P =
1 2 3 4 5 6 7 8 91 0 1/2 0 1/2 0 0 0 0 02 1/3 0 1/3 0 1/3 0 0 0 03 0 0 1 0 0 0 0 0 04 1/3 0 0 1/3 1/3 0 0 0 05 0 1/4 0 1/4 0 1/4 0 1/4 06 0 0 1/3 0 1/3 1/3 0 0 07 0 0 0 0 0 0 1/2 1/2 08 0 0 0 0 0 0 1/3 1/3 1/39 0 0 0 0 0 0 0 1/2 1/2
1. Starting from 1, what is the probability the rat will be shocked?
Define the MC Yn with state 1,2,3,4,5,6 and A, where
{Yn = k} = {Xn = k}, for k = 1, ..., 6, and {Yn = A} = {Xn = 7, 8, 9}
11
Page 53
Then its probability matrix is
P =
1 2 3 4 5 6 A1 0 1/2 0 1/2 0 0 02 1/3 0 1/3 0 1/3 0 03 0 0 1 0 0 0 04 1/3 0 0 1/3 1/3 0 05 0 1/4 0 1/4 0 1/4 1/46 0 0 1/3 0 1/3 1/3 0A 0 0 0 0 0 0 1
Let T = minn{Yn = 3 or A} and ui = P (YT = 3|Y0 = i). Then
uA = 0, u3 = 1
Then by the first step analysis
ui = pi1u1 + pi2u2 + pi3u3 + pi4u4 + pi5 + pi6u6 + piAuA
for i = 1, 2, 4, 5, 6. it gives
u1 = 0.6552
2. In the long run, what is the probability that the rat will be in 7 starting from 1?
P (Xn = 7|X0 = 1) = P (Xn = 7|YT = A)P (YT = A|X0 = 1)
Note that {Yn} with states 7, 8, 9 is an irreducible MC with transition probability matrix
PY =
7 8 97 1/2 1/2 08 1/3 1/3 1/39 0 1/2 1/2
Let
πk = limn→∞P (Xn = k|YT = A)
Then
(π7, π8, π9) = (π7, π8, π9)PY
π7 + π8 + π9 = 1
It gives
π7 = 0.2857
Therefore, starting from 1 the limiting probility that the rat is in 7 is
0.2857 ∗ (1− 0.6552) = 0.0985
12
