Date post:  16Dec2015 
Category: 
Documents 
Author:  lillianjackson 
View:  262 times 
Download:  15 times 
Markov Chains
Summary
Markov Chains Discrete Time Markov Chains
Homogeneous and nonhomogeneous Markov chains
Transient and steady state Markov chains
Continuous Time Markov Chains Homogeneous and nonhomogeneous Markov
chains Transient and steady state Markov chains
Markov Processes
Recall the definition of a Markov Process The future a a process does not depend on its past, only on its
present
1 1 0 0
1 1
Pr  ,...,
Pr 
k k k k
k k k k
X t x X t x X t x
X t x X t x
Since we are dealing with “chains”, X(t) can take discrete
values from a finite or a countable infinite set. For a discretetime Markov chain, the notation is also
simplified to
1 1 0 0 1 1Pr  ,..., Pr k k k k k k k kX x X x X x X x X x
Where Xk is the value of the state at the kth step
ChapmanKolmogorov Equations
Define the onestep transition probabilities
1Pr ij k kp X j X ik Clearly, for all i, k, and all feasible transitions from state i
1ijj i
p k
xi
x1
xR
… xj
k u k+n
Pr ,ij k n kp X j X ik k n
Define the nstep transition probabilities
ChapmanKolmogorov Equations
xi
x1
xR
… xj
k u k+n
1
Pr  , Pr ,R
ij k n u k u kr
p X j X r X i X r X ik k n
Using total probability
Pr  , Pr k n u k k n uX j X r X i X j X r
Using the memoryless property of Marckov chains
1
, , , ,R
ij ir rjr
p p p k u k nk k n k u u k n
Therefore, we obtain the ChapmanKolmogorov Equation
Matrix Form
Define the matrix
We can rewrite the ChapmanKolmogorov Equation
Choose, u = k+n1, then
,, ijp k k nk k n H
, , ,k k n k u u k n H H H
, , 1 1,
, 1 1
k k n k k n k n k n
k k n k n
H H H
H P
One step transition probability
Forward ChapmanKolmogorov
Matrix Form
Choose, u = k+1, then
, , 1 1,
1,
k k n k k k k n
k k nk
H H H
P H
One step transition probability
Backward ChapmanKolmogorov
Homogeneous Markov Chains
The onestep transition probabilities are independent of time k.
1 or Pr ij k kp X j X ik P P
Even though the one step transition is independent of k, this does not mean that the joint probability of Xk+1 and Xk is also independent of k Note that
1 1Pr , Pr  Pr
Pr
k k k k k
ij k
X j X i X j X i X i
p X i
Example
Consider a two processor computer system where, time is divided into time slots and that operates as follows At most one job can arrive during any time slot and this can
happen with probability α. Jobs are served by whichever processor is available, and if
both are available then the job is given to processor 1. If both processors are busy, then the job is lost When a processor is busy, it can complete the job with
probability β during any one time slot. If a job is submitted during a slot when both processors are
busy but at least one processor completes a job, then the job is accepted (departures occur before arrivals).
Describe the automaton that models this system. Describe the Markov Chain that describe this model.
Example: Automaton
Let the number of jobs that are currently processed by the system by the state, then the State Space is given by X= {0, 1, 2}.
Event set: a: job arrival, d: job departure
Feasible event set: If X=0, then Γ(X)= a If X= 1, 2, then Γ(Χ)= a, d.
State Transition Diagram
0 1 2
a
 / a,d
a
d d / a,d,d
dd
/a/ad
Example: Alternative Automaton
Let (X1,X2) indicate whether processor 1 or 2 are busy, Xi= {0, 1}. Event set:
a: job arrival, di: job departure from processor i Feasible event set:
If X=(0,0), then Γ(X)= a If X=(0,1) then Γ(Χ)= a, d2. If X=(1,0) then Γ(Χ)= a, d1. If X=(0,1) then Γ(Χ)= a, d1, d2.
State Transition Diagram
a
 / a,d1
a

d2 d1
d1,d2
/a/ad1/ad2

d1
a,d2
a,d1,d2
00
10
11
01
Example: Markov Chain
For the State Transition Diagram of the Markov Chain, each transition is simply marked with the transition probability
0 1 2
p01
p11
p12
p00p10
p21
p20
p22
00 1p 01p 02 0p
10 1p 11 1 1p 12 1p
220 1p 2
21 2 1 1p 2
22 21 1p
Example: Markov Chain
Suppose that α = 0.5 and β = 0.7, then,
0 1 2
p01
p11
p12
p00p10
p21
p20
p22
0.5 0.5 0
0.35 0.5 0.15
0.245 0.455 0.3ijp
P
State Holding Times
Suppose that at point k, the Markov Chain has transitioned into state Xk=i. An interesting question is how long it will stay at state i.
Let V(i) be the random variable that represents the number of time slots that Xk=i.
We are interested on the quantity Pr{V(i) = n}
1 1
1
1 1
Pr Pr , ,..., 
Pr  ,...,
Pr ,..., 
k n k n k k
k n k n k
k n k k
V n X i X i X i X ii
X i X i X i
X i X i X i
1
1 2
2 1
Pr 
Pr  ...,
Pr ,..., 
k n k n
k n k n k
k n k k
X i X i
X i X X i
X i X i X i
State Holding Times
This is the Geometric Distribution with parameter pii. Clearly, V(i) has the memoryless property
1
1 2
2 1
Pr Pr 
Pr  ...,
Pr ,..., 
k n k n
k n k n k
k n k k
V n X i X ii
X i X X i
X i X i X i
1Pr 1 nii iiV n p pi
1 2
2 3
3 1
1 Pr 
Pr  ,...,
Pr ,..., 
ii k n k n
k n k n k
k n k k
p X i X i
X i X i X i
X i X i X i
State Probabilities
An interesting quantity we are usually interested in is the probability of finding the chain at various states, i.e., we define
Pri kX ik For all possible states, we define the vector
0 1, ...k k k π Using total probability we can write
1 1Pr  Pr
1
i k k kj
ij jj
X i X j X jk
p k k
In vector form, one can write
1k k k π π P 1k k π π POr, if homogeneous Markov Chain
State Probabilities Example
Suppose that
1 0 00 π
Find π(k) for k=1,2,…
with
0.5 0.5 0
0.35 0.5 0.15
0.245 0.455 0.3
P
0.5 0.5 0
1 0 0 0.35 0.5 0.15 0.5 0.5 010.245 0.455 0.3
π
Transient behavior of the system: MCTransient.m In general, the transient behavior is obtained by solving the difference equation
1k k π π P
Classification of States
Definitions State j is reachable from state i if the probability to go
from i to j in n >0 steps is greater than zero (State j is reachable from state i if in the state transition diagram there is a path from i to j).
A subset S of the state space X is closed if pij=0 for every iS and j S
A state i is said to be absorbing if it is a single element closed set.
A closed set S of states is irreducible if any state jS is reachable from every state iS.
A Markov chain is said to be irreducible if the state space X is irreducible.
Example
Irreducible Markov Chain
0 1 2
p01 p12
p00p10
p21
p22
Reducible Markov Chain p01 p12
p00p10
p14
p224
p23
p32
p33
0 1 2 3
Absorbing State
Closed irreducible set
Transient and Recurrent States
Hitting Time
Recurrence Time Tii is the first time that the MC returns to state i.
Let ρi be the probability that the state will return back to i given it starts from i. Then,
0min 0 : ,ij kT k X i X j
1
Pri iik
T k
The event that the MC will return to state i given it started
from i is equivalent to Tii < ∞, therefore we can write
1
Pr Pri ii iik
T k T
A state is recurrent if ρi=1 and transient if ρi<1
Theorems
If a Markov Chain has finite state space, then at least one of the states is recurrent.
If state i is recurrent and state j is reachable from state i then, state j is also recurrent.
If S is a finite closed irreducible set of states, then every state in S is recurrent.
Positive and Null Recurrent States
Let Mi be the mean recurrence time of state i
A state is said to be positive recurrent if Mi<∞. If Mi=∞ then the state is said to be nullrecurrent.
Theorems If state i is positive recurrent and state j is reachable
from state i then, state j is also positive recurrent. If S is a closed irreducible set of states, then every
state in S is positive recurrent or, every state in S is null recurrent, or, every state in S is transient.
If S is a finite closed irreducible set of states, then every state in S is positive recurrent.
1
Pri ii iik
M E T k T k
Example
p01 p12
p00p10
p14
p224
p23
p32
p33
0 1 2 3
Recurrent State
Transient States Positive
Recurrent States
Periodic and Aperiodic States
Suppose that the structure of the Markov Chain is such that state i is visited after a number of steps that is an integer multiple of an integer d >1. Then the state is called periodic with period d.
If no such integer exists (i.e., d =1) then the state is called aperiodic.
Example1 0.5
0.5
0 1 2
1
Periodic State d = 2
0 1 0
0.5 0 0.5
0 1 0
P
Steady State Analysis
Recall that the probability of finding the MC at state i after the kth step is given by
Pri kX ik 0 1, ...k k k π An interesting question is what happens in the “long run”,
i.e., limi kk
Questions: Do these limits exists? If they exist, do they converge to a legitimate
probability distribution, i.e., How do we evaluate πj, for all j.
1i
This is referred to as steady state or equilibrium or stationary state probability
Steady State Analysis
Recall the recursive probability 1k kπ π P
If steady state exists, then π(k+1) π(k), and therefore the steady state probabilities are given by the solution to the equations
If an Irreducible Markov Chain the presence of periodic states prevents the existence of a steady state probability
Example: periodic.m
π πP and 1ii
0 1 0
0.5 0 0.5
0 1 0
P 1 0 00 π
Steady State Analysis
THEOREM: In an irreducible aperiodic Markov chain consisting of positive recurrent states a unique stationary state probability vector π exists such that πj > 0 and
1limj jk
j
kM
where Mj is the mean recurrence time of state j
The steady state vector π is determined by solving
π πP and 1ii
Ergodic Markov chain.
BirthDeath Example
1p 1p
pp
1p
p
0 1 i
p1 0
0 1
0 0
p p
p p
p
P
Thus, to find the steady state vector π we need to solve
π πP and 1ii
BirthDeath Example
0 0 1p p In other words
1 1 , 1, 2,...1j j j p jp
1 0
1 p
p
Solving these equations we get2
2 0
1 p
p
In general 0
1j
j
p
p
Summing all terms we get
0 00 0
1 11 1
i i
i i
p p
pp
BirthDeath Example
Therefore, for all states j we get
0
1i
i
p
p
If p<1/2, then 0, for all j j
0
1 1j i
ji
p p
p p
All states are transient
0
10
2 1
i
i
p pp p
If p>1/2, then 12 1
, for all j
j
ppj
pp
All states are positive recurrent
BirthDeath Example
If p=1/2, then
0
1i
i
p
p
0, for all j j
All states are null recurrent
Reducible Markov Chains
In steady state, we know that the Markov chain will eventually end in an irreducible set and the previous analysis still holds, or an absorbing state.
The only question that arises, in case there are two or more irreducible sets, is the probability it will end in each set
Transient Set T
Irreducible Set S1
Irreducible Set S2
Transient Set T
Reducible Markov Chains
Suppose we start from state i. Then, there are two ways to go to S. In one step or Go to r T after k steps, and then to S.
Define
Irreducible Set S
i
rs1
sn
0Pr  , 1, 2,...i kX S X i kS
Reducible Markov Chains
Next consider the general case for k=2,3,…
1 0Pr X S X i ijj S
p
1 1 1 0Pr , ..., k k kX S X r T X r T X i
1 1 1 0
1 0
Pr , ..., ,
Pr 
k k kX S X r T X r T X i
X r T X i
i ij r irj S r T
p pS S
First consider the onestep transition
1 1 1Pr , ...,k k k irX S X r T X r T p r irpS
ContinuousTime Markov Chains
In this case, transitions can occur at any time Recall the Markov (memoryless) property
1 1 0 0
1 1
Pr  ,...,
Pr 
k k k k
k k k k
X t x X t x X t x
X t x X t x
where t1 < t2 < … < tk
Recall that the Markov property implies that X(tk+1) depends only on X(tk) (state memory)
It does not matter how long the state at X(tk) (age memory).
The transition probabilities now need to be defined for every time instant as pij(t), i.e., the probability that the MC transitions from state i to j at time t.
Transition Function
Define the transition function Pr  , ,ijp X j X i s ts t t s
The continuoustime analogue of the ChapmanKolmokorov equation is
Using the memoryless property
,
Pr  , Pr 
ij
r
p s t
X j X r X i X r X it u s u s
Pr  Pr ,ijr
p X j X r X r X is t t u u s Define H(s,t)=[pij(s,t)], i,j=1,2,… then
, , , , s u ts t s u u t H H H Note that H(s, s)= I
Transition Rate Matrix
Consider the ChapmanKolmogorov for s ≤ t ≤ t+Δt
, , ,s t t s t t t t H H H
Subtracting H(s,t) from both sides and dividing by Δt
, ,, , s t t t ts t t s t
t t
H H IH H
Taking the limit as Δt0
,
,s t
s t tt
H
H Q
where the transition rate matrix Q(t) is given by
0
,lim
t
t t tt
t
H I
Q
Homogeneous Case
In the homogeneous case, the transition functions do not depend on s and t, but only on the difference ts thus
,ij ijp ps t t s
It follows that
,s t t s H H P
and the transition rate matrix
0 0
,lim lim , constant
t t
t t t tt
t t
H I H IQ Q
Thus
1 if with 0
0 if ij
i jt pti jt
PP Q tet QP
State Holding Time
The time the MC will spend at each state is a random variable with distribution
1 iiG et
where
i j
j i
Explain why…
Transition Rate Matrix Q.
Recall that t
tt
P
P Q
ijir rj
r
p tp qt
t
First consider the qij, i ≠ j, thus the above equation can be written as
ij
ii ij ir rjr i
p tp q p qt t
t
Evaluating this at t = 0, we get that
0
ijij
t
p t qt
pij(0)= 0 for all i ≠ j
The event that will take the state from i to j has exponential residual lifetime with rate λij, therefore, given that in the interval (t,t+τ) one event has occurred, the probability that this transition will occur is given by Gij(τ)=1exp{λijτ}.
Transition Rate Matrix Q.
0
ijij
pq
Since Gij(τ)=1exp{λijτ}.
0
ij
ijije
In other words qij is the rate of the Poisson process that activates the event that makes the transition from i to j.
Next, consider the qjj, thus
ijij jj ir rj
r j
p tp q p qt t
t
Evaluating this at t = 0, we get that
0
ijjj
t
p t qt
0
1 ij jjt
p qtt
Probability that chain leaves state j
Transition Rate Matrix Q.
0ijj
q Note that for each row i, the sum
The event that the MC will transition out of state i has exponential residual lifetime with rate Λ(i), therefore, the probability that an event will occur in the interval (t,t+τ) given by Gi(τ)=1exp{ Λ(i)τ}.
0
ijjq e ii
Transition Probabilities P.
1Pr ij k kP X j X i
Recall that in the case of the superposition of two or more Poisson processes, the probability that the next event is from process j is given by λj/Λ.
Suppose that state transitions occur at random points in time T1 < T2 <…< Tk <…
Let Xk be the state after the transition at Tk
Define
In this case, we have
,ijij
ii
qP i j
q
0iiP and
Example
Assume a computer system where jobs arrive according to a Poisson process with rate λ.
Each job is processed using a First In First Out (FIFO) policy.
The processing time of each job is exponential with rate μ. The computer has buffer to store up to two jobs that wait
for processing. Jobs that find the buffer full are lost. Draw the state transition diagram. Find the Rate Transition Matrix Q. Find the State Transition Matrix P
Example
The rate transition matrix is given by
a
d
0 1 2 3
a a
a
d d
0 0
0
0
0 0
Q
0 0 0
0 01
0 0
0 0 0
P
The state transition matrix is given by
State Probabilities and Transient Analysis
Similar to the discretetime case, we define Prj X jt t
In vector form 1 2, ,...t t t π
With initial probabilities 1 2, ,...0 0 0 π
Using our previous notation (for homogeneous MC)
0t t π π P 0teQπ
Obtaining a general solution is not easy!
Differentiating with respect to t gives us more “inside” d t
tdt
π
π Q
jjj j ij i
i j
d tq qt t
dt
“Probability Fluid” view
We view πj(t) as the level of a “probability fluid” that is stored at each node j (0=empty, 1=full).
j
jj j ij ii j
d tq qt t
dt
Change in the probability fluid
inflowoutflow
ri
j
qij… qjr…
Inflow Outflow
jj jrr j
q q
Steady State Analysis
Often we are interested in the “longrun” probabilistic behavior of the Markov chain, i.e.,
limj jtt
As with the discretetime case, we need to address the following questions Under what conditions do the limits exist? If they exist, do they form legitimate probabilities? How can we evaluate these limits?
These are referred to as steady state probabilities or equilibrium state probabilities or stationary state probabilities
Steady State Analysis
Theorem: In an irreducible continuoustime Markov Chain consisting of positive recurrent states, a unique stationary state probability vector π with
limj jtt
These vectors are independent of the initial state probability and can be obtained by solving
and 1jj
πQ = 0
0 jj j ij ii j
q qt t
0 Change
inflow
outflow ri
j
qij… qjr…
Inflow Outflow
Using the “probability fluid” view
0j t
dt
Example
For the previous example, with the above transition function, what are the steady state probabilities
a
d
0 1 2 3
a a
a
d d
0 1 2 3
0 0
0
0
0 0
πQ 0
0 1 2 3 1
Solve
Example
The solution is obtained
0 1 2 3 1
0 1 0 1 0
0 1 2 0 2
2 0
1 2 3 0 3
3 0
0 2 3
1
1
BirthDeath Chain
Find the steady state probabilities
0 0
1 11 1
2 2 2
0
0
Q
Similarly to the previous example,
λ0
0 1 i
λ1 λi1 λi
μ1μi μi+1
And we solve
πQ 00
1ii
and
Example
The solution is obtained
0 0 1 1 0 01 0
1
0 0 1 2 21 1 0 0 12 0
1 2
In general
1 1 1 1 0j jj j j j j 0
1 01 1
...
...j
jj
Making the sum equal to 1
0 1
01 1
...1 1
...j
j j
Solution exists if
0 1
1 1
...1
...j
j j
S
Uniformization of Markov Chains
In general, discretetime models are easier to work with, and computers (that are needed to solve such models) operate in discretetime
Thus, we need a way to turn continuoustime to discretetime Markov Chains
Uniformization Procedure Recall that the total rate out of state i is –qii=Λ(i). Pick
a uniform rate γ such that γ ≥ Λ(i) for all states i. The difference γ  Λ(i) implies a “fictitious” event that
returns the MC back to state i (self loop).
Uniformization of Markov Chains
Uniformization Procedure Let PU
ij be the transition probability from state I to state j for the discretetime uniformized Markov Chain, then
if
if
ij
Uij
ijj i
qi j
Pq
i j
i
j
k
……
qij
qik
Uniformization i
j
k
……
ijq
ikq
iiq