Date post: | 07-Mar-2016 |
Category: |
Documents |
Upload: | rasull-saimon |
View: | 230 times |
Download: | 0 times |
of 22
Markov Chain
Stochastic process (discrete time):{X1, X2, ..., }
Markov chain Consider a discrete time stochastic process with
discrete space. Xn {0, 1, 2, ...}. Markovian property
P{Xn+1 = j | Xn = i,Xn1 = in1, ..., X0 = i0}= P{Xn+1 = j | Xn = i} = Pi,j
Pi,j is the transition probability: the probability ofmaking a transition from i to j.
Transition probability matrix
P =
P0,0 P0,1 P0,2 P1,0 P1,1 P1,2
.
.
.
.
.
.
.
.
.
.
.
.
Pi,0 Pi,1 Pi,2 ...
.
.
.
.
.
.
.
.
.
1
Example
Suppose whether it will rain tomorrow depends onpast weather condition ony through whether it rainstoday. Consider the stochastic process {Xn, n = 1, 2, ...}
Xn =
{0 rain on day n1 not rain on day n
P (Xn+1|Xn, Xn1, ..., X1) = P (Xn+1 | Xn) State space {0, 1}. Transition matrix:(
P0,0 P0,1P1,0 P1,1
)
P0,0 = P (tomorrow rain|today rain) = . Then P0,1 =1 .
P1,0 = P (tomorrow rain|today not rain) = . ThenP1,1 = 1 .
Transition matrix: ( 1 1
)
2
Transforming into a Markov Chain
Suppose whether it will rain tomorrow depends onwhether it rained today and yesterday.
P (Xn+1|Xn, Xn1, ..., X1) = P (Xn+1|Xn, Xn1). Theprocess is not a first order Markov chain.
Define Yn:
Yn =
0 Xn = 0, Xn1 = 0 RR1 Xn = 0, Xn1 = 1 NR2 Xn = 1, Xn1 = 0 RN3 Xn = 1, Xn1 = 1 NN
P (Yn+1|Yn, Yn1, ...)
= P (Xn+1, Xn|Xn, Xn1, ...)= P (Xn+1, Xn|Xn, Xn1)= P (Yn+1|Yn)
{Yn, n = 1, 2, ...} is a Markov chain.
P0,0 P0,1 P0,2 P0,3P1,0 P1,1 P1,2 P1,3P2,0 P2,1 P2,2 P2,3P3,0 P3,1 P3,2 P3,3
3
P0,1 = P (Yn+1 = 1|Yn = 0) = P (Xn+1 = 0, Xn =1|Xn = 0, Xn1 = 0) = 0.
P0,3 = P (Yn+1 = 3|Yn = 0) = P (Xn+1 = 1, Xn =1|Xn = 0, Xn1 = 0) = 0.
Similarly, P1,1 = P1,3 = 0, P2,0 = P2,2 = 0, P3,0 =P3,2 = 0.
Transition matrix
P0,0 0 P0,2 0P1,0 0 P1,2 00 P2,1 0 P2,30 P3,1 0 P3,3
=
P0,0 0 1 P0,0 0P1,0 0 1 P1,0 00 P2,1 0 1 P2,10 P3,1 0 1 P3,1
The Markov chain is specified by P0,0, P1,0, P2,1, P3,1.1. P0,0 = P (tomorrow will rain|today rain, yesterday rain).2. P1,0 = P (tomorrow will rain|today rain, yesterday not rain).3. P2,1 = P (tomorrow will rain|today not rain, yesterday rain).4. P3,1 = P (tomorrow will rain|today not rain, yesterday not rain).
4
Chapman-Kolmogorov Equations
Transition after nth steps:P ni,j = P (Xn+m = j | Xm = i).
Chapman-Kolmogorov Equations:
P n+mi,j =k=0
P ni,kPmk,j, n,m 0 for all i, j.
Proof (by Total probability formula):P n+mi,j = P (Xn+m = j|X0 = i)
=k=0
P (Xn+m = j,Xn = k|X0 = i)
=k=0
P (Xn = k|X0 = i)
P (Xn+m = j|Xn = k,X0 = i)=
k=0
P ni,kPmk,j
5
n-step transition matrix:
P(n) =
P n0,0 Pn0,1 P
n0,2
P n1,0 Pn1,1 P
n1,2
.
.
.
.
.
.
.
.
.
.
.
.
P ni,0 Pni,1 P
ni,2
.
.
.
.
.
.
.
.
.
.
.
.
Chapman-Kolmogorov Equations:P
(n+m) = P(n) P(m), P(n) = Pn. Weather example:
P =
( 1 1
)=
(0.7 0.30.4 0.6
)Find P (rain on Tuesday | rain on Sunday) andP (rain on Tuesday and rain on Wednesday | rain on Sunday).Solution:
P (rain on Tuesday | rain on Sunday) = P 20,0P
(2) = P P =(
0.7 0.30.4 0.6
)(
0.7 0.30.4 0.6
)
=
(0.61 0.390.52 0.48
)P (rain on Tuesday | rain on Sunday) = 0.61
6
P (rain on Tuesday and rain on Wednesday | rain on Sunday)= P (Xn = 0, Xn+1 = 0 | Xn2 = 0)= P (Xn = 0|Xn2 = 0)P (Xn+1 = 0|Xn = 0, Xn2 = 0)= P (Xn = 0|Xn2 = 0)P (Xn+1 = 0|Xn = 0)= P 20,0P0,0= 0.61 0.7 = 0.427
7
Classification of States
Accessible: State j is accessible from state i if P ni,j >0 for some n 0. i j. Equivalent to: P (ever enter j|start in i) > 0.
P (ever enter j|start in i)= P (n=0{Xn = j}|X0 = i)
n=0
P (Xn = j | X0 = i)
=n=0
P ni,j
Hence if P ni,j = 0 for all n, P (ever enter j|start in i) =0. On the other hand,
P (ever enter j|start in i)= P (n=0{Xn = j}|X0 = i) P ({Xn = j}|X0 = i) for any n= P ni,j .
If P ni,j > 0 for some n, P (ever enter j|start in i) P ni,j > 0.
Examples
8
Communicate: State i and j communicate if they areaccessible from each other. i j. Properties:
1. P 0i,i = P (X0 = i|X0 = i) = 1. Any state icommunicates with itself.
2. If i j, then j i.3. If i j and j k, then i k.
Proof:
i j = P ni,j > 0 and P n
j,i > 0
j k = Pmj,k > 0 and Pm
k,j > 0
P n+mi,k =l=0
P ni,lPml,k Chapman-Kolmogorov Eq.
> P ni,j Pmj,k> 0
Similarly, we can show P n+mk,i > 0. Hence ik.
9
Class: Two states that communciate are said to bein the same class. A class is a subset of states thatcommunicate with each other. Different classes do NOT overlap. Classes form a partition of states.
Irreducible: A Markov chain is irreducible if there isonly one class. Consider the Markov chain with transition proba-
bility matrix:
P =
12 12 01
214
14
0 1323
The MC is irreducible. MC with transition probability matrix:
P =
12
12
0 012
12 0 0
14
14
14
14
0 0 0 1
Three classes: {0, 1}, {2}, {3}.
10
Recurrent and Transient States
fi: probability that starting in state i, the MC will everreenter state i.
Recurrent: If fi = 1, state i is recurrent. A recurrent states will be visited infinitely many
times by the process starting from i. Transient: If fi < 1, state i is transient.
Starting from i, the MC will be in state i for ex-actly n times (including the starting state) is
fn1i (1 fi) , n = 1, 2, ...This is a geometric distribution with parameter 1fi. The expected number of times spent in state iis 1/(1 fi).
11
A state is recurrent if and only if the expected numberof time periods that the process is in state i, startingfrom state i, is infinite.
Recurrent E(number of visits to i|X0 = i) = Transient E(number of visits to i|X0 = i)
Consider a MC with
P =
0 0 1212
1 0 0 00 1 0 00 1 0 0
The MC is irreducible and finite state, hence all statesare recurrent.
Consider a MC with
P =
12
12 0 0 0
12
12
0 0 00 0 12
12 0
0 0 12
12
014
14 0 0
12
Three classes: {0, 1}, {2, 3}, {4}. State 0, 1, 2, 3 arerecurrent and state 4 is transient.
14
Random Walk
A Markov chain with state space i = 0,1,2, .... Transition probability: Pi,i+1 = p = 1 Pi,i1.
At every step, move either 1 step forward or 1 stepbackward.
Example: a gambler either wins a dollar or loses adollar at every game. Xn is the number of dollars hehas when starting the nth game.
For any i < j, P jii,j = pji > 0, P jij,i = (1 p)ji >0. The MC is irreducible.
Hence, either all the states are transient or all thestates are recurrent.
15
Under which condition are the states transient or re-current? Consider State 0.
n=1
P n0,0 =
{ recurrentfinite transient
Only for even m, Pm0,0 > 0.
P 2n0,0 =
(2nn
)pn(1 p)n = (2n)!
n!n!(p(1 p))n
n = 1, 2, 3, ...
By Stirlings approximation
n! nn+1/2en
2pi
P 2n0,0 (4p(1p))n
pin
.
When p = 1/2, 4p(1 p) = 1.n=0
(4p(1 p))npin
=n=0
1pin
,
The summation diverges. Hence, all the states arerecurrent.
When p 6= 1/2, 4p(1 p) < 1. n=0 (4p(1p))npinconverges. All the states are transient.
16
Limiting Probabilities
Weather example
P =
(0.7 0.30.4 0.6
)
P(4) = P4 =
(0.5749 0.42510.5668 0.4332
)
P(8) = P(4)P(4) =
(0.572 0.4280.570 0.430
)
P(4) and P(8) are close. The rows in P(8) are close. Limiting probabilities?
17
Period d: For state i, if P ni,i = 0 whenever n is notdivisible by d and d is the largest integer with thisproperty, d is the period of state i. Period d is the greatest common divisor of all them such that Pmi,i > 0.
Aperiodic: State i is aperiodic if its period is 1. Positive recurrent: If a state i is recurrent and the ex-
pected time until the process returns to state i is finite.
If i j and i is positive recurrent, then j is posi-tive recurrent.
For a finite-state MC, a recurrent state is also pos-itive recurrent.
A finite-state irreducible MC contains all positiverecurrent states.
Ergodic: A positive recurrent and aperiodic state isan ergodic state.
A Markov chain is ergodic if all its states are ergodic.
18
Theorem 4.1: For an irreducible ergodic Markov chain,limnP ni,j exists and is independent of i. Let pij =limnP ni,j, j 0, then pij is the unique nonnegativesolution of
pij =i=0
piiPi,j j 0j=0
pij = 1 .
The Weather Example:
P =
( 1 1
)The MC is irreducible and ergodic.
pi0 + pi1 = 1
pi0 = pi0P0,0 + pi1P1,0 = pi0 + pi1
pi1 = pi0P0,1 + pi1P1,1 = pi0(1 ) + pi1(1 )Solve the linear equations, pi0 = 1+, pi1 =
11+.
19
Gamblers Ruin Problem
At each play, a gambler either wins a unit with prob-ability p or loses a unite with probability q = 1 p.Suppose the gambler starts with i units, what is theprobability that the gamblers fortune will reach Nbefore reaching 0 (broke)?
Solution: Let Xt be the number of units the gambler has at
time t. {Xt; t = 0, 1, 2, ...} is a Markov chain. Transition probabilities:
P0,0 = PN,N = 1
Pi,i+1 = p, Pi,i1 = 1 p = q . Denote the probability that starting from i units,
the gamblers fortune will reach N before reaching0 by Pi, i = 0, 1, ..., N .
Condition on the result of the first game and applythe total probability formula:
Pi = pPi+1 + qPi1 .
20
Changing forms:(p + q)Pi = pPi+1 + qPi1
q(Pi Pi1) = p(Pi+1 Pi)Pi+1 Pi = q
p(Pi Pi1)
Recursion:P0 = 0
P2 P1 = qp(P1 P0) = q
pP1
P3 P2 = qp(P2 P1) =
(q
p
)2P1
.
.
.
Pi Pi1 = qp(Pi1 Pi2) =
(q
p
)i1P1
Add up the equations:Pi = P1[1 +
q
p+ + (q
p)i1]
=
P1
1(qp)i1qp
if qp 6= 1iP1 if qp = 1
We know PN = 1
PN =
P1
1(qp)N1qp
if qp 6= 1NP1 if qp = 1
21
Hence,
P1 =
{1qp
1(qp)Nif p 6= 1/2
1/N if p = 1/2
In summary,
Pi =
1(qp)i1(qp)N
if p 6= 1/2i/N if p = 1/2
Note, if N ,
Pi =
{1 (qp)i if p > 1/20 if p 1/2
When p > 1/2, there is a positive probability that thegambler will win infinitely many units.
When p 1/2, the gambler will surely go broke(with probability 1) if not stop at a finite fortune (as-suming the opponent is infinitely rich).
22