Date post: | 20-Jan-2016 |
Category: |
Documents |
Upload: | loraine-lyons |
View: | 217 times |
Download: | 0 times |
1
Introduction to Stochastic ModelsIntroduction to Stochastic ModelsGSLM 54100GSLM 54100
2
OutlineOutline
limiting distribution connectivity
types of states and of irreducible DTMCs transient, recurrent, positive recurrent, null
recurrent
periodicity
limiting behavior of irreducible chains
3
ConnectivityConnectivity
4
Connectivity of a Connectivity of a DTMCDTMC
connectivity: one factor that determines the limiting behavior of a DTMC
1 2
1
0.010.99
A
0.99 0.01
0 1
P
1 2
0.99
0.010.99
B
0.01
0.99 0.01
0.01 0.99
P
5
Connectivity of a Connectivity of a DTMCDTMC
1 2
1
0.010.99
A
0.99 0.01
0 1
P
(4) 0.960596 0.03940399
0 1
P
(16) 0.8514576 0.1485422
0 1
P
(64) 0.5255965 0.4744035
0 1
P
(256) 0.07631498 0.923685
0 1
P
(1024) 0.0000339187 0.9999661
0 1
P
( ) 0 1
0 1
P
6
Connectivity of a Connectivity of a DTMCDTMC
1 2
0.99
0.010.99
B
0.01
0.99 0.01
0.01 0.99
P
(4) 0.96118408 0.03881592
0.03881592 0.96118408
P
(16) 0.8618989 0.1381011
0.1381011 0.8618989
P
(64) 0.6372268 0.3627732
0.3627732 0.6372268
P
(256) 0.5028369 0.4971631
0.4971631 0.5028369
P
(1024) 0.5 0.5
0.5 0.5
P
( ) 0.5 0.5
0.5 0.5
P
7
Connectivity of a Connectivity of a DTMCDTMC
1 2
0.99
0.010.99
B
0.01
0.99 0.01
0.01 0.99
P
( ) 0.5 0.5
0.5 0.5
P
1 2
1
0.010.99
A
0.99 0.01
0 1
P
( ) 0 1
0 1
P
8
Connectivity of a Connectivity of a DTMCDTMC
1 2 0.6
0.10.9
C
1
30.4
0.9 0.1 0
0 0.6 0.4
1 0 0
P
( )
0.7407407 0.1851852 0.07407407
0.7407407 0.1851852 0.07407407
0.7407407 0.1851852 0.07407407
P
9
Connectivity of a Connectivity of a DTMCDTMC
1 2
1
0.010.99
A
0.99 0.01
0 1
P
( ) 0 1
0 1
P
1 2
0.8
0.10.9
D
0.20.9 0.1
0.2 0.8
P
( ) 0.666667 0.333333
0.666667 0.333333
P
( )11lim 0n
np
( )
12lim 1n
np
( )21lim 0n
np
( )
22lim 1n
np
( )11lim 2 / 3n
np
( )
12lim 1/ 3n
np
( )21lim 2 / 3n
np
( )
22lim 1/ 3n
np
10
Connectivity of a Connectivity of a DTMCDTMC
rows of P() may not be the same as in the previous examples
1 2
1
0.5
3 4
0.5
0.5
0.5
0.5
5
0.51
1 0 0 0 0
0.5 0 0.5 0 0
0 0.5 0 0.5 0
0 0 0.5 0 0.5
0 0 0 0 1
P
( )
1 0 0 0 0
0.75 0 0 0 025
0.5 0 0 0 0.5
0.25 0 0 0 0.75
0 0 0 0 1
P
11
Connectivity of a Connectivity of a DTMCDTMC
1 2
1
F
1
0 1
1 0
P
(even) 1 0
0 1
P
(odd) 0 1
1 0
P
limit of P(m) may not exist
12
Connectivity of a Connectivity of a DTMCDTMC0 1 0
0.75 0 0.25
0 1 0
P
even0.75 0 0.25
0 1 0
0.75 0 0.25
P 0 1 0
0.75 0 0.25
0 1 0
odd
P
1 2 3
0.25
1
1
0.75
1 2 3
0.25
0.99
1
0.75
0.01
0 1 0
0.75 0 0.25
0.01 0.99 0
P
0.3757803 0.4993758 0.1248439
0.3757803 0.4993758 0.1248439
0.3757803 0.4993758 0.1248439
P
13
Types of States and of Irreducible DTMCs
14
Limiting Results for a Limiting Results for a DTMCDTMC
depending on the type of states and the chain
type: transient, positive recurrent, null recurrent
connectivity and periodicity
15
Transient State Transient State
1 2
1
0.010.99
A
1 2
1
0.5
3 4
0.5
0.5
0.5
0.5
5
0.51
( ) 0 1
0 1
P( )11lim 0
np
( )
1 0 0 0 0
0.75 0 0 0 025
0.5 0 0 0 0.5
0.25 0 0 0 0.75
0 0 0 0 1
P
( )22lim 0
np
( )33lim 0
np
( )44lim 0
np
16
Recurrent State Recurrent State
state i is recurrent if P(return to i|X0 = i) = 1
( )
0
nii
np
17
Recurrent State Recurrent State
two types of recurrent states positive recurrent:
E(# of transitions to return to i|X0 = i) <
null recurrent:
E(# of transitions to return to i|X0 = i) =
( )
0lim
N nii
n
N
p
N
( )
0lim
N nii
n
N
p
N
18
Periodicity
19
PeriodicityPeriodicity
state i is of period d if Xn can return to state i in multiples of d
states 1, 2, 3 are of period 2
state i of period d
1 2 3
0.25
1
1
0.75
( ) 0 for 1nd kiip k d
20
PeriodicityPeriodicity
period of states 1, 2, 3, and 4 = ?
state 4 of period 2 state 4 can return to itself in 2 steps
1 2 3
0.25
0.99
1
0.754
0.01
1
21
Communicating States Communicating States
communicating states are of the same type transient, positive recurrent, null recurrent at the
same time of the same period
states in an irreducible chain are of the same type transient, positive recurrent, null recurrent at the
same time of the same period
22
Limiting Behavior of Irreducible Chains
23
Limiting Behavior Limiting Behavior of a Positive Irreducible Chainof a Positive Irreducible Chain
j = fraction of time at state j
N: a very large positive integer
# of periods at state j j N
balance of flow j N i (i N)pij j = i ipij
[ ]ijpP
24
Limiting Behavior Limiting Behavior of a Positive Irreducible Chainof a Positive Irreducible Chain
j = fraction of time at state j
j = i ipij
1 = 0.91 + 0.22
2 = 0.11 + 0.82
linearly dependent
normalization equation: 1 + 2 = 1
solving: 1 = 2/3, 2 = 1/3
1 2
0.8
0.10.9
C
0.2
( ) 0.666667 0.333333
0.666667 0.333333
P
25
Limiting Behavior Limiting Behavior of a Positive Irreducible Chainof a Positive Irreducible Chain
1 = 0.752 + 0.013
3 = 0.252
1 + 2 + 3 = 1
1 = 301/801, 2 = 400/801, 3 = 100/801
1 2 3
0.25
0.99
1
0.75
0.01
0 1 0
0.75 0 0.25
0.01 0.99 0
P
0.3757803 0.4993758 0.1248439
0.3757803 0.4993758 0.1248439
0.3757803 0.4993758 0.1248439
P
26
Limiting Behavior Limiting Behavior of a Positive Irreducible Chainof a Positive Irreducible Chain
an irreducible DTMC {Xn} is positive there exists a unique nonnegative solution to
j: stationary (steady-state) distribution of {Xn}
0
0
1 (normalization eqt)
, for all , (balance eqts)
jj
j i iji
p j
27
Limiting Behavior Limiting Behavior of a Positive Irreducible Chainof a Positive Irreducible Chain
j = fraction of time at state j
j = fraction of expected time at state j
average cost cj for each visit at state j
random i.i.d. Cj for each visit at state j
for aperiodic chain:
1
0lim
k
n
Xk
j jn j
E cc
n
1
0lim ( )
k
n
Xk
j jn j
E CE C
n
0lim ( | )n jn
P X j X
28
Limiting Behavior Limiting Behavior of a Positive Irreducible Chainof a Positive Irreducible Chain
1 = 301/801, 2 = 400/801, 3 = 100/801
profit per state: c1 = 4, c2 = 8, c3 = -2
average profit
1 2 3
0.25
0.99
1
0.75
0.01
0 1 0
0.75 0 0.25
0.01 0.99 0
P 301 400 100801 801 801
4201801
(4) (8) (2)
29
Limiting Behavior Limiting Behavior of a Positive Irreducible Chainof a Positive Irreducible Chain
1 = 301/801, 2 = 400/801, 3 = 100/801
C1 ~ unif[0, 8], C2 ~ Geo(1/8), C3 = -4 w.p. 0.5; and = 0 w.p. 0.5 E(C1) = 4, E(C2) = 8, E(C3) = -2
average profit
1 2 3
0.25
0.99
1
0.75
0.01
0 1 0
0.75 0 0.25
0.01 0.99 0
P
301 400 100801 801 801
4201801
(4) (8) (2)
30
Different Interpretations of Different Interpretations of
balance equations: balance of rates
total rate into a group of states = total rate out of a group of states
{0}: 0 = q1
{0, 1}: p1 = q2
{0, 1, …, n}: pn = qn+1 , n 1
0 1
q
2 3
p
q
p
q
p1
…q
0j i ij
ip
31
Example: Condition for the Example: Condition for the Following Chain to be PositiveFollowing Chain to be Positive
{0}: 0 = q1
{0, 1}: p1 = q2 1
{0, 1, …, n}: pn = qn+1 , n 11
positive {j} exists
0 + 1 + 2 + … = 1 has solution
.
p < q 0 1
q
2 3
p
q
p
q
p1
…q
11 0q
12 1 0
p pq q q
11
0
npn q q
21 1 1
0 1 ... 1p pq q q q q
32
Example 4.24 of RossExample 4.24 of Ross four-state production process states {1, 2, 3, 4} up states {3, 4}, down states {1, 2} find E(up time) & E(down time)
time
1
state2
3
4
time
downstate
up
33
Example 4.24 of RossExample 4.24 of Ross
1 = 3/16, 2 = 1/4, 3 = 7/24, 4 = 13/48
how to find E(up time) E(down time)
0.25 0.25 0.5 0
0 0.25 0.5 0.25
0.25 0.25 0.25 0.25
0.25 0.25 0 0.5
P
34
Example 4.24 of RossExample 4.24 of Ross
fraction of up time = 3 + 4 =
rate of turning from up to down
= (p31+p32)3 + (p41+p42) 4
= rate of turning from down to up
= p131 + (p23+p24) 2
=
0.25 0.25 0.5 0
0 0.25 0.5 0.25
0.25 0.25 0.25 0.25
0.25 0.25 0 0.5
P
(up time)
(up time) (up time)
E
E E
1
(up time) (up time)E E