'
lim ( )
lim ( ) lim ( ) ( )
0
j ijt
ij kj ik j ijk jtt
kj k j jk j
kj k j j j j kj kk j k j
P P t
P t q P t v P t
q P v P
q P v P v P q P
Limiting probabilities
The limiting probabilities can be obtained by solving the following
system of equations:
0
1
kj k j jk j
jj
q P v P
P
rate at which the process leaves state
rate at which the process enters state
rate out of state rate into state
j j
kj kk j
j j kj kk j
v P j
q P j
v P q P
j j
The limiting probabilities Pj exist if
(a) all states of the Markov chain communicate (i.e., starting in state i, there is a positive probability of ever being in state j, for all i, j and
(b) the Markov is positive recurrent (i.e, starting in any state, the mean time to return to that state is finite).
When do the limiting probabilities exist?
The M/M/1 queue
10 2
3
0 1
1 2 0
State rate out of state rate into state
0
1 ( )
2
j j
P P
P P P
2 3 1
1 1
( )
1 ( ) n n n
P P P
n P P P
The M/M/1 queue
0 1
1 2
2 3
By adding to each equation the equation preceding it, we obtain
0
1
2
P P
P P
P P
n
11 n nP P
The M/M/1 queue
0
1 0
22 0
33 3
Solving in terms of yields
0 ( / )
1 ( / )
2 ( / )
1 ( /n
P
P P
P P
P P
n P
0)nP
The M/M/1 queue
0
0 0 1
0
1
Using the fact that 1,we obtain
( / ) 1
11
1 ( / )
/ (1 / )
Note that we must have / 1.
nn
n
n
n
n
n
n
P
P P
P
P
The expected number in the system
0 0
20
( ) / (1 / )
(use the fact )(1 )
n
nn n
n
n
E n nP n
xnx
x
The birth and death process
0 0 1 1
1 1 1 2 2 0 0
State rate out of state rate into state
0
1 ( )
2
j j
P P
P P P
2 2 2 3 3 1 1
1 1 1 1
( )
1 ( )n n n n n n n
P P P
n P P P
10 2
3
0 0 1 1
1 1 2 2
2 2 3 3
By adding to each equation the equation preceding it, we obtain
0
1
2
P P
P P
P P
1 1 1 n n n nn P P
0
1 0 1 0
2 1 2 1 1 0 2 1 0
3 2 3 2 2 1 0 3 2 1 0
Solving in terms of yields
0 ( / )
1 ( / ) ( / )
2 ( / ) ( / )
P
P P
P P P
P P P
1 1 1 2 1 0 1 2 1 0
1 ( / ) ( ... / ... )n n n n n n n nn P P P
0
1 2 1 00 0 1
1 2 1
01 2 1 0
11 2 1
1 2 1 0
1 2 1 01 2 1
1 2 1
Using the fact that 1,we obtain
...1
...
1...
1...
...
...... 1
...
Note that
nn
n n
nn n
n nn
n n
n nn
n nn n n
n n
P
P P
P
P
1 2 1 0
11 2 1
... we must have .
...n n
nn n
011 2 1 0
1 11 2 1
The / / queue is a birth and death process with
if
if
1 1...
1 1... ! !
n
n
n nkn n
n n n k nn n kn n
M M k
n n k
k n k
P
n k k
The M/M/m queue
0
0
( / ),
!
( / )
!
The stability condition is / 1 .
n
n n
n k
P n knP
P n kk k
k
The M/M/m queue
020
The expected number of customers in a / / queue
( / )( ) / where / .
!(1 )
k
nn
M M k
E n nP P kk
The M/M/m queue
A machine repair model
A system with M machines and one repairman. The time between machine is exponentially distributed with mean 1/. Repair times are also exponentially distributed with mean 1/. What is the average number of working machines? What is the fraction of time each machine is in use?
The system is in state if there are failed machines
0,1,2,...,
n n
n M
The machine repairman problem
The system is in state if there are failed machines
0,1,2,...,
The corresponding process is a birth and death process with
( ) if
0 if n
n n
n M
M n n M
n M
, 1n n
The machine repairman problem
The system is in state if there are failed machines
0,1,2,...,
The corresponding process is a birth and death process with
( ) if
0 if n
n n
n M
M n n M
n M
01 2 1 0
111 2 1
1
, 1
1 1... ( 1) ...( 1)
11...
1
1 ( / ) !/( )!
n
MM n nnnn
n n
M n
n
n
PM M M n
M M n
The machine repairman problem
1 2 1 00
1 2 1
1
...
...
( / ) !/( )! = , 0,1,...,
1 ( / ) !/( )!
n nn
n n
n
M n
n
P P
M M nn M
M M n
The machine repairman problem
1 2 1 00
1 2 1
1
1
1
1
...
...
( / ) !/( )! = , 0,1,...,
1 ( / ) !/( )!
( / ) !/( )!( )
1 ( / ) !/( )!
n nn
n n
n
M n
n
M nM n
n Mn n
n
P P
M M nn M
M M n
n M M nE n nP
M M n
The machine repairman problem
0{machine is working} {machine is working | not working}
M
nnP P n P
The machine repairman problem
0
0
0
{machine is working} {machine is working | not working}
=
1 1 ( ) /
M
nn
M
nn
M
nn
P P n P
M nP
M
nPE n M
M
The machine repairman problem
The automated teller machine (ATM) problem
Customers arrive to an ATM according to a Poisson process with rate . If the customer finds more than N other customers at the machine, he/she does not wait and goes away. Machine transaction times are exponentially distributed with mean 1/. What is the probability that a customer goes away? What is the average number of customers at the ATM? If the machine earns $h per customer served, what is the average revenue the machine generates per unit time?
The M/M/1/N queue
0 0 1 1
1 1
State rate out of state rate into state
0
1 -1 ( )
1 n n n
j j
P P
n N P P P
N
1 N NP P
The M/M/1/N queue
0
0 10
1
( / )
1 /1
1 ( / )
( / ) (1 / )
1 ( / )
nn
N
n Nn
n
n N
P P
P P
P
The M/M/1/N queue
0
0 10
1
10 0
1
1
( / )
1 /1
1 ( / )
( / ) (1 / )
1 ( / )
1 /( ) ( / )
1 ( / )
[1 ( / ) ( 1)( / ) ]
( )[1 ( / ) ]
nn
N
n Nn
n
n N
nn Nn n
N N
N
P P
P P
P
E n nP n
N N
The M/M/1/N queue
1
( / ) (1 / )The probability that a customer goes away is
1 ( / )
The average number of customers at the ATM is ( )
The average revenue per unit time (revenue rate) is (1 )
N
N N
N
P
E n
P h
The production inventory problem
Consider a production system that manufacturers a single product. Production times are exponentially distributed with mean 1/. The production facility can produce ahead of demand by holding finished goods inventory. Orders from customers arrives according to a Poisson process with rate . If there is inventory on-hand, the order is satisfied immediately. Otherwise, the order is backordered. The production system incurs a holding cost $h per unit of held inventory per unit time and a backorder cost $b per unit backordered per unit time. The production system manages its finished goods inventory using a base-stock policy with base-stock level s.
The production inventory problem
• What is the expected inventory level?
• What is expected backorder level?
• What is the expected total cost?
• What is the optimal base-stock level?
I: level of finished goods inventoryB: number of backorders (backorder level)IO: inventory on order.
Three basic processes
Under a base-stock policy, the arrival of each customer order triggers the placement of an order with the production system
s = I + IO – B
s = E[I] + E[IO] – E[B]
Three basic processes
I and B cannot be positive at the same time
I = max(0, s - IO) = (s – IO)+
E[I] = E[(s – IO)+]
B = max(0, IO - s) = (IO - s)+
E[B] = E [(IO - s)+]
Three basic processes
The production system behaves like an M/M/1 queue, with IO corresponding to the number of customers in the system.
Pr( ) (1 )
[ ]1
nIO n
E IO
0
0
1
[ ] [max(0, )]
( ) (1 )
(1 )
(1 )
1
n
n s
n s
n
s n
n
s
E B E IO s
n s
n
n
Expected backorder level
1
[ ] ( ) [ ]1 1
s
E I s E IO E B s
Expected inventory level
1 1
( ) : expected cost (holding cost + backorder cost)
( ) [ ] [ ] ( )1 1 1
s s
z s
z s hE I bE B h s b
Expected cost
Optimal base-stock level
2 2 1 1
2 1 2 1
1
1
1
( 1) ( ) ( 1 ) ( ) 01 1 1 1 1 1
(1 ) ( ) 01 1 1 1
(1 )( ) 01
( ) 0
s s s s
s s s s
s
s
s
r r r r r rz s z s h s b h s b
r r r r r r
r r r rh b
r r r r
rh r h b
r
h r h b
hr
h b
Optimal base-stock level
1
ln1
ln
ln ln* 1
ln ln
lnIf we ignore the integrality of *, then * .
ln
s hr
h bhh bsrh hh b h bsr r
hh bs sr