Introduction Stochastic processes SIS model DTMC CTMC
Stochastic modelling of epidemic spread
Julien Arino
Department of Mathematics
University of Manitoba
Winnipeg
Julien [email protected]
19 May 2012
Introduction Stochastic processes SIS model DTMC CTMC
1 Introduction
2 Stochastic processes
3 The SIS model used as an example
4 DTMC
5 CTMC
Introduction Stochastic processes SIS model DTMC CTMC
1 Introduction
2 Stochastic processes
3 The SIS model used as an example
4 DTMC
5 CTMC
Introduction Stochastic processes SIS model DTMC CTMC
Deterministic SIR
Use the standard KMK as in Fred’s lectures:
S
0 = ��SII
0 = �SI � ↵I
R
0 = ↵I
Basic reproduction number
R0
=�
↵S
0
• If R0
< 1, no outbreak
• If R0
> 1, outbreak
Use ↵ = 1/4, N = S + I + R = 100, I (0) = 1, R0
= {0.5, 1.5}and � so that this is true
Introduction Stochastic processes SIS model DTMC CTMC
R0
= 0.5 R0
= 1.5
0 50 100 1500
10
20
30
40
50
60
70
80
90
100
Time (days)
SIR
0 50 100 1500
10
20
30
40
50
60
70
80
90
100
Time (days)
SIR
0 50 100 1500
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Time (days)
Num
ber i
nfec
tious
0 50 100 1500
1
2
3
4
5
6
7
8
Time (days)
Num
ber i
nfec
tious
Introduction Stochastic processes SIS model DTMC CTMC
Stochastic DTMC SIR
Consider S(t) and I (t) (R(t) = N � S(t)� I (t)). Definetransition probabilities
p
(s
0,i 0),(s,i)(�t) =
P{(S(t +�t), I (t +�t)) = (s + s
0, i + i
0)|(S(t), I (t)) = (s, i)}
by
p
(s
0,i 0),(s,i)(�t) =
8>>>><
>>>>:
�si�t (s 0, i 0) = (�1, 1)↵i�t (s 0, i 0) = (0,�1)1� [�si + ↵i ]�t (s 0, i 0) = (0, 0)
0 otherwise
Parameters as previously and �t = 0.01
Introduction Stochastic processes SIS model DTMC CTMC
R0
= 0.5 R0
= 1.5
0 50 100 1500
1
2
3
4
5
6
7
Time (days)
Num
ber i
nfec
tious
0 50 100 1500
5
10
15
20
25
Time (days)
Num
ber i
nfec
tious
0 50 100 1500
0.2
0.4
0.6
0.8
1
1.2
1.4
Time (days)
Num
ber i
nfec
tious
0 50 100 1500
0.5
1
1.5
2
2.5
3
3.5
Time (days)
Num
ber i
nfec
tious
Introduction Stochastic processes SIS model DTMC CTMC
Number of outbreaks
0 10 20 30 40 50 60 70 80 90 1000
100
200
300
400
500
600
700
800
900
1000
Value of I0
Num
ber o
f sim
ulat
ions
with
an
outb
reak
(/10
00)
R0=0.5
R0=1.5
outbreak: 9t s.t. I (t) > I (0)
Introduction Stochastic processes SIS model DTMC CTMC
Why stochastic models?
Important to choose right type of model for given application
Deterministic models
• Mathematically easier
• Perfect reproducibility
• For given parameter set,one sim su�ces
Stochastic models
• Often harder
• Each realization di↵erent
• Needs many simulations
In early stage of epidemic, very few infective individuals:
• Deterministic systems: behaviour entirely governed byparameters
• Stochastic systems: allows “chance” to play a role
Introduction Stochastic processes SIS model DTMC CTMC
In the context of the SIR model
Deterministic
• R0
strict threshold
• R0
< 1 ) I (t)! 0 monotonically
• R0
> 1 ) I (t)! 0 after a bump (outbreak)
Stochastic
• R0
threshold for mean
• R0
< 1 ) I (t)! 0 monotonically (roughly) on average butsome realizations have outbreaks
• R0
> 1 ) I (t)! 0 after a bump on average (outbreak) butsome realizations have no outbreaks
Introduction Stochastic processes SIS model DTMC CTMC
1 Introduction
2 Stochastic processes
3 The SIS model used as an example
4 DTMC
5 CTMC
Introduction Stochastic processes SIS model DTMC CTMC
Theoretical setting of stochastic processes
Stochastic processes form a well defined area of probability, asubset of measure theory, itself a well established area ofmathematics
Strong theoretical content
Very briefly presented here for completeness, will not be used inwhat follows
Used here: A. Klemke, Probability Theory, Springer 2008
DON’T PANIC!!!
Introduction Stochastic processes SIS model DTMC CTMC
�-algebras
Definition 1 (�-algebra)
Let ⌦ 6= ; be a set and A ⇢ 2⌦, the set of all subsets of ⌦, be aclass of subsets of ⌦. A is called a �-algebra if:
1 ⌦ 2 A2 A is closed under complements, i.e., Ac := ⌦ \ A 2 A for any
A 2 A3 A is closed under countable unions, i.e., [1
n=1
A
n
2 A for anychoice of countably many sets A
1
,A2
, . . . 2 A
⌦ is the space of all elementary events and A the system ofobservable events
Introduction Stochastic processes SIS model DTMC CTMC
Probability space
Definition 21 A pair (⌦,A), with ⌦ a nonempty set and A ⇢ 2⌦ a
�-algebra is a measurable space, with sets A 2 Ameasurable sets. If ⌦ is at most countably infinite and ifA = 2⌦, then the measurable space (⌦, 2⌦) is discrete
2 A triple (⌦,A, µ) is a measure space if (⌦,A) is ameasurable space and µ is a measure on A
3 If in addition, µ(⌦) = 1, then (⌦,A, µ) is a probability space
and the sets A 2 A are events. µ is then usually denoted P(or Prob)
Introduction Stochastic processes SIS model DTMC CTMC
Topological spaces and Borel �-algebras
Definition 3 (Topology)
Let ⌦ 6= ; be an arbitrary set. A class of sets ⌧ ⇢ ⌦ is a topology
on ⌦ if:
1 ;,⌦ 2 ⌧
2
A \ B 2 ⌧ for any A,B 2 ⌧
3 ([A2FA) 2 ⌧ for any F 2 ⌧
(⌦, ⌧) is a topological space, sets A 2 ⌧ are open and A 2 ⌦with A
c 2 ⌧ are closed
Definition 4 (Borel �-algebra)
Let (⌦, ⌧) be a topological space. The �-algebra B(⌦, ⌧)generated by the open sets is the Borel �-algebra on ⌦, withelements A 2 B(⌦, ⌧) the Borel sets
Introduction Stochastic processes SIS model DTMC CTMC
Polish spaces
Definition 5A separable topological space whose topology is induced by acomplete metric is a Polish space
Rd , Zd , RN, (C ([0, 1]), k · k1) are Polish spaces
Introduction Stochastic processes SIS model DTMC CTMC
Stochastic process
Definition 6 (Stochastic process)
Let (⌦,F ,P) be a probability space, (E , ⌧) be a Polish space withBorel �-algebra E and T ⇢ R be arbitrary (typically, T = N,T = Z, T = R
+
or T an interval).A family of random variables X = (X
t
, t 2 T ) on (⌦,F ,P) withvalues in (E , E) is called a stochastic process with index set (ortime set) T and range E .
Introduction Stochastic processes SIS model DTMC CTMC
OKAY, DONE WITH HARD THEORY!!!
Introduction Stochastic processes SIS model DTMC CTMC
Types of processes
So a stochastic process is a collection of random variables (r.v.)
{X (t,!), t 2 T ,! 2 ⌦}
The index set T and r.v. can be discrete or continuous
•Discrete time Markov chain (DTMC) has T and X
discrete (e.g., T = N and X (t) 2 N)•Continuous time Markov chain (CTMC) has T continuousand X discrete (e.g., T = R
+
and X (t) 2 N)
In this course, we focus on these two. Also exist Stochasticdi↵erential equations (SDE), with both T and X (t) continuous
Introduction Stochastic processes SIS model DTMC CTMC
Markov property
Definition 7 (Markov property)
Let E be a Polish space with Borel �-algebra B(E ), T ⇢ R and(X
t
)t2T an E -valued stochastic process. Assume that
(Ft
)t2T = �(X ) is the filtration generated by X . X has the
Markov property if for every A 2 B(E ) and all s, t 2 T withs t,
P[Xt
2 A|Fs
] = P[Xt
2 A|Xs
]
Theorem 8 (The usable one)
If E is countable then X has the Markov property if for all n 2 N,all s
1
< · · · < s
n
< t and all i
1
, . . . , in
, i 2 E with
P[Xs
1
= i
1
, . . . ,Xs
n
= i
n
] > 0, there holds
P[Xt
= i |Xs
1
= i
1
, . . . ,Xs
n
= i
n
] = P[Xt
= i |Xs
n
= i
n
]
Introduction Stochastic processes SIS model DTMC CTMC
1 Introduction
2 Stochastic processes
3 The SIS model used as an example
4 DTMC
5 CTMC
Introduction Stochastic processes SIS model DTMC CTMC
SIS model
We consider an SIS model with demography. In ODE,
S
0 = b � dS � �SI + �I
I
0 = �SI � (� + d)I
Total populationN
0 = b � dN
asymptotically constant (N(t)! b/d =: N⇤ as t !1). Canwork with asymptotically autonomous system where N = N
⇤ (andS = N
⇤ � I )
Introduction Stochastic processes SIS model DTMC CTMC
S
0 = b � �SI � dS + �I
I
0 = �SI � (� + d)I
DFE: (S , I ) = (b/d , 0). EEP: (S , I ) =⇣�+d
� , bd
� �+d
�
⌘
VdD&W: F = D
I
[�S ]|DFE
= �N⇤, V = D
I
[(� + d)I ]|DFE
= � + d
) R0
= ⇢(FV�1) =�
� + d
N
⇤
• If R0
< 1, limt!1(S(t), I (t)) = (N⇤, 0)
• If R0
> 1, limt!1(S(t), I (t)) =
⇣N
⇤
R0
,N⇤ � N
⇤
R0
⌘
Introduction Stochastic processes SIS model DTMC CTMC
Reduction to one equation
As N is asymptotically constant, we can write S = N
⇤ � I and soonly need
I
0 = �(N⇤ � I )I � (� + d)I
We can then reconstruct dynamics of S from that of I
Introduction Stochastic processes SIS model DTMC CTMC
1 Introduction
2 Stochastic processes
3 The SIS model used as an example
4 DTMC
5 CTMC
Introduction Stochastic processes SIS model DTMC CTMC
Discrete time Markov chain
Definition 9 (DTMC – Simple definition)
An experiment with finite number of possible outcomes S1
, . . . ,SN
is repeated. The sequence of outcomes is a discrete time Markov
chain if there is a set of N2 numbers {pij
} such that theconditional probability of outcome S
j
on any experiment givenoutcome S
i
on the previous experiment is pij
, i.e., for 1 i , j N,t = 1, . . .,
p
ji
= Pr(Sj
on experiment t + 1|Si
on experiment t).
The outcomes S1
, . . . ,SN
are the states, and the p
ij
are thetransition probabilities. The matrix P = [p
ij
] is the transition
matrix.
Introduction Stochastic processes SIS model DTMC CTMC
Homogeneity
One major distinction:
• If pij
does not depend on t, the chain is homogeneous
• If pij
(t), i.e, transition probabilities depend on t, the chain isnonhomogeneous
Introduction Stochastic processes SIS model DTMC CTMC
Markov chains operate on two level:
• Description of probability that the system is in a given state
• Description of individual realizations of the process
Because we assume the Markov property, we need only describehow system switches (transitions) from one state to the next
We say that the system has no memory, the next state dependsonly on the current one
Introduction Stochastic processes SIS model DTMC CTMC
States
The states are here the di↵erent values of I . I can take integervalues from 0 to N
The chain links these states
I=0 I=1 I=2 I=3 I=N-1 I=N
Recoveryor Death
Recoveryor Death
Recoveryor Death
Recoveryor Death
Infection Infection Infection
Introduction Stochastic processes SIS model DTMC CTMC
Probability vector
Letp
i
(t) = P(I (t) = i)
p(t) =
0
BBB@
p
0
(t)p
1
(t)...
p
N
(t)
1
CCCA
is the probability distribution
We must haveP
i
p
i
(t) = 1
Initial condition at time t = 0: p(0)
Introduction Stochastic processes SIS model DTMC CTMC
Transition probabilities
We then need description of probability of transition
In general, suppose that Si
is the current state, then one ofS1
, . . . ,SN
must be the next state. If
p
ji
= P{X (t + 1) = Sj
|X (t) = Si
}
then we must have
p
i1
+ p
i2
+ · · ·+ p
iN
= 1, 1 i N
Some of the p
ij
can be zero
Introduction Stochastic processes SIS model DTMC CTMC
Transition matrixThe matrix
P =
0
BB@
p
11
p
12
· · · p
1N
p
21
p
22
· · · p
2N
p
N1
p
N2
· · · p
NN
1
CCA
has
• nonnegative entries, pij
� 0
• entries less than 1, pij
1
• column sum 1, which we write
NX
i=1
p
ij
= 1, j = 1, . . . ,N
or, using the notation 1l = (1, . . . , 1)T ,
1l
T
P = 1l
T
Introduction Stochastic processes SIS model DTMC CTMC
In matrix form
p(t + 1) = Pp(t), t = 1, 2, 3, . . .
where p(t) = (p1
(t), p2
(t), . . . , pN
(t))T is a (column) probabilityvector and P = (p
ij
) is a N ⇥ N transition matrix,
P =
0
BB@
p
11
p
12
· · · p
1N
p
21
p
22
· · · p
2N
p
N1
p
N2
· · · p
NN
1
CCA
Introduction Stochastic processes SIS model DTMC CTMC
Stochastic matrices
Definition 10 (Stochastic matrix)
The nonnegative N ⇥N matrix M is stochastic ifP
N
i=1
a
ij
= 1 forall j = 1, 2, . . . ,N. In other words, 1lTM = 1l
T
Theorem 11Let M be a stochastic matrix M. Then the spectral radius
⇢(M) 1, i.e., all eigenvalues � of M are such that |�| 1.Furthermore, � = 1 is an eigenvalue of M
M has all column sums 1 , 1l
T
M = 1l
T
� e.value of M with associated left e.vector v , v
T
M = �vT
) 1l
T
M = 1l
T : left e.vector 1lT and e.value 1..
Introduction Stochastic processes SIS model DTMC CTMC
Long “time” behaviorLet p(0) be the initial distribution (column) vector. Then
p(1) = Pp(0)
p(2) = Pp(1)
= P(Pp(0))
= P
2
p(0)
Iterating, we get that for any t 2 N,
p(t) = P
t
p(0)
Therefore,
limt!+1
p(t) = limt!+1
P
t
p(0) =
✓lim
t!+1P
t
◆p(0)
if this limit exists (it does if P is constant and chain is regular)
Introduction Stochastic processes SIS model DTMC CTMC
Regular Markov chainRegular Markov chains not covered here, as most chains in thedemographic context are absorbing. For information, though..
Definition 12 (Regular Markov chain)
A regular Markov chain is one in which P
k is positive for someinteger k > 0, i.e., Pk has only positive entries, no zero entries.
Theorem 13If P is the transition matrix of a regular Markov chain, then
1
the powers P
t
approach a stochastic matrix W ,
2
each column of W is the same (column) vector
w = (w1
, . . . ,wN
)T ,
3
the components of w are positive.
So if the Markov chain is regular,
limt!+1
p(t) = Wp(0)
Introduction Stochastic processes SIS model DTMC CTMC
Interesting properties of matrices
Definition 14 (Irreducibility)
The square nonnegative matrix M is reducible if there exists apermutation matrix P such that PT
MP is block triangular. If M isnot reducible, then it is irreducible
Definition 15 (Primitivity)
A nonnegative matrix M is primitive if and only if there is aninteger k > 0 such that Mk is positive
Introduction Stochastic processes SIS model DTMC CTMC
Directed graphs (digraphs)
Definition 16 (Digraph)
A directed graph (or digraph) G = (V,A) consists in a set V ofvertices and a set A of arcs linking these vertices
1 2
3
5
4
Introduction Stochastic processes SIS model DTMC CTMC
Connecting graphs and matrices
Adjacency matrix M has mij
= 1 if arc from vertex j to vertex i ,0 otherwise
1 2
3
5
4
) M =
0
BBBB@
0 1 0 0 01 0 0 0 01 0 0 1 00 1 0 1 00 0 0 0 0
1
CCCCA
Column of zeros (except maybe diagonal entry): vertex is a sink
(nothing leaves)
Introduction Stochastic processes SIS model DTMC CTMC
Paths of length exactly 2
Given adjacency matrix M, M2 has number of paths of lengthexactly two between vertices
1 2
3
5
4
) M
2 =
0
BBBB@
1 0 0 0 00 1 0 0 00 2 0 1 00 0 0 1 00 0 0 0 0
1
CCCCA
Write m
k
ij
the (i , j) entry in M
k
m
2
32
= 2 since 2! 1! 3 and 2! 4! 3..
Introduction Stochastic processes SIS model DTMC CTMC
Connecting matrices, graphs and regular chains
Theorem 17A matrix M is irreducible i↵ the associated connection graph is
strongly connected, i.e., there is a path between any pair (i , j) ofvertices
Irreducibility: 8i , j , 9k , mk
ij
> 0
Theorem 18A matrix M is primitive if it is irreducible and there is at least one
positive entry on the diagonal of M
Primitivity: 9k , 8i , j , mk
ij
> 0
Theorem 19Markov chain regular , transition matrix P primitive
Introduction Stochastic processes SIS model DTMC CTMC
Strong connectedness (1)
4
5
7
8
2
6
3
1
Not strongly connected
Introduction Stochastic processes SIS model DTMC CTMC
Strong connectedness (2)
4
5
7
8
2
6
3
1
Strongly connected
Introduction Stochastic processes SIS model DTMC CTMC
Strong components
4
5
7
8
2
6
3
1
Strong component 1 Strong component 2
Introduction Stochastic processes SIS model DTMC CTMC
Absorbing states, absorbing chains
Definition 20A state S
i
in a Markov chain is absorbing if whenever it occurs onthe n
th generation of the experiment, it then occurs on everysubsequent step. In other words, S
i
is absorbing if pii
= 1 andp
ij
= 0 for i 6= j
Definition 21A Markov chain is said to be absorbing if it has at least oneabsorbing state, and if from every state it is possible to go to anabsorbing state
In an absorbing Markov chain, a state that is not absorbing iscalled transient
Introduction Stochastic processes SIS model DTMC CTMC
Absorbing strong components
4
5
7
8
2
6
3
1
Absorbingstrong component Transient states
Here, we focus on absorbing strong components containing onlyone vertex
Introduction Stochastic processes SIS model DTMC CTMC
Some questions on absorbing chains
1 Does the process eventually reach an absorbing state?
2 Average number of times spent in a transient state, if startingin a transient state?
3 Average number of steps before entering an absorbing state?
4 Probability of being absorbed by a given absorbing state,when there are more than one, when starting in a giventransient state?
Introduction Stochastic processes SIS model DTMC CTMC
Reaching an absorbing state
Answer to question 1:
Theorem 22In an absorbing Markov chain, the probability of reaching an
absorbing state is 1
Introduction Stochastic processes SIS model DTMC CTMC
Standard form of the transition matrix
For an absorbing chain with k absorbing states and N � k
transient states, the transition matrix can be written as
P =
✓Ik
0
R Q
◆
with following meaning,
Absorbing states Transient statesAbsorbing states I
k
0
Transient states R Q
with Ik
the k ⇥ k identity matrix, 0 an k ⇥ (N � k) matrix of zeros,R an (N � k)⇥ k matrix and Q an (N � k)⇥ (N � k) matrix.
Introduction Stochastic processes SIS model DTMC CTMC
The matrix IN�k
� Q is invertible. Let
•F = (I
N�k
� Q)�1 be the fundamental matrix of theMarkov chain
•T
i
be the sum of the entries on row i of F
•B = FR
Answers to our remaining questions:
2
F
ij
is the average number of times the process is in the jthtransient state if it starts in the ith transient state
3
T
i
is the average number of steps before the process enters anabsorbing state if it starts in the ith transient state
4
B
ij
is the probability of eventually entering the jth absorbingstate if the process starts in the ith transient state
Introduction Stochastic processes SIS model DTMC CTMC
Back to the SIS model
ODE:I
0 = �(N⇤ � I )I � (� + d)I
This equation has 3 components:
1 �(N⇤ � I )I new infection
2 �I recovery
3
dI death
In the DTMC, want to know what transitions mean and what theirprobabilities are
I=0 I=1 I=2 I=3 I=N-1 I=N
Recoveryor Death
Recoveryor Death
Recoveryor Death
Recoveryor Death
Infection Infection Infection
Introduction Stochastic processes SIS model DTMC CTMC
Consider small amount of time �t, su�ciently small that only oneevent occurs: new infection, loss of infectious individual, nothing..Let
B(i) := �(N⇤ � i)i D(i) := (� + d)i
be rates of “birth” and “death” of infectious. Then
p
ji
(�t) = P {I (t +�) = j |I (t) = i}
=
8>>>><
>>>>:
B(i)�t j = i + 1
D(i)�t j = i � 1
1� [B(i) + D(i)]�t j = i
0 otherwise
I=0 I=1 I=2 I=3 I=N-1 I=N
Recoveryor Death
Recoveryor Death
Recoveryor Death
Recoveryor Death
Infection Infection Infectionβ(N−1)Δ t β(N−2)2Δ t β(N−1)Δ t
(γ+d )Δ t (γ+d )2Δt (γ+d )3Δ t (γ+d )N Δ t
Introduction Stochastic processes SIS model DTMC CTMC
Transition matrix
0
BBBBB@
1 D(1)�t0 1� (B(1) + D(1))�t0 B(1)�t...
. . .D(N)�t
0 0 0 · · · B(N � 1)�t 1� D(N)�t
1
CCCCCA
Note that indices in the matrix range from 0 to N here
Introduction Stochastic processes SIS model DTMC CTMC
Properties
p
0
is an absorbing state
For any initial distribution p(0) = (p0
(t), p1
(0), . . . , pN
(0)),
limt!1
p(t) = (1, 0, . . . , 0)T limt!1
p
0
(t) = 1
Exists quasi-stationary distribution conditioned on non-extinction
q
i
(t) =p
i
(t)
1� p
0
(t), i = 1, . . . ,N
Introduction Stochastic processes SIS model DTMC CTMC
Consequence of absorption
We have
limt!1
p(t) = (1, 0, . . . , 0)T limt!1
p
0
(t) = 1
but time to absorption can be exponential
Introduction Stochastic processes SIS model DTMC CTMC
1 Introduction
2 Stochastic processes
3 The SIS model used as an example
4 DTMC
5 CTMC
Introduction Stochastic processes SIS model DTMC CTMC
Several ways to formulate CTMC’s
A continuous time Markov chain can be formulated in terms of
• infinitesimal transition probabilities
• branching process
• time to next event
Here, time is in R+
Introduction Stochastic processes SIS model DTMC CTMC
For small �t,
p
ji
(�t) = P {I (t +�) = j |I (t) = i}
=
8>>>><
>>>>:
B(i)�t + o(�t) j = i + 1
D(i)�t + o(�t) j = i � 1
1� [B(i) + D(i)]�t + o(�t) j = i
o(�t) otherwise
with o(�t)! 0 as �t ! 0
Introduction Stochastic processes SIS model DTMC CTMC
Forward Kolmogorov equations
Assume we know I (0) = k . Then
p
i
(t +�t) = p
i�1
(t)B(i � 1)�t + p
i+1
(t)D(i + 1)�t
+ p
i
(t)[1� (B(i) + D(i))�t] + o(�t)
Compute (pi
(t +�t)� p
i
(t))/�t and take lim�t!0
, giving
d
dt
p
0
= p
1
D(1)
d
dt
p
i
= p
i�1
B(i � 1) + p
i+1
D(i + 1)� p
i
[B(i) + D(i)] i = 1, . . . ,N
Forward Kolmogorov equations associated to the CTMC
Introduction Stochastic processes SIS model DTMC CTMC
In vector form
Write previous system as
p
0 = Qp
with
Q =
0
BBBBBB@
0 D(1) 0 · · · 00 �(B(1) + D(1)) D(2) · · · 00 B(1) �(B(2) + D(2)) · · · 0
D(N)�D(N)
1
CCCCCCA
Q generator matrix. Of course,
p(t) = e
Qt
p(0)
Introduction Stochastic processes SIS model DTMC CTMC
Linking DTMC and CTMC for small �t
DTMC:p(t +�t) = P(�t)p(t)
for transition matrix P(�t). Let �t ! 0, obtain Kolmogorovequations for CTMC
d
dt
p = Qp
where
Q = lim�t!0
P(�t)� I�t
= P
0(0)
Introduction Stochastic processes SIS model DTMC CTMC
Going ODE to CTMC
Easy to convert compartmental ODE model to CTMC!!
Introduction Stochastic processes SIS model DTMC CTMC
The general setting: flow diagram
S Ib
βSI
γ I
dS dI
Introduction Stochastic processes SIS model DTMC CTMC
ODE approach
S Ib
βSI
γ I
dS dI
Focus on compartments
Sb
βSI
γ I
dS
I
βSI
γ I
dI
Introduction Stochastic processes SIS model DTMC CTMC
CTMC approach
S Ib
βSI
γ I
dS dI
Focus on arrows (transitions)
S Ib
βSI
γ I
dS dI
Introduction Stochastic processes SIS model DTMC CTMC
Focus on transitions
S Ib
βSI
γ I
dS dI
•b: S ! S + 1 birth
•dS : S ! S � 1 death of susceptible
• �SI : S ! S � 1, I ! I + 1 new infection
• �I : S ! S + 1, I ! I � 1 recovery
•dI : I ! I � 1 death of infectious
Will use S = N
⇤ � I and omit S dynamics
Introduction Stochastic processes SIS model DTMC CTMC
Gillespie’s algorithm (SIS model)
1: Set t t
0
and I (t) I (t0
)2: while t t
f
do
3: ⇠t
�(N⇤ � i)i + �i + di
4: Draw ⌧t
from T s E(⇠t
)5: v [�(N⇤ � i)i/⇠
t
,�(N⇤ � i)i/⇠t
+ �i/⇠t
, 1]6: Draw ⇣
t
from U([0, 1])7: Find pos such that ⇣
t
� v(pos)8: switch (pos)9: case 1: New infection, I (t + ⌧
t
) = I (t) + 110: case 2: End of infectious period, I (t + ⌧
t
) = I (t)� 111: case 3: Death, I (t + ⌧
t
) = I (t)� 112: end switch
13: t t + ⌧t
14: end while
Introduction Stochastic processes SIS model DTMC CTMC
IMPORTANT: in Gillespie’s algorithm, we do not consider pii
, theevent “nothing happens”
Introduction Stochastic processes SIS model DTMC CTMC
Drawing at random from an exponential distributionWant ⌧
t
from T s E(⇠t
), i.e., T has probability density function
f (x , ⇠t
) = ⇠t
e
�⇠t
x
1
x�0
Use cumulative distribution function F (x , ⇠t
) =Rx
�1 f (s, ⇠t
) ds
F (x , ⇠t
) = (1� e
�⇠t
x)1x�0
which has values in [0, 1]. So draw ⇣ from U([0, 1]) and solveF (x , ⇠
t
) = ⇣ for x :
F (x , ⇠t
) = ⇣ , 1� e
�⇠t
x = ⇣
, e
�⇠t
x = 1� ⇣
, ⇠t
x = � ln(1� ⇣)
, x =� ln(1� ⇣)
⇠t