Lesson 11Markov Chains

Math 20

October 15, 2007

Announcements

I Review Session (ML), 10/16, 7:30–9:30 Hall E

I Problem Set 4 is on the course web site. Due October 17

I Midterm I 10/18, Hall A 7–8:30pm

I OH: Mondays 1–2, Tuesdays 3–4, Wednesdays 1–3 (SC 323)

I Old exams and solutions on website

The Markov Dance

Divide the class into three groups A, B, and C .

Upon my signal:

I 1/3 of group A goes to group B, and 1/3 of group A goes togroup C .

I 1/4 of group B goes to group A, and 1/4 of group A goes togroup C .

I 1/2 of group C goes to group B.

The Markov Dance

Divide the class into three groups A, B, and C . Upon my signal:

I 1/3 of group A goes to group B, and 1/3 of group A goes togroup C .

I 1/4 of group B goes to group A, and 1/4 of group A goes togroup C .

I 1/2 of group C goes to group B.

The Markov Dance

Divide the class into three groups A, B, and C . Upon my signal:

I 1/3 of group A goes to group B, and 1/3 of group A goes togroup C .

I 1/4 of group B goes to group A, and 1/4 of group A goes togroup C .

I 1/2 of group C goes to group B.

The Markov Dance

Divide the class into three groups A, B, and C . Upon my signal:

I 1/3 of group A goes to group B, and 1/3 of group A goes togroup C .

I 1/4 of group B goes to group A, and 1/4 of group A goes togroup C .

I 1/2 of group C goes to group B.

Math 20 - October 15, 2007.GWBMonday, Oct 15, 2007

Page2of9

Another Example

Suppose on any given class day you wake up and decide whether tocome to class. If you went to class the time before, you’re 70%likely to go today, and if you skipped the last class, you’re 80%likely to go today.

Some questions you might ask are:

I If I go to class on Monday, how likely am I to go to class onFriday?

I Assuming the class is infinitely long (the horror!),approximately what portion of class will I attend?

Another Example

Suppose on any given class day you wake up and decide whether tocome to class. If you went to class the time before, you’re 70%likely to go today, and if you skipped the last class, you’re 80%likely to go today. Some questions you might ask are:

I If I go to class on Monday, how likely am I to go to class onFriday?

I Assuming the class is infinitely long (the horror!),approximately what portion of class will I attend?

Another Example

Suppose on any given class day you wake up and decide whether tocome to class. If you went to class the time before, you’re 70%likely to go today, and if you skipped the last class, you’re 80%likely to go today. Some questions you might ask are:

I If I go to class on Monday, how likely am I to go to class onFriday?

I Assuming the class is infinitely long (the horror!),approximately what portion of class will I attend?

Many times we are interested in the transition of somethingbetween certain “states” over discrete time steps. Examples are

I movement of people between regions

I states of the weather

I movement between positions on a Monopoly board

I your score in blackjack

DefinitionA Markov chain or Markov process is a process in which theprobability of the system being in a particular state at a givenobservation period depends only on its state at the immediatelypreceding observation period.

Many times we are interested in the transition of somethingbetween certain “states” over discrete time steps. Examples are

I movement of people between regions

I states of the weather

I movement between positions on a Monopoly board

I your score in blackjack

DefinitionA Markov chain or Markov process is a process in which theprobability of the system being in a particular state at a givenobservation period depends only on its state at the immediatelypreceding observation period.

Common questions about a Markov chain are:

I What is the probability of transitions from state to state overmultiple observations?

I Are there any “equilibria” in the process?

I Is there a long-term stability to the process?

DefinitionSuppose the system has n possible states. For each i and j , let tijbe the probability of switching from state j to state i . The matrixT whose ijth entry is tij is called the transition matrix.

Example

The transition matrix for the skipping class example is

T =

(0.7 0.80.3 0.2

)

DefinitionSuppose the system has n possible states. For each i and j , let tijbe the probability of switching from state j to state i . The matrixT whose ijth entry is tij is called the transition matrix.

Example

The transition matrix for the skipping class example is

T =

(0.7 0.80.3 0.2

)

Math 20 - October 15, 2007.GWBMonday, Oct 15, 2007

Page3of9

Math 20 - October 15, 2007.GWBMonday, Oct 15, 2007

Page4of9

The big idea about the transition matrix reflects an important factabout probabilities:

I All entries are nonnegative.

I The columns add up to one.

Such a matrix is called a stochastic matrix.

DefinitionThe state vector of a Markov process with n-states at time step kis the vector

x(k) =

p

(k)1

p(k)2...

p(k)n

where p

(k)j is the probability that the system is in state j at time

step k .

Example

Suppose we start out with 20 students in group A and 10 studentsin groups B and C . Then the initial state vector is

x(0) =

0.50.250.25

.

Math 20 - October 15, 2007.GWBMonday, Oct 15, 2007

Page5of9

DefinitionThe state vector of a Markov process with n-states at time step kis the vector

x(k) =

p

(k)1

p(k)2...

p(k)n

where p

(k)j is the probability that the system is in state j at time

step k .

Example

Suppose we start out with 20 students in group A and 10 studentsin groups B and C . Then the initial state vector is

x(0) =

0.50.250.25

.

DefinitionThe state vector of a Markov process with n-states at time step kis the vector

x(k) =

p

(k)1

p(k)2...

p(k)n

where p

(k)j is the probability that the system is in state j at time

step k .

Example

Suppose we start out with 20 students in group A and 10 studentsin groups B and C . Then the initial state vector is

x(0) =

0.50.250.25

.

Math 20 - October 15, 2007.GWBMonday, Oct 15, 2007

Page6of9

Example

Suppose after three weeks of class I am equally likely to come to

class or skip. Then my state vector would be x(10) =

(0.50.5

)The big idea about state vectors reflects an important fact aboutprobabilities:

I All entries are nonnegative.

I The entries add up to one.

Such a vector is called a probability vector.

Example

Suppose after three weeks of class I am equally likely to come to

class or skip. Then my state vector would be x(10) =

(0.50.5

)

The big idea about state vectors reflects an important fact aboutprobabilities:

I All entries are nonnegative.

I The entries add up to one.

Such a vector is called a probability vector.

Example

Suppose after three weeks of class I am equally likely to come to

class or skip. Then my state vector would be x(10) =

(0.50.5

)The big idea about state vectors reflects an important fact aboutprobabilities:

I All entries are nonnegative.

I The entries add up to one.

Such a vector is called a probability vector.

LemmaLet T be an n × n stochastic matrix and x an n × 1 probabilityvector. Then Tx is a probability vector.

Proof.We need to show that the entries of Tx add up to one. We have

n∑i=1

(Tx)i =n∑

i=1

n∑j=1

tijxj

=n∑

j=1

(n∑

i=1

tij

)xj

=n∑

j=1

1 · xj = 1

LemmaLet T be an n × n stochastic matrix and x an n × 1 probabilityvector. Then Tx is a probability vector.

Proof.We need to show that the entries of Tx add up to one. We have

n∑i=1

(Tx)i =n∑

i=1

n∑j=1

tijxj

=n∑

j=1

(n∑

i=1

tij

)xj

=n∑

j=1

1 · xj = 1

TheoremIf T is the transition matrix of a Markov process, then the statevector x(k+1) at the (k + 1)th observation period can bedetermined from the state vector x(k) at the kth observationperiod, as

x(k+1) = Tx(k)

This comes from an important idea in conditional probability:

P(state i at t = k + 1)

=n∑

j=1

P(move from state j to state i)P(state j at t = k)

That is, for each i ,

p(k+1)i =

n∑j=1

tijp(k)j

TheoremIf T is the transition matrix of a Markov process, then the statevector x(k+1) at the (k + 1)th observation period can bedetermined from the state vector x(k) at the kth observationperiod, as

x(k+1) = Tx(k)

This comes from an important idea in conditional probability:

P(state i at t = k + 1)

=n∑

j=1

P(move from state j to state i)P(state j at t = k)

That is, for each i ,

p(k+1)i =

n∑j=1

tijp(k)j

Illustration

Example

How does the probability of going to class on Wednesday dependon the probabilities of going to class on Monday?

go skip

go skip go skip

p(k)1

t11 t21

p(k)2

t12 t22

Monday

Wednesday

p(k+1)1 = t11p

(k)1 + t12p

(k)2

p(k+1)2 = t21p

(k)1 + t22p

(k)2

Example

If I go to class on Monday, what’s the probability I’ll go to class onFriday?

Solution

We have x(0) =

(10

). We want to know x(2). We have

x(2) = Tx(1) = T (Tx(0)) = T 2 = Tx(0)

=

(0.7 0.80.3 0.2

)2(10

)=

(0.7 0.80.3 0.2

)(0.70.3

)=

(0.730.27

)

Example

If I go to class on Monday, what’s the probability I’ll go to class onFriday?

Solution

We have x(0) =

(10

). We want to know x(2). We have

x(2) = Tx(1) = T (Tx(0)) = T 2 = Tx(0)

=

(0.7 0.80.3 0.2

)2(10

)=

(0.7 0.80.3 0.2

)(0.70.3

)=

(0.730.27

)

Let’s look at successive powers of the probability matrix. Do theyconverge? To what?

Let’s look at successive powers of the transition matrix in theMarkov Dance.

T =

0.333333 0.25 0.0.333333 0.5 0.50.333333 0.25 0.5

T 2 =

0.194444 0.208333 0.1250.444444 0.458333 0.50.361111 0.333333 0.375

T 3 =

0.175926 0.184028 0.1666670.467593 0.465278 0.4791670.356481 0.350694 0.354167

Let’s look at successive powers of the transition matrix in theMarkov Dance.

T =

0.333333 0.25 0.0.333333 0.5 0.50.333333 0.25 0.5

T 2 =

0.194444 0.208333 0.1250.444444 0.458333 0.50.361111 0.333333 0.375

T 3 =

0.175926 0.184028 0.1666670.467593 0.465278 0.4791670.356481 0.350694 0.354167

Let’s look at successive powers of the transition matrix in theMarkov Dance.

T =

0.333333 0.25 0.0.333333 0.5 0.50.333333 0.25 0.5

T 2 =

0.194444 0.208333 0.1250.444444 0.458333 0.50.361111 0.333333 0.375

T 3 =

0.175926 0.184028 0.1666670.467593 0.465278 0.4791670.356481 0.350694 0.354167

T 4 =

0.17554 0.177662 0.1753470.470679 0.469329 0.4722220.353781 0.353009 0.352431

T 5 =

0.176183 0.176553 0.1765050.470743 0.47039 0.4707750.353074 0.353057 0.35272

T 6 =

0.176414 0.176448 0.1765290.470636 0.470575 0.4705830.35295 0.352977 0.352889

Do they converge? To what?

T 4 =

0.17554 0.177662 0.1753470.470679 0.469329 0.4722220.353781 0.353009 0.352431

T 5 =

0.176183 0.176553 0.1765050.470743 0.47039 0.4707750.353074 0.353057 0.35272

T 6 =

0.176414 0.176448 0.1765290.470636 0.470575 0.4705830.35295 0.352977 0.352889

Do they converge? To what?

T 4 =

0.17554 0.177662 0.1753470.470679 0.469329 0.4722220.353781 0.353009 0.352431

T 5 =

0.176183 0.176553 0.1765050.470743 0.47039 0.4707750.353074 0.353057 0.35272

T 6 =

0.176414 0.176448 0.1765290.470636 0.470575 0.4705830.35295 0.352977 0.352889

Do they converge? To what?

A transition matrix (or corresponding Markov process) is calledregular if some power of the matrix has all nonzero entries. Or,there is a positive probability of eventually moving from every stateto every state.

Theorem 2.5If T is the transition matrix of a regular Markov process, then

(a) As n→∞, T n approaches a matrix

A =

u1 u1 . . . u1

u2 u2 . . . u2

. . . . . . . . . . . .un un . . . un

,

all of whose columns are identical.

(b) Every column u is a a probability vector all of whosecomponents are positive.

Theorem 2.6If T is a regular∗ transition matrix and A and u are as above, then

(a) For any probability vector x, Tnx→ u as n→∞, so that u isa steady-state vector.

(b) The steady-state vector u is the unique probability vectorsatisfying the matrix equation Tu = u.

Finding the steady-state vector

We know the steady-state vector is unique. So we use the equationit satisfies to find it: Tu = u.

This is a matrix equation if you put it in the form

(T− I)u = 0

Finding the steady-state vector

We know the steady-state vector is unique. So we use the equationit satisfies to find it: Tu = u.This is a matrix equation if you put it in the form

(T− I)u = 0

Math 20 - October 15, 2007.GWBMonday, Oct 15, 2007

Page9of9

Example (Skipping class)

If the transition matrix is T =

(0.7 0.80.3 0.2

), what is the

steady-state vector?

SolutionWe can combine the equations (T − I )u = 0, u1 + u2 = 1 into asingle linear system with augmented matrix −3/10 8/10 0

3/10 −8/10 01 1 1

1 0 8/11

0 1 3/11

0 0 0

So the steady-state vector is

(8/11

3/11

). You’ll go to class about 72%

of the time.

Example (Skipping class)

If the transition matrix is T =

(0.7 0.80.3 0.2

), what is the

steady-state vector?

SolutionWe can combine the equations (T − I )u = 0, u1 + u2 = 1 into asingle linear system with augmented matrix −3/10 8/10 0

3/10 −8/10 01 1 1

1 0 8/11

0 1 3/11

0 0 0

So the steady-state vector is

(8/11

3/11

). You’ll go to class about 72%

of the time.

Example (Skipping class)

If the transition matrix is T =

(0.7 0.80.3 0.2

), what is the

steady-state vector?

SolutionWe can combine the equations (T − I )u = 0, u1 + u2 = 1 into asingle linear system with augmented matrix −3/10 8/10 0

3/10 −8/10 01 1 1

1 0 8/11

0 1 3/11

0 0 0

So the steady-state vector is

(8/11

3/11

). You’ll go to class about 72%

of the time.

Example (The Markov Dance)

If the transition matrix is T =

1/3 1/4 01/3 1/2 1/2

1/3 1/4 1/2

, what is the

steady-state vector?

SolutionWe have

−2/3 1/4 0 01/3 −1/2 1/2 01/3 1/4 −1/2 0

1 1 1 1

1 0 0 3/17

0 1 0 8/17

0 0 1 6/17

0 0 0 0

Example (The Markov Dance)

If the transition matrix is T =

1/3 1/4 01/3 1/2 1/2

1/3 1/4 1/2

, what is the

steady-state vector?

SolutionWe have

−2/3 1/4 0 01/3 −1/2 1/2 01/3 1/4 −1/2 0

1 1 1 1

1 0 0 3/17

0 1 0 8/17

0 0 1 6/17

0 0 0 0