CSE 736: Markov Chains Lecturer: Hung Q. Ngo SUNY Buffalo, Spring 2003 Scribe: Sumesh Philip

Lecture 4: Markov Chains - Basics & Classification of States

1 Preliminaries

Definition 1.1. A discrete-time Markov chain is a discrete-time stochastic process {X_{0}, X_{1}, … } whose state space I is countable (i.e. finite or enumerable), and for all i, j ≥ 0 there is a given probability p_{ij} such that

P [X_{n+1} = j | X_{n} = i, X_{n−1} = i_{n−1}, … X_{0} = i_{0}] = p_{ij},

for all i_{0}, …, i_{n−1}, n ≥ 0.

The state space I is normally assumed to be a subset of ℕ or ℤ. Basically, if the current state at time n is i, then there is a fixed probability that the next state is j, no matter what states the process was in before time n. Hence,

p_{ij} ≥ 0, ∀i, j ∈ I, and ∑_{j∈I} p_{ij} = 1, ∀i ≥ 0.

The matrix P = (P_{ij}) is called the (one-step) transition probability matrix.

Definition 1.2 (Measures and distributions). A vector λ = (λ_{i} : i ∈ I) consisting of coordinates λ_{i} ≥ 0, i ∈ I, is called a measure on I. If the sum of all coordinates is 1, then it is called a distribution. Thus, each row of the transition probability matrix P of a Markov chain is a distribution.

It is easy to see that P^{n} is the n-step transition probability matrix; namely, (P^{n})_{ij} is the probability p_{ij}^{(n)} that the process from state i will be in state j after n transitions. To see that p_{ij}^{(n)} is the same as (P^{n})_{ij}, we only need to observe that they both satisfy the Chapman-Kolmogorov Equations:

p_{ij}^{(m+n)} = ∑_{k=0}^{∞} p_{ik}^{(n)}p_{kj}^{(m)}, ∀n, m ≥ 0, i, j ∈ I.

Remark 1.3. If P is diagonalizable by a matrix S, i.e. P = SΛS^{−1} for some diagonal matrix Λ consisting of P’s eigenvalues on the diagonal, then the entries (P^{n})_{ij} can easily be calculated since P = SΛ^{n}S^{−1}, and the (Λ^{n})_{ii} = λ_{ii}^{n}.

Given the distribution λ of X_{0}, we can easily calculate the distribution at time n:

P [X_{n} = j] = ∑_{i∈I} P [X_{n} = j | X_{0} = i]P [X_{0} = i]
= ∑_{i∈I} λ_{i}p_{ij}^{(n)}.

In other words, X_{n} has distribution λP^{n}. Here, we think of λ as a row vector.

We shall often use the notation P_{i}[F] to denote the probability of event F conditional on X_{0} = i, i.e. P_{i}[F] = P [F | X_{0} = i].

2 Classification of states

Definition 2.1. Given a Markov chain P, state j is said to be reachable from state i if p_{ij}^{(n)} > 0 for some n ≥ 0. We also write i ⇝ j to denote the fact that j is reachable from i. If i ⇝ j and j ⇝ i, then states i and j are said to communicate. We use i ↔ j to denote communication.

Proposition 2.2. Communication is an equivalent relation.

Proof. A property is equivalent if it is reflexive, symmetric and transitive. By default, p_{ii}^{(0)} = 1 > 0, hence i ⇝ i: reflexivity holds. If i ⇝ j, and j ⇝ k, then there are n and m such that p_{ij}^{(n)} > 0 and

p_{jk}^{(m)} > 0. Thus,

p_{ik}^{(n+m)} = ∑_{j′∈I} p_{ij′}^{(n)}p_{j′k}^{(m)} ≥ p_{ij}^{(n)}p_{jk}^{(m)} > 0,

implying i ⇝ k. Similarly, k ⇝ j and j ⇝ i imply k ⇝ i. Thus, the communication relation is transitive. The fact that it is also symmetric is trivial.

An equivalent relation partition the state space into communication classes, where two states are in the same class if and only if they communicate. We now investigate some class properties.

Definition 2.3. A (communication) class C is closed if for all i ∈ C, the fact that i ⇝ j implies j ∈ C. A state i is absorbing if {i} is a closed class by itself.

Example 2.4 (Gambler’s Ruin Problem). A gambler goes to a casino where he bets 1 dollar every hand. He wins with probability p, and loses with probability 1 − p. He stops playing when he gets broke, or when he gets to N dollars. Figure 1 illustrates the process. In Figure 1, {0}, {N}, and {1, 2, ...N −1}

Figure 1: A Markov chain for the Gambler’s ruin problem

are communication classes. States 0 and N are absorbing since the classes {0} and {N} are closed.

Denote by f_{ij}^{(n)} the probability that the process visits state j from state i for the first time in n (≥ 1) steps, namely

f_{ij}^{(n)} = P [X_{n} = j | X_{0} = i, X_{k} ≠ j, k ∈ I].  (1)

Then, the probability of ever getting from i to j is

f_{ij} = ∑_{n=1}^{∞} f_{ij}^{(n)},  (2)

and the mean number of steps required to reach j from i is

μ_{ij} = ∑_{n=1}^{∞} nf_{ij}^{(n)}.  (3)

Definition 2.5. A state is said to be recurrent if f_{ii} = 1 and transient if f_{ii} < 1. Furthermore, a recurrent state i is said to be positive recurrent if μ_{ii} < ∞ and null recurrent if μ_{ii} = ∞. The reader might also see persistent used instead of recurrent.

Example 2.6 (Positive Recurrence). For the example in Figure 2, let’s calculate f_{11} and μ_{11}.

Figure 2: A Markov chain with two states

It is easy to see that

f_{11}^{(1)} = 1 − p, f_{11}^{(2)} = p^{2}, and f_{11}^{(3)} = p^{2}(1 − p).

In general,

f_{11}^{(n)} = p^{2}(1 − p)^{(n−2)},
f_{11} = ∑_{n=1}^{∞} f_{11}^{(n)}

= (1 − p) + p^{2} ∑_{n=2}^{∞}(1 − p)^{n−2}
= 1.

Thus, state 1 is recurrent. Moreover,

μ_{11} = ∑_{n=1}^{∞} nf_{11}^{(n)}

= (1 − p) + p^{2} ∑_{n=2}^{∞} n(1 − p)^{n−2}

= (1 − p) + \frac{p^{2}}{(1 − p)^{2}} (∑_{n=1}^{∞} n(1 − p)^{n} − (1 − p))
= 2.

Thus state 1 is positive recurrent.

Example 2.7 (Null Recurrence). For the example in Figure 3, let’s calculate f_{00} and μ_{00}.

f_{00}^{(n)} = \{ \begin{matrix}0 & n = 1 \\ \frac{1}{n(n−1)} & n ≥ 2 \end{matrix}

Hence,

f_{00} = ∑_{n=1}^{∞} f_{00}^{(n)} = ∑_{n=2}^{∞} \frac{1}{n(n − 1)} = ∑_{n=2}^{∞} (\frac{1}{n − 1} − \frac{1}{n}) = 1.

Figure 3: A null recurrent example

Thus state 0 is recurrent. Moreover, since

μ_{00} = ∑_{n=1}^{∞} nf_{00}^{(n)} = ∑_{n=2}^{∞} \frac{1}{n − 1} = ∞

state 0 is null recurrent.

Proposition 2.8. Given a DTMC P,

(i) a state i is recurrent if and only if ∑_{n=1}^{∞} p_{ii}^{(n)} = ∞,

(ii) a state i is transient if and only if ∑_{n=1}^{∞} p_{ii}^{(n)} < ∞.

Proof. It is sufficient to show either (i) or (ii), and the other follows. We shall prove the later. Let V_{i} denote the number of visits to i starting from X_{0} = i. Note that starting from i, the probability of visiting i exactly n times is f_{ii}^{n−1}(1 − f_{ii}). (Here, we make implicit use of the so-called strong Markov property to be introduced later.) Thus, V_{i} is a geometric random variable with parameter (1 − f_{ii}). The expected number of visits to a state i starting from i is then E_{i}[V_{i}] = \frac{1}{1−f_{ii}} < ∞.

Now, we calculate E_{i}[V_{i}] in another way. Denote by I_{n} the random variable that indicates if the state of the process after n steps is i, namely

I_{n} = \{ \begin{matrix}1 & X_{n} = i \\ 0 & otherwise \end{matrix}

Then, we have

E_{i}[V_{i}] = E_{i}[∑_{n=1}^{∞} I_{n}] = ∑_{n=1}^{∞} E_{i}[I_{n}] = ∑_{n=1}^{∞} P_{i}[I_{n} = 1] = ∑_{n=1}^{∞} P_{i}[X_{n} = i] = ∑_{n=1}^{∞} p_{ii}^{(n)},

which completes the proof.

Proposition 2.9. Recurrence and transience are class properties. In other words, given a class C of a DTMC P, if i, j ∈ C, and i ↔ j, then i and j have to be both transient or recurrent.

Proof. Given communicating states i, j, we shall prove that j is transient if i is transient by showing that ∑_{n=1}^{∞} p_{jj}^{(n)} < ∞. Since i is transient, we know that

∑_{n=1}^{∞} p_{ii}^{(n)} < ∞.  (4)

Since i and j communicate, we know that ∃n_{0}, m_{0} such that p_{ij}^{(n_{0})} > 0, and p_{ji}^{(m_{0})} > 0. It is easy to see that

p_{ii}^{(n_{0}+n+m_{0})} ≥ p_{ij}^{(n_{0})}p_{jj}^{(n)}p_{ji}^{(m_{0})},

which implies

p_{jj}^{(n)} ≤ \frac{p_{ii}^{(n_{0}+n+m_{0})}}{p_{ij}^{(n_{0})}p_{ji}^{(m_{0})}} .

Summing the inequality above over all n, we get

∑_{n=1}^{∞} p_{jj}^{(n)} ≤ \frac{1}{p_{ij}^{(n_{0})}p_{ji}^{(m_{0})}} ∑_{n=1}^{∞} p_{ii}^{(n_{0}+n+m_{0})} ≤ \frac{1}{p_{ij}^{(n_{0})}p_{ji}^{(m_{0})}} ∑_{n=1}^{∞} p_{ii}^{(n)} < ∞.

The next two results essentially say that recurrent classes and closed classes are the same for finite DTMCs.

Proposition 2.10. Every recurrent class is closed.

Proof. We will show this by contradiction. Given a recurrent class C, and state i ∈ C, we wish to show that if there is a state j and an integer n such that p_{ij}^{(n)} > 0 and j ∉ C, then i cannot be recurrent.

The probability that a recurrent state i is visited exactly n times, starting from i, is f_{ii}^{n−1}(1−f_{ii}) = 0. Hence, a recurrent state is visited infinitely many times, or the probability that a recurrent state is visited infinite number of times is 1.

It is thus sufficient to show that there is a positive probability that i is visited only a finite number of times. Denote by i∞ the event that i is visited an infinite number of times. We have

P_{i}[i∞] = ∑_{k∈I} P_{i}[i∞ | X_{n} = k]P_{i}[X_{n} = k]
= P_{i}[i∞ | X_{n} = j]P_{i}[X_{n} = j] + ∑_{k∈I−{j}} P_{i}[i∞ | X_{n} = k]P_{i}[X_{n} = k]

The first term in the right hand side is zero, since the process cannot return to i once it reaches j. Thus

P_{i}[i∞] = ∑_{k∈I−{j}} P_{i}[i∞ | X_{n} = k]P_{i}[X_{n} = k]
≤ ∑_{k≠j} p_{ik}^{(n)}
= 1 − p_{ij}^{(n)}
< 1

which is a contradiction, since we assumed that i is recurrent.

Exercise 2.11. Show the previous proposition by showing that given a class C, and a state i ∈ C, if i ⇝ j and j ∉ C, then

f_{ii} < 1,

namely C is not recurrent.

Exercise 2.12. Show the previous proposition by showing that given a class C, and a state i ∈ C, if i ⇝ j and j ∉ C, then

∑_{n=1}^{∞} p_{ii}^{(n)} < ∞,

namely C is not recurrent.

Proposition 2.13. Every finite closed class is recurrent.

Proof. If a finite closed class C is transient, then each state i in C can only be visited finite number t_{i} of times. But then at time 1 + ∑_{i∈C} t_{i} the process has to be somewhere. Contradiction!

Infinite closed classes could be recurrent or transient as the next example shows.

Example 2.14 (Random walk on ℤ). Consider the random walk as shown in Figure 4.

Figure 4: A random walk illustrating that an infinite closed class could either be transient or recurrent.

For any state i, we have

p_{ii}^{(2n+1)} = 0, p_{ii}^{(2n)} = \binom{2n}{n}p^{n}q^{n}.

(We cannot return to i in an odd number of steps.) Therefore, the probability of ever coming back to the state i, f_{ii}, is

∑_{n=0}^{∞} p_{ii}^{(2n)} = ∑_{n=0}^{∞} \binom{2n}{n}p^{n}q^{n}

= ∑_{n≥n_{0}} \frac{\sqrt{4πn}(2n/e)^{2n}(1 + o(1))}{2πn(n/e)^{2n}(1 + o(1))}p^{n}q^{n}

= ∑_{n=0}^{∞} \frac{1}{\sqrt{πn}}(4pq)^{n}(1 + o(1)).

Here, we have used Stirling approximation for n!.

When p = q = 1/2, the sum tends to ∞ as ∑_{n} 1/\sqrt{n} → ∞. On the other hand, when p ≠ q we have 4pq < 1 and the sum does converge to some constant.

Thus, the random walk could either be transient or recurrent.