20/001 Existence of Trembling hand perfect and sequential equilibrium in Stochastic Games

Sofia Moroni

February, 2020

Existence of Trembling hand perfect and sequentialequilibrium in Stochastic Games

Sofia Moroni*

University of Pittsburgh

moroni@pitt.edu

February 2020

Abstract

In this paper we define notions of trembling hand and sequential equilibrium and

show that both types of equilibria exist in a large class of stochastic games that may

feature incomplete and imperfect information. These equilibria do not necessitate the

use of a public correlating device. Under further regularity assumptions each stochas-

tic game has a sequence of approximating finite games whose equilibria approximate

equilibria of the limit game.

KEYWORDS: existence, stochastic games, trembling hand perfect equilibrium,

sequential equilibrium, revision games.

*I am grateful to Juan Escobar, Faruk Gul, Adam Kapor, Roger Myerson, Wolfgang Pesendorfer, PhilReny, Luca Rigotti, Larry Samuelson, Ali Shourideh, Can Urgun and Richard Van Weelden for their insight-ful comments.

1

1 Introduction

In this paper we propose notions of trembling hand perfect and sequential equilibrium thatextend Selten (1975) and Kreps and Wilson (1982)’s definitions to a large class of stochas-tic games with informational asymmetries. This class encompasses games with an infinitestate space and finite actions in which players receive signals that may be informative aboutstates visited, about past play or about their opponents’ observations. We show that undertwo assumptions—one restricting the payoff function and another requiring a noisy obser-vation of the opponents’ signals—trembling hand perfect equilibria exist and, under furtherregularity assumptions, these equilibria are also sequential equilibria. These results are thefirst proofs of existence for a general class of stochastic games with infinite state spaceswhich do not require the existence of a public correlating device. Two particular cases thatsatisfy the restriction on payoffs are stochastic games with discounted payoffs and gameswith stochastic move opportunities, such as revision games, that feature a finite expectedlength.1

A trembling hand perfect equilibrium is defined as a limit of totally mixed strategiesthat are constrained equilibria, in which players must put positive weight on every availableaction. This limit is in weak convergence of strategies, which requires the convergence ofprobabilities of histories of play for each player.2 Our assumptions guarantee that underthe resulting topology the space of strategies is compact and the players’ payoffs are con-tinuous in strategies. These properties guarantee existence of a trembling hand equilibriumand imply that it is a Nash Equilibrium. Under further regularity assumptions the equilib-rium is, for each player, a best response to a limit of beliefs that are consistent with totallymixed approximating strategies.3 A strategy that satisfies the latter property of sequentialrationality is what we call a weak sequential equilibrium. Weak limits coincide with stan-dard limits in countable spaces, and, therefore, our notion coincides with Kreps and Wilson(1982)’s in finite and countably infinite games.

Our assumption on payoffs requires that a sequence of bounds for each period t’s ex-pected payoff be summable. It includes, in addition to discounted games, games in whichpayoffs decrease more slowly over time. It also applies to most of the games with stochastic

1This class includes repeated games and games of finite length.2The function from player’s signal realizations to probabilities over actions can be viewed as a function

in an L2 space. The convergence is in the weak topology of L2, starting from each deviation from a limitstrategy.

3The limit of beliefs is taken in the weak-* of the dual of L∞ and it results in a finitely additive measure.

2

move opportunities with a deadline that have been studied in the literature. The assumptionon the (un)observability of the opponents signals is more delicate. It requires that the jointdistribution of the players’ signals up to each period t be absolutely continuous with re-spect to the product measure of the marginals of each player’s signal sequence up to t. Thisassumption—which we dub noisy observability—is a generalization of the absolute conti-nuity assumption in Milgrom and Weber (1985) to the context of dynamic games. Noisyobservability holds, for example, in games with countable signal spaces and in Markovstochastic games with private monitoring with uncountable signal spaces in which eachplayer’s signal has some idiosyncratic noise.4

Our final set of results relates to sequences of games that may approximate the originalstochastic game. We define a notion of an approximating game sequence that has the fea-ture that its constrained equilibria converge to constrained equilibria of the original game.It requires, among other conditions, that there be finer and finer countable partitions of thestate and signal spaces in which the available action set and the payoffs are well behaved.We also show that any game that is Markov, has compact state and signal spaces and pay-offs continuous in the state has a finite approximating game sequence.5 These results donot require the noisy observability assumption and may be useful to prove existence insettings in which the assumption fails. They imply, in particular, that perfect informationasynchronous moves games with compact state spaces and continuous payoffs (as a func-tion of the state), with either discounted payoffs or a finite expected length have Markovperfect equilibria.

As a motivation for our interest in these games, consider a setting in which playersreceive stochastic opportunities to move at times between 0 and a deadline at time T . At anopportunity players choose an available action after observing a signal that is informativeabout a payoff relevant state, the rate at which moves will realize in the future, and/orpast play of the opponents. In particular, after some actions players may not learn that anopponent received an opportunity at all.6 This type of game can model realistic settings inwhich players have uncertainty about the timing of their moves and their opponents’. Asthe rate at which the players receive move opportunities grows large, the game approachesa continuous time game while avoiding the technical issues with the definition of strategies

4See section 4.15The key is the finite subcovers of the compact state and signal spaces define a suitable sequence of

approximating partitions.6In games in which players can choose to “keep their own action” and opportunities arrive frequently,

this assumption—which requires imperfect information—is the most natural.

3

that arise in continuous time settings. In these games it is crucial that the timing of play bea part of the state, which must, therefore, be uncountable. Thus, due to the dimensionalityof the space of histories and the informational assumptions, existing proofs of existence donot apply.

Several applied settings have been studied using games with stochastic move oppor-tunities. To give some examples, in online auctions, players must submit bids before adeadline if they want to win the good, and make inferences about opposing players’ bidsand valuations as the auction proceeds. Due to delays and failures of information trans-mission the players cannot in practice move at arbitrary times. A player who attempts tomove at the very last instant may fail to do so.7 When labor and management of a firmbargain over pay, if the two sides do not reach a deal before a deadline a strike may occur.Politicians in congress may need to reach an agreement before a deadline at which the debtceiling binds.8 Candidates must choose their stance, announce their policy proposals anddisclose opposition research at strategic times before an election.9

The literature on stochastic games, stemming from the seminal work by Shapley (1953)focuses mostly on Markov games with almost perfect information with discounted payoffsand shows existence of Markov perfect equilibria.10 Due to the potentially non-stationarynature of our setting the techniques developed in the stochastic games literature with al-most complete information games cannot be readily applied to our setting.11 Fink (1964),Sobel (1971) and Takahashi (1964) show existence of Markov perfect equilibria under afinite state space. Parthasarathy (1973) shows existence in a two player game with a count-able set of states. Federgruen (1978) and Whitt (1980) show existence of Markov perfect

7Ambrus et al. (2014), Hopenhayn and Saeedi (2016), and Kapor and Moroni (2016) are examples ofstochastic move models of online auctions. Relatedly, Roth and Ockenfels (2002); Ockenfels and Roth(2006) model online auctions with discrete moving times but randomness in the realization of the final moveopportunity.

8Ambrus and Lu (2014) consider a model of bargaining with stochastic moving times.9Kamada and Sugaya (2020) and Kamada and Kandori (2017) consider stochastic-timing models of

election campaigns.10Players may move simultaneously but they observe the present state and all previous moves in each

period.11In rough terms, the approach of the stochastic games literature is to define a correspondence that takes

expected payoffs, as a function of observed states, to expected payoffs. It is then shown that under the as-sumptions the correspondence has a fixed point. An equilibrium is found as a measurable selection that yieldsthe expected payoffs of the fixed point. Under incomplete or imperfect information, however, the players’expected payoffs are not fully determined by any observable state. An unobserved action, for instance, af-fects payoffs. If there is incomplete information about types, players’ previous actions (which one can makepart of the state) may signal an opponent’s type and affect expected payoffs endogenously in equilibrium.See, for example, Duggan (2012) and Nowak and Raghavan (1992).

4

equilibria when the state space is countable. When the state space is uncountable, however,examples of nonexistence have been constructed. Simon (2003) provides an example of aone stage game with three players and finite actions in which there is no equilibrium. Inthe three stage game in Harris et al. (1995) there is no subgame perfect equilibrium in agame with a continuum of actions.12 The standard approach for proving existence—whichinvolves using Kakutani-Fan-Glicksberg over a compact set of actions—does not work. Ifone defines a topology in which strategies live in a compact space the limits of these strate-gies may not be strategies themselves but instead correlated strategies.13 Many authors“close” the space of strategies by assuming the game has some way of correlating the play-ers’ strategies. Duggan (2012) adds potentially payoff relevant noise that is conditionallyindependent of the current state and actions. Harris et al. (1995) add a public signal thatserves as a correlating device. Manelli (1996) adds cheap talk to a signaling game to obtainexistence. Our approach, in contrast, is to assume that there is noise in the observation ofthe opponents’ observations.

There is a growing literature that considers stochastic games with imperfect and incom-plete information. Altman et al. (2008) study stochastic games with imperfect and incom-plete information with finite actions and states. They show that, under some assumptionson the nature of the transition matrix, the set of feasible strategies and non-observabilityof costs, a stationary equilibrium exists. Balbus et al. (2013) show existence of a station-ary Markov Nash equilibrium in a setting with private signals in stochastic games withstrategic complementarities. It has also been shown that in some cases equilibria may notexist: Flesch et al. (2003) provides an example of a game with unobservable actions andobservable payoffs that does not have an ε-equilibrium in a setting with the average pay-offs criterion. Ours is the first paper, to our knowledge, to show existence of tremblinghand perfect and sequential equilibria in a class of stochastic games with imperfect andincomplete information and an infinite state space.

Our paper also relates to a literature that proposes notions of sequentially rational equi-libria for dynamic games. The closest paper, Myerson and Reny (2019), considers gameswith infinite state and action spaces with finite horizon and defines an equilibrium notion, aperfect conditional equilibrium which is sequentially rational. A perfect conditional equi-librium is defined as a distribution over states and actions that is the limit of a net of perfect

12Their game can be expressed as a stochastic game in which the action is part of the state in period 3.13This issue has been pointed out, for example, by Harris et al. (1995), Börgers (1991) and Simon and

Stinchcombe (1989). Myerson and Reny (2019) call it “strategic entanglement”.

5

conditional ε-equilibria (in which players optimize their payoffs up to ε conditional onevery measurable set of private histories and are close to ε-equilibria of slightly perturbedgames). A perfect conditional equilibrium may involve correlated strategies. The maindifference with our paper is that we consider the limit of equilibria in which players areconstrained to utilize perfectly mixed strategies and show that this limit—a trembling handperfect equilibrium—is sequentially rational under further technical assumptions. Becausewe assume a finite action space—while theirs is infinite—and make different informationalassumptions we are able to show existence of an equilibrium without correlations.

2 Stochastic Games with Incomplete and Imperfect Infor-mation

We consider a game in which players choose actions in each period t ∈ N. The game canbe described by the list Γ = (N,(Ω,F ),(Si,Si),(Xi,Ai,gi),(µ

ω ,µs)) where N is a finiteset of n players, Ω is a state space with Borel measurable sets in F , Si is a space of signalsfor each player i with Borel measurable sets Si, Xi is a finite set of actions for each player i,X = Πi∈NXi with generic element—an action profile—a = (a1, . . . ,an) ∈ X , Ai : ∪t∈NΩt×X t−1 ⇒ Xi is a measurable correspondence that yields the actions available to player i asa function of the history realized states and the history of play, gi : ∪t∈NΩt ×X t → R ismeasurable in ω t ∈ Ωt and represents the flow payoff received by player i as a functionof history up to the period of play, and µω and µs are transition probabilities which wedescribe in what follows.14

The state transitions from one period to the next according to the state transition prob-

ability function µω : ∪t∈NΩt ×X t ×F → [0,1]. This function determines the probabilitywith which each state is drawn as a function of the states that were visited previously andthe actions taken by the players. That is, µω(Z|ω t ,at) is the probability that the state drawnat time t +1 is in the set Z ∈F given that states ω t = (ω1, . . . ,ωt) were drawn and actionprofiles at = (a1, . . .at) were played in periods 1 through t, and it is measurable as functionof (ω t ,at).

We are interested in games with imperfect and incomplete information in which eachplayer may receive signals about previous play, about players’ types, and about their pay-offs. Specifically, at each time t each player i observes a signal si,t ∈ Si before choosing her

14We use the notation Y t :=×ti=1Y for any set Y .

6

action in Ai(ωt ,at−1) at time t. Player i’s signals are drawn according to the signal transi-

tion probability function µsi :(∪t∈NΩt×X t−1

)×Si→ [0,1], where µs

i (Z|ω t ,at−1) is the

probability that player i’s period-t signal si,t satisfies si,t ∈ Z ∈ Si after ω t ∈Ωt and at−1 ∈X t−1, and it is measurable as a function of (ω t ,at−1).15 The measurable space of jointsignals of all players is (S :=×i∈NSi,SSS :=⊗iSi) and µs :

(∪t∈N(Ω

t×X t−1))×SSS→ [0,1]

denotes the joint transition probability on that space. For each player i, Si may contain anelement ?, which represents “no observation”. When ? is drawn a player is unaware that amove took place. In an application a player may be drawn to play in some periods and notothers and may be unaware of the moves that took place in periods in which she was notcalled to play. To make our notation consistent, whenever si = ?, we assume i takes actiona? with probability 1 (and does not remember it).16,17

Player i’s signal si ∈ Si may contain information about s j ∈ S j for j 6= i. For example,in a game of almost complete information it is common knowledge that Si = S j = Ω×X and at each period t, the period-t signals are given by si,t = s j,t = (ωt ,at−1) for eachplayer i, j ∈ N, with a0 = /0.18 More generally, our model encompasses many differentinformational settings. Players may have types that affect their likelihood of drawing stateswhich they learn privately at time 1. Players may not observe past actions but may receivean informative signal about the action profiles realized in previous periods. Additionally,they may observe partitional or other information regarding an opponent’s payoff relevantvariable.

Histories A time-t state history consists of all the states that were visited up to time t. Ageneric state history is written as ω t = (ω1,ω2, . . .ωt). A time-t history of play contains theaction profiles selected by the players up to time t. A generic history of play is written asat = (a1,a2, . . . ,at). A time-t history of signals consists of all the signals realized up to timet. A generic time-t history of signals is written as st = (s1,s2, . . . ,st), where sl =

(si,l)

i∈Nis the profile of signals realized at time l ≤ t. A generic time-t history h = (ω t ,at ,st)

is composed of a state history, a history of play, and a history of signals. We use ω(h),a(h), s(h) to denote the history of states, play and signals in history h, respectively. For

15Because µsi depends on all previous states, each player may learn more about states that were realized

before time t, at time t.16Moves that a player is unaware of do not enter her private histories, as defined below.17Formally, whenever a player’s signal at time t is ?, the state is such that Ai(ω

t ,at−1) = a?.18These games are called “games of almost perfect information” because players observe all past moves

and move simultaneously.

7

at ∈ X t , ati,τ denotes player i’s action in the action profile (at)τ , for τ ≤ t. ω t

i,τ and sti,τ

are defined analogously. at,(l) for l ≤ t denotes the truncation of at up to and includingperiod l. ω t,(l) and st,(l) are defined analogously. The length of a history h is denoted |h|.The set of all time-t histories is denoted H t and it is endowed with the product measure.H :=

⋃t∈NH t is the set of histories. The σ -algebra of measurable sets in H is denoted

H.19

A private history of a player i at time t, hti =((si,l)l≤t,si,l 6=?,(ai,l)l≤t−1,ai,l 6=a?

)consists

of the signals si,l , other than ?, received at each period l ≤ t, and player i’s actions, otherthan a?, up to period t−1. The set of player-i, time-t private histories is denoted H t

i andthe set of player i private histories is given by Hi :=∪t∈NH t

i .20 The set of measurable setsin Hi is denoted Hi.21 There is a unique private history of player i associated with eachgame history h ∈H t and it is denoted by hi(h).22 Since a player’s private history dependsonly on her signals and actions we may also write hi(st

i,at−1i ) for i’s private history after

signal realization sti ∈ St

i and sequence of actions profiles at−1i ∈ X t−1

i . For h ∈H t , thetruncation of h up to period l ≤ t is denoted h(l) and player i’s period-l private history isgiven by h(l)i (h) := hi(h(l)).

The correspondence Ai must be measurable with respect to each player i’s private his-tory. That is, we can write Ai : Hi ⇒ Xi.

Strategies A strategy of player i, σi : Hi→ ∆Xi, is a function from i’s private histories toprobability distributions over i’s actions satisfying suppσi(ht

i)⊆ Ai(hti). The set of player i

strategies is denoted Σi. A strategy profile is of the form σ :=×iσi ∈ ×iΣi := Σ.

Expected Payoffs Consider a strategy σ =×i∈Nσi and a history (ω t ,at ,st). The strategyinduces a probability, prob(at |ω t ,st) over histories (of actions) conditional on the realiza-tions in (ω t ,st). For history h = (ω t ,at ,st),

prob(at |ω t ,st ,σ) := ∏l≤t

∏i∈N

σi(ati,l|h

(l)i (h)),

19The measurable sets in H are defined by the product σ−algebras.20Notice that a player may be unaware of the number of moves that have occurred at the time that she

observes her private history. Therefore a private history hi ∈Hi may belong to ∩k∈IHk

i for some set I with|I|> 1.

21The σ -algebra is defined by product σ -algebras.22hi is H/Hi measurable.

8

where σi(ati,l|h

(l)i (h)) is the probability that player i assigns to action at

i,l at private history

h(l)i (h) when she moves at time l. We call the function prob(at |ω t ,st ,σ) the probability

function associated to σ .Abusing notation we also write prob(h,σ) for prob

(a(h)|ω(h), s(h),σ

). We de-

fine also, prob−i(at |ω t ,st ,σ) := ∏l≤t ∏ j∈N\iσ j(a j,l|h(l)j (h)) and, probi(at |ω t ,st ,σ) :=

∏l≤t σi(ai,l|h(l)j (h))which we also denote prob−i(h,σ) and probi(h,σ), respectively.

Player i’s payoff when players follow strategy σ is given by23

Ui(σ) :=∞

∑t=1

∫h∈H t

gi(h) · prob(h,σ) dµt (h) ,

where µ t is the measure over H t induced by µω and µs and the counting measure overX .24

3 Equilibrium

Each function prob(·,σ) associated with a strategy σ ∈ Σ can be viewed as a function inan L2 space endowed with a measure µ over H . The measure µ is such that the innerproduct between two functions ψ,φ : H → R in L2(H , µ) is given by

〈ψ,φ〉=∞

∑t=1

∫h∈H t

δt ·ψ(h) ·φ(h)dµ

t(h),

for fixed δ ∈ (0,1), and the norm of f ∈ L2(H , µ) is given by‖ f‖2 = 〈 f (h), f (h)〉1/2.25

We endow each L2(H , µ) space with the topology of weak convergence.26 Notice that

23gi(h) denotes gi(ω(h), a(h)

).

24Formally, let µω,s(·|ω l ,al) be the joint measure over Ω×S induced by µω(·|ω l ,al) and µs(·|ω l+1,al).The measure µ t(·) on H t is given by

µt(Z) = ∑

at∈X t

∫Ωt×St

1(ω t ,st ,at) ∈ Zdµω,s(ωt ,st |ω t−1,st−1,at−1) · · ·dµ

ω,s(ω1,s1| /0).

for each measurable set Z in H t .25The measure µ over H t that yields the inner product is given by

µ(Z) =∞

∑t=1

δt∫

h∈H t1h ∈ Z, h(t) = hdµ

t(h),

for each measurable set Z ⊆H .26The topology is characterized by its convergent nets: a net fα converges to f ∗ in the weak topology if

9

for every σ ∈ Σ, prob(·,σ) is in L2(H , µ) and∥∥prob(·,σ)

∥∥2 <

(δ

1−δ

)1/2. Thus, by the

Banach-Alaoglu Theorem if the set

Λ(B) = prob(·,σ)|σ ∈ B,

for B⊆ Σ is closed in the weak topology of L2(H , µ), then it is compact.

Trembling hand perfect equilibrium Given ε,ν > 0, such that ε > ν , a function εi :

(hi,ai)|hi ∈Hi,ai ∈Ai(hi)→ [ν ,ε] is called an (ε,ν)-tremble of player i, and ε =(εi)i∈N

is called an (ε,ν)-tremble profile.Given an ε-tremble profile ε , a strategy profile is a ε-constrained strategy if at each

private history hi ∈ Hi each player puts weight at least εi(hi,ai) on action ai ∈ Ai(hi).Σi(ε) denotes the set of ε-constrained strategies of player i.27 Σ(ε) denotes the set ofε-constrained strategy profiles.

Definition 1. Let ε be a (ε,ν)-tremble profile. An ε-constrained equilibrium σ ε is anε-constrained strategy profile such that for every player i ∈ N

σεi ∈ argmax

Ui(σ

′i ,σ

ε−i)|σ ′i ∈ Σi(ε)

We define for each t and h ∈H and H ⊆ 1, . . . , |h| the function

probi(h,σi, H) =

∏l∈H σi(a(h)i,l|h(l)i (h)) if H 6= /0

1 otherwise.

Notice that probi(h,σ) = probi(h,σi,1, . . . , t).Define for h ∈H

Hi(h, σ) =

l ≤ |h|

∣∣σi

(a(h)i,l|hi(h(l))

)> 0,∀l ∈ l . . . , |h|−1

.

Hi(h, σ) contains the indices of moves that occur after the “last deviation” of i from σi to azero probability action in history h.

and only if 〈 fα ,ψ〉→ 〈 f ∗,ψ〉 for every ψ ∈ L2(H , µ). This topology coincides with the weak star topologydue to the reflexivity of L2.

27We allow the lower bound on the probability on each action to depend on action and the private history,as in the standard definition, so that at “off-path” histories of the THPE (defined below) beliefs may benon-uniform over actions.

10

We say that a net of strategy profiles σα converges to σ weakly in strategies if forevery player i if (a) for every time t ∈ N

〈probi(·,σα , Hi(·, σ)),ϕ〉 → 〈probi(·, σ , Hi(·, σ)),ϕ〉,

for every ϕ ∈ L2(H , µ), with suppϕ ∈ H t , and (b)σα

i (ai,l|hi(h))1

h|σi(ai,l|hi(h)

)= 0

converges to zero almost surely.In words, a sequence of strategy profiles σm converges weakly in strategies to a limit

σ if (a) the probability of i’s history of play under σm after i’s last zero probability action(according to σ ) converges weakly to that probability under σ , and (b) σm converges tozero almost surely on the set of histories in which σ is equal to zero.28

The motivation for using this form of convergence is two-fold. First, it generates, bydefinition, continuity of the probability of i’s histories of play with respect to strategies.29

In contrast, if we were to take convergence of each σ εi (·|·) individually, this continuity

would not necessarily obtain due the players’ ability to “entangle” their future actions withtheir previous play as function of a signal that lives in the continuum (see footnotes 13and 43). Second, the convergence is “reset” after a zero probability action so that thelimit is able to recover the strategies after histories with a deviation.30 Indeed, without thisproperty, the information about play after a deviation, which is present in the ε-constrainedstrategies due to mixing, would be lost.31

Definition 2. A strategy profile σ∗ is a trembling hand perfect equilibrium (THPE) if thereexist a sequence (εm)m∈N of (εm,νm)-tremble profiles, with εm > νm > 0 and limm→∞ εm =

0, and εm-constrained equilibria, σm, such that σm converges to σ∗ weakly in strategies.

This definition is in the spirit of Perfect Equilibrium for finite extensive form gamesdefined by Selten (1975). An important difference is that we assume weak convergence of

28Notice that the convergence “drops information” regarding actions that occurred before a given periodabout each player i’s strategies. Since every player knows the actions she has taken, this does not precludesequential rationality with respect to beliefs that pertain to the opposing players’ past actions (see section5.3).

29As discussed below, this property will be essential for existence: in conjunction with Assumption 2 itimplies compactness and continuity of payoffs.

30The strategy can be calculated as:

σi(a(h)i,|h||hi(h)) = probi(h, σ , Hi(h, σ))/probi(h(|h|−1), σ , Hi(h(|h|−1), σ)).

31Notice that the reset does not generate issues with continuity because σ εi must converge to 0 almost

surely, whenever σ = 0 and, therefore, no “entanglement” is lost from the truncation.

11

strategies instead of pointwise convergence of behavioral strategies. In finite games bothtypes of convergence coincide. The advantage of our choice of topology is that, underour assumptions,32 weak convergence of strategies is strong enough to imply convergenceof expected payoffs of strategies (Lemma 2 and Proposition 2) but weak enough that thespace of strategies is closed (and therefore compact). The former condition guarantees thatthe best response has closed graph, which together with the latter condition implies theexistence of ε-constrained equilibria via Kakutani-Fan-Glicksberg.

Under our weak notion of convergence it is also not straightforward that the limit ofa sequence should inherit desirable properties from its approximating elements. For ex-ample, even though it is fairly straightforward that a THPE is Nash (Lemma 4), it is notobvious that it should satisfy a notion of sequential rationality. We address this questionin section 5.3 where we define a weak sequential equilibrium, as a natural extension ofKreps and Wilson (1982)’s definition for finite games to our setting, and show that a THPEis a weak sequential equilibrium under some regularity conditions. Our notion of weakconvergence in strategies is chosen purposely to obtain this property.

3.1 Existence of a trembling hand perfect equilibrium

We now introduce two assumptions—1 and 2 below—that will be allow us to show exis-tence of an equilibrium.

Define for each player i ∈ N, gt,max,t−1i : Ωt×St×X t−1→ R as

gt,max,t−1i (ω t ,st ,at−1) = max

at∈X|gi(ω

t , at ,at−1)|,

and recursively, for each j ≤ t−2, define gt,max, ji : Ω j+1×S j+1×X j→ R as

gt,max, ji (ω j+1,s j+1,a j)= max

a j+1∈X

∫Ω×S

gt,max, j+1(ω j+2,s j+2,a j+1)dµω,s(ω j+2,s j+2|ω j+1,s j+1,a j+1),

where µω,s(·|ω j+1,s j+1,a j+1) denotes the joint measure over Ω×S induced by µω andµs. As the maximum of finitely many measurable functions, each gt,max, j

i is measurable.

Assumption 1. For each i ∈ N, we have

∞

∑t=1

∫Ω×S

gt,max,1(ω1)dµ(ω1,s1| /0)< ∞.

32See Assumptions 1 and 2 below.

12

Assumption 1 is satisfied if for each i ∈ N, the following stronger condition holds

∞

∑t=1

suph∈H t

|gi(h)|< ∞. (1)

Notice that (1) holds in a stochastic game with discounted payoffs of the form gi(h) =

δ tvi(h) for h ∈H t where vi(h) uniformly bounded in H for each i ∈ N. Condition (1)also holds for payoffs that decrease slower than geometrically in t, such as a time t flowpayoff given by gi(h) = vi(h) · 1

t2 for h ∈H t and i ∈ N. Assumption 1, however, is weakerthan condition (1). It holds, for instance, in games with stochastic move opportunities—described in section 4.2—that have bounded payoffs and a finite expected length. Condi-tion (1), in contrast, does not hold in such settings.

The main role of Assumption 1 is to ensure that weak convergence of strategies impliesconvergence of expected payoffs.33

The following definition relates to the (un)observability of the opponents’ signals.

Definition 3 (Noisy Observability). We say a game of imperfect and incomplete informa-tion has noisy observability of the opponents’ signals if for each at−1 ∈ X t−1,

µt(ω t ,st ,at−1) = µ

ω(ω t |st ,at−1)× µs(st |at−1),

where µω : St ×X t × (⊗tSi)→ [0,1] is a transition probability, µs is the marginal over St

of µ t(ω t ,st ,at−1) (for fixed at−1) and

dµs(st |at−1) = f s(st ,at−1)dµ

t1(s

t1|at−1)dµ

t2(s

t2|at−1) · · ·dµ

tn(s

tn|at−1),

where µ tj(s

tj|at−1) is the marginal of µ t on st

j ∈ Stj, for fixed action profile at−1, and f s is a

function in L1(St , µs(·|at−1)).

The noisy observability of opponents’ signals condition holds if the spaces of signalsare countable. When the state space is uncountable the condition precludes settings inwhich all players observe the state, as, crucially, players are allowed to observe their op-ponents’ observation only with noise.

Noisy observability is related to the uniform continuity condition that guarantees ex-istence in Milgrom and Weber (1985) and Balder (1988)’s extension in a static Bayesian

33This result is implied by the dominated convergence theorem (see Proposition 2 and 8).

13

game.34,35 In fact, in a static Bayesian game our condition holds precisely if the distribu-tion satisfies Milgrom and Weber (1985)’s absolute continuity.36 Thus, noisy observationgeneralizes absolute continuity for the context of a dynamic stochastic games, in whichplayers can receive signals that are informative about past play, payoff types, the proba-bilistic evolution of the game, etc.

Assumption 2. The game has noisy observability of the opponents’ signals.

We are now ready to state our main result.

Theorem 1. Let Γ be a stochastic game that satisfies Assumptions 1 and 2. Γ has a trem-

bling hand perfect equilibrium.

When assumptions fail: Let us discuss examples of non-existence when Assumptions 1and 2 fail.

It is easy to construct an example of non-existence when Assumption 1 fails, even ina one player setting. Suppose there is one player who must choose an action in the setC,E at each t. If she chooses C (continue) n times and E (end) at time n+1 her payoffis ∑

nj=0

12 j . If she chooses C at every t her payoff is zero. This game does not have an

equilibrium because for any stopping date in which E is chosen the player increases herpayoff by choosing C for one period. This stochastic game does not satisfy Assumption 1as for each t, gt,max,1 = maxa∈X t gi ≥ 1 and, therefore, the sum in Assumption 1 is infinite.

Simon (2003) provides an example of non-existence that fails the noisy observabilityproperty in a one period game, with three players and finite actions. The state space isthe set of binary sequences indexed by the integers. At each information set each player

34Their focus on distributional strategies is also similar to our use of “prob” functions as an intermediatestep to obtain existence.

35Milgrom and Weber (1985) consider a one-shot game in which the signal is a player’s type and thestate is the combination of all signals. States and signals live in a metric spaces. Balder (1988) extendsMilgrom and Weber (1985) existence result to general measure (not necessarily metric) spaces, like the oneswe consider in our setting. Both papers’ approach is—instead of viewing strategies as functions in Lp, as wedo—to associate strategies to transitional probabilities and use the compactness implied by (extensions) ofProkhorov’s theorem to obtain existence. It is likely that a similar approach can be used to show existence oftrembling hand perfect equilibrium in our dynamic setting. However, it would also require the definition ofcombined transition probabilities for sequences of actions taken over time (much like our “prob” functions).It is also not clear how to obtain our results on sequential equilibrium with their alternative approach. In ourview, an advantage of using weak convergence in Lp spaces is that it is a convergence of functions, which isoftentimes easier to ascertain.

36In a Bayesian game the state is composed by the collection of types and µω is the Dirac measure.

14

believes that two sequences are possible, each probability 1/2. These two sequences are de-fined by measure preserving involutions. Players 2 and 3 observe the same signal whereasplayer 1 observes a different signal. The three players play a version of matching penniesin which 1 wants to mismatch 2 and 3 while players 2 and 3 want to match player 1. Player1’s payoff depends also on the state. Their example fails Assumption 2 because players 2and 3 observe the same signal that lives in an uncountable space. Therefore, the distribu-tions of players’ signals are not absolutely continuous with respect to the product measureof marginals.

Lifting other assumptions in our definition of a stochastic game can also lead to non-existence. Harris et al. (1995) provides an example in which the space of actions is infinitethat does not have a subgame perfect equilibrium. They show existence by allowing forpublicly correlated strategies.37 In a setting in which players evaluate payoffs accordingto the average reward, Gillette (1957) provides an example of non-existence in a gamecommonly referred to as the “Big Match”. The form of expected payoffs in our settingdoes not include average reward payoffs.

4 Applications

The bulk of the literature on stochastic games considers Markov settings with perfect al-most perfect information and discounted payoffs. Ours is, to our knowledge, the first proofof existence of notions of trembling hand perfect and sequential equilibrium for generalstochastic games without perfect information.

We now provide two important applications of our main theorem. The first applicationillustrates how a small amount of noise in the observation of the state can ensure the exis-tence of an equilibrium in a stochastic game. The second application applies our results togames with stochastic move opportunities.

4.1 Markov stochastic games with imperfect signals

Consider a stochastic game in which the distribution of the period t state µω(ωt |ω t−1,at−1)

depends only on the previous state visited and action (i.e. we can write µω(ωt |ωt−1,at−1)).

37Our methods do not accommodate infinite action spaces. The natural extension would be to allow fordistributions over actions and consider the space of “prob(·,σ)” functions as we have done. This space,however—even abstracting of the issue of strategic entanglement—need not be closed (i.e. the limits ofthese functions may not be probability distributions).

15

Players observe all past play but they can only observe a noisy signal of the state beforethey can move in each period. More specifically, suppose that the state space Ω is a Polishvector space with distance d and player i’s signal at time t is given by

si,t = ωt + εi,

where ωt is the state at time t and εi,t is a “noisy” random variable with support in a set Ξi,satisfying Assumption 3, below. Let the joint distribution of (εi)i∈N be denoted µε and themarginal of εi be denoted µεi . We call a stochastic game as described a discounted Markov

stochastic game with imperfect signals.38

Assumption 3. The joint measure of (εi)i∈N is (a) independent of ω , independent across

periods and absolutely continuous with respect to µε1 × . . .× µεn and (b) for each j ∈N and each measurable B ⊆ Ξ := ×i∈N\ jΞi if µε(B|ε j) > 0 there is η > 0 such that

µ(B− z|ε j− z)> 0 where z = (z)i∈N\ j for z ∈Ω such that d(0,z)< η .

Assumption 3 is satisfied if Ω ⊆ Rl for l ≥ 1, and dµε(ε1, . . . ,εn) =

f ε(ε1,ε2, . . . ,εn)dλ (ε1) . . .dλ (εn), where λ is the Lebesgue measure and almost-surelyin ε , f (ε) > 0 implies f (ε) > 0 for ε in a vicinity of ε . In applied work it is common toassume imperfect information of this form.

Notice that the restrictions imposed by Assumption 3 are different than the require-ments in Duggan (2012), where all players observe the state perfectly but the state has acomponent which is sufficiently noisy.

Proposition 1. A stochastic game with imperfect signals that satisfies Assumption 1 has a

trembling hand perfect equilibrium.

The result follows from Theorem 1 and the fact that Assumption 3 implies Assumption2.39

4.2 Games with stochastic move opportunities

There is a growing literature that considers games with stochastic move opportunities. Inthese games, players are drawn to move in an interval [0,T ) with T ∈ R+∪∞. Many of

38This information structure allows perfect or imperfect observation of previous actions. In fact, informa-tion about previous play can, for example, be conveyed by a coordinate of the state.

39The proof shows that a weaker assumption (Assumption 6 which is implied by Assumption 3) thatrequires absolute continuity of the joint distribution of the period t realization of the state and the signalrealization profiles up to period t, with respect to the product measure of the marginals of these two objects.

16

these games can be modeled as a stochastic game as follows.40 In each period j a movingtime t j ∈ [0,T ), a set of players N j, and an underlying state γ j in a set of states ϒ are drawn.These are part of the state of the stochastic game. There is an absorbing state γend ∈ ϒ thatrepresents the end of the game. Once the game ends players receive zero flow payoffs. Theevolution of the state is as follows. Let γ0 ∈ ϒ denote the initial underlying state and it isdrawn from a distribution α(·| /0) : ϒ→ [0,1]. The first moving time is drawn from a densityf1(·|γ0) : [0,T ]→R, with respect to the Lebesgue measure over [0,T ], and the set of playerswho move at time t1, N1, are drawn according to a probability distribution χ1(·|γ0) ∈ ∆2N .After t1 is drawn a new state γ1 is drawn according to distribution α(·|γ0, t1). Recursively,the j’th timing of play is drawn from a density f j(·|γ j−1, t j−1) : (t j−1,T ]→ R, the subsetof players who move, N j, are drawn according to χ j(·|γ j−1, t j−1) and the state γ j is drawnaccording to α j(·|γ j−1, t j). In our notation the period 0 state is given by ω0 = γ0 andthe period j state is given by ω j = (γ j, t j,N j). Let µ(|σ) denote the measure over H t

generated by f j j∈N, α , and a strategy profile σ .The set of histories in which the game ends at time j is given by H j = h ∈

H j|ω(h) j,1 = γend, ω(h) j−1,1 6= γend. If each gi is bounded, the following Lemma showsthat Assumption 1 holds if the expected number of moves is finite.

Lemma 1. Assumption 1 holds if suph∈H |gi(h)|< ∞ and supσ∈Σ ∑ j∈N j ·µ(H j|σ)< ∞.

Assumption 2 holds if, for example, only one player is drawn to move at each movingtime (the game is asynchronous), each player observes her own moving time but not heropponents’, and at most one player observes the realization of the state variable γl for eachl ∈ N if ϒ is uncountable. Players may observe also a countable signal that is informativeabout their opponents’ moving times and the state. As an example, the signal space ofplayer i can be given by Si = [0,T )×Pi, where Pi is a countable partition of [0,T )n−1.

These examples accommodate settings in which players move “almost” simultaneouslyas each player could observe their own moving time but observe others’ moving times upto an arbitrarily small unit of time.

From Theorems 1 and 2 any game with stochastic moves that satisfies Assumptions 1and 2 has a trembling hand perfect equilibrium that is a weak sequential equilibrium.

Revision Games A leading example of games with stochastic move opportunities is re-vision games. A revision game is a game with stochastic move opportunities in which

40Games in continuous time are generally not representable as stochastic games. We show that gameswith stochastic move opportunities are an exception.

17

there is an underlying normal form game that is fixed throughout. Players can revise theirchoices in the normal form game at their move opportunities that arrive before a fixeddeadline (T < ∞) at constant Poisson rates. State γend is reached when a time larger thanthe deadline is drawn.

Revision games model settings in which, even though players do not move simultane-ously at prescribed fixed times as in a repeated game, they also cannot move at every timein the continuum. Some reasons for these assumptions, in applications, include the possi-bility that there is randomness in the players’ response time or that players’ face unforeseenevents that can prevent them from acting at any time at will. As the rate of arrival of op-portunities tends to infinity, however, a revision game resembles a continuous time gameas players are able to move often. Our results show that revision games can be a suitableoption to model continuous time settings, as strategies are well-defined, and Assumption 1holds—as discussed above41—and, therefore, Assumption 2 guarantees that sequentiallyrational equilibria exist.

Kamada and Kandori (2019) were first to introduce revision games. Calcagno et al.(2014) study equilibrium selection in a revision game built on an opposing interest stagegame. Gensbittel et al. (2017) study revision games with an underlying zero-sum stagegame. Kamada and Kandori (2019) show that cooperation can arise in several applicationsof revision games, such as games of price competition, exchange of goods and electioncampaigns. Kamada and Sugaya (2020) consider a model of dynamic posturing before anelection.

In concurrent work, Lovo and Tomala (2015) show existence of a Markov Perfect Equi-librium in games with stochastic timing of moves with almost perfect information. Theirresult holds for a finite state space ϒ and Poisson distributed opportunities.

5 Existence of Equilibrium: Analysis

In this section we show existence of a trembling hand perfect equilibrium. We then showthat a THPE is a Nash Equilibrium that is sequentially rational with respect to beliefs.

41The expected number of moves is bounded by maxi∈N λiT where λi is the rate at which player i’sopportunities to move realize.

18

5.1 Existence of trembling hand perfect equilibrium

The bulk of the proof of existence of trembling hand perfect equilibria consists of provingthe existence of ε-constrained equilibrium. The key steps are showing that under Assump-tion 2 the space of strategies is closed in L2(H , µ), and hence compact—by Banach-Alaoglu as it is norm-bounded—and that players’ expected payoffs are continuous withrespect to the topology of weak convergence of strategies.

Fix an (ε,ν)-tremble profile ε = (εi)i∈N .The following lemma establishes an important consequence of the noisy observability

of opponents’ signals assumption.

Lemma 2. Suppose Assumption 2 holds. Let σα ⊆ Σ(ε) and let f α := prob(·,σα) be a

net that converges in the weak topology of L2(H , µ) to f ∗. Then there is a strategy σ∗ such

that f = prob(·,σ∗) and the net σα has a subnet that converges weakly in strategies to

σ∗. Conversely, if σα converges weakly in strategies to σ∗ then prob(·,σα) has a subnet

that converges to prob(·,σ∗) in the weak topology of L2(H , µ).

Lemma 2 shows that weak convergence in probabilities and weak convergence in strate-gies are closely related. The Lemma would be immediate under almost sure convergence,but is not under weak convergence. In fact, the assumption that players observe others’information noisily plays an important role. It ensures that the candidate limit strategythat corresponds to the limit of probability functions is indeed a strategy—i.e. it does notfeature correlation between unobserved actions, which would make a player’s strategy notmeasurable with respect to her information.42,43 Under the noisy observation assumption,

42This issue, sometimes called “strategic entanglement”, has been discussed by other authors. See ex-ample 2 in Milgrom and Weber (1985), and discussions in Harris et al. (1995), Börgers (1991), Simon andStinchcombe (1989) and Myerson and Reny (2019).

43For example, consider a game with two players and 2 actions, A and B. Both players receive a signalthat exactly coincides with a state that uniformly distributed in Ω = [0,1]. Consider a sequence of strategiesin which both players choose A when the state is in an interval

[j

m ·T,j+1m ·T

]for odd j, and choose B,

otherwise. In every set C ∈F∫C

prob((A,A)|ω

)dµ(ω) =

∫C

prob((B,B)|ω

)dµ(ω) = 1/2.

This shows that 1) the strategy associated to the limit does not condition on the timing of moves and putsprobability 1/2 on both players choosing A, and probability 1/2 on both players choosing B in all historiesin Ω. This strategy is not measurable with respect to the players’ private histories, and the expected payoffis not continuous with respect to each player’s strategy (their limit strategy is to put probability 1/2 in eachaction, independently of the other player’s choice). This example is analogous to Example 2 in Milgrom andWeber (1985), Example 2.1 in Cotter (1991) and Example 2.1 in Stinchcombe (2011).

19

players cannot condition their strategies on the exact signal of their opponents becausethe do not observe them precisely, and, therefore, as shown in Lemma 7 in the appendix,the convergence of each player’s strategy inherits the desirable properties of almost sureconvergence with respect to the opponents’ signals.

An important consequence of Lemma 2 is that the space of probability functions asso-ciated to strategies is closed. It also follows from the Lemma that the set

Σprob(ε) = prob(·,σ)|σ ∈ Σ(ε)

is compact.We will say that a net σα ⊆ Σ(ε) converges to σ ∈ Σ(ε) weakly in probabilities if

prob(·,σα) converges to prob(·,σ) in the weak topology of L2(H , µ).44 This propertydefines a topology in Σ(ε), under which, from our previous discussion, Σ(ε) is closed andcompact. Also the function σ → prob(·,σ) from Σ(ε) to L2(H , µ)—the latter endowedwith the topology of weak convergence—is a homeomorphism. In fact, it is invertible aswe can back out σ from prob(·,σ) by setting, for h ∈H t ,

σi(a(h)i,t |hi(h)

)=

1prob(h(t−1),σ)

∑a−i∈X−i

prob((

a(h)i,t ,a−i, ω(h)t , s(h)t ,h(t−1)),σ

),

where the denominator is non-zero µ-a.s. as σ is ε-constrained.The following is a useful consequence of Lemma 2.

Proposition 2. Suppose assumption 1 holds. Then, if prob(·,σα) converges in the weak

topology of L2(H , µ) to prob(·,σ∗), then for every player i ∈ N, Ui(σα) converges to

Ui(σ∗).

Proposition 2 shows that convergence of the probability functions associated to strate-gies implies convergence of each player’s corresponding expected utility.45 Thus, Ui iscontinuous in σ under the topology of weak convergence in probabilities.

Define player i’ s best response correspondence ri : Σε−i ⇒ Σε

i as

ri(σ−i) ∈ argmaxUi(σi,σ−i)|σi ∈ Σi(ε).44Notice that weak convergence in probabilities does not coincide with the notion of convergence in

probability for random variables.45The argument relies on Assumption 1, which guarantees that convergence of the expected payoffs at

each time t implies convergence of the infinite sum by the dominated convergence theorem.

20

Let r : Σ(ε) ⇒ Σ(ε) be the Cartesian product of the ri and let Σ(ε) be endowed with thetopology defined by weak convergence in probabilities.

Lemma 3. The correspondence r is non-empty, convex-valued, and has closed graph.

The proof of Lemma 3 uses Lemma 2 and Proposition 2 to establish that Ui(·,σ−i) iscontinuous and therefore attains its maximum in σi, and to establish that the best responsecorrespondence, r, is closed.

Lemma 3 and the compactness of Σ(ε) imply directly, by the Kakutani-Fan-Glicksberg’s fixed point Theorem, that an ε-constrained equilibrium exists.46

Finally, by Lemma 2 the set Σprob = prob(·,σ)|σ ∈ Σ is compact. Therefore, ex-istence of ε-constrained equilibria for every (ε,ν)-tremble profiles implies existence of aTHPE. This concludes the proof of Theorem 1.

5.2 Nash Equilibrium

Definition 4. A strategy profile σ∗ =×i∈Nσi is a Nash Equilibrium if

Ui(σ∗)≥Ui(σ

′i ,σ∗−i)

for every player i ∈ N.

Lemma 4. A trembling hand equilibrium is a Nash Equilibrium.

The proof is straightforward. There must be a sequence of (εm,νm)-tremble profiles,εm, and εm-constrained strategies, σm, that converge weakly in strategies to a THPE, σ .By Lemma 2 and Proposition 2, the players’ payoffs from σm converge to the payoffs fromσ . Thus, the fact that each σm does not have a profitable deviation implies that σ cannothave profitable deviations.

5.3 Weak Sequential Equilibrium

We now turn to the question of whether a THPE is sequentially rational with respect tobeliefs.

Throughout this section we assume that Ω and Sii∈N are Polish spaces and that if ,? ∈ Si, for a player i ∈ N, then ? is an isolated point of Si.

46Corollary 17.55 in Aliprantis and Border (1999).

21

The set of histories in H t that lead to player i’s private history hi at some time t ≤ t isgiven by,

H ti (hi) :=

h ∈H t |hi(h(t)) = hi, for some t ≤ t

.

For a history h ∈H ti (hi), t(h, hi) is defined as the minimum t such that hi(h(t)) = hi.47

Let H t− := Ωt ×St ×X t−1 be the set of time-t histories before players choose their time-t

actions and let H− := ∪t∈NH t−. The set of histories in H− that generate private history hi

(at the time of observation) is given by

Hi(hi) :=

h− ∈H−|hi(h−) = hi

,

where hi(h−) is the private history of i after h− ∈ H− is realized. We define also,H t

i (hi) := Hi(hi)∩H t−.

We define an assessment as a pair (σ , µ) with σ ∈ Σ, and µ = ×i∈N µ−i a system of

beliefs, with µ−i = ×t∈Nµ t−i. The latter is a probability measure over histories of states,

signals and opponents’ actions—present and future—given each player i’s private history.That is, the belief µ t

−i(Z|hi) is the probability that i assigns to the realization of historiesin a set Z ⊆H t

i (hi) after observing private history hi, assuming she assigns probability 1to her observed actions in each history h ∈ Z.48 Thus, µ t

i is required to be the countingmeasure with respect to player i’s actions. Notice that a system of beliefs is over historiesof any length, not past histories consistent with hi—i.e. histories in Hi(hi)—as is usuallydefined.

Player i’s payoff from assessment (σ , µ) at private history hi is given by

Ui(σi|hi, µ) =∞

∑l=0

∫h∈H l

i (hi)gi(h) · probi(h,σi, hi)dµ

l−i(h|hi),

where probi(h,σi, hi) = probi(h,σi,Hi(h, hi)) and where Hi(h, hi) = l|l ≥ t(h, hi). Inwords, probi(h,σi, hi) is the product of the probability of i’s actions in h, given strategy σi,after private history hi.

A strategy σ induces a distribution over the opponents’ moves in H−, denoted α(·|σ)

and is given by dα(ω t ,st ,at−1|σ) = prob−i(at−1|ω t,(t−1),st,(t−1))dµ t−(ω

t ,st ,at−1), for(ω t ,st ,at−1) ∈ H t

−, where µ t− is defined analogously to µ t , except that µ t does not

47Due to perfect recall t corresponds to the timing of the last signal different from ? at the time of obser-vation.

48By observed actions we mean actions other than a?

22

“count” over the time t actions.49,50 Let αhi(·|σ) : Hi → [0,1] defined as αhi(Hi|σ) =

α

(h ∈H : hi(h) ∈ Hi|σ

). That is, αhi(·|σ) ascribes to each set of private histories

Hi ∈Hi, the measure of the histories in H− (induced by σ ) that lead to private histories inHi.

We now define a notion of Bayesian updating of beliefs requiring the analogue of anapplication of “Bayes’ rule when possible” to our infinite setting.

Definition 5. We say that an assessment (σ , µ) satisfies the generalized Bayes’ rule if forevery player i and t ∈ N

dα(h|σ) = dµt−i(h|hi,σ)×dαhi(hi|σ), (BR)

for h ∈H , hi ∈Hi. If (σ , µ) satisfies the generalized Bayes’ rule we also say that µ is theBayes belief of σ .51

Thus, a belief system of player i satisfies the generalized Bayes’ rule if it correspondsto the conditional distribution over histories induced by the strategy σ , given i’s privatehistory. In a finite setting, our generalized Bayes’ rule coincides with standard Bayesianupdating. In a larger game, however, Bayesian updating is not always possible due to thepossibility that for a positive measure of a player i’s private histories, Hi, for each hi ∈ Hi,Hi(hi) has zero measure even when strategies are totally mixed. The natural analogy tostandard Bayes’ rule is that players should form a conditional belief about their opponent’spast play given their observations in a manner that is consistent with the objective beliefover history realizations given their opponents’ strategies. This is the requirement in equa-tion (BR). By definition, player i’s belief does not contain information about i’s previousplay. Intuitively, whenever i is called to play she recalls her previous actions, and thus herbelief can simply assign probability one to them.

For every strategy profile σ there is a belief µ such that (σ , µ) is weakly consistent.This belief is unique almost surely in hi.52 If σ is ε-constrained, every H ∈H such that

49Notice that by the definition of µ t−, α(·|σ) is the counting measure with respect to i’s actions.

50Formally, the measure µ t−(·) on H t

− is given by

µt(Z) = ∑

at−1∈X t−1

∫Ωt×St

1(ω t ,st ,at−1) ∈ Zdµω,s(ωt ,st |ω t−1,st−1,at−1) · · ·dµ

ω,s(ω1,s1| /0).

for each measurable set Z in H t−.

51As we explain below this belief is unique, almost surely in hi.52See theorem 5.9 in Pollard (2002).

23

µ t(H)> 0, for some t, satisfies αhi(hi(H)|σ)> 0. In analogy to the finite case, due to thealmost sure uniqueness, the distribution µ that makes (σ , µ) weakly consistent is pinneddown almost surely at any such set H. In fact, under Assumption A5.b, µ can be calculatedexplicitly.53

Definition 6. We say that (σ ε , µε), where σ ε is an ε-constrained equilibrium is a ε-

constrained sequentially rational if ∀i,∀hi ∈Hi and ∀σ εi ∈ Σi(ε)

Ui(σεi |hi, µ

ε)≥Ui(σεi |hi, µ

ε).

We say that (σ , µ), with σ ∈ Σ, is sequentially rational if ∀i,∀hi ∈Hi and ∀σ ′i ∈ Σi

Ui(σi|hi, µ)≥Ui(σ′i |hi, µ).

The following Proposition shows that an ε-constrained equilibrium is part of a con-strained sequentially rational assessment.

Proposition 3. Let ε be an (ε,ν)-tremble, for ε,ν > 0, let σ ε be a ε-constrained equilib-

rium, and let µε be the belief such that (σ ε , µε) satisfies the generalized Bayes’ rule. The

assessment (σ ε , µε) is constrained sequentially rational.

From Proposition 3 a THPE is arbitrarily close to a strategy that is sequentially ratio-nal over a set strategies that is arbitrarily close to Σi. In many cases—including in everycountable game—this property will imply that a THPE is sequentially rational with respectto the limit belief. However, in the general case, a difficulty arises due to the discontinuityof players’ payoffs with respect to beliefs over the opponents’ actions. In fact, each playeri’s strategy may become strategically entangled with her belief in the limit, thus, yieldingthe discontinuity. Since the limit of beliefs may yield a distribution that cannot be repre-sented by an L1 function, assumption 2 is no longer sufficient to ensure the “detangling”of the limits. In what follows we define convergence of beliefs and introduce a continuityand regularity “by parts” requirement on the densities of players’ signals that ensures thedesired continuity of payoffs.

We say that a system of beliefs µn converges to µ∗ in the weak−∗ topology of (L∞)∗

if for every t ∈ N, each player i, µn,t−i (·|hi) converges to µ

∗,t−i (·|hi) in the weak-* topology

53See Lemma 12.

24

of (L∞(H ti (hi),µ

t))∗ for every hi in a full measure subset of Hi.54 µ∗(·|hi) is a finitelyadditive measure for each hi ∈Hi.55

Definition 7. An assessment (σ , µ) is quasi-consistent if there is a sequence of assess-ments (σn, µn) weakly consistent, with σn totally mixed, such that σn converges to σ

weakly in strategies, and µn converges to µ in the weak−∗ topology of (L∞)∗.

Definition 8. An assessment (σ , µ) is a weak sequential equilibrium if it is quasi-consistent and sequentially rational.

Our definition of sequential equilibrium is closely related to Kreps and Wilson (1982)’s.One important difference is that we define beliefs to be over past and future play of the

opponents at the time of observation. Kreps and Wilson (1982) in contrast, define thebelief over histories of play up to the present. In a finite game as in Kreps and Wilson(1982), this distinction is irrelevant, as defining the belief in either way yields the sameset of equilibria. In fact, the two notions coincide in countable games. Because eachplayer’s own strategy can be “entangled” with their future play in the limit, however, ourdefinition is needed in order to find existence outside of countable games. Finally, becausethe topology that is used for convergence matters for issues of compactness and continuity,we had to require “weaker” types of convergence than Kreps and Wilson (1982), who usealmost-sure convergence.

Let gti := suph∈H t |gi(h)| and define

gti(h) =

gti if |gi(h)|> 0

0 otherwise.

Assumption 4. For each i ∈ N, ∑∞l=1 maxal∈X l

∫Ωl×Sl gl

i(ωl,al,sl)dµ l(ω l,al,sl)< ∞.

Assumption 4 implies Assumption 1.We now introduce a technical assumption that requires continuity and boundedness

of the probability density introduced in Assumption 2. Define Ti(sti|at−1) := hi(st

i,at−1i ),

where hi(sti,a

t−1i ) is the private history of i after signal history st

i and i’s action his-tory ,at−1

i . Let γhi(·|at−1) : Hi → [0,1] be the measure defined by γhi(Hi|at−1) =

54(L∞(H ti (hi),µ

t))∗ denotes the dual space of L∞(H ti (hi),µ

t)). The unit ball in (L∞)∗, characterized bythe norm‖T‖(L∞)∗ = supT (x)|x ∈ L∞,‖x‖

∞≤ 1, is compact in the weak-∗ topology.

55See, for example, Dunford and Schwartz (1957) for the characterization of (L∞)∗. It is implicit in thisdefinition of convergence that µn(·|hi) belongs to (L∞(Hi(hi), µ))

∗ for each hi.

25

∫St

i1T−1

i (Hi|at−1)dµ ti (s

ti|at−1). That is to say, γhi measures the set of i’s signals that

induce a set of i’s private histories for a fixed sequence of action profiles. For t, t ∈ N andat−1 ∈ X t ,st

i ∈ Sti define At(at−1) = at−1 ∈ X t−1|T−1

i (hi(sti,a

t−1)|at−1) 6= /0.56 At(at−1)

is the set of action profiles that are possible to have realized given an observation of playeri induced by a sequence of signals and actions. At(at−1) may not be empty for t 6= t in agame with past unobserved moves (that is a game in which signal ? may be drawn). Sup-pose that Assumption 2 holds, we define for t, t, t ∈ N with t ≤ t, st

i ∈ Sti , st−i ∈ St

i, andat−1 ∈ X t−1 the function

¯st,t(s

ti,a

t−1,st−i, a

t−1) = log f s(st−i,T

−1i (hi(s

t,(t)i ,at,(t−1))|at−1), at−1). (2)

Assumption 5. Suppose Assumption 2 holds and for each i and for each t there is a

countable collection of sets Pt,i = Pkt,ik∈N with Pk

t,i ⊆ Sti for each k ∈ N and such that

µ ti (S

ti|at−1) = µ t

i (∪k∈NPkt,i|at−1) and,

A5.a ∀ν > 0,k ∈ N,sti ∈ Pk

t,i there is δ > 0 such that if sti ∈ B(st

i,δ )∩Pkt,i then

|`st,t(s

ti,a

t ,st−i, a

t−1)− `st,t(s

ti,a

t ,st−i, a

t−1)|< ν

∀t ≤ t, t ∈ N, at−1 ∈At(at),st−i ∈ St

−i.57

A5.b There is a measure βi : Hi→ [0,1] such that γhi(hi|at−1) has Radon-Nikodym deriva-

tive bi(hi,at−1) with respect to βi, and for every k ∈ N, bi(hi(sti,a

t−1i ),at−1) is con-

tinuous in sti ∈ Pk

t,i, for all at−1 ∈ X t−1.58

A5.c For every k ∈ N, t ∈ N and at−1 ∈ X t−1 there is Rk(·,at−1, t) ∈ L1(Pkt,i,µ

ti (·|at−1))

such that ∣∣∣∣∣∣∣f s(st

i,st−i,a

t−1)

f s(

st,(t)i ,st,(t)

−i ,at−1,(t−1))

bi,t(sti,at−1)

∣∣∣∣∣∣∣≤ Rk(sti,a

t−1, t), (3)

for every sti ∈ Pk

t,i ∩ Sti(a

t−1) and st−i ∈ St

−i, where bi,t(sti,a

t−1) :=

56Notice that At(at−1) does not depend on the choice of sti ∈ St

i .57We use the convention that `s

t,t(sti,at ,st

−i, at−1) − `st,t(s

ti,at ,st

−i, at−1) = 0 if

`st,t(s

ti,at ,st

−i, at−1), `st,t(s

ti,at ,st

−i, at−1) =−∞.58Due to the Radon-Nikodym Theorem such βi exists. The restrictiveness of the assumption lies in the

continuity over sti in each cell element Pk

t,i.

26

bi

(hi

(st,(t)

i ,at−1,(t−1)),at−1,(t−1)

)and St

i(at) :=

st

i ∈ Sti|bi,t(st

i,at) 6= 0

.

A stochastic move opportunity game with partitional observation of the opponents’moving times and Poisson arrivals, such as a revision game, satisfies Assumption 5. Theassumption requires that in each cell of an almost-sure countable partition of each player’ssignal space, a) the log of the density of the players’ signals satisfies a continuity require-ment, b) the density over private histories of each player’s individual signal is continuouswith respect to the signals that generate the private histories, and c) there is “sufficientboundedness” (above and below) of the previously mentioned densities.

Theorem 2. Let Γ be a stochastic game that satisfies Assumptions 4 and 5. Then every

trembling hand perfect equilibrium of Γ is a weak sequential equilibrium.

6 Approximating Games

In this section we study conditions under which stochastic games admit approximatinggames whose equilibria approach equilibria of the original game.

Let us assume that Ω and Si for i ∈ N are Polish spaces.Define the correspondence A : ∪t∈NΩt → ∪t∈NX t as A(ω t) =

at ∈ X t |(at)i,τ ∈ Ai

(ω t,(τ),at,(τ−1)

), for τ ≤ t

, if ω t ∈ Ωt . A is a correspon-

dence that yields for every length-t vector of realized states, the set of length-t actionprofiles that are feasible given each player i’s feasible actions correspondence, Ai.

Definition 9. We say that a stochastic game Γ has an approximating game sequence ifthere are sequences of countable partitions of the state space Ω, PΩ

n n∈N, and of eachspace of signals Si, PSi

n n∈N, such that

(a) A(ω t) = A(ω t) for each ω t ∈ PΩ,tn (ω t), where PC ,t

n (ω t), for C ∈ Ω,Sii∈N, de-notes the element of the partition PC ,t

n :=×tl=1P

Cn to which ω t belongs to.

(b) For every ε > 0, and ω t ∈Ωt , at ∈ X t there is n such that

|gi(ωt ,at)−gi(ω

t ,at)|< ε,

for every ω t ∈ PΩ,tn (ω t), i ∈ N.

27

(c) For every at ∈ X t , B1 ⊆ Ωt , B2 ⊆ St , open sets, the sets B1,n = ω t |ω t ∈ P1 ∈PΩ,t

n ,P1⊆B1 and B2,n = st |st ∈P2 ∈PS,tn ,P2⊆B2, with PS,t

n :=×tl=1×i∈N PSi

n ,satisfy59

limn→∞

µω,s(B1,n,B2,n|at) = µ

ω,s(B1,B2|at)

In words, a game has an approximating game sequence if there are sequences of un-countable partitions of the state and signal spaces such that (a) along the sequence theavailable actions are the same at every element of the partition, (b) each player’s flow pay-off gets closer and closer within each partition cell as the sequence evolves and (c) foropen sets B1 ⊆ Ωt and B2 ⊆ St , the measure of the partition cells contained in B1 and B2

converges to the measure of B1 and B2, respectively.We say that the stochastic game has a finite approximating game sequence if PΩ

n andPSi

n are finite for each i ∈ N and n ∈ N.

Consider a game Γ that has an approximating game sequence. We define a n’th approx-

imating game of Γ, as a stochastic game Γn = (N,(Ωn,Fn),(Si,n,Si),(Xi,Ai,gi),(µωn ,µs

n)),such that the state space Ωn and signal spaces, Si,n, are of the form Ωn = ωpp∈PΩ

n

and Si,n = si,pipi∈PSin

, with ωp ∈ p and si,pi ∈ pi for each p ∈PΩn and pi ∈PSi

n . The

measure over Ωtn and St

n, induced by µωn and µs

n, denoted µω,sn , satisfies µ

ω,sn (ω t , st |at) =

µω,s(PΩ,tn (ω t),PS,t

n (st)|at) for each t ∈ N.Given a strategy in the approximating game Γn, σn, we can define an associated strat-

egy in Γ, σi,n(ai|hi(ωt ,st ,at)) = σi,n

(ai|hi(ωn(ω

t),sn(st),at)), where ωn(ω

t)∈ PΩ,tn (ω t)∩

Ωtn and sn(st) ∈ Ps,t

n (st)∩Stn .

In the following Proposition convergence is in the weak topology of L2(H , µ)

Proposition 4. Let Γ be a stochastic game that satisfies Assumption 1 and let Γn be a

sequence of approximating games. For each n, let σn be a ε-constrained equilibrium of Γn

for some (ε,ν)-tremble, ε , and let σn be its associated strategy in Γ. If there is a strategy σ∗

in Γ such that prob(·, σn)→ prob(·,σ∗), and for each i ∈ N, probi(·, σni )→ probi(·,σ∗i )

and prob−i(·, σn−i)→ prob−i(·,σ∗−i) then σ∗ is a ε-constrained equilibrium of Γ.

Proposition 4 can be useful in settings where it is easier to derive properties of equilibriaof an approximating game that—due to the convergence in the Proposition—translate toproperties of the equilibria of the original stochastic game. It may also be a step to showexistence. For example, if a game has an approximating sequence our results imply that

59µω,s(·|ω l ,al) is the joint measure over Ω×S induced by µω(·|ω l ,al) and µs(·|ω l+1,al).

28

equilibria exist along the sequence under Assumption 1. Thus, convergence to a strategy asrequired by the Proposition, implies existence of an ε-equilibrium in the orginal stochasticgame. By Lemma 7 in the appendix the noisy observability condition implies that theequilibria of an approximating sequence have a convergent subsequence that converges toa strategy as required by Proposition 4.

We say that a game is Markovian in payoffs and actions if for each player i the payofffunction gi depends only on the last state visited and action profile, and the available actioncorrespondence Ai depends only on the present state.60 That is, for ω t ∈ Ωt and at ∈ X t ,we can write gi(ω

t ,at) ≡ gi((ωt)t ,(at)t) and Ai(ω

t ,at−1) ≡ Ai((ωt)t ,(at−1)t) for some

gi : Ω×X → R and Ai : Ω→ Xi.61

The following result establishes that if a game is Markovian in payoffs and actions,and the payoffs and the measure of states and signals are well behaved, then the stochasticgame has a finite approximating game sequence.

Proposition 5. Let Γ = (N,(Ω,F ),(Si,Si),(Xi,Ai,gi),(µω ,µs)) be a stochastic game

Markovian in payoffs and actions. If Γ satisfies Assumption 1, Ω is compact and, for

each i ∈ N, Si is compact, Ai is upper-hemicontinuous and gi is continuous, then Γ has a

finite approximating game sequence.

For the proof we exploit the fact that by compactness and continuity, the state and signalspaces can be covered by finitely many sets chosen so that within each set the players’ flowpayoffs are at most at any given (small) distance apart. These sets define the approximatingpartitions as we take the distance between payoffs within each of these sets to zero.

Propositions 4 and 5 suggest an alternative approach to show existence of a tremblinghand perfect equilibrium in stochastic games under Assumption 2.62 The alternative ap-proach, however, requires stronger continuity and metrizability conditions than we need toobtain our main result.

A stochastic game is said to be Markov if it is Markovian in payoffs and actions and themeasure over states in each period depend only on the previous period’s state and actions.Strategies are Markov if they only condition on the period’s state. A Markov game has

60Traditionally, in a Markovian game the distribution of states only depends on the last state that is drawn(and there are no signals due to an almost perfect information assumption). We do not need these conditionsfor Proposition 5.

61Notice that one can always define a non-Markovian game to be Markovian by enlarging the state spaceto comprise all sequences of states visited. The Markovian assumption does have bite, however, in our nextresult because it requires a compact state space.

62Assumption 2 implies the convergence requirements of Proposition 4.

29

discounting if for each player i, gi(ωt ,at) = δ tvi((ω

t)t ,(at)t) for some bounded functionvi : Ω×X → R.

Corollary 1. Let Γ be a asynchronous Markov stochastic game of almost perfect informa-

tion that satisfies the assumptions in Proposition 5. If either Γ has discounting or has a

finite expected length then it has a trembling hand perfect equilibrium in Markov strategies.

The result follows from the fact that finite Markov stochastic games have Markov Per-fect equilibria.63 By Proposition 5 the game has a finite approximating game sequence. ByProposition 4 the ε-constrained Markov equilibria of the approximating game converge toequilibria of the original game. The convergence required in Proposition 4 follows fromthe asynchronicity of the players’ strategies and the independence of player’s strategies onprevious realized states along the approximating sequence.

Corollary 1 implies, in particular, that asynchronous revision games have a MarkovPerfect Equilibrium.

A Appendix: Proofs

PROOF OF PROPOSITION 1We will show that the joint measure of the players’ signals up to period t is absolutely

continuous with respect to the product measure of player’s marginals over their signals,thus satisfying assumption 2.

We will show that the following weaker assumption, is sufficient for Assumption 2 andwe will then show that Assumption 3 implies Assumption 6,

Assumption 6. (a) Assumption 3 (a) holds and

(b) for each j ∈ N and at−1 ∈ X t−1, the joint measure of the period t realization of the

state ωt , and the sequence of profiles of signal realizations up to time t, st , condi-

tional of j’s signal realizations stj, µΩ,St

(·|stj,a

t−1), is absolutely continuous with

63By Sobel (1971) a finite Markov game of almost perfect information with discounting has a MarkovPerfect equilibrium. A simple extension of Sobel (1971) shows that a finite approximating game has an ε-constrained Markov equilibrium in a game that ends in finite time with probability 1. In fact, the expectedpayoff (vi

δin his notation) is continuous in ε-constrained strategies. To see this, notice that the end of the

game can be modelled as an absorbing state. Thus, as in Sobel (1971), the expected payoff can be computedfrom the geometric series of the Markov matrix of the stochastic game and this expected payoff is continuous,even if there is no discounting, because the geometric series of a Markov matrix with an absorbent state isconvergent.

30

respect to the product measure of the corresponding marginals, µΩ(·|stj,a

t−1) and

µSt(·|st

j,at−1).

In the rest of the proof of Proposition 1, we omit the dependence on at−1 for ease ofnotation.

Lemma 5. Assumption 6 implies Assumption 2 in a Markov stochastic game with imperfect

signals.

Proof. Let µP denote the product measure of the marginals and let µJ be the joint measure.Suppose set B is such that µJ(B)> 0. Let us show that under Assumption 6, µP(B)> 0.

As B is a measurable set B⊆ St it can be written as

B = (s1,s2, . . . ,st) ∈ St |s1 ∈ S1,s2 ∈ S2(s1), . . . ,st ∈ St(s1, . . . ,st−1),

where for each τ ≤ t, (s1, . . . ,sτ−1) ∈ Sτ , Sτ(s1, . . . ,sτ−1) is a measurable set in S.64 Simi-larly, Sτ ca be written as

Sτ(s1, . . . ,sτ−1) = (s1, . . . ,sn) ∈ S|si ∈ Sτi (s1, . . . ,sτ−1)(s1, . . . ,si−1), i ∈ 1, . . . ,n.

Slightly abusing language, we will say that a set B is contained in ×kj=1S j if B = ˆB×

(×nj=k+1S j) for some ˆB⊆×k

j=1S j. We argue by induction on t ∈ N and the largest k suchthat St(s1, · · · ,st−1) is contained in ×k

j=1S j for every (s1, · · · ,st−1) ∈ St−1.Suppose t = 1 and k = 1 and let B be such that B = B× (×n

j=1S j). By the definition ofthe marginal distribution µJ(B) =

∫B dµ1

1 (s1) = µP(B).Suppose that the result holds true for time t and k−1 or time t−1 and k = n. We will

show that µP(B)> 0 for t and k, and t and k = 1, respectively.Let Bt−1 = (s1,s2, . . . ,st−1) ∈ St−1|s1 ∈ S1,s2 ∈ S2(s1), . . . ,st ∈

St−1(s1, . . . ,st−2) and let Bt−1,k−1 = (s1,s2, . . . ,sk−1) ∈ S1 × S2 × . . .Sk−1|si ∈St

i(s1, . . . ,st−1)(s1, . . . ,si−1), i ∈ 1, . . . ,k−1 and Bt−1k (s1, . . . ,st−1)(s1, . . . ,sk−1) =

Stk(s1, . . . ,st−1)(s1, . . . ,sk−1). In what follows µ(·) denotes the joint measure, according

to the primitives, of the evaluated variables and µ(·|·) denotes a conditional measure.

64See Lemma 4.46 in Aliprantis and Border (1999), as a Polish Space is second countable.

31

µJ(B)> 0 implies

0 <∫

st−1∈Bt−1

∫sk−1∈Bt−1,k−1(st−1)

∫sk∈Bk(sk−1)

dµ(sk|sk−1,st−1)dµ(sk−1|st−1)dµ(st−1) =∫st−1∈Bt−1

∫sk−1∈Bt−1,k−1(st−1)

∫sk∈Bk(sk−1)

dµ(sk|sk−1,st−1) f (sk−1,st−1)dµ11 (s

k−11 |st−1

1 ) . . .µ1k−1(s

k−1k−1|s

t−1k−1),

for some measurable function f : St−1× (× j≤k−1S j)→ R, by the Radon Nikodym Theo-rem and the induction hypothesis. Now,

∫sk−1∈Bk(sk−1)

dµ(sk|sk−1,st−1) =∫ωt∈Ω

∫εk∈Bk(sk−1)−ωt

f ε(sk−11 −ωt , . . . ,sk−1

k−1−ωt ,εk)dµεk(εk)dµ(ωt |sk−1,st−1) =∫

ωt∈Ω

∫εk∈Bk(sk−1)−ωt

f ε(sk−11 −ωt , . . . ,sk−1

k−1−ωt ,εk) f (ωt ,sk−1,st−1)dµεk(εk)dµ

ω(ωt |st−1k ),

where the second equality follows by Assumption 6, which implies dµ(ωt |sk−1,st−1) =

f (ωt ,sk−1,st−1)dµω(ωt |st−1k ), for some measurable function f : Ω×S1×. . .Sk−1×St−1→

R. Now, since∫ωt∈Ω

∫εk∈Bk(sk−1)−ωt

dµεk(εk)dµ

ω(ωt |st−1k ) = µ

1k (Bk(sk−1)|st−1

k ),

it follows that

µP(B)=

∫st−1∈Bt−1

∫sk−1∈Bt−1,k−1(st−1)

∫sk∈Bk(sk−1)

dµ1k (sk|st−1

k )dµ1k−1(s

k−1k−1|s

t−1k−1) . . .µ

11 (s

k−11 |st−1

1 )·

dµt−1k (st−1

k ) . . .dµt−11 (st−1

1 )> 0.

Lemma 6. Assumption 3 implies Assumption 6.

Proof. Let B = (ω,st− j)|ω ∈ Ω,st

− j ∈ St− j(ω) where Ω⊆ Ω and St(ω)⊆ St

− j for eachω ∈ Ω. The joint measure of B, conditional on st

j, is given by,

∫ω t−1∈Ωt−1

∫ωt∈Ω

∫st− j∈St

− j(ωt)dµ

(st− j|ωt ,ω

t−1,stj

)dµ

(ωt |ω t−1,st

j

)dµ(ω t−1|st

j) =∫ω t−1∈Ωt−1

∫ωt∈Ω

∫ε t− j∈St

− j(ωt)−ω(ω t−1,ωt)∏τ≤t

f ε(ε t− j,τ ,s

tj,τ−ω

tτ)dµ

ε

(ε

t− j,τ

)dµ

(ωt |ω t−1,st

j

)dµ(ω t−1|st

j),

32

where ω(ω t−1,ωt) =((ω t−1

1 )i∈N\ j, . . . ,(ωt−1t−1 )i∈N\ j,(ωt)i∈N\ j

)and ω t =

(ω t−1,ωt). This implies∫ωt−1∈Ωt−1

∫ωt∈Ω

dµ(ωt |ω t−1,stj)dµ(ω t−1|st

j)> 0.

It also implies that it is without loss to assume that for each ωt ∈ Ω, there is ω t−1(ωt) ∈Ωt−1 such that∫

εt− j∈St

− j(ωt)−ω(ωt−1(ωt),ωt)∏τ≤t

f ε(ε t− j,τ ,s

tj,τ −ω

tτ(ωt))dµ

ε

(ε

t− j,τ

)> 0,

where ω t(ωt) = (ω t−1(ωt),ωt).By Assumption 3 (b), for each ωt ∈ Ω there is δ (ωt) such that∫

εt− j∈St

− j(ωt)−ω(ωt−1,ωt)∏τ≤t

f ε(ε t− j,τ ,s

tj,τ −ω

tτ)dµ

ε

(ε

t− j,τ

)> 0,

if d((ω t−1, ωt),(ω

t−1(ωt),ωt))< δ (ωt) which implies, for ωt ∈ Ω,

µ(St− j(ωt)|st

j) =∫

ωt∈Ωt

∫εt− j∈St

− j(ωt)−ω(ωt)∏τ≤t

f ε(ε t− j,τ ,s

tj,τ − ω

tτ)dµ

ε

(ε

t− j,τ

)> 0.

This completes our proof.

PROOF OF LEMMA 1Let M be such that suph∈H |gi(h)| < M. Recall that the state at time j is given by

ω j = (γ j, t j,N j). Define

g(at−1,ω t ,st) =

M if γt 6= γend

0 otherwise.

Then, for each player i, |gt,max,t−1i (at−1,ω t ,st)| ≤ g(at−1,ω t ,st).

Let Lσ be a random variable that represents the length of the game given µ(|σ). Wehave

∞

∑t=1

∫Ω×S

gt,max,1(ω1)dµ(ω1,s1| /0)≤M · supσ∈Σ

∞

∑t=1

∑t>t

P(Lσ = t) =M · supσ∈Σ

∞

∑t=1

(t−1)P(Lσ = t),

33

where the inequality obtains from the definition of gt,max, j, and the equality is obtainedby changing the order of the summations. The right hand side is finite under the secondhypothesis of Lemma 1.

PROOF OF LEMMA 2Let us first establish a useful Lemma. Let Y =×i∈NYi be a measurable space with Borel

σ -algebra BY =⊗i∈NBYi , where BYi is the Borel σ -algebra in Yi, and measure µ such that

dµ(y1,y2, . . . ,yn) = g(y1,y2, . . . ,yn)dκ1(y1)dκ2(y2) · · ·dκn(yn),

with each κi : Yi→ [0,1] a finite measure and g ∈ L1(Y, µ). Let κ−i =⊗ j∈N\iκ j.In what follows convergence is in the weak topology of L2(Y, µ).

Lemma 7. Let f αi α , with f α

i : Yi→R, be a uniformly bounded net of functions (indexed

by α) that converges to f ∗i for each i. Then

∏i∈N f αi

αhas a subnet that converges to

∏i∈N f ∗i .

Proof of Lemma 7. We need to show that for every ψ ∈ L2(Y, µ)∫y∈Y

∏i∈N

f αi (yi)ψ(y)dµ(y)→

∫y∈Y

∏i∈N

f ∗i (yi)ψ(y)dµ(y).

Since f αi is uniformly bounded and simple functions are dense in L2 it is sufficient to show

that the equality holds for test functions of the form ψ(y) = ∏ni=1 1Bi for Bi ∈BYi for

each i ∈ 1, . . . ,n.65

We argue by induction. If f α1 converges weakly to f ∗1 then, by the definition of weak

convergence we have∫y∈Y

f α1 (y1)ψ(y)dµ(y)→

∫y∈Y

f ∗1 (y1)ψ(y)dµ(y).

Suppose ∏j−1i=1 f α

i (yi) converges to ∏j−1i=1 f ∗i (yi) in the weak topology of L2(Y, µ). Let’s

see that there is a subnet of ∏ji=1 f α

i (yi) that converges to ∏ji=1 f ∗i (yi).

65Suppose∫( f α − f ∗)ϕsdµ for every simple function ϕs. Let ϕ ∈ L2, there is a sequence of simple

functions ϕn such that ϕn→ ϕ in L2. Thus,∫( f α − f )ϕdµ =

∫( f α − f )(ϕn−ϕ)dµ +

∫( f α − f )ϕndµ. (∗)

The first integral converges to zero uniformly in α since f α − f is uniformly bounded and ϕn → ϕ in L2.Therefore, (∗) converges to zero.

34

Since f αj is uniformly bounded and µ is finite, it is in L∞(Yj,κ j) = (L1(Yj,κ j))

∗ and—by Banach-Alaoglu—it is contained in a compact set. Thus, there is a subnet of f α

j thatconverges weakly to some function f ∗∗j in L∞(Yj,κ j). Passing to the subnet yields,

∫y j∈Y j

f αj (y j)φ(y j)dκ j(y j)→

∫y j∈Y j

f ∗∗j (y j)φ(y j)dκ j(y j), (4)

Since g(·,y− j) is in L1(Yj,κ j), evaluating (4) on φ = g(·,y− j)1B for B ∈ BY andintegrating over Y− j under measure κ− j shows that we must have f ∗j = f ∗∗j , µ-almostsurely.66

By the induction hypothesis it is sufficient to show that

∫y∈Y

(∏

j−1i=1 f α

i (yi)[

f αj (y j)− f ∗j (y j)

]ψ(y)

)dµ(y)→ 0.

Rewriting the left hand side of the previous expression using the Radon-Nikodym deriva-tives yields

Λα(g) =

∫y− j∈Y− j

∏j−1i=1 f α

i (yi)

(∫y j∈Yj

[f α

j (y j)− f ∗j (y j)]

ψ(y j,y− j)g(y j,y− j)dκ j(y j)

)dκ− j(y− j).

(5)

Let ε > 0, and let M > 0 be a bound on∣∣∣∏ j−1

i=1 f αi (yi)

[f α

j (y j)− f ∗j (y j)]∣∣∣. The set of

simple functions with support on B1× ·· · ×Bn|Bi ∈ BYi is dense in L1(Y, µ). Thereis ϕ(y) = ∑

Kk=1 ak1Bk

j1Bk− j, with ak ∈ R, Bk

j ∈ BY j and Bk− j ∈ BY− j for each k ∈

1, . . . ,K, such that‖g−ϕ‖1 <ε

2M . The expression in (5) can be rewritten as

Λα(g)−Λ

α(ϕ)+Λα(ϕ)≤

K

∑k=1

ak

(∫y− j∈Y− j

1Bk− j∏

j−1i=1 f α

i (yi)dκ− j(y− j)

)(∫y j∈Yj

[f α

j (y j)− f ∗j (y j)]

1Bkjdκ j(y j)

)+ M‖g−ϕ‖1 , (6)

where Bk− j = Bk

− j ∩B1× . . .B j−1 and Bkj = Bk

j ∩B j. The first parenthesis in each termof the summation is bounded. Therefore, by (4) there is α such that for every α ≥ α thesummation in (6) is less than ε/2. This shows that for every ε > 0 there is α such that forevery α ≥ α , Λα(g)≤ ε .

66By Theorem 4.48 in Aliprantis and Border (1999), φ(·,y− j) is measurable in Yj.

35

Next is the proof of Lemma 2.

Claim 1. Let σα ⊆ Σ be a net. There is σ∗ and a subnet σ α such that

〈probi(·,σ αi , Hi(·,σ∗)),ψ〉 → 〈probi(·,σ∗i , Hi(·,σ∗)),ψ〉 (7)

for every ψ ∈H t , t ∈ N and i ∈ N (i.e. σ α converges weakly in strategies to σ∗).

Proof of Claim 1. We will argue by induction on the length of the history of play.Consider first the set of histories of length one, denoted H 1, and let µ1 denote the

restriction of µ to H 1. Since µ1 is a finite measure, the restriction of probi to H 1,denoted prob1

i , is in L2(H 1, µ1). Therefore, there is a subnet of σα and a functionprob∗,1i ∈ L2(H 1, µ1), such that 〈prob1

i (·,σαi ),ψ〉 = 〈probi(·,σα

i ),ψ〉 → 〈prob∗,1i (·),ψ〉for ψ with support in H 1. Each player i’ s strategy after the empty history is defined asσ∗i ((a(h))1,i| /0) = prob∗,1i (h). Thus, condition (7) is satisfied for ψ with support in H 1.

Now suppose we have defined σ∗i (·|hi(h)) at each h ∈H t−1 and that there is a subnetof σα

i (which slightly abusing notation we also denote σαi ) such that for every j ≤ t− 1,

〈probi(·,σαi , Hi(h,σ∗)),ψ〉 → 〈probi(·,σ∗i , Hi(h,σ∗)),ψ〉 for ψ with support in H j.

As before, we can find a subnet of σα and a function prob∗,ti such that, passing tothe subnet, 〈probi(·,σα

i , Hi(h,σ∗)),ψ〉 → 〈prob∗ti (·),ψ〉 for every ψ with support in H t

(notice that Hi(h,σ∗) depends on h via h(t−1) and, therefore, it is well defined by theinduction hypothesis).

We now define σ∗(·|hi(h)) for h ∈H t by setting

σ∗i ((a(h))t,i|hi(h)) =

prob∗,ti (h) if prob∗,t−1i (h(t−1)) = 0

prob∗,ti (h)/prob∗,t−1i (h(t−1)) if prob∗,t−1

i (h(t−1))> 0.

By definition, 〈probi(·,σαi , Hi(h,σ∗)),ψ〉 → 〈probi(·,σ∗i , Hi(h,σ∗)),ψ〉 for ψ with sup-

port in H t . Furthermore, the resulting σ∗i is a strategy. In fact, σαi measurable with

respect to i’s information implies that σ∗i is measurable with respect to i’s information al-most surely. Also, since σα

i is a probability over available actions, so is σ∗i . Thus, weconclude that σα converges weakly in strategies to σ∗.

Claim 2. If σα converges weakly in strategies to σ∗ then prob(·,σα) has a subnet that

converges to prob(·,σ∗) in the weak topology of L2(H , µ).

36

Proof. Proof of Claim 2We first show that

〈prob(·,σα),ψ〉 → 〈prob(·,σ∗),ψ〉 (8)

holds for ψ with support in H t for each t.Since prob(h,σα) is uniformly bounded, it is enough to show (8) holds for ψ = 1B

for B measurable under µ .67

To see that condition (8) holds for ψ with support in H t , notice that by Assumption 2we can write for every σ ∈ Σ∫

Ht

prob(h, σ)ψ(h)dµt(h) =

∫X t×St

prob(h, σ)∫

Ωtψ(h)dµ

ω(ω t |st ,at)dµs(st |at).

Since∫

Ωt ψ(h)dµω(ω t |st ,at) is bounded and measurable with respect to st for each at ,Lemma 7 implies that prob(h,σα) has a subnet such that,∫

Ht

prob(h,σα)ψ(h)dµt(h)→

∫Ht

prob(h,σ∗)ψ(h)dµt(h).

Now, to see that prob(·,σα) converges in the weak topology of L2(H , µ) notice that

δt∫H t

prob(h,σα)ψ(h)dµt(h)≤ δ

t .

Therefore, the dominated convergence theorem yields the desired result.

Lemma 7 and Claims 1 and 2 establish the second statement of Lemma 2. Thefirst statement follows from Claim 1 that establishes the existence of the limit σ∗.Claim 2 implies that that a subnet of f α converges to prob(·,σ∗). Thus, we must havef ∗ = prob(·,σ∗).

PROOF OF PROPOSITION 2Define for each player i∈N and t ∈N, gi|H t (h) := gi(h) ·1h∈H t for h∈H . Since

gi|H t ∈ L2(H , µ) we have

〈prob(·,σα),gi|H t 〉 → 〈prob(·,σ∗),gi|H t 〉.

Notice that we can write Ui(σ) = ∑t∈N1δ t 〈prob(·,σ),gi|H t 〉. Lemma 8 (below) shows

67See footnote 65

37

that | 1δ t 〈prob(·,σ),gi|H t 〉| ≤

∫Ω×S gt,max,1(ω1,s1)dµ(ω1,s1| /0) for every σ ∈ Σ and t ∈ N.

Therefore, since by Assumption 1, ∑∞t=1∫

Ω×S gt,max,1(ω1,s1)dµ(ω1,s1| /0) < ∞. By thedominated convergence theorem Ui(σ

α)→Ui(σ∗).

Lemma 8. For every σ ∈ Σ, we have∣∣∣∣ 1δ t 〈prob(·,σ),gi|H t 〉

∣∣∣∣≤ ∫Ω×S

gt,max,1(ω1,s1)dµω,s(ω1,s1| /0)

Proof of Lemma 8. First note that

∣∣∣∣ 1δ t 〈prob(·,σ),gi|H t 〉

∣∣∣∣=∣∣∣∣∣∣ ∑at−1∈X t−1

∫Ωt×St

∑at∈X

gi(ωt ,at)prob(at |st ,σ)dµ

ω,s(ω t ,st |at−1)

∣∣∣∣∣∣≤ ∑

at−1∈X t−1

∫Ωt×St

gt,max,t−1i (ω t ,st ,at−1)prob(at−1|st−1,σ)dµ

ω,s(ω t ,st |at−1),

where in the previous equation at = (at−1,at).We argue by induction, assume that

∣∣∣ 1δ t 〈prob(·,σ),gi|H t 〉

∣∣∣≤ Gl where

Gl = ∑al∈X l

∫Ωl+1×Sl+1

gt,max,li (ω l+1,sl+1,al)prob(al|sl,σ)dµ

ω,s(ω l+1,sl+1|al).

for some l ∈ 2, . . . , t. We now show that∣∣∣∣ 1δ t 〈prob(·,σ),gi|H t 〉

∣∣∣∣≤ Gl−1.

In fact, we will show that Gl ≤ Gl−1. We have

Gl = ∑al−1∈X l−1

∫Ωl×Sl

∑al∈X

prob(al|sl,σ)∫

Ω×Sgt,max,l

i (ω l+1,sl+1,al)dµω,s(ωl+1,sl+1|ω l,sl,al)dµ

ω,s(ω l,sl|al−1)

≤ ∑al−1∈X l−1

∫Ωl×Sl

∑al∈X

prob(al|sl,σ)gt,max,l−1(ω l,sl,al−1)dµω,s(ω l,sl|al−1) = Gl−1.

Thus, we obtain,∣∣∣∣ 1δ t 〈prob(·,σ),gi|H t 〉

∣∣∣∣≤ G1 =∫

Ω×Sgt,max,1(ω,s)dµ

ω,s(ω,s| /0).

38

PROOF OF LEMMA 3It is immediate that ri(σ−i) is convex for each i ∈ N and σ−i ∈ Σ−i(ε). Let us now

show that r is non-empty and that it has closed graph.r is non-empty:Define the set Λi(σ−i, ε) := prob(·,σi,σ−i)|σi ∈ Σi(ε), for σ−i ∈ Σ−i(ε). Λi(σ−i, ε)

is closed. In fact, let prob(·,σα ,σ−i) ⊆ Λi(σ−i, ε) be a net that converges toprob(·, σi, σ−i) (By Lemma 2 such a strategy , σi, σ−i, exists). Also by Lemma 2, (σα

i ,σ−i)

has a subnet that converges weakly in strategies to (σi, σ−i). This implies σ−i = σ−i almostsurely.

Since Λi(σ−i, ε) is closed, it is compact. Now, by Proposition 2, Ui(·,σ−i) is continu-ous in prob(·,σi,σ−i) ∈ Λi(σ−i, ε) (with the topology of weak convergence of L2(H , µ)).Thus, Ui(·,σ−i) attains its maximum over Λi(σ−i, ε) (which is homeomorphic to Σi(ε)).

r has closed graph:Let σα and σα be nets such that σα ∈ r(σα), σα → σ∗ and σα → σ∗ weakly inprobabilities. Let us show that σ∗ ∈ r(σ∗) (i.e. r has closed graph).

σα ∈ r(σα) implies that for every σi ∈ Σi(ε)

Ui(σαi ,σα

−i)≥Ui(σi,σα−i) (9)

Also, by Lemma 7 there are subnets of σα and σα that converge weakly in strategiesto σ∗ and σ∗, respectively.

Now, by the definition of weak convergence in strategies, for each player i ∈ N andσi ∈ Σi(ε), passing to the subnet, (σα

i ,σα−i) and (σi,σ

α−i) converge weakly in strategies

to (σ∗i ,σ∗−i) and (σi,σ

∗−i) respectively. Then, by Lemma 7, there are subnets of σα and

σα such that passing to the subnet, prob(·, σαi ,σα

−i) and prob(·,σi,σα−i) converge to

prob(·, σ∗i ,σ∗−i) and prob(·,σi,σ∗−i), respectively, in the weak topology of L2(H , µ).

Finally, by Proposition 2, equation (9) implies σ∗i ∈ ri(σ∗i ) for each i ∈ N.

PROOF OF LEMMA 4Let σ∗ be a THPE and let σm be a sequence of εm-constrained equilibria with each εm

an (εm,νm)-tremble with εm→ 0 converging weakly in strategies to σ∗.

39

Then for every player i and every σ εi ∈ Σi(ε

m)

Ui(σm)≥Ui(σ

εi ,σ

m−i). (10)

Let σi ∈ Σi. There is a sequence σ εm

i m, with σ εm

i ∈ Σi(εm), that converges a.s. to σi.

By Proposition 2 σm and (σm,σm−i) converge weakly in probabilities to σ∗ and (σi,σ

∗−i),

respectively. By Proposition 2 Ui(σm)→Ui(σ

∗) and Ui(σmi ,σm

−i)→Ui(σi,σ−i), which,by (10), yields

Ui(σ∗)≥Ui(σi,σ

∗−i).

PROOF OF PROPOSITION 3The following Lemma establishes Proposition 3

Lemma 9. Let ε be a (ε,ν)-tremble, let σ ε be an ε-constrained equilibrium, and let µε

its corresponding weakly consistent belief, then

Ui(σε |hi(h), µε)≥Ui(σ

′i ,σ

ε−i|hi(h), µε). (11)

for every σ ′i ∈ Σi(ε), µ-almost surely in h ∈H .

Proof of Lemma 9. Suppose there is a set H ⊆Hi, with µ(h ∈H |hi(h) ∈ H

)> 0 and

a strategy σ ′i for each hi ∈ H, such that

Ui

(σ

ε |hi, µε

)<Ui

(σ′i ,σ

ε−i|hi, µ

ε

), (12)

for hi ∈ H.Let dµε

i (hi) =(

∑∞l=0∫

h∈H li (hi)

prob−i(h,σ ε , hi)dµ l−i(h|hi)

)· probi(hi,σ

εi )dαhi(hi)

where probi(hi,σεi ) := probi(h,σ ε

i ) for h such that hi(h) = hi.Define the strategy σ ′i to be equal to σ ε

i for every h 6∈ H and σ ′i (hi) after hi for eachhi ∈ H. Since prob−i(h,σ ε , hi) > 0 and probi(h,σ ε

i ) > 0 for every strategy σ ε ∈ Σi(ε),equation (12) implies∫

hi∈HUi(σi|hi, µ

ε)dµεi (hi)<

∫hi∈H

Ui(σ′i (hi),σ

ε−i|hi, µ

ε)dµεi (hi).

However, from from the definition of αhi and µ l−i(h|hi) the previous expression can be

40

written as

∞

∑t=0

∫h∈Hi(H)∩H t

gi(h)· prob(h,σ ε)dµt(h)<

∞

∑t=0

∫h∈Hi(H)∩H t

gi(h)· prob(h,σ ′i ,σε−i)dµ

t(h),

which contradicts σ ε is ε-constrained equilibrium: strategy σ ′i is a profitable deviation forplayer i.

B Online Appendix

PROOF OF THEOREM 2Let us first establish two useful Lemmas. In what follows Y1 and Y2 are Polish spaces

with Borel sets Y1 and Y2, and η1 and η2 are finite Borel measures over Y1 and Y2, respec-tively.

Lemma 10. Suppose that

L.10.a φ mm∈N is a sequence of functions from Y1×Y2 to R, where φ m(y1, ·) ∈ L1(Y2,η2)

for each y1 ⊆ Y1 ⊆ Y1, and such that the family of functions F = pm : Y1 →L1(Y2,η2) | pm(y1) = φ m(y1, ·),m ∈ N is equicontinuous in y1 ∈ Y1.68

L.10.b There is M : Y1→ R such that ,∥∥φ m(y1, ·)

∥∥L1(Y2,η2)

≤M(y1) for each y1 ∈ Y1.

Then there exists a subsequence mrr∈N and a family of finitely additive finite measures

η∗(·,y1)y1∈Y1 with η∗(·,y1) : Y2→ [0,1], for each y1 ∈ Y1, such that

L.10.1 dηmr(·,y1) := φ mr(y1, ·)dη2(·)→ dη∗(·,y1) for every y1 ∈ Y1 in the weak-* topology

of (L∞(Y2,η2))∗,

L.10.2 η∗(B2,y1) is measurable in y1 for each B2 ∈ Y2

Proof. First notice that by condition L.10.b, due to the compactness of the unit ball in theweak-* topology of

(L∞(Y2,η2)

)∗, for each y1 ∈ Y1 there is a subsequence mrr∈N (thatmay depend on y1) and a finitely additive measure η∗(·,y1) such that∫

Y2

1B2dηmr(y2,y1)→

∫Y2

1B2dη∗(y2,y1). (13)

68For the purposes of this equicontinuity, L1(Y2,η2) is endowed with its norm topology.

41

for every B2 ∈ Y2.Since Y2 is a Polish space, by L.10.a, for each k ∈ N there is a countable open cover of

Y1, V= V kn n∈N, such that69

∥∥φm(y1, ·)−φ

m(y′1, ·)∥∥

L1(Y2,η2)<

1k

for every y1,y′1 ∈V kn ∩ Y1 and m ∈ N.

Let Y = yk,nk,n∈N be such that ykn ∈ V k

n ∩ Y1 for k,n ∈ N. We can find a subsequencemrr∈N such that (13) holds for a limit measure η(·,y1) for every y1 ∈ Y .70 Defineη∗(·,y1)= η∗(·,y1) for y1 ∈ Y . Now, since (13) also holds for any subsequence of mr, wedefine η∗(·,y1), for y1 ∈ Y1 \Y , as the limit, η(·,y1), in (13) of a subsequence mr(y1)r∈N

of mrr∈N. Define η∗(·,y1) as the zero measure for y1 ∈ Y1 \ Y1.The following claim establishes L.10.2 as continuous functions are Borel measurable.

Claim 3. For every ε > 0 and y1 ∈ Y1, if y′1,y1 ∈V kn ∩ Y1 for k ≥ d6/εe then |η∗(B2,y1)−

η∗(B2,y′1)| < ε . Therefore, η∗(B2,y1), restricted to Y1, is continuous in y1 for every B2 ∈Y2.

Proof of Claim 3.Let ε > 0, let y1 ∈ Y1 and k ≥ d6/εe. Let us see that |η∗(B2,y1)−η∗(B2,y′1)| < ε fory1,y′1 ∈V k

n ∩ Y1.We have

|η∗(B2,y1)−η∗(B2,y′1)| ≤ |η∗(B2,y1)−η

∗(B2,ykn)|+ |η∗(B2,y′1)−η

∗(B2,ykn)|.

Let us show that |η∗(B2,y1) − η∗(B2,ykn)| ≤ ε

2 . The proof that |η∗(B2,y′1) −η∗(B2,yk

n)| ≤ ε

2 is analogous.

|η∗(B2,y1)−η∗(B2,yk

n)| ≤ |η∗(B2,y1)−ηmr(y1)(B2,y1)|+

|η mr(y1)(B2,y1)−ηmr(y1)(B2,yk

n)|+ |η mr(y1)(B2,ykn)−η

∗(B2,ykn)|.

By the definition of mr(y1), there is r1 ∈ N such that for every r ≥ r1, |η∗(B2,y1)−η mr(y1)(B2,y1)| ≤ ε

6 . Since mr(y1) is a subsequence of mr there is r2 such that

69A Polish space is Lindelöf.70This follows straightforwardly by means of a Cantor diagonal argument.

42

|η mr(y1)(B2,ykn)−η∗(B2,yk

n)| ≤ ε

6 , ∀r ≥ r2. Finally, since y1,ykn ∈V k

n ∩ Y1, by L.10.a,

|η mr(y1)(B2,y1)−ηmr(y1)(B2,yk

n)| ≤∥∥∥φ

mr(y1)(y1, ·)−φmr(y1)(y′1, ·)

∥∥∥L1(Y2,η2)

≤ ε

6,

∀r ≥maxr1, r2. This concludes the proof of Claim 3.The following Claim establishes L.10.1 by L.10.b and footnote 65.

Claim 4. For every B2 ∈ Y2 and y1 ∈ Y1,∫Y2

1B2dηmr(y2,y1)→

∫1B2dη

∗(y2,y1). (14)

Proof of Claim 4.

Let ε > 0, y1 ∈ Y1 and k ≥ d12εe. Let n be such that y1 ∈ V k

n . We will show that there isr ∈ N such that for r ≥ r, |ηmr(B2,y1)−η∗(B2,y1)|< ε (which corresponds to (14)).

|ηmr(B2,y1)−η∗(B2,y1)| ≤ |ηmr(B2,y1)−η

mr(B2,ykn)|+ |ηmr(B2,yk

n)−η∗(B2,yk

n)|

+ |η∗(B2,ykn)−η

∗(B2,y1)|

First, |ηmr(B2,y1)− ηmr(B2,ykn)| ≤

∥∥∥φ mr(y1, ·)−φ mr(ykn, ·)∥∥∥

L1(Y2,η2)≤ ε/12 by the def-

inition of V kn and yk

n. Second, by the definition of mrr, there is r ∈ N such that|ηmr(B2,yk

n)− η∗(B2,ykn)| ≤ ε/4 for r ≥ r and, finally, by Claim 3, since k ≥ d12

εe,

|η∗(B2,ykn)−η∗(B2,y1)| ≤ ε/2. This concludes the proof of Claim 4.

Lemma 11. Suppose that

L.11.a f mm∈N is a bounded sequence of functions in L∞(Y1,η1) such that f m→ f ∗ in the

weak-* topology of L∞(Y1,η1),

L.11.b Conditions L.10.1 and L.10.2 hold with Y1 a full η1-measure set and L.10.b holds for

M ∈ L1(Y1,η1).

Then for every h ∈ L∞(Y1×Y2,η1⊗η2)∫Y 1×Y2

h(y1,y2) f mr(y1)dηmr(y2,y1)dη1(y1)→

∫Y 1×Y2

h(y1,y2) f ∗(y1)dη∗(y2,y1)dη1(y1),

(15)as r→ ∞.

43

Proof. We start by proving a simple consequence of L.11.a and L.11.b.

Claim 5. For every B1 ∈ Y1 and B2 ∈ Y2∫Y 1×Y2

1B1×B2 f mr(y1)dη∗(y2,y1)dη1(y1)→

∫Y 1×Y2

1B1×B2 f ∗(y1)dη∗(y2,y1)dη1(y1),

(16)

Proof of Claim 16.∫

Y21B2dη∗(y2,y1) = η∗(B2,y1) is a measurable function in

L1(Y1,η1) by L.11.b. Therefore, by L.11.a, (16) follows.

Finally, notice that is it enough to prove to that (15) holds for h(y1,y2) = 1y1 ∈B1,y2 ∈ B2 for B1 ∈ Y1 and B2 ∈ Y2 (by footnote 65 and L.11.b). From this observationand Claim 5, (15) is equivalent to∫

Y 11B1 f mr(y1)

(η

mr(B2,y1)−η∗(B1,y1)

)dη1(y1)→ 0. (17)

Let `mr(y1) = ηmr(B2,y1)− η∗(B1,y1). From L.11.b, `mr(y1)→ 0 and |`mr(y1)| ≤2M(y1) for every y1 ∈ Y1. By the boundedness condition in L.11.a, equation (17) thenfollows by the the dominated convergence theorem, as Y1 is of full measure.

Define Ti(sti|at−1) := hi(st

i,at−1i ), where hi(st

i,at−1i ) is the private history of i after signal

history sti and i’s action history ,at−1

i .Let bi(hi,at−1) be the Radon-Nikodym derivative of γhi(hi|at−1) with respect to a mea-

sure βi which exist by Assupmtion 5.

Lemma 12. Let σ be an ε-constrained strategy for (ε,ν) tremble ε . Under Assumption 5

the following holds

L.12.1 the measure αhi(·|σ) in Definition 5 is given by

dαhi(hi|σ) = υ(hi,σ)dβi(hi),

where

υ(hi,σ) = ∑t∈N

at−1∈X t−1

∫St−i

prob−i(at−1−i |s

t−1−i ,σ) f s(st

−i, hi, at−1)dµ−i(st−i|at−1),

and f s(st−i, hi, at−1) = f s(st

−i,T−1

i (hi|at−1), at−1)bi(hi, at−1) ·1T−1i (hi|at−1) 6= /0.

44

L.12.2 The measure µ t−i(·|hi,σ) is given by,

µt−i(h|hi,σ)=ϕ(at ,st , hi,σ)dµ

ω(ω t |st , hi,at,(t−1))dµ−i(st−i|at,(t−1))dµ

ti (s

ti|hi,at,(t−1)),

(18)for h = (ω t ,at ,st), where

ϕ(at ,st , hi,σ)=1

υ(hi,σ)prob−i(h,σ) f s(st ,at,(t−1))bi(hi,at,(t(h,hi)−1))1h∈H t

i (hi),

(19)and µ t

i (·|hi,at,(t−1)) : Sti → [0,1] is the measure such that µ t(st

i|at,(t−1)) =

µ ti (·|hi,at,(t−1))× γhi(hi|at,(t−1)), where t(h, hi) is the minimum t ∈ N such that

hi(st,(t)i ,at,(t)) = hi.

Proof. To establish L.12.1 notice that from the definition of αhi(·|σ) we have

αhi(Hi|σ)= ∑t∈N

∫(ωt ,st ,at−1)∈H t

i (Hi)prob−i(at−1|st−1

−i ,σ) f s(st−i,s

ti,a

t−1)dµω(ω t |st , hi,at−1)dµ−i(st

−i|at−1)dµti (s

ti|at−1)

(20)

Notice first that the integral with respect to ω t integrates to 1. Notice also that we canwrite

dµti (s

ti|at−1) = dγi(st

i|at−1, hi)×dγhi(hi|at−1)

with γi(sti|at−1, hi) = 1T−1

i (hi|at−1) = sti. Replacing into (20), we obtain αhi(Hi|σ) =∫

Hiυ(hi,σ)dβi(hi).An analogous argument shows L.12.2 by the definition of γhi and µ t

i (·|hi,at,(t−1)).

Let σ be THPE and let σm be a sequence that converges to σ weakly in strategies,where σm is an εm-constrained equilibrium, with εm an (εm,νm)-tremble profile such thatεm→ 0. Let µm be the belief profile, weakly consistent with σm. By Lemma 12, µ

m,t−i is

given by equation (18).Fix t and t ≤ t. Define the function

φmt,t(a

t ,st) =1

υ

(hi

(st,(t)

i ,at,(t)),σm

) prob−i(at−i|st−i,σ

m) · f s(st ,at), (21)

with υ defined in L.12.1.

Lemma 13. Under Assumption 5, for each t, player i, Pt,i ∈ Pt,i, t ≤ t and at ∈ X t ,

45

L.13.1 the family of functions

Fφ=

pm : Pt ∩ Sti(a

t)→ L1(St−i,µ−i(·|at,(t−1)))|pm(st

i) = φmt,t(a

t ,sti,s

t−i),m ∈ N

is equicontinuous at every st

i ∈ Pt,i∩ Sti(a

t), with Sti(a

t) :=

sti ∈ St

i|bi,t(sti,a

t) 6= 0

,

and bi,t(sti,a

t) := bi

(hi

(st,(t)

i ,at,(t−1)),at,(t−1)

).

L.13.2 There is M ∈ L1(Sti,µ

ti (·|at,(t−1))) such that

∥∥∥φ mt,t(s

ti, ·)∥∥∥

L1(St−i,µ−i(·|at,(t−1)))

< M(sti) for

every sti ∈ Pt ∩ St

i(at).

Proof. In what follows ‖·‖1 denotes ‖·‖L1(St−i,µ−i(·|at−1)). For t ∈ N and at−1 ∈ X t−1,

‖·‖∞,t,at−1 denotes the norm in L∞(St

−i,µt−i(·|at−1)).

The following Claim establishes L.13.1.

Claim 6. Let ε > 0. For every sti ∈ Pt ∩ St

i(at) there is δ > 0 such that for s′ti ∈ B(st

i,δ )∥∥∥φ mt,t(s

ti, ·)− φ m

t,t(s′ti , ·)∥∥∥

1≤ ε , ∀m ∈ N.

Proof of Claim 6.For st

i ∈ Pt ∩ Sti(a

t), t ≤ t and t ∈ N, let

f st (s

ti,s

t−i, a

t−1) := f s(

st−i,hi

(st,(t)

i , at,(t−1)i

), at−1

).

Notice we can write

φmt,t(a

t , sti,s

t−i) =

prob−i(at−i|st

−i,σm) f s(st

i,st−i,at,(t−1))

∑t∈N,at−1∈X t−1∫

st−i∈St

−iprob−i(at−1|st−1

−i ,σm) f s

t (sti, st−i, at−1)dµ t

−i(st−i|at−1)

. (22)

Let υ(sti,σ

m) denote the denominator on the right hand side of (22) as a function of sti and

σm.From Assumption 5, there is R(st

i,at,(t−1), t) ∈ L1(Pt,i,µ

ti (·|at,(t−1))) such that∣∣∣∣∣∣∣

f s(sti,s

t−i,a

t,(t−1))

f s(

st,(t)i ,st,(t)

−i ,at,(t−1))

bi,t(sti,at)

∣∣∣∣∣∣∣≤ R(sti,a

t,(t−1), t), (23)

46

for every sti ∈ Pt ∩ St

i(at), st

−i ∈ St−i and t ≤ t. From (22) we have

∥∥∥φmt,t(a

t , sti,s

t−i)∥∥∥

1≤

∥∥∥∥∥∥∥prob−i(at

−i|st−i,σ

m) f s(st,(t)i ,st,(t)

−i ,at,(t−1))

υ(sti,σ

m)·

f s(sti,st−i,at,(t−1))

f s(

st,(t)i ,st,(t)

−i ,(at,(t−1))∥∥∥∥∥∥∥

1

≤

∥∥∥∥∥∥ prob−i(at−i|st

−i,σm) f s(st,(t)

i ,st,(t)−i ,at,(t−1))bi,t(st

i,at)

υ(sti,σ

m)

∥∥∥∥∥∥1

·R(sti,a

t,(t−1), t)≤ R(sti,a

t,(t−1), t),

(24)

where the third inequality follows the the fact that the numerator inside the norm is a termin the sum in υ(st

i,σm).

Let t ∈ N and sti ∈ Pt ∩ St

i(at). From Assumption 5, for every ν > 0 there is δ > 0 such

that for s′ti ∈ B(sti,δ )∩Pt ∩ St

i(at), we can write

f s(s′ti ,st−i,a

t,(t−1)) = (1+ϕ(st−i, s

′ti )) · f s(st

i,st−i,a

t,(t−1))

f st (s′ti ,s

t−i,a

t−1) = (1+ϕ′t (s

t−i, s

′ti )) · f s

t (sti,s

t−i,a

t−1), (25)

where∥∥∥ϕ(st

−i, s′ti )∥∥∥

∞,t,at−1≤ ν and

∥∥∥ϕ ′t (st−i, s

′ti )∥∥∥

∞,t,at−1≤ ν for every t ∈ N. Let ν is small

enough that 2ν

1−ν·R(st

i,at,(t−1), t)≤ ε .

∥∥∥φmt,t(s

′ti , ·)− φ

mt,t(s

ti, ·)∥∥∥

1≤∥∥∥∥φ

mt,t(s

ti, ·)

1+ν

1−ν− φ

mt,t(s

ti, ·)∥∥∥∥

1=

2ν

1−ν·∥∥∥φ

mt,t(s

ti, ·)∥∥∥

1≤ ε,

where the first inequality follows from equations (22) and (25) and the last inequality fol-lows from the choice of ν and equation (24).

L.13.2 follows from equation (24) and R(sti,a

t,(t−1), t) ∈ L1(Pt,i,µti (·|at,(t−1))).

For each t ∈ N, t ≤ t and at ∈ X t , we define the measure over St−i, ηm

t,t(·, sti,a

t) asdηm

t,t(st−i, s

ti,a

t) = φ mt,t(a

t , sti,s

t−i)dµ−i(st

−i|at,(t−1)).Lemma 13 implies that for Y2 = St

−i and Y1 = Sti , φ m(·) = φ m

t,t(·,at), conditions L.10.a

and L.10.b hold. Therefore, by Lemma 10, there is a subsequence mrr and a finitelyadditive measure η∗t,t such that η

mrt,t (·, s

ti,a

t) converges to η∗t,t in the weak-∗ topology ofL∞(St

−i,µt(·|at)) for every st

i ∈ Pt,i∩ Sti(a

t),Pt,i ∈ Pt,i.Notice that we can write,

µt−i(ω

t ,at ,st |hi,σmr) = µ

ω(ω t |st ,at,(t−1))×ηmr

t,ˆt(st ,at)×b(hi,at,(ˆt−1)) ·µ t

i (sti|hi,at,(t−1)), (26)

47

where ˆt := ˆt(at ,st , hi) is the smallest t such that hi(at ,st) = hi.We define, analogously,

µ∗,t−i (ω

t ,at ,st |hi) = µω(ω t |st ,at,(t−1))×η

∗t,ˆt(st ,at)×b(hi,at,(ˆt−1)) ·µ t

i (sti|hi,at,(t−1)).

Lemma 14. Under Assumption 5, µ(·|·,σmr) converges to µ∗ in the weak-∗ topology of

(L∞)∗.

Proof. Let f ∈ L∞(H ti (hi),µ

t). Let us show that∫f (h)dµ

t−i(h|hi,σ

mr)→∫

f (h)dµ∗,t−i (h|hi), (27)

almost surely in hi. The integral over Ωt via µω(|st ,at,(t−1)) in the definitions of µ and µ∗

yields a function in L∞(Sti,µ

t−i(·|at,(t−1))) µ t

i (·|at,(t−1))-almost surely in sti. Therefore, by

Lemma 10, for every at ∈ X t , Pt,i ∈ Pt,i71 (omitting the dependence inside ˆt(at ,st , hi))

∫St−i×Ωt

f (ω t ,at ,st−i,s

ti)dµ

ω(ω t |st−i,s

ti,a

t,(t−1))dηmr

t,ˆt(st−i,s

ti,a

t)→∫St−i×Ωt

f (ω t ,at ,st−i,s

ti)dµ

ω(ω t |st−i,s

ti,a

t,(t−1))dη∗t,ˆt(st−i,s

ti,a

t),

∀sti ∈ Pt,i ∩ St

i(at), and hence, also b(hi,at,(t(at ,hi)−1)) · µ t

i (·|hi,at,(t−1))-almost surely, βi-almost surely in hi.

Thus, by L.13.2 and Lemma 11 (where η2(·) = b(hi,at,(t(at ,hi)−1)) · µ ti (·|hi,at,(t−1))),

equation (27) follows.

Recall that σ is a THPE and σmr is a sequence that converges to σ weakly in strategies,such that σmr is an εmr-constrained equilibrium, with εmr an (εmr ,νmr)-tremble profilesuch that εmr → 0. µmr is the system of beliefs that is weakly consistent with σmr . Wehave shown that there is a system of beliefs µ∗ such that µmr converges to µ∗ in (L∞)∗.Then, by definition, (σ ,µ∗) is quasi-consistent. Let us now show that σ is sequentiallyrational, and hence, that (σ ,µ∗) is a weak sequential equilibrium.

By introduction, suppose that there is t, at ∈ X t and Si ⊆ Sti , with µ t

i (Si|at) > 0, and a

71Notice that the equicontinuity required by L.10.a holds, by L.13.1 after replacing t by ˆt(at ,st , hi) because? is an isolated point of the signal space, and therefore, ˆt is constant in a small enough vicinity of st

i .

48

player i strategy σi such that

Ui(σi|hi(sti,a

t),µ∗)<Ui(σi|hi(sti,a

t),µ∗), (28)

for sti ∈ Si.

By weak consistency we have for every σmri ∈ Σi(ε

mr) and hi ∈Hi

Ui(σmri |hi, µ

mr) =∞

∑l=0

∫h∈H l

i (hi)gi(h) · probi(h, σ

mri , hi)dµ

mr,l−i (h|hi). (29)

Let σmi be a sequence of εm-constrained strategies that approaches σi, µ-almost surely.

Since each σm is a ε-constrained equilibrium, by Proposition 3,

Ui(σmri |hi(st

i,at), µmr)≥Ui(σ

mri |hi(st

i,at), µmr),

for every sti ∈ Si.

Define Hi(ati,s

ti,σ) = l ≤ t|σi(at

i,l|hi(a

t,(l)i ,st,(l)

i ) > 0,∀l ∈ l, . . . , t. We define :=probi((at

i,sti),σ

mri , Hi(at

i,sti,σ

mr)). Notice that prob<,ti (at

i,sti,σ

mr) > 0 almost surely inst

i.72 Therefore,

∫st

i∈Si

prob<,ti (at

i,sti,σ

mr) ·Ui(σmri |hi(st

i,at), µm)dµ

ti (s

ti|at)≥∫

sti∈Si

prob<,ti (at

i,sti,σ

mr) ·Ui(σmi |hi(st

i,at), µm)dµ

ti (s

ti|at),

Replacing the definition of Ui(σmri |hi(st

i,at), µmr) in the previous expression, we can

re-write it as

∑l∈N

∫st

i∈Si

∫h∈H l

i (hi(sti ,a

ti))

gi(h) · probti(h,σ

mri )dµ

mr,l−i (h|hi(st

i,ati))dµ

ti (s

ti|at)≥

∑l∈N

∫sti∈Si

∫h∈H l

i (hi(sti ,a

ti))

gi(h) · probti(h, ˆσmr

i )dµmr,l−i (h|hi(st

i,ati))dµ

ti (s

ti|at), (30)

where for strategy σ ∈ Σi, probti(h, σi) := probi

(h, σi, Hi((at

i,sti),σi)∪t +1, . . . , |h|

)and ˆσmr

i is the strategy that coincides with σmri at all private histories in Hi \hi(st

i,at)|st

i ∈72Notice that we are abusing notation as probi() was previously defined over h ∈H . However, the new

notation is also valid as its value depends only on i’s actions and signals.

49

Si, and coincides with σmr after private histories in hi(sti,a

t)|sti ∈ Si.

Passing to a subsequence if necessary, probti(h,σ

mri ) converges to probt

i(h,σi) in theweak-∗ topology of L∞(H , µ) and probt

i(h, ˆσmr) converges to probti(h, ˆσi) in the same

topology, where ˆσi is the strategy that coincides with σi at all private histories in Hi \hi(st

i,at)|st

i ∈ Si, and coincides with σi after private histories in hi(sti,a

t)|sti ∈ Si. These

statements follow from the weak convergence in strategies of σmri to σ .

Now, the following Claim yields a contradiction of (28)

Claim 7. Equation (30) implies

∑l∈N

∫st

i∈Si

∫h∈H l

i (hi(sti ,a

ti))

gi(h) · probti(h,σi)dµ

∗l−i(h|hi(st

i,ati))dµ

ti (s

ti|at)≥

∑l∈N

∫st

i∈Si

∫h∈H l

i (hi(sti ,a

ti))

gi(h) · probti(h, ˆσi)dµ

∗,l−i (h|hi(st

i,ati))dµ

ti (s

ti|at),

Proof of Claim 7.Notice that by equation (26), the right hand side of (30) is equal to

∑l∈N

∫st

i∈Si

∫h∈H l

i (hi(sti ,a

ti))

gi(h)· probti(h,σ

mri )dµ

ω(ω l|sl,al,(l−1))dηmr(sl,al)dµ

ti (s

li|al,(l−1)).

Now, for each l

G(σmr) :=∫

sti∈Si

∫h∈H l

i (hi(sti ,a

ti))

gi(h)· probti(h,σ

mri )dµ

ω(ω l|sl,al,(l−1))dηmr(sl,al)dµ

ti (s

li|al,(l−1))

→∫

sti∈Si

∫h∈H l

i (hi(sti ,a

ti))

gi(h) · probti(h,σi)dµ

ω(ω l|sl,al,(l−1))dη∗(sl,al)dµ

ti (s

li|al,(l−1)).

The convergence follows from Lemmas 11 and 13 since the integral of g with respect toω t is bounded almost surely in sl and that µ l(∪p∈Pt,i p) = 1

Now, to see that the sum over l converges to the corresponding limit notice that foreach l, G(σmr)≤maxal∈X l

∫Ωl×Sl gl

i(ωl,al,sl)dµ l(ω l,al,sl) for every r∈N, and therefore,

the dominated convergence theorem applies. The convergence of the right hand side ofequation (30) follows analogously.

PROOF OF PROPOSITION 4Let Σi,n(ε) denote the set of player i’s ε constrained strategies in Γn and let Σn =

×i∈NΣi,n(ε).

50

Let ε be a (ε,ν)-tremble and let σn ⊆ Σn(ε) be a sequence of ε-constrained strate-gies. Let σn be the associated game-Γ strategy and suppose that prob(·|σn)→ prob(·|σ∗),probi(·|σn)→ probi(·|σ∗) and prob−i(·|σn)→ prob−i(·|σ∗) weakly in strategies for someσ∗ ∈ Σ.

Define gi,n(ωt ,at) := gi(ω

t ,at) for each ω t ∈ PΩ,tn (ω t) with ω t ∈Ωt

n. Abusing notationwe also write, for h ∈H , gi,n(h), for gi,n(ω

t(h), at(h)).Player i’s payoff in game Γn from strategy profile σn can be written as a function of σ

as,

Ui,n(σn) := ∑t∈N

∫H

gi,n(h)prob(h|σn)dµt(h)

We now show that σ∗ is a ε-constrained equilibrium. That is, let us show that

Ui(σ∗)≥Ui(σ

′i ,σ∗−i),

for every σ ′i ∈ Σi(ε). Fix σ ′i ∈ Σi(ε).

Lemma 15. There is a sequence σ ′i,nkk∈N ⊆ Σi,nk(ε) and associated sequence of strate-

gies in Γ, σ ′i,nkk∈N, such that

∥∥∥σ ′i,nk−σ ′i

∥∥∥2→ 0

Proof of Lemma 15. We will show that we can construct a sequence of strategies, σ ′i,nk∈

Σi,nk(ε), and associated sequence of strategies σ ′i,nk∈ Σi(ε) such that for each n,∥∥∥σ ′i,nk

−σ ′i

∥∥∥2< 1

k , which yields the desired result.

First, σ ′i : Hi→RXi is a measurable function in L2(Hi, µi), where µi = µ hi(·). There-fore, σ ′i can be approximated by a sequence of simple functions. Let αi,kk∈N be a se-quence of simple functions that converges to σ ′i in the norm of L2. We can assume thateach αi,k is a strategy. Each αi,k is of the form

αi,k = ∑l∈Ik

clk ·1C

lk,

for some finite set of indices Ik, real constants clkl∈Ik and open sets Cl

kl∈Ik with Clk ⊆Hi

for each k ∈ N and l ∈ Ik.73

73σ ′i can be approximated by sequences of simple functions with indicators over measurable sets. Due tothe outer regularity of a measure over Borel sets of a metric space, there is also an approximating sequenceof simple functions with indicators over open sets.

51

Let h = (ω t ,st ,at) ∈H and define

PHn (h) =

h ∈H |h ∈ PΩ,t

n (ω t)×PS,tn (st)×at

and PHi

n (hi) = hi

(PH

n (h))

for any h satisfying hi(h). PHin denotes the partition of Hi

comprised by the collection of sets PHin (hi) for hi ∈Hi.

Define

αi,k,n(hi) =

αi,k(hi) PHin (hi)⊆Cl for some l ∈ Ik

1|Ai(hi)| otherwise.

Notice that αi,k,n is constant in every cell of PHin . Therefore, it has a “natural” associated

strategy in Γn.Due to the L2 convergence of αi,k to σ ′i , for each k ∈ N, there is k1(k) such that∥∥∥αi,k1(k)−σ ′i

∥∥∥2< 1

2k . Now, due to condition (c) in Definition 9, there is also k2(k) such

that∥∥∥αi,k1(k),k2(k)−αi,k1(k)

∥∥∥2< 1

2k . Defining the sequence of strategies σi,nk := αi,k1(k),k2(k)

concludes the proof of Lemma 15.

Let σ ′i,nkk∈N ⊆ Σi(ε) be the sequence that exists from Lemma 15.

Since each σn is a Σn(ε)-constrained equilibrium

Ui,nk

(σnk

)≥Ui,nk

(σ′i,nk

, σ−i,nk

)(31)

Point (b) in Definition 9 implies gi,n converges to gi almost surely. Thus, Ui,nk

(σnk

)→

Ui(σ∗) and Ui,nk

(σ ′i,nk

, σ−i,nk

)→ Ui(σ

∗i ,σ

∗−i) which by equation (31) yields the desired

conclusion.

PROOF OF PROPOSITION 5Because the game is Markovian in payoffs and actions the correspondence A is only a

function of the state. Also, A is upper hemicontinuous so for each C ⊆ X , the set Λ(C) =

ω ∈Ω|C = A(ω) is closed and hence compact.74

Let ε > 0. Define the set BΩ(ω,ε)=

ω|∣∣gi(ω,a)−gi(ω,a)

∣∣< ε ∀i ∈ N,d(ω, ω)< ε

where d is the distance in Ω. Define also, for each i ∈ N, ε > 0, BSi(si,ε) =

si ∈ Si|d(si, si)< ε

.

74This follows directly from A’s closed graph.

52

For every C ∈ X and ε > 0,

BΩ(ω,ε)

ω∈Ω

is a an open cover of Λ(C). By

compactness, there is an open subcover

BΩ(ωCi ,ε)

i∈1,...,I(C)

for a finite set of states

ωCi i∈1,...,I(C). We can define a measurable partition, Pi(C,ε)i∈1,...,I(C), of Λ(C) re-

cursively as follows. Define

P1(C,ε) = BΩ(ωC1 ,ε)∩Λ(C)

and for each j ∈ 2, . . . , I(C),

Pj(C,ε) = BΩ(ωCj ,ε)∩Λ(C)\

⋃l< j

Pl(C,ε).

The collection of sets P(ε) = Pl(C,ε)l∈1,...,I(C),C∈2X is a partition of Ω. A mea-surable partition of Si, Pi(ε), in which elements in a cell of partition are at a distance of atmost ε is defined analogously.

Define the sequences of partitions PΩn n∈N where PΩ

n = P(1/n) and PSin n∈N

where PSin = P(1/n) for each i ∈ N. The sets Ωn and Si,n are any selection from the

corresponding partitions.Let us see that Γ has an approximating game sequence. Conditions (a) and (b) in

Definition 9 are satisfied by construction. To see that (c) holds, let B1 ⊆Ωt and B2 ⊆ St beopen sets, and at ∈ X t . Let B1,n = ω t |ω t ∈ P1 ∈PΩ,t

n ,P1 ⊆ B1 and B2,n = st |st ∈ P2 ∈PS,t

n ,P2 ⊆ B2. We can write

µ(B1,B2|at) =∫

1B11B2dµ(ω t ,st |at), (32)

where 1B1 is an indicator function that yields 1 if ω t ∈ B1 and zero otherwise, and 1B2is defined analogously. We also can write

µ(B1,n,B2,n|at) =∫

1B1,n1B2,ndµ(ω t ,st |at). (33)

Let us show that1B1,n1B2,n→ 1B11B2 (34)

almost surely. In fact, if ω t /∈ B1 or st /∈ B2 then 1B11B2= 1B1,n1B2,n= 0 for ev-ery n∈N. Let ω t ∈ B1 and st ∈ B2. For ν > 0 and j≤ t, let B

((ω t) j, ν

)and let B

((st) j, ν

)

53

be the balls centered at (ω t) j and (st) j, respectively, of radius ν . Because B1 and B2 areopen sets there is ν small enough that × j≤tB

((ω t) j, ν

)⊆ B1 and × j≤tB

((st) j, ν

)⊆ B2.

Let n be such that 1n ≤

ν

4 . By the triangle inequality PΩ,tn(ω t) ⊆ × j≤tB

((ω t) j, ν

)⊆

B1 and PS,tn(st) ⊆ × j≤tB

((st) j, ν

)⊆ B2 for every n ≥ n. Therefore, 1B11B2 =

1B1,n1B2,n= 1 for every n≥ n.By the Dominated Convergence Theorem, and equations (32) and (33), condition (c)

in Definition 9 follows.

References

ALIPRANTIS, C. D. AND K. C. BORDER (1999): Infinite dimensional analysis, vol.32006, Springer.

ALTMAN, E., K. AVRACHENKOV, N. BONNEAU, M. DEBBAH, R. EL-AZOUZI, AND

D. S. MENASCHE (2008): “Constrained cost-coupled stochastic games with indepen-dent state processes,” Operations Research Letters, 36, 160–164.

AMBRUS, A., J. BURNS, AND Y. ISHII (2014): “Gradual bidding in ebay-like auctions,”Working Paper, 1–50.

AMBRUS, A. AND S. E. LU (2014): “A continuous model of multilateral bargaining withrandom arrival times,” American Economics Journal: Microeconomics.

BALBUS, Ł., K. REFFETT, AND Ł. WOZNY (2013): “Markov Stationary Equilibria inStochastic Supermodular Games with Imperfect Private and Public Information,” Dy-

namic Games and Applications, 3, 187–206.

BALDER, E. J. (1988): “Generalized equilibrium results for games with incomplete infor-mation,” Mathematics of Operations Research, 13, 265–276.

BÖRGERS, T. (1991): “Upper hemicontinuity of the correspondence of subgame-perfectequilibrium outcomes,” Journal of Mathematical Economics, 20, 89–106.

CALCAGNO, R., Y. KAMADA, S. LOVO, AND T. SUGAYA (2014): “Asynchronicity andcoordination in common and opposing interest games,” Theoretical Economics, 9, 409–434.

54

COTTER, K. D. (1991): “Correlated equilibrium in games with type-dependent strategies,”Journal of Economic Theory, 54, 48–68.

DUGGAN, J. (2012): “Noisy Stochastic Games,” Econometrica, 80, 2017–2045.

DUNFORD, N. AND J. T. SCHWARTZ (1957): Linear Operators. Part 1: General Theory,New York Interscience.

FEDERGRUEN, A. (1978): “On n-person stochastic games by denumerable state space,”Advances in Applied Probability, 10, 452–471.

FINK, A. M. (1964): “Equilibrium in a stochastic $n$-person game,” J. Sci. Hiroshima

Univ. Ser. A-I Math., 28, 89–93.

FLESCH, J., F. THUIJSMAN, AND O. J. VRIEZE (2003): “Stochastic games with non-observable actions,” Mathematical Methods of Operations Research, 58, 459–475.

GENSBITTEL, F., S. LOVO, J. RENAULT, AND T. TOMALA (2017): “Zero-sum revisiongames,” Games and Economic Behavior.

GILLETTE, D. (1957): “Stochastic games with zero stop probabilities,” Contributions to

the Theory of Games, 3, 179–187.

HARRIS, C., P. RENY, AND A. ROBSON (1995): “The existence of subgame-perfect equi-librium in continuous games with almost perfect information: A case for public random-ization,” Econometrica: Journal of the Econometric Society, 507–544.

HOPENHAYN, H. AND M. SAEEDI (2016): “Bidding Dynamics in Auctions,” .

KAMADA, Y. AND M. KANDORI (2017): “Revision Games, Part II: Applications andRobustness,” Working Paper.

——— (2019): “Revision Games,” Econometrica: Journal of the Econometric Society.

Forthcoming.

KAMADA, Y. AND T. SUGAYA (2020): “Optimal Timing of Policy Announcements inDynamic Election Campaigns,” Quarterly Journal of Economics, Forthcoming.

KAPOR, A. AND S. MORONI (2016): “Sniping in Proxy Auctions with Deadlines,” .

55

KREPS, D. M. AND R. WILSON (1982): “Sequential equilibria,” Econometrica: Journal

of the Econometric Society, 863–894.

LOVO, S. AND T. TOMALA (2015): “Markov Perfect Equilibria in Stochastic RevisionGames,” .

MANELLI, A. M. (1996): “Cheap talk and sequential equilibria in signaling games,”Econometrica: Journal of the Econometric Society, 917–942.

MILGROM, P. R. AND R. J. WEBER (1985): “Distributional strategies for games withincomplete information,” Mathematics of operations research, 10, 619–632.

MYERSON, R. B. AND P. J. RENY (2019): “Perfect Conditional ε-Equilibria of Multi-Stage Games with Infinite Sets of Signals and Actions,” Econometrica: Journal of the

Econometric Society, Forthcoming.

NOWAK, A. S. AND T. E. S. RAGHAVAN (1992): “Existence of stationary correlatedequilibria with symmetric information for discounted stochastic games,” Mathematics

of Operations Research, 17, 519–526.

OCKENFELS, A. AND A. E. ROTH (2006): “Late and multiple bidding in second priceInternet auctions: Theory and evidence concerning different rules for ending an auction,”Games and Economic Behavior, 55, 297–320.

PARTHASARATHY, T. (1973): “Discounted, positive, and noncooperative stochasticgames,” International Journal of Game Theory, 2, 25–37.

POLLARD, D. (2002): A user’s guide to measure theoretic probability, vol. 8, CambridgeUniversity Press.

ROTH, A. AND A. OCKENFELS (2002): “Last minute bidding and the rules for ending sec-ond price auctions: evidence from eBay and Amazon auctions on the Internet,” American

Economic Review.

SELTEN, R. (1975): “Reexamination of the Perfectness Concept for Equilibrium Points inExtensive Games,” International journal of game theory.

SHAPLEY, L. S. (1953): “Stochastic games,” Proceedings of the national academy of

sciences, 39, 1095–1100.

56

SIMON, L. K. AND M. B. STINCHCOMBE (1989): “Extensive form games in continuoustime: Pure strategies,” Econometrica: Journal of the Econometric Society, 1171–1214.

SIMON, R. S. (2003): “Games of incomplete information, ergodic theory, and the measur-ability of equilibria,” Israel Journal of Mathematics, 138, 73–92.

SOBEL, M. J. (1971): “Noncooperative Stochastic Games,” The Annals of Mathematical

Statistics, 42, 1930–1935.

STINCHCOMBE, M. B. (2011): “Correlated equilibrium existence for infinite games withtype-dependent strategies,” Journal of Economic Theory, 146, 638–655.

TAKAHASHI, M. (1964): “Equilibrium points of stochastic non-cooperative $ n $-persongames,” Journal of Science of the Hiroshima University, Series AI (Mathematics), 28,95–99.

WHITT, W. (1980): “Representation and approximation of noncooperative sequentialgames,” SIAM Journal on Control and Optimization, 18, 33–48.

57