arX

iv:1

511.

0539

0v1

[q-

bio.

PE]

17

Nov

201

5

STOCHASTIC SELECTION PROCESSES

ALEX MCAVOY

Abstract. We propose a mathematical framework for natural selection in finite populations. Traditionally,many of the selection-based processes used to describe cultural and genetic evolution (such as imitationand birth-death models) have been studied on a case-by-case basis. Over time, these models have grown

in sophistication to include population structure, differing phenotypes, and various forms of interactionasymmetry, among other features. Furthermore, many processes inspired by natural selection, such asevolutionary algorithms in computer science, possess characteristics that should fall within the realm of a“selection process,” but so far there is no overarching theory encompassing these evolutionary processes. Theframework of stochastic selection processes we present here provides such a theory and consists of three maincomponents: a population state space, an aggregate payoff function, and an update rule. A population statespace is a generalization of the notion of population structure, and it can include non-spatial informationsuch as strategy-mutation rates and phenotypes. An aggregate payoff function allows one to generically talkabout the fitness of traits without explicitly specifying a method of payoff accounting or even the natureof the interactions that determine payoff/fitness. An update rule is a fitness-based function that updatesa population based on its current state, and it includes as special cases the classical update mechanisms(Moran, Wright-Fisher, etc.) as well as more complicated mechanisms involving chromosomal crossover,mutation, and even complex cultural syntheses of strategies of neighboring individuals. Our frameworkcovers models with variable population size as well as with arbitrary, measurable trait spaces.

1. Introduction

Evolutionary game theory has proven itself extremely useful for modeling both cultural and geneticevolution (Maynard Smith, 1982; Hofbauer and Sigmund, 1998; Dugatkin, 2000; Nowak, 2006). Traits, whichare represented as strategies, determine the fitness of the players. An individual’s fitness might be determinedsolely by his or her strategy (frequency-independent fitness) or it might also depend on the traits of the otherplayers in the population (frequency-dependent fitness). The population evolves via a fitness-based updatemechanism, and the long-run behavior of this process can be studied to determine which traits are moresuccessful than others.

Evolutionary game theory was first used to study evolution in infinite populations via deterministicreplicator dynamics (Taylor and Jonker, 1978). More recently, the dynamics of evolutionary games have alsobeen studied in finite populations (Nowak et al., 2004; Taylor et al., 2004). Finite-population evolutionarydynamics typically have two timescales: one for interactions and one for updates. A player has a strategy(trait) and interacts with his or her neighbors in order to receive a payoff. In a birth-death process, forinstance, this payoff is converted to reproductive fitness and used to update the population as follows:First, a player is chosen from the population for reproduction with probability proportional to (relative)fitness. Next, a player is chosen uniformly at random from the population for death, and the offspring of thereproducing player replaces the deceased player. This birth-death process is a frequency-dependent versionof the classical Moran process (Moran, 1958; Nowak, 2006).

Whereas replicator dynamics are deterministic, finite-population models of evolution are inherently sto-chastic and incorporate principles of both natural selection and genetic drift. All biological populations arefinite, and we focus here on stochastic evolutionary games in finite populations. In particular, we focus onselection processes, which, informally, are evolutionary processes in which the update step depends on fitness.Selection processes typically resemble birth-death processes in that there is reproduction and replacement,with “fitter” players more likely to reproduce than those with lower fitness. Of course, the order and numberof births and deaths may vary, and the update might instead be based on imitation instead of reproduction.One of our goals here is to precisely define selection process in a way that captures all of the salient featuresof the classical models of evolution in finite populations.

1

Nowak et al. (2009) state that “There is (as yet) no general mathematical framework that would encom-pass evolutionary dynamics for any kind of population structure.” We propose here such a framework, whichwe term stochastic selection processes, to describe the evolutionary games used to model natural selection infinite populations. Stochastic selection processes model evolutionary processes with two timescales: one forinteractions (which determine fitness) and one for updates (selection, mutation, etc.). Our framework takesinto account arbitrary population structures, as well as non-spatial information about the population suchas phenotypes and strategy mutations. Moreover, this framework encompasses all types of strategy spaces,games (matrix, asymmetric, multiplayer, etc.), and fitness-based update rules.

An example of ambiguity in evolutionary game theory is that classical games, such as two-player matrixgames, are often used to define evolutionary processes in populations, and in this context the term “game”can refer to either the classical game or to the evolutionary process. Moreover, classical multiplayer games,such as the public goods game, can result in processes in which each player in the population derives a payofffrom several multiplayer interactions, and some of these interactions might involve players who are notneighbors (see McAvoy and Hauert, 2015a). Even further, when a player is involved in multiple interactions,the total payoff to this player may be derived in more than one way: payoffs from individual encounters maybe accumulated (added) or averaged, for instance, and the nature of this accounting can strongly influencethe evolutionary process (Maciejewski et al., 2014).

In order to accommodate the many ways of deriving “total payoff” from a classical game, one may distillfrom these methods a common feature: each player has a strategy and receives an aggregate payoff froma sequence of interactions. That is, if S is the strategy space available to each of the N players in thepopulation, then there is an aggregate payoff function, u : SN Ñ R

N , such that the ith coordinate function,ui, is the payoff to player i for all of the (microscopic) interactions in which this player is involved. As anexample, consider the two-player game defined by the matrix

˜A B

A a b

B c d

¸. (1)

Suppose that the population is well mixed so that each player interacts with every other player. Let S “tA,Bu and let s P SN be a strategy profile consisting of k ` 1 players using A and N ´ 1 ´ k players usingB (the ordering is not important since the population is well mixed). If player i is an A-player, then thepayoff to this player is

uacci psq :“ ka` pN ´ 1 ´ kq b (2)

if payoffs are accumulated and

uavei psq :“ka` pN ´ 1 ´ kq b

N ´ 1(3)

if payoffs are averaged. These two methods of calculating payoffs from pairwise interactions give essentiallyequivalent evolutionary dynamics since the population is well mixed, but this phenomenon need not hold formore complicated methods of payoff accounting or (spatial) population structures. Evolutionary dynamicsaside, this example illustrates how one can define an aggregate payoff function, u, from a classical game suchas a 2 ˆ 2 matrix game; different methods of obtaining total payoff from a series of interactions result indifferent aggregate payoff functions.

A selection process in a finite population typically has for a state space the set of all strategy profiles, i.e.the set of all N -tuples of strategies (Allen and Tarnita, 2012). A strategy profile indicates a strategy for eachplayer in the population, and often an evolutionary process updates only these strategies. The populationstructure may be fixed (Lieberman et al., 2005; Szabo and Fath, 2007) or dynamic (Tarnita et al., 2009;Wardil and Hauert, 2014). (In the latter case, one must also account for the population structure in thestate space of the process.) An aggregate payoff function, which takes into account both strategies andpopulation structure, assigns a payoff to each player in the population. The payoff to player i, ui, is thenconverted to fitness, fi “ f puiq, where f is some payoff-to-fitness function (e.g. f puiq “ exp tβuiu for someβ ě 0). A fitness-based update rule such as birth-death (Moran, 1958; Nowak et al., 2004); death-birthor imitation (Ohtsuki et al., 2006; Ohtsuki and Nowak, 2006); pairwise comparison (Szabo and Toke, 1998;

2

Traulsen et al., 2007); or Wright-Fisher (Ewens, 2004; Imhof and Nowak, 2006) is then repeatedly applied tothe population, at each step updating the strategies of the players and (possibly) the population structure.

Based on this pattern, it seems reasonable to define the state of an evolutionary process to be a pair, ps,Γq,where s is an N -tuple of strategies and Γ is a population structure. However, there may be more to the stateof a population than its spatial structure. For example, Antal et al. (2009) consider evolution in phenotype

space, a model in which each player has both a strategy and a phenotype, and phenotypes influence theeffects of strategies on interactions. In the Prisoner’s Dilemma, for instance, a cooperator (whose strategyis C) cooperates only with other players who are phenotypically similar and defects otherwise. In thisinstance, we show how one can consider phenotypes as a part of the “population state” in the sense thatthey contain information about the players in the population. In general, a state of the evolutionary processcan be represented by a pair, ps,Pq, where s is a strategy profile and P is a population state (which wemake mathematically precise). Notably, the population state is distinct from the strategies of the players; itdescribes all of the non-strategy information about the players.

When viewed from this perspective, the effects of phenotypes on strategies can be implemented directlyin the aggregate payoff function, u: a player facing a cooperator receives the payoff for facing a cooperatorif and only if they are phenotypically similar. Similarly, strategy mutations can also be considered as apart of the population state and accounted for directly in the update step of the process. As a part of ouranalysis, we formally define aggregate payoff function and update rule and show how they are influenced bythe components of the population state.

This setup, which involves a series of interactions as the population transitions through various states,is reminiscent of a stochastic game (Shapley, 1953). A stochastic game is played in stages, with each stageconsisting of a normal-form game determined by some “state.” The game played in the subsequent stage isdetermined probabilistically by the current state as well as the strategies played in the current stage. Wewill see that, in general, selection processes are not necessarily stochastic games, and neither are stochasticgames necessarily selection processes. However, these two types of processes do share some common features,and we use some of the components of a stochastic game as inspiration for our framework.

Many problems in evolution, such as how and why cooperation evolves, depend on the specifics of the up-date rule, population structure, mutation rates, etc. (Ohtsuki et al., 2006; Taylor et al., 2007; Traulsen et al.,2009; Debarre et al., 2014; Rand et al., 2014). We clarify how these pieces (among others) fit together. Ourobjective here is threefold: (1) to compare and contrast evolutionary and stochastic games, drawing inspira-tion from the latter to describe the former; (2) to propose a general mathematical framework encompassingnatural selection models in finite populations; and (3) to examine the components of several classical evo-lutionary processes and demonstrate how they fit into our framework. Many (if not most) of the existingmodels of evolution in finite populations involve a fixed population size. Therefore, our framework is firststated in terms of a fixed population size since this setting most readily allows for a comparison to thetheory of stochastic games and for illustrative examples placing several standard evolutionary processes intoa broader context. However, the assumption that the population size is fixed is not crucial to our theory,and we conclude by extending our framework to processes with variable population size.

2. Stochastic games

Prior to outlining the basic theory of stochastic games, we first need to recall some definitions and notation.A measurable space consists of a set, X , and a σ-algebra of sets, F pXq, on X . Often, we refer to X itself asa “measurable space” and suppress F pXq. If X is a measurable space, then we denote by ∆ pXq the spaceof probability measures on X ; that is, if M pXq is the space of all measures on pX,F pXqq, then

∆ pXq :“ tµ P M pXq : µ pXq “ 1u . (4)

For measurable spacesX and Y , denote byK pX,Y q the set of Markov kernels fromX to Y ; that is, K pX,Y qis the set of functions κ : X ˆF pY q Ñ r0, 1s such that (i) κ px,´q : F pY q Ñ r0, 1s is in ∆ pY q for each x P Xand (ii) κ p´, Eq : X Ñ r0, 1s is measurable for each E P F pY q. We also write κ : X Ñ ∆ pY q to denote sucha kernel.

Shapley (1953) considers a collection of normal-form games, together with a probabilistic rule for transi-tioning between these games, which he refers to as a stochastic game. A stochastic game is a generalizationof a repeated game that, formally, consists of the following components:

3

(i) N players, labeled 1, . . . , N ;(ii) for each player, i, a measurable strategy space, Si;(iii) a measurable state space, P;(iv) a “single-period” payoff function, u : S ˆ P Ñ R

N , where ui is the payoff to player i and S :“S1 ˆ ¨ ¨ ¨ ˆ SN is the set of all strategy profiles ;

(v) a transition kernel, T : S ˆ P Ñ ∆ pPq.

A Markov decision process is a stochastic game with one player (Puterman, 1994; Neyman, 2003), and arepeated game is a stochastic game whose state space, P, consists of just a single element (McMillan, 2001;Mertens et al., 2015).

Examples of strategies for stochastic games are the following (see Neyman and Sorin, 2003):

(1) Pure strategies : Let H denote the set of all possible histories, i.e.

H :“ t∅u Yğ

tě1

pS ˆ Pqt, (5)

where ∅ denotes the “null” history andŮ

tě1pS ˆ Pq

tis the disjoint union of the spaces of t-tuples,

pS ˆ Pqt, for t ě 1. A pure strategy for player i is a map

si : H ÝÑ Si, (6)

indicating an action in Si for each t and history ht P pS ˆ Pqt

Ď H. We denote by Map pH, Siq theset of all such maps, i.e. the set of player i’s pure strategies.

(2) Mixed strategies: A mixed strategy for player i is a probability distribution over the set of purestrategies for player i, i.e. an element σi P ∆ pMap pH, Siqq.

(3) Behavioral strategies : A behavioral strategy for player i is a map

σi : H ÝÑ ∆ pSiq , (7)

indicating a distribution over Si for each t and history ht P pS ˆ Pqt

Ď H.(4) Markov strategies : A Markov strategy for player i is a behavioral strategy, σi, such that σi phtq “

σi pktq for each t and ht, kt P pS ˆ Pqt

Ď H with htt “ ktt . In other words, a Markov strategy is a“memory-one” behavioral strategy, i.e. a behavioral strategy that depends on only the last strategyprofile, state, and t.

(5) Stationary strategies : A stationary strategy is a Markov strategy that is independent of t, i.e. abehavioral strategy that depends on only the last strategy profile and state.

Of these five classes of strategies, behavioral strategies are the most general. Indeed, pure, mixed, Markov,and stationary strategies are all instances of behavioral strategies. In the context of repeated games, amemory-one strategy of the repeated game (Press and Dyson, 2012) is equivalent to a stationary strategyof the stochastic game, and a longer-memory strategy of the repeated game (Hauert and Schuster, 1997) isequivalent to a behavioral strategy.

2.1. Evolutionary processes as stochastic games. At first glance, stochastic games seem to providea reasonable framework for evolutionary games: a stochastic game transitions through states in stages(“periods”), which could be population structures or states, and in each stage the players receive payoffsbased on a single-period payoff function, u. However, it is in the dynamics that the differences betweenstochastic and evolutionary games become evident:

The combination of a stochastic game and a strategy for each player defines a stochastic process on SˆP,although this stochastic process might or might not be a Markov chain. For example, let T be the transitionkernel for a stochastic game, and suppose that σi is a stationary strategy for player i for i “ 1, . . . , N . Letκ be the transition kernel on S ˆ P defined by the product measure, i.e.

κ : S ˆ P ÝÑ ∆ pS ˆ Pq

: ps,Pq ÞÝÑ σ1 rs,Ps ˆ ¨ ¨ ¨ ˆ σN rs,Ps ˆ T rs,Ps . (8)

Thus, the stochastic game–together with this profile of stationary strategies–defines a time-homogeneousMarkov chain on S ˆ P. In general, if these strategies are instead Markov strategies, then the resulting

4

Markov chain might be time-inhomogeneous. If these strategies are pure, mixed, or behavioral, then thestochastic process on S ˆ P defined by the game need not even be a Markov chain.

Evolutionary processes are typically defined as Markov chains on S ˆ P, where P is chosen to be a statespace appropriate for the evolutionary process (such as the space of population structures, mutation-rateprofiles, or phenotype profiles). In light of the previous remarks, it is not unreasonable to expect that manyof these processes are equivalent to stochastic games combined with stationary strategies. As it turns out,evolutionary processes generally possess correlations between updates of strategies and population states,forbidding an equivalence between an evolutionary process and a Markov chain constructed from a stochasticgame via Eq. (8). These correlations are evident already in one of the most basic models of evolution infinite populations:

Example 1 (Moran process). Suppose that S “ tA,Bu, where strategy A represents the mutant type andstrategy B represents the wild type. A mutant type has fitness r ą 0 relative to the wild type (whose fitnessrelative to itself is just 1). Let m be a finite subset of r0, 1s consisting of a number of “mutation rates,” andlet P :“ m

N . In a well-mixed population of size N , the Moran process proceeds as follows: In each time step,an individual (“player”) is chosen for reproduction with probability proportional to relative fitness. Anotherplayer (including the one chosen for reproduction) is then chosen uniformly at random from the populationfor death. The offspring of the player chosen for reproduction replaces the deceased player. If player i ischosen for reproduction and εi P m is this player’s mutation rate, then the offspring of this player inheritsthe type of the parent with probability 1 ´ εi and takes on a type uniformly at random from tA,Bu withprobability εi. The mutation rate of the offspring is εi, which is inherited directly from the parent. Thus,a state of this process consists of a profile of types (“strategies”), s P SN , and a profile of mutation rates,ε P P “ m

N .A transition between states ps, εq and ps1, ε1q is possible only if there exists j such that sℓ “ s1

ℓ and εℓ “ ε1ℓ

for each ℓ ‰ j. If player i is selected for reproduction and player j is chosen for death, then it must be thecase that εi “ ε1

j . If δs,t “ 1 when s “ t (and is 0 otherwise), then the probability that player i is selectedfor reproduction is

rδsi,A ` δsi,BřNℓ“1

prδsℓ,A ` δsℓ,Bq. (9)

The probability that player j is chosen for death is 1{N . If the offspring of player i inherits the strategy ofthe parent, then it must be true that si “ s1

j . Otherwise, the offspring of player i “mutates” and adopts

strategy s1j with probability 1{2. Therefore, the probability of transitioning between states ps, εq and ps1, ε1q

is

Tps,εq,ps1,ε1q “Nÿ

i,j“1

˜ź

ℓ‰j

δsℓ,s1ℓδεℓ,ε1

ℓ

¸δεi,ε1

j

ˆ

˜rδsi,A ` δsi,BřN

ℓ“1prδsℓ,A ` δsℓ,Bq

¸ˆ1

N

˙„p1 ´ εiq δsi,s1

j` εi

ˆ1

2

˙. (10)

By Eq. (10), the distributions on SN and P, respectively, are not independent.

More formally, consider a Markov chain on S ˆ P defined by some evolutionary process such as theMoran process of Example 1, and let κ be its transition kernel. The projection maps Π1 : S ˆ P Ñ S andΠ2 : S ˆ P Ñ P produce pushforward maps

pΠ1q˚ : ∆ pS ˆ Pq ÝÑ ∆ pSq

: µ ÞÝÑ µ ˝ Π´1

1; (11a)

pΠ2q˚ : ∆ pS ˆ Pq ÝÑ ∆ pPq

: µ ÞÝÑ µ ˝ Π´1

2. (11b)

From κ, we obtain a transition kernel for a stochastic game, T , defined by

T rs,Ps :“ pΠ2q˚ κ rs,Ps (12)

5

for each s P S and P P P. Similarly, we obtain the (stationary) strategy profile

σ rs,Ps :“ pΠ1q˚ κ rs,Ps . (13)

However, one typically loses information in passing from such a Markov chain to the combination of astochastic game and a profile of stationary strategies. First of all, the transition kernel κ generally cannotbe reconstructed from T and σ since κ rs,Ps need not be in ∆ pSq ˆ ∆ pPq; a priori, we know only thatκ rs,Ps P ∆ pS ˆ Pq for s P S and P P P (see Eq. (10), for example). Moreover, σ need not be of theform pσ1, . . . , σN q for stationary strategies σ1, . . . , σN ; in particular, σ is a correlated stationary profile (seeAumann, 1987; Fudenberg and Tirole, 1991). In other words, whereas a sequence of independent strategychoices produce an element

´σ1 rs,Ps , . . . , σN rs,Ps

¯P ∆ pS1q ˆ ¨ ¨ ¨ ˆ ∆ pSN q , (14)

it might be the case that σ rs,Ps P ∆ pS1 ˆ ¨ ¨ ¨ ˆ SN q ´´∆ pS1q ˆ ¨ ¨ ¨ ˆ ∆ pSN q

¯.

In §3, we present a framework for stochastic evolutionary processes used to model natural selection.These processes, which we call stochastic selection processes, illustrate more clearly the correlations betweendistributions on S and P arising in many evolutionary processes. We saw in Example 1 that, despite thesimilarities between stochastic games and evolutionary processes, there are important differences betweenthe two frameworks. However, our notion of a stochastic selection process draws inspiration from the theoryof stochastic games. Namely, we appropriate the concepts of (i) state space, P; (ii) single-period payofffunction, u; and (iii) update step.

3. Stochastic selection processes with fixed population size

Here we focus on a type of evolutionary process that we term a stochastic selection process. Stochasticselection processes seek to model processes with two timescales: one for interactions and one for selection.In the interaction step, players interact with one another and receive payoffs based on their strategies. In theselection step, the population is updated probabilistically based on the current population and the players’payoffs. Selection processes provide a general framework for the evolutionary games used to model processesbased on natural selection (Maynard Smith, 1982).

Roughly speaking, the processes modeled by evolutionary game theory may be split into two classes:cultural and genetic (McAvoy and Hauert, 2015c). In cultural processes, there is a fixed set of playerswho repeatedly revise their strategies based on some update rule (such as imitation). In genetic processes,strategies are updated via reproduction and genetic inheritance. Naturally, a process need not be one orthe other; there may be both cultural and genetic components in an evolutionary process. Unlike purelycultural processes, those with a genetic component have the property that the players themselves, as wellas the size of the population, may actually change via births and deaths, thus it does not make sense tospeak of a fixed population of players as in requirement (i) of a stochastic game. However, one can choosean arbitrary enumeration of the players in each step of the process and refer to the player labeled i at timen as “player i.” Of course, player i at time m and player i at time n ‰ m might be different, but a stochasticselection process is a Markov chain and the transition kernel can be defined for any enumeration of theplayers at a given time. The implicit property that natural selection does not depend on the enumeration ofthe players must be stated formally in terms of the update rule. Informally, the update rule for a stochasticselection process must satisfy a symmetry condition that guarantees the dynamics do not depend on theseenumerations.

As an example, an evolutionary process based on a game with two-strategies in a well-mixed populationmay be modeled as a Markov chain whose state space is t0, 1, . . . , Nu (Nowak, 2006). If S “ tA,Bu isthe strategy set, then the state of the population is determined by the number of A-players, which is justan integer i P t0, 1, . . . , Nu. Alternatively, one may choose an arbitrary enumeration of the players andrepresent the state of the population as an element ps1, . . . , sNq P SN . Since the population is well mixed,evolutionary dynamics depend on only the frequency of each strategy in the population, so any two statesps1, . . . , sN q and ps1

1, . . . , s1N q consisting of the same number of A-players should be indistinguishable. In

other words, if T is the transition matrix for an evolutionary game in this population, then Tπs,τs1 “ Ts,s1

6

for each s, s1 P SN and π, τ P SN , where SN acts on SN by permuting the coordinates. Thus, the Markovchain is more naturally defined on the quotient space

S :“ SN{SN , (15)

which is isomorphic to t0, 1, . . . , Nu when S “ tA,Bu. (Recall that an action of a group, G, on a set, X , givesan equivalence relation, „, on X , which is defined by x „ x1 if and only if there exists g P G with x1 “ gx.The quotient space, X{G, is defined as the set of equivalence classes under „, and the class containingx is denoted by “x mod G.”) Of course, one may consider strategy spaces with more than two strategies:Suppose that S “ tA1, . . . , Anu, and, for each r “ 1, . . . , n, let ψr : SN Ñ t0, 1, . . . , Nu be the map sendinga strategy profile, s P SN , to the number of players using strategy Ar in s. Since ϕr pπsq “ ϕr psq for eachπ P SN and s P SN , the map

Ψ : SN ÝÑ!

pk1, . . . , knq P t0, 1, . . . , NuN : k1 ` ¨ ¨ ¨ ` kn “ N)

: s ÞÝÑ´ψ1 psq , . . . , ψn psq

¯(16)

descends to an isomorphism

rΨ : SN{SN ÝÑ!

pk1, . . . , knq P t0, 1, . . . , NuN

: k1 ` ¨ ¨ ¨ ` kn “ N). (17)

Since an evolutionary update rule may be defined on the space of strategy-frequency profiles,!

pk1, . . . , knq P t0, 1, . . . , NuN : k1 ` ¨ ¨ ¨ ` kn “ N), (18)

we see once again that the Markov chain defined by an evolutionary process in this population naturally hasSN{SN for a state space. We show here that this phenomenon generalizes to arbitrary types of populationsand update rules. In the process of establishing this general construction, we must formally define populationstate (§3.1) and update rule (§3.3).

We first assume that the population size, N , is fixed. This assumption allows us to place many of theclassical (fixed population size) stochastic evolutionary processes into the context of our framework. Afterdiscussing the components of a selection process and giving several examples, we formally define stochastic

selection process in its full generality (covering populations of variable size) in §4.Just like a stochastic game, a stochastic selection process consists of a measurable state space, P, and a

strategy space for each player. We assume that

S1 “ S2 “ ¨ ¨ ¨ “ SN “: S, (19)

so that S “ SN . This assumption that the players all have the same strategy space is not restrictive sincethe dynamics of the process define the evolution of strategies; one can just enlarge each player’s strategyspace if necessary and let the dynamics ensure that those strategies that are not available to a given playerare never used. Before discussing these dynamics, we first need to explore the state space, P:

3.1. Population states. We seek to appropriate the idea of a state space, P, of a stochastic game in orderto introduce population structure into an evolutionary process. In fact, a population’s spatial structure (suchas a graph) is just one component of this space; mutation rates or phenotypes may also be included in thisspace. Therefore, rather than declaring P to be a space of population structures, we say that P is the spaceof population states. A population state indicates properties of the players and relationships between theplayers. (Note that “population state” in this context does not include the strategies of the players in thepopulation.) If one enumerates these players differently, then there should be a corresponding “relabeling” ofthe population state so that these properties and relationships are preserved. Since changing the enumerationof the population amounts to applying an element of SN (the symmetric group on N letters) to t1, . . . , Nu,it follows that P must be equipped with a group action of SN . Thus, if s P SN is a strategy profile of thepopulation and P P P is a population state, then the pair ps,Pq represents the same population of playersas pπs, πPq whenever π P SN . In other words, the population state space, P “ PN , which we write with asubscript to indicate the population size, is a measurable SN -space. More formally:

7

Definition 1. A population state space for a set of N players is a measurable space, PN , equipped with anaction of SN in such a way that the map π : PN Ñ PN is measurable for each π P SN . If PN is a populationstate space for a set of N players, then a population state for these players is simply an element P P PN .

3.1.1. Examples.

Example 2 (Graphs). Consider the set of N ˆN , nonnegative matrices over R,

PG

N :“ Γ P R

NˆN : Γij ě 0 for each i, j “ 1, . . . , N(, (20)

equipped with an action of SN defined by pπΓqij “ Γπpiqπpjq. An element Γ P PG

N defines a directed, weighted

graph whose vertices are t1, . . . , Nu, with an edge from i to j if and only if Γij ‰ 0 (the weight of the edge

is then Γij). Γ is undirected if Γij “ Γji for each i and j, and Γ is unweighted if Γ P t0, 1uNˆN

.

Example 3 (Sets). A set-structured population consists of a finite number of sets, each containing somesubset of the population, such that each player is in at least one set (Tarnita et al., 2009). Set-structuredpopulations may be modeled using relations:

PS

N :“!R Ď t1, . . . , Nu ˆ t1, . . . , Nu : R is reflexive and symmetric

). (21)

That is, if R P PS

N , then pi, iq P R for each i (“reflexive”) and pi, jq P R if and only if pj, iq P R (“symmetric”).R P PS

N defines a set-structure with i and j in a common set if and only if pi, jq P R. There is a naturalaction of SN on PS

N defined by

pi, jq P πR ðñ pπi, πjq P R, (22)

which makes PS

N into a population state space.

Example 4 (Demes). A deme-structure on a population is a subdivision of the population into subpop-ulations, or “demes” (Taylor et al., 2001; Wakeley and Takahashi, 2004; Hauert and Imhof, 2012). Similarto set-structured populations, deme-structured populations may be modeled using relations (but with thestronger notion of equivalence relation):

PD

N :“!R Ď t1, . . . , Nu ˆ t1, . . . , Nu : R is reflexive, symmetric, and transitive

). (23)

PD

N Ď PS

N , and whereas a set-structured population may have overlapping sets, a deme-structured populationhas disjoint sets. The additional transitivity requirement guarantees that these sets partition t1, . . . , Nu.The action of SN on PD

N is the one inherited from PS

N , making PD

N into a population state space.

So far, we have considered population structures that describe spatial and qualitative relationships betweenthe players. One could also associate to the players quantities such as mutation rates or phenotypes :

Example 5 (Mutation rates). Consider a process in which updates are based on births and deaths (such as aMoran or Wright-Fisher process). Moreover, suppose that the spatial structure of the population is a graph.If player i reproduces, then with probability εi the offspring adopts a novel strategy uniformly at random(“mutates”), and with probability 1 ´ εi the offspring inherits the strategy of the parent. The mutationrate, εi, is passed on directly from parent to offspring. The population state space for this process is then

PN :“ PG

N ˆr0, 1sN; a population state consists of (i) a graph, indicating the spatial relationships between the

players, and (ii) a profile of mutation rates, ε P r0, 1sN, with εi indicating the probability that the offspring

of player i mutates. For update rules based on imitation, mutation rates might more appropriately be called“exploration rates,” and they are implemented slightly differently. In general, mutation rates appear indifferent forms and help to distinguish cultural and genetic update rules; we give several examples in §3.3.The upshot of this discussion is that a population state may consist of a spatial structure, such as a graphin PG

N , as well as some extra information pertaining to the players, such as mutation rates.

Example 6 (Phenotype space). In addition to strategies, the players may also have phenotypes that affectinteractions with other players in the population (Antal et al., 2009; Nowak et al., 2009). If the spatialstructure of the population is a graph and the phenotype of each player is a one-dimensional discrete quantity,then one may define the population state space to be PG

N ˆ ZN . Just as with mutation rates, the rest of

the process determines how these phenotypes affect the dynamics. In §3.2, we continue this example and go8

into the details of how the inclusion of phenotypes in the population state space affects the payoffs of theplayers, which can then be used to recover a model of evolution in phenotype space of Antal et al. (2009).

3.1.2. Symmetries of population states. The action of SN on PN can be used to formally define a notion ofpopulation symmetry:

Definition 2 (Automorphism of population state). For a population state, P, in a population state space,PN , an automorphism of P is an element π P SN such that πP “ P. The group of automorphisms of P

is Aut pPq :“ StabSNpPq, where StabSN

pPq denotes the stabilizer of P under the group action of SN

on PN .

If Γ P PG

N , for example, then Aut pΓq is the group of graph automorphisms of Γ in the classical sense, i.e.the set of π P SN such that Γπpiqπpjq “ Γij for each i and j. Such automorphisms have played an importantrole in the study of evolutionary games on graphs (see Taylor et al., 2007; Debarre et al., 2014). We discusssymmetries of population states further in §5.

Now that we have a formal definition of population state space, we explore how this space influencesevolutionary dynamics. The processes we seek to model have two timescales: interactions and updates. In§3.2, we consider the influence of the population state on interactions, and in §3.3, we define update rule andshow how the population state fits into the update step of an evolutionary process.

3.2. Aggregate payoff functions. Prior to stating the definition of a general payoff function for a sto-chastic selection process, we consider a motivating example that is based on a popular type of game used tomodel frequency-dependent fitness in evolutionary game theory:

Example 7. Consider the symmetric, two-player game whose payoff matrix is

˜ A1 A2

A1 a11 a12

A2 a21 a22

¸. (24)

If the population structure is a graph and the population state space is PG

N , then one can construct a function

u : t1, 2uN

ˆ PG

N Ñ RN by letting ui be defined as

ui

´ps1, . . . , sN q ,Γ

¯:“

Nÿ

j“1

Γijasisj . (25)

That is, ui is the “aggregate payoff” function for player i since it produces the total payoff from player i’sinteractions with all of his or her neighbors, weighted appropriately by the edge weights of the populationstructure. If π P SN , then

ui

´ `sπp1q, . . . , sπpNq

˘, πΓ

¯“

Nÿ

j“1

Γπpiqπpjqasπpiqsπpjq

“Nÿ

j“1

Γπpiqjasπpiqsj

“ uπpiq

´ps1, . . . , sNq ,Γ

¯. (26)

Therefore, although uΓ :“ u p´,Γq : t1, 2uN

Ñ RN need not be symmetric in the sense that uΓ pπsq “ πuΓ psq

for each s P t1, 2uN, it is symmetric in the sense that

u pπs, πΓq “ πu ps,Γq (27)

for each s P t1, 2uN and Γ P PN “ PG

N . In other words, all of the information that results in payoff asymmetryis contained in the population state space, PN .

Using the function u and Eq. (27) of the previous example as motivation, we have:9

Definition 3 (Aggregate payoff function). An aggregate payoff function is a map

u : SN ˆ PN ÝÑ RN (28)

that satisfies u pπs, πPq “ πu ps,Pq for each π P SN , s P SN , and P P PN .

The symmetry condition in Definition 3, u pπs, πPq “ πu ps,Pq, implies that an aggregate payoff func-tion, u, is completely determined by the map u1 : SN ˆPN Ñ R. Indeed, if u1 is known and π P SN sends 1to i, then ui ps,Pq “ u1 pπs, πPq for each s P SN and P P PN , which recovers the map u : SN ˆPN Ñ R

N .This symmetry condition must hold even for individual encounters that are asymmetric (Example 8).

3.2.1. Examples.

Example 8. In place of (24), one could consider a collection of bimatrices,

Mij :“

˜ A1 A2

A1 aij11, a

ji11

aij12, a

ji21

A2 aij21, a

ji12

aij22, a

ji22

¸, (29)

indexed by i, j P t1, . . . , Nu (McAvoy and Hauert, 2015c). For each i and j, Mij is the payoff matrix forplayer i against player j, with the first coordinate of each entry denoting the payoff to player i and the

second coordinate denoting the payoff to player j. The collection Mij

(Ni,j“1

is equivalent to an element of`R

2ˆ2˘NˆN

, i.e. a 2ˆ2 real matrix indicating the payoff to player i against player j for each i, j “ 1, . . . , N .In this case, the population state space consists of more than spatial structures; it also includes the detailsof the payoff-asymmetry appearing in individual encounters. In other words, if the population is graph-structured, then a population state consists of a graph and a collection of payoff matrices of the form of Eq.(29), i.e.

PN :“ PG

N ˆ`R

2ˆ2˘NˆN

, (30)

where the action of π P SN on`R

2ˆ2˘NˆN

is π`Mij

˘Ni,j“1

:“`Mπpiqπpjq

˘Ni,j“1

. The aggregate payoff

function, u : SN ˆ PN Ñ RN , is defined by

ui

´ps1, . . . , sN q ,

´Γ,

`Mij

˘Ni,j“1

¯¯:“

Nÿ

j“1

Γijaijsisj

. (31)

For π P SN , we see that

ui

´ `sπp1q, . . . , sπpNq

˘, π

´Γ,

`Mij

˘Ni,j“1

¯¯“

Nÿ

j“1

Γπpiqπpjqaπpiqπpjqsπpiqsπpjq

“Nÿ

j“1

Γπpiqjaπpiqjsπpiqsj

“ uπpiq

´ps1, . . . , sN q ,

´Γ,

`Mij

˘Ni,j“1

¯¯, (32)

so u defines an aggregate payoff function in the sense of Definition 3.

Remark 1. For a fixed collection,`Mij

˘Ni,j“1

, we could have instead let PN “ PG

N and

ui

´ps1, . . . , sN q ,Γ

¯:“

Nÿ

j“1

Γijaijsisj

. (33)

However, for π P SN , we would then have

ui

´ `sπp1q, . . . , sπpNq

˘, πΓ

¯“

Nÿ

j“1

Γπpiqjaiπ´1pjqsπpiqsj

; (34a)

10

uπpiq

´ps1, . . . , sN q ,Γ

¯“

Nÿ

j“1

Γπpiqjaπpiqjsπpiqsj

. (34b)

The only way (34a) and (34b) are the same for each s P t1, 2uN

and Γ P PG

N is if Mij “ Mπpiqπpjq for eachi, j “ 1, . . . , N , and this equality need not hold (which would mean that u, when defined in this way, is notan aggregate payoff function in the sense of Definition 3). Therefore, by enlarging PN via (30) and definingu via (31), we can essentially “factor out” the asymmetry present in the payoff function defined by (33). Inother words, PN contains all of the non-strategy information that distinguishes the players’ payoffs.

Due to the separation of timescales in the selection processes we consider here, it often happens that u isindependent of a portion of the population state space. More specifically, the population state space can bedecomposed into an interaction state space, EN , and a dispersal state space, DN , such that PN “ EN ˆ DN

and

u´s, pE ,Dq

¯“ u

´s,`E ,D 1

˘ ¯(35)

for each s P SN , E P EN , and D ,D 1 P DN . This decomposition generalizes models with separate interactionand dispersal graphs (Taylor et al., 2007; Ohtsuki et al., 2007a,b; Pacheco et al., 2009; Debarre et al., 2014).The interaction state, for instance, might consist of a population structure and other information (such asphenotypes):

Example 9 (Phenotype space, continued). Antal et al. (2009) study the evolution of cooperation in phe-notype space. In terms of (24), each player has a strategy, A1 (“cooperate”) or A2 (“defect”), as well asa one-dimensional phenotype, which is simply an integer. If an interaction state has a graph as its spatialstructure, then the interaction state space is EN :“ PG

N ˆ ZN . Thus, an interaction structure consists of a

graph, Γ P PG

N , and an N -tuple of phenotypes r “ pr1, . . . , rN q P ZN , where ri is the phenotype of player i

in the population. The phenotypes affect the strategies of the players as follows: cooperators cooperate withother neighbors with whom they share a phenotype, and they defect otherwise. Defectors always defect,

regardless of phenotypic similarities. For each i, the payoff function, u : t1, 2uN

ˆ EN Ñ RN , satisfies

ui

´ps1, . . . , sN q , pΓ, rq

¯“

Nÿ

j“1

Γij

´δri,rjasisj `

`1 ´ δri,rj

˘a22

¯, (36)

where δri,rj is 1 if ri “ rj and 0 otherwise. Therefore, one can directly implement the influence of phenotypeon strategy using the interaction state space, EN , and u.

3.3. Update rules. We saw at the beginning of §3 that for a game with n strategies in a well-mixedpopulation, the state space for an evolutionary process is

SN{SN –!

pk1, . . . , knq P t0, 1, . . . , NuN

: k1 ` ¨ ¨ ¨ ` kn “ N). (37)

On the other hand, between update steps, one can simply fix some enumeration of the population andrepresent the state of the evolutionary process by an element of SN , i.e. a representative of the statespace, SN{SN . In populations with spatial structure, mutation rates, phenotypic differences, etc., thisrepresentative contains more information than simply a strategy profile; it also contains information aboutthe population state. In other words, at a fixed point in time, the state of the evolutionary process can bedescribed by an element of SN ˆ PN , where PN is a population state space. Of course, the evolutionarydynamics of the process should not be affected by the choice of enumeration of the players in each time step,which means that the state space for the evolutionary process is naturally the quotient space

S :“`SN ˆ PN

˘{SN , (38)

generalizing the state space of Eq. (37) to structured populations.We now wish to describe the update step of an evolutionary process on S. This update rule should not

depend on how the players in the updated population are labeled. For example, if a player dies and is replacedby the offspring of another player, then the result of this death and replacement is a new element of S. Inother words, the new population does not lie in SN ˆ PN in a natural way; we must choose an enumerationof the players in order to get an element of SN ˆ PN , and this enumeration may be arbitrary. Therefore,

11

given the current strategy profile and population state, an update rule should give a probability distributionover the state space of the process, S (not SN ˆPN). On the other hand, in order to update the state of thepopulation, one needs to speak of the likelihood that each player in the current state is updated. To do so,one may choose a representative of the current state of the process, ps,Pq P SN ˆPN , which is equivalent tochoosing a labeling of the players at that point in time. Again, the distribution over S (conditioned on thecurrent state of the process) should not depend on the labeling of the current state. Finally, this distributionover S is a function of the fitness profile of the population; each player has a real-valued fitness, and theupdate rule depends on these values. Putting these components together, we have:

Definition 4 (Update rule). An update rule is a map,

U : RN ÝÑ K`SN ˆ PN , S

˘, (39)

that satisfies the symmetry condition

U rπxs´

pπs, πPq , E¯

“ U rxs´

ps,Pq , E¯

(40)

for each π P SN , x P RN , ps,Pq P SN ˆ PN , and E P F pSq, where F pSq is the quotient σ-algebra on

S “`SN ˆ PN

˘{SN derived from F pSq and F pPN q.

That is, an update rule is a family of Markov kernels,!U rxs

)

xPRNĎ K

`SN ˆ PN , S

˘, (41)

parametrized by the fitness profiles of the population, x P RN , and satisfying Eq. (40). Eq. (40) says that

the update does not depend on how the current population is represented. In other words, if ps,Pq andpπs, πPq are two representatives of the same population at time t, then the update rule treats ps,Pq andpπs, πPq as the same population. (If x is the fitness profile corresponding to the representative ps,Pq, thenπx is the fitness profile corresponding to the representative pπs, πPq.)

Together, an update rule and aggregate payoff function define a Markov chain on S whose kernel, κ, isconstructed as follows: Let f : R Ñ R be a payoff-to-fitness map, i.e. a function that converts a player’spayoff to fitness. Consider the function

F : RN ÝÑ RN

: px1, . . . , xN q ÞÝÑ´f px1q , . . . , f pxN q

¯, (42)

which converts payoff profiles to fitness profiles. If u : SN ˆPN Ñ RN is an aggregate payoff function, then,

for ps,Pq P SN ˆ PN and E P F pSq, we let

κ´

ps,Pq mod SN , E¯:“ U

”F´u ps,Pq

¯ı´ps,Pq , E

¯. (43)

κ is well defined since, for each π P SN ,

κ´

pπs, πPq mod SN , E¯

“ U

”F´u pπs, πPq

¯ı´pπs, πPq , E

¯

“ U

”πF

´u ps,Pq

¯ı´pπs, πPq , E

¯

“ U

”F´u ps,Pq

¯ı ´ps,Pq , E

¯

“ κ´

ps,Pq mod SN , E¯, (44)

where the second and third lines come from Eqs. (27) and (40), respectively.

3.3.1. Update pre-rules. Despite the fact that the evolutionary processes we seek to model here naturallyhave S “

`SN ˆ PN

˘{SN for a state space, many evolutionary processes in the literature are defined directly

on SN ˆPN (see Allen and Tarnita, 2012). Update rules are sometimes cumbersome to write out explicitly,and defining a Markov chain on SN ˆ PN instead of on

`SN ˆ PN

˘{SN can simplify the presentation of

the transition kernel. In this context, the notion of “update rule” still makes sense, but we instead call it anupdate pre-rule to distinguish it from the update rule of Definition 4:

12

Definition 5 (Update pre-rule). An update pre-rule is a map,

U0 : RN ÝÑ K`SN ˆ PN , S

N ˆ PN

˘, (45)

such that for each π P SN , x P RN , ps,Pq P SN ˆ PN , and E P F pSq,

U0 rπxs´

pπs, πPq , E¯

“ U0 rxs´

ps,Pq , τE¯

(46)

for some τ P SN .

In many cases, the permutation τ is just π´1, i.e.

U0 rπxs´

pπs, πPq , πE¯

“ U0 rxs´

ps,Pq , E¯

(47)

for each π P SN . However, all that an evolutionary process on SN ˆ PN really requires is that if statepπs, πPq is updated to ps1,Pq, then state ps,Pq is updated to pτs1, τP 1q for some τ . To relate Definitions4 and 5, consider the projection map,

Π : SN ˆ PN ÝÑ`SN ˆ PN

˘{SN “ S

: ps,Pq ÞÝÑ ps,Pq mod SN . (48)

Π gives rise to a pushforward map on measures,

Π˚ : ∆`SN ˆ PN

˘ÝÑ ∆ pSq

: µ ÞÝÑ µ ˝ Π´1, (49)

which can be used to naturally derive an update rule from an update pre-rule:

Proposition 1. An update pre-rule canonically defines an update rule.

Proof. Let U0 be an update pre-rule and consider the map

Π˚U0 : RN ÝÑ K`SN ˆ PN , S

˘

: x ÞÝÑ!

ps,Pq ÞÑ Π˚U0 rxs´

ps,Pq ,´¯). (50)

For π P SN , x P R, ps,Pq P SN ˆ PN , and E P F pSq, there exists τ P SN such that

pΠ˚U0q rπxs´

pπs, πPq , E¯

“ U0 rπxs´

pπs, πPq ,Π´1E¯

“ U0 rxs´

ps,Pq , τΠ´1E¯

“ U0 rxs´

ps,Pq ,Π´1E¯

“ pΠ˚U0q rxs´

ps,Pq , E¯, (51)

so Π˚U0 is an update rule, which completes the proof. �

In other words, an update pre-rule can be “pushed forward” to an update rule. If S and PN are finite,then an update rule can also be “pulled back” to an update pre-rule:

Proposition 2. If U is an update rule and S and PN are finite, then there exists an update pre-rule, U0,such that Π˚U0 “ U. That is, U can be “pulled back” to U0.

Proof. From U, we define a map, U0, as follows:

U0 rxs´

ps,Pq ,`s1,P 1

˘ ¯“

1

|orbSNps1,P 1q|

U rxs´

ps,Pq ,`s1,P 1

˘mod SN

¯. (52)

For π P SN , we see from Eq. (40) that

U0 rπxs´

pπs, πPq ,`s1,P 1

˘ ¯

“1

|orbSNps1,P 1q|

U rπxs´

pπs, πPq ,`s1,P 1

˘mod SN

¯

13

“1

|orbSNps1,P 1q|

U rxs´

ps,Pq ,`s1,P 1

˘mod SN

¯

“ U0 rxs´

ps,Pq ,`s1,P 1

˘ ¯. (53)

Therefore, U0 satisfies Eq. (46) and defines an update pre-rule. Since

pΠ˚U0q rxs´

ps,Pq ,`s1,P 1

˘mod SN

¯

“ U0 rxs´

ps,Pq ,Π´1

´ `s1,P 1

˘mod SN

¯¯

“ÿ

ps2,P2qPorbSNps1,P1q

U0 rxs´

ps,Pq ,`s2,P2

˘ ¯

“ÿ

ps2,P2qPorbSNps1,P1q

1

|orbSNps2,P2q|

U rxs´

ps,Pq ,`s2,P2

˘mod SN

¯

“ÿ

ps2,P2qPorbSNps1,P1q

1

|orbSNps1,P 1q|

U rxs´

ps,Pq ,`s1,P 1

˘mod SN

¯

“ U rxs´

ps,Pq ,`s1,P 1

˘mod SN

¯, (54)

it follows that Π˚U0 “ U, which completes the proof. �

Remark 2. The proof of Proposition 2 requires that

E0 P F`SN ˆ PN

˘ùñ ΠE0 P F pSq . (55)

In the case that SN ˆPN is finite, the singletons generate the canonical (i.e. discrete) σ-algebra on SN ˆPN ,and the image Π ps1,P 1q “ ps1,P 1q mod SN is measurable for each ps1,P 1q P SN ˆ PN . In general, theimage of a measurable set need not be measurable. However, the purpose of introducing an update pre-ruleis to provide an alternative way to obtain an update rule. An update rule defines a Markov chain on thetrue space of the evolutionary process, S; pulling this chain back to SN ˆ PN is not necessary.

An update pre-rule defines a Markov chain on SN ˆ PN whose kernel, κ0, satisfies

κ0

´ps,Pq , E0

¯:“ U0

”F´u ps,Pq

¯ı´ps,Pq , E0

¯(56)

for each ps,Pq P SN ˆPN and E0 P F`SN ˆ PN

˘. Denote by Π˚κ0 the kernel of the Markov chain defined

by Π˚U0. The stationary distribution(s) of κ0 can be “pushed forward” to stationary distribution(s) of Π˚κ0via the pushforward map of Eq. (49):

Proposition 3. If µ is a stationary distribution of the Markov chain defined by κ0, then Π˚µ is a stationarydistribution of the Markov chain defined by Π˚κ0.

Proof. Suppose that µ is a stationary distribution of κ0, i.e.

µ pE0q “

ż

sPSN ˆPN

κ0 ps, E0q dµ psq (57)

for each E0 P F`SN ˆ PN

˘. For each E P F pSq, it follows thatż

s mod SN PS

pΠ˚κ0q ps mod SN , Eq d pΠ˚µq ps mod SNq

“

ż

sPSN ˆPN

pΠ˚κ0q ps mod SN , Eq dµ psq

“

ż

sPSN ˆPN

κ0`s,Π´1E

˘dµ psq

“ µ`Π´1E

˘

“ pΠ˚µq pEq (58)

by the change of variables formula and Eq. (57), which completes the proof. �

14

Thus, the passage from an update pre-rule to an update rule via Π˚ is compatible with the steady statesof the chains defined by κ0 and Π˚κ0, respectively.

3.3.2. Examples. We now give several classical examples of update pre-rules and rules:

Example 10 (Death-birth process). Suppose that S is finite and that PN is the finite subset of PG

N consistingof the undirected, unweighted graphs on N vertices (with no other restrictions–this set contains regulargraphs, scale-free networks, etc.). In each step of a death-birth process, a player is selected uniformly atrandom from the population for death. The neighbors (determined by a graph, Γ P PN) then compete to fillthe vacancy: a neighbor–say, player j–is chosen for reproduction with probability proportional to relativefitness, xj . The offspring of this player inherits the strategy of the parent and fills the vacancy left by thedeceased player. The population state (i.e. the graph) is left unchanged by this process. We define anupdate pre-rule for this process by giving transition probabilities from SN ˆ PN to itself when the N ´ 1surviving players in each round retain their labels. That is, if ps,Γq is the state of the process and playeri is chosen for death, Γ remains the same and only the ith coordinate of s is updated in order to obtain anew state, ps1,Γq. Thus, for two states, ps,Γq , ps1,Γ1q P SN ˆ PN , it must be the case that Γ “ Γ1 for thereto be a nonzero probability of transitioning from ps,Γq to ps1,Γ1q. The probability of choosing player i fordeath is 1{N , and, if this player is chosen for death, a transition is possible only if sj “ s1

j for each j ‰ i.The probability that player i is replaced by the offspring of a player using si’ is

ÿ

j‰i

δsj ,s1i

˜Γjixjřj‰i Γjixj

¸, (59)

where δsj ,s1iis 1 if sj “ s1

i and 0 otherwise. Thus, for x P RN , the transition probability from ps,Γq to ps1,Γ1q

is given by the update pre-rule, U0, defined by

U0 rxs´

ps,Γq ,`s1,Γ1

˘ ¯:“ δΓ,Γ1

Nÿ

i“1

˜ź

j‰i

δsj ,s1j

¸ˆ1

N

˙˜řj‰i δsj ,s1

iΓjixjř

j‰i Γjixj

¸. (60)

U0 is indeed an update pre-rule since, for each π P SN ,

U0 rπxs´

pπs, πΓq ,`πs1, πΓ1

˘¯

“ δπΓ,πΓ1

Nÿ

i“1

˜ź

j‰i

δsπpjq,s1πpjq

¸ˆ1

N

˙˜řj‰i δsπpjq,s

1πpiq

Γπpjqπpiqxπpjqřj‰i Γπpjqπpiqxπpjq

¸

“ δΓ,Γ1

Nÿ

i“1

¨˝

ź

j‰πpiq

δsj ,s1j

˛‚ˆ

1

N

˙˜řj‰πpiq δsj ,s1

πpiqΓjπpiqxjř

j‰πpiq Γjπpiqxj

¸

“ δΓ,Γ1

Nÿ

i“1

˜ź

j‰i

δsj ,s1j

¸ˆ1

N

˙˜řj‰i δsj ,s1

iΓjixjř

j‰i Γjixj

¸

“ U0 rxs´

ps,Γq ,`s1,Γ1

˘ ¯. (61)

This example verifies in detail the symmetry condition of an update pre-rule, Eq. (46). Calculations forother processes are similar.

Example 11 (Wright-Fisher process). In contrast to the Moran process of Example 1 and the death-birth process of Example 10, one could also consider a process in which the entire population is updatedsynchronously. For example, in the Wright-Fisher process (Ewens, 2004; Imhof and Nowak, 2006), thepopulation is updated as follows: A player–say, player i–is first selected for reproduction with probabilityproportional to fitness. The offspring of this player inherits the strategy of the parent with probability 1´ εiand takes on a novel strategy uniformly at random with probability εi. The mutation rate of the offspringis inherited from the parent (so that the offspring’s offspring will also mutate with probability εi). Thisprocess is then repeated until there are N new offspring, and these offspring constitute the new population.Thus, one update step of the Wright-Fisher process involves updating the entire population.

15

Let S be finite and let m be some finite subset of r0, 1s. The population state space for this version of theWright-Fisher process is PN :“ m

N . That is, a population state is an N -tuple of strategy-mutation rates,ε, with εi the mutation rate for player i. We define an update pre-rule, U0, as follows: for x P R

N andps, εq , ps1, ε1q P SN ˆ PN , let

U0 rxs´

ps, εq ,`s1, ε1

˘ ¯“

Nź

i“1

Nÿ

j“1

δε1i,εj

ˆxj

x1 ` ¨ ¨ ¨ ` xN

˙„δs1

i,sj p1 ´ εjq ` εj

ˆ1

n

˙. (62)

For each π, τ P SN , x P RN , and ps, εq , ps1, ε1q P SN ˆ PN , it is readily verified that

U0 rπxs´

pπs, πεq ,`τs1, τε1

˘ ¯“ U0 rxs

´ps, εq ,

`s1, ε1

˘¯. (63)

Therefore, the resulting update rule, U :“ Π˚U0, satisfies

U rxs´

ps, εq mod SN ,`s1, ε1

˘mod SN

¯

“ˇorbSN

`s1, ε1

˘ˇ Nź

i“1

Nÿ

j“1

δε1i,εj

ˆxj

x1 ` ¨ ¨ ¨ ` xN

˙„δs1

i,sjp1 ´ εjq ` εj

ˆ1

n

˙. (64)

Consider the simple case ε “ ε1 “ 0 (meaning there are no strategy mutations). If k1r denotes the frequency

of strategy r in state s1 for r “ 1, . . . , n, then |StabSNps1,0q| “ k1

1! ¨ ¨ ¨ k1n!, which means that

U rxs´

ps,0q mod SN ,`s1,0

˘mod SN

¯

“ˇorbSN

`s1,0

˘ˇ Nź

i“1

Nÿ

j“1

ˆxj

x1 ` ¨ ¨ ¨ ` xN

˙δs1

i,sj

“|SN |

|StabSNps1,0q|

Nź

i“1

Nÿ

j“1

ˆxj

x1 ` ¨ ¨ ¨ ` xN

˙δs1

i,sj

“

ˆN

k11, . . . , k1

n

˙ Nź

i“1

Nÿ

j“1

ˆxj

x1 ` ¨ ¨ ¨ ` xN

˙δs1

i,sj , (65)

where the third line was obtained using the orbit-stabilizer theorem (see Knapp, 2006). Eq. (65) is just theclassical formula for the transition probabilities of the Wright-Fisher process based on multinomial sampling(Kingman, 1980; Durrett, 2002; Der et al., 2011).

Example 12 (Pairwise comparison process). In each of our examples so far, both S and PN have beenfinite. Since our theory allows for these sets to be measurable, we now give an example of an evolutionaryprocess whose strategy space is continuous. Let S “ r0,Ks for some K ą 0. This interval might be thestrategy space for a public goods game, for instance, with K the maximum amount any one player maycontribute to the public good. As in Examples 1 and 11, let PN :“ m

N for some finite subset, m, of r0, 1s;an element of PN is just a profile of mutation rates, ε.

In each step of a pairwise comparison process, a player–say, player i–is selected uniformly at random fromthe population to evaluate his or her strategy. Another player–say, player j–is then chosen uniformly atrandom from the rest of the population as a model player. With probability 1 ´ εi, the focal player takesinto account the model player and probabilistically updates his or her strategy as follows: if xi (resp. xj)is the fitness of the focal (resp. model) player, and if β ě 0 is the selection intensity, then the focal playerimitates the model player with probability

1

1 ` e´βpxj´xiq(66)

and retains his or her current strategy with probability

1

1 ` e´βpxi´xjq(67)

(Szabo and Toke, 1998). On the other hand, with probability εi the focal player ignores the model playercompletely. In this case, the focal player “explores” and adopts a new strategy from the interval r0,Ks

16

probabilistically according to a truncated Gaussian distribution centered at si (the current strategy of thefocal player). For some specified variance, σ2, this truncated Gaussian distribution has for a density function

φsi pxq :“

˜ż K

0

exp

˜´

py ´ siq2

2σ2

¸dy

¸´1

exp

˜´

px´ siq2

2σ2

¸. (68)

The parameter σ may be interpreted as a measure of how venturesome a player is, with cautious explorationcorresponding to small σ and risky exploration corresponding to large σ. The density function, φsi , definesa probability measure,

Φsi : F pSq ÝÑ r0, 1s

: E ÞÝÑ

ż

E

φsi pxq dx. (69)

If player i ignores the model player, then he or she adopts a strategy from E P F pSq with probability Φsi pEq.Thus, a player who explores is more likely to adopt a strategy close to his or her current strategy than onefarther away. For β ě 0, let

gβ pxq :“1

1 ` e´βx(70)

be the logistic function. (In terms of this function, the probability that a focal player with fitness xi imitatesa model player with fitness xj is gβ pxj ´ xiq.) We assemble these components into an update pre-rule, U0,

as follows: For x P RN , ps, εq P SN ˆPN , and a measurable rectangle, E1 ˆ ¨ ¨ ¨ ˆEN ˆE1 P F pSq

NˆF pPN q,

let

U0 rxs´

ps, εq , E1 ˆ ¨ ¨ ¨ ˆ EN ˆ E1¯

:“ δε`E1

˘ Nÿ

i“1

ˆ1

N

˙˜ź

j‰i

δsj pEjq

¸ÿ

j‰i

ˆ1

N ´ 1

˙#εiΦsi pEiq

` p1 ´ εiq”δsj pEiq gβ pxj ´ xiq ` δsi pEiq gβ pxi ´ xjq

ı+, (71)

and extend this definition additively to disjoint unions of measurable rectangles. For each x P RN and

ps, εq P SN ˆ PN , one can verify that U0 rxs´

ps, εq ,´¯extends to a measure on SN ˆ PN by the Hahn-

Kolmogorov theorem, which we also denote by U0 rxs´

ps, εq ,´¯. It is readily verified that U0 is an update

pre-rule, so U0 extends to an update rule, Π˚U0, by Proposition 1. This example illustrates how the strategymutations might themselves depend on the strategies (as opposed to simply being uniform random variableson S as they were in the previous examples).

In Examples 1, 11, and 12, we considered processes with heterogeneous mutation rates (meaning εi dependson i). In Examples 1 and 11, there is a nonzero probability of transitioning from a state with heterogeneousmutation rates to a state with homogeneous mutation rates. For instance, in the Wright-Fisher processof Example 11, if ps, εq is a state such that εℓ ‰ εℓ1 for some ℓ and ℓ1, and if ps1, ε1q is a state satisfyings11 “ s1

2 “ ¨ ¨ ¨ “ s1N “ sℓ and ε1

1 “ ε12 “ ¨ ¨ ¨ “ ε1

N “ εℓ, then there is a nonzero probability of transitioningfrom ps, εq to ps1, ε1q provided xℓ ą 0. Thus, the population state, which is simply of profile of mutationrates, can change from generation to generation. In contrast, the population state of Example 12 cannot

change from generation to generation since strategies are imitated and mutation rates are not inherited. Inother words, much of the biological meaning behind the quantities appearing in the population state areencoded in the dynamics of the process via the update rule.

In a more formal setting, let κ be the transition kernel obtained from an update rule via Eq. (43). Fromthe projection Π2 : SN ˆ PN Ñ PN , we obtain a map

rΠ2 :`SN ˆ PN

˘{SN ÝÑ PN{SN

: ps,Pq mod SN ÞÝÑ P mod SN . (72)

17

rΠ2 gives us a pushforward map,´rΠ2

¯

˚: ∆ pSq Ñ ∆ pPN{SNq, which we use to formalize the intuition

behind “static” and “dynamic” population states:

Definition 6 (Static and dynamic population states). A population state, P, in a population state space,PN , is static relative to κ if, for each s P SN ,

´rΠ2

¯

˚κ´

ps,Pq mod SN ,´¯

“ δP mod SN, (73)

where δP mod SNdenotes the Dirac measure on ∆ pPN{SN q centered at P mod SN . Otherwise, if κ is not

static relative to κ, we say that P is dynamic relative to κ.

In Examples 10 and 12, every population state is static. In Examples 1 and 11, only the population states(i.e. mutation profiles) with ε1 “ ε2 “ ¨ ¨ ¨ “ εN are static.

4. Stochastic selection processes with variable population size

Suppose now that the population size is dynamic, and let N Ď t0, 1, 2, . . . u be the set of admissiblepopulation sizes. As in §3, let S be the strategy space for each player. Instead of having a single populationstate space, we now require the existence of a population state space, PN , for each N P N. The state spacefor such a process is

S :“ğ

ℓPN

`Sℓ ˆ Pℓ

˘{Sℓ, (74)

whereŮ

ℓPN

`Sℓ ˆ Pℓ

˘{Sℓ denotes the disjoint union of the spaces

`Sℓ ˆ Pℓ

˘{Sℓ. Instead of a single

aggregate payoff function and update rule, we now require that there be an aggregate payoff function,uN : SN ˆ PN Ñ R

N , and an update rule, UN : RN Ñ K`SN ˆ PN , S

˘, for each admissible population

size, N P N. If the population currently has size N , then uN determines the payoffs to the players in theinteraction step, and U

N updates the population (possibly to one of a different size). Of course, for eachπ P SN , x P R

N , s P SN , P P PN , and E P F pSq, these functions must satisfy

uN pπs, πPq “ πuN ps,Pq ; (75a)

UN rπxs

´pπs, πPq , E

¯“ U

N rxs´

ps,Pq , E¯

(75b)

just as they did in Definitions 3 and 4, respectively.Finally, we have the definition of a stochastic selection process in its full generality:

Definition 7 (Stochastic selection process). A stochastic selection process consists of the following compo-nents:

(1) a set of admissible population sizes, N Ď t0, 1, 2, . . . u;(2) a measurable strategy space, S;(3) for each N P N, a population state space, PN ;(4) for each N P N, an aggregate payoff function, uN : SN ˆ PN Ñ R

N ;(5) a payoff-to-fitness function, f : R Ñ R;(6) for each N P N, an update rule, UN : RN Ñ K

`SN ˆ PN , S

˘, where

S :“ğ

ℓPN

`Sℓ ˆ Pℓ

˘{Sℓ. (76)

The components of a stochastic selection process produce a Markov chain on S whose kernel, κ, is definedas follows: for N P N, ps,Pq P SN ˆ PN , and E P S,

κ´

ps,Pq mod SN , E¯

“ UN”F´uN ps,Pq

¯ı ´ps,Pq , E

¯. (77)

It is readily verified that κ is well defined (see Eq. (44)).

Remark 3. The payoff-to-fitness function, f , in requirement (5) of a stochastic selection process, is notstrictly necessary. It could instead be absorbed into either the payoff function (which would then be a fitness

function) or the update rule (which would then be a family of transition kernels parametrized by payoff

profiles rather than by fitness profiles). We include this function as a part of a stochastic selection process18

for three reasons: (1) having an aggregate payoff function instead of an aggregate fitness function allowsfor a more straightforward comparison to the theory of stochastic games; (2) having an update rule be afamily of transition kernels parametrized by fitness simplifies its presentation (see §3.3.2); and (3) payoff-to-fitness functions are often explicitly mentioned in models of evolutionary games in the literature. Tuning theselection strength of a process, for instance, amounts to modifying the payoff-to-fitness function, so includingthis function in a stochastic selection process allows one to more explicitly separate the various componentsof a selection process.

Remark 4. The notion of update pre-rule also makes sense for populations of variable size, although onemust define the symmetry condition, Eq. (46), with greater care. If N is the set of admissible populationsizes, then we require–for each N P N–a map

UN0 : RN ÝÑ K

˜SN ˆ PN ,

ğ

ℓPN

Sℓ ˆ Pℓ

¸. (78)

For each N P N, there is an action of SN onŮ

ℓPN Sℓ ˆ Pℓ defined by

π ps,Pq “

#pπs, πPq ps,Pq P SN ˆ PN ;

ps,Pq ps,Pq R SN ˆ PN .(79)

From the set of groups tSNuNPN, one can construct the free product,

SN :“ ˚NPN

SN , (80)

which is just the analogue of disjoint union in the category of groups (see Knapp, 2006). Collectively, theactions of SN ˆ PN on

ŮℓPN S

ℓ ˆ Pℓ defined by Eq. (79) (over all N P N) result in a (measurable) action

of SN onŮ

ℓPN Sℓ ˆ Pℓ. For

UN

0

(NPN

to define a collection of update pre-rules, we require that for each

N P N, π P SN , x P RN , ps,Pq P SN ˆ PN and E0 P F

`ŮℓPN S

ℓ ˆ Pℓ

˘, there exists τ P SN such that

U0 rπxs´

pπs, πPq , E0

¯“ U0 rxs

´ps,Pq , τE0

¯. (81)

The reason we require τ to be in the free product, SN, instead of just in SN for some N P N, is that it neednot hold that E0 P F

`SN ˆ PN

˘for some N P N. E0 could be some complicated measurable set consisting

of elements of SN ˆPN for several N P N, so we need some way of relabeling elements SN ˆ PN for severalvalues of N P N simultaneously. Extending the action of SN on SN ˆPN to

ŮℓPN S

ℓ ˆPℓ via (79), in orderto form the free product, SN, via Eq. (80), accomplishes this task.

Analogues of Propositions 1, 2, and 3 also hold in this context, but we do not go through the details here;the proofs are essentially the same as they were in §3.3.1.

5. Discussion

We use the term “selection process” instead of “evolutionary game” in order to emphasize that theupdate step is based on the principles of natural selection. Several types of adaptive processes appearingin the economics literature have been referred to as evolutionary games. Best-response dynamics of Ellison(1993) is a procedure in which, at each round, the players update their strategies based on the best responsesto their opponents in the previous round. This process is known to converge to a Nash equilibrium of thegame. Hart and Mas-Colell (2000) define a similar process called regret matching that leads to a correlatedequilibrium of the game. These processes can be phrased as stochastic games (along with appropriatestrategies), but they are not stochastic selection processes, as we now illustrate with best-response dynamics:

Suppose N “ 2. Let S “ tA,Bu and let u : S2 Ñ R2 be the payoff function for a game between two

players. If best-response dynamics in this population defines a stochastic selection process, then there existsa population state space, P2, and an update rule, U, such that the transition kernel of the resulting Markovchain, κu, satisfies

κu

´ps,Pq , E

¯“ U ru psqs

´ps,Pq , E

¯(82)

19

for each ps,Pq P S2 ˆ P2 and E P F pSq. In other words, κu depends on u insofar as U depends on R2.

Consider the two payoff functions, u, v : S2 Ñ R2, defined by

ˆu pA,Aq u pA,Bqu pB,Aq u pB,Bq

˙“

ˆ2 21 1

˙; (83a)

ˆv pA,Aq v pA,Bqv pB,Aq v pB,Bq

˙“

ˆ2 01 1

˙. (83b)

Since u pB,Bq “ v pB,Bq “ 1, it follows that for s “ pB,Bq and any P P P2,

κu

´´pB,Bq ,P

¯, E

¯“ U ru pB,Bqs

´´pB,Bq ,P

¯, E

¯

“ U rv pB,Bqs´´

pB,Bq ,P¯, E

¯

“ κv

´´pB,Bq ,P

¯, E

¯. (84)

Thus, the strategy profile pB,Bq must be updated by best-response dynamics in the same way for bothfunctions, u and v. However, best-response dynamics actually results in different updates of pB,Bq for thesetwo games: for u, the profile pB,Bq is updated to pA,Aq; for v, the profile pB,Bq is updated to pB,Bq (itis already a Nash equilibrium). The key observation is that, while the Markov chain defined by a stochasticselection process depends on the aggregate payoff function, u, the update rule is independent of u. Incontrast, the update step in best-response dynamics clearly depends on u. In a stochastic selection process,the only role of the aggregate payoff function is to determine the fitness profile (which is then passed to theupdate rule).

An update rule in the classical sense, vaguely speaking, generally consists of information about births anddeaths (or imitation). Choosing to represent the strategies of the players in the population as an N -tuple,s P SN , is just a mathematical convenience; the update rule is independent of how the players are labeled.We defined the notion of update pre-rule (Definition 5) in order to relate stochastic selection processes to theway in which evolutionary processes are frequently modeled–as Markov chains on SN (or, more generally,on SN ˆ PN ). In many cases, an update pre-rule is simpler to write down explicitly than an update rulesince one can choose a convenient enumeration of the players in each time step. We showed that an updatepre-rule can always be “pushed forward” to an update rule, so it is sufficient to give an update pre-rule inplace of an update rule in the definition of stochastic selection process. However, an update rule is the truemathematical formalization of an evolutionary update in this context since it is independent of the labelingof the players.

Stochastic selection processes encompass existing models of selection such as the evolutionary game Markov

chain of Wage (2010) and the evolutionary Markov chain of Allen and Tarnita (2012). Wage (2010) considersa Markov chain arising from probability distributions over a collection of inheritance rules. An inheritancerule is a map, I : t1, . . . , Nu Ñ t1, . . . , N,mu, that designates the source of a player’s strategy: if I piq “ j ‰m, then player i inherits his or her strategy from player j; if I piq “ m, then player i’s strategy is the result ofa random mutation (assumed to be uniform over the strategy set). Similarly, Allen and Tarnita (2012) modelevolution in populations with fixed size and structure using replacement events. A replacement event is a pair,pR,αq, consisting of a collection, R Ď t1, . . . , Nu, of players who are replaced and a rule, α : R Ñ t1, . . . , Nu,indicating the parent of each offspring involved in the replacement. These frameworks provide good modelsfor many classical evolutionary processes, but they do not account for genetic processes with crossover, forinstance. Moreover, one could imagine a cultural process in which a player updates his or her strategybased on some complicated synthesis of many strategies in the population. Our framework generalizes thesemodels, taking into account arbitrary strategy spaces, payoff functions, population structures, and updaterules.

McAvoy and Hauert (2015b) define the notion of a homogeneous evolutionary game, which, informally,means that any two states consisting of a single A-mutant in a monomorphic B-population are equivalent.More specifically, suppose that a population state consists of a graph and a profile of mutation rates; that is,PN “ PG

N ˆmN for some finite subset, m, of r0, 1s. If s, s1 P SN , and if π P SN satisfies πΓ “ Γ and πε “ ε for

some Γ P PG

N and ε P mN , then McAvoy and Hauert (2015b) argue that the states ps, pΓ, εqq and pπs, pΓ, εqq

are evolutionarily equivalent in the sense that π induces an automorphism of the Markov chain on SN ˆPN

20

that sends ps, pΓ, εqq to pπs, pΓ, εqq. This result is a special case of an observation that is completely obviousin the context of stochastic selection processes: if π P Aut pPq for some P in a population state space,PN , then the representatives ps,Pq and pπs,Pq “ pπs, πPq define exactly the same state in S. Therefore,working on the true state space of an evolutionary process, S, helps to elucidate structural symmetries thatare not as clear when working with a Markov chain on SN ˆ PN that is defined by an update pre-rule.

In the definition of a population state space (Definition 1), we require a measurable space, PN , along witha measurable action of SN on PN . Naturally, this setup raises the question of what types of SN -actionsone can put on PN while still retaining the desired properties of an evolutionary process. If one were totake an update rule, U, and then arbitrarily change the action of SN on PN , then U need not remain anupdate rule under this new action. For example, let Γ0 P PG

N be the Frucht graph, which is an undirected,unweighted, regular graph with N “ 12 vertices and no nontrivial symmetries (Frucht, 1939). In other words,if Γ0

πpiqπpjq “ Γ0ij for each i, j “ 1, . . . , 12, then π “ id. Instead of the standard action of SN on PG

N , one could

instead declare that SN acts trivially on PG

N ; in particular, π‹Γ “ Γ for each π P SN and Γ P PG

N . If U is theupdate rule for the death-birth process (see Example 10), then it follows that the representatives

`s,Γ0

˘and`

πs, π ‹ Γ0˘

“`πs,Γ0

˘define the same point in the state space, S. For the (frequency-dependent) Snowdrift

Game, all 12 states consisting of a single cooperator in a population of defectors give rise to different fixationprobabilities for cooperators (see McAvoy and Hauert, 2015b). Therefore, it cannot be the case that U

defines a Markov chain on S via Eq. (43); in particular, U is no longer a well-defined update rule under thenew action, ‹, of SN on PN .

The dynamics of a stochastic selection process, which are obtained via its update rule, encode much ofthe biological meaning of the components that constitute the population state space. In both of Examples11 and 12, the population state space was PN :“ m

N for some finite subset, m, of r0, 1s. On the other hand,the interpretations of these mutation rates were completely different in these two processes: in Example 11,the mutation applied to the offspring of reproducing players; in Example 12, the mutation was interpretedas “exploration” and applied to a player who was chosen to update his or her strategy. One could evenconsider different implementations of mutation rates in the same process: in a genetic process based onreproduction, a player’s strategy-mutation rate may be inherited from the parent (as in Examples 1 and11), or, alternatively, it may be determined by a player’s spatial location (see McAvoy and Hauert, 2015b).These details are encoded entirely in the update rule.

Although the framework we present here is clearly aimed at evolutionary games used to describe naturalselection, related processes that are not technically “games” may also constitute stochastic selection pro-cesses. Evolutionary algorithms, for example, form an important subclass of stochastic selection processes.These algorithms seek to apply the principles of natural selection to solve search and optimization problems(Back, 1996). Evolutionary algorithms typically do not have population state spaces, which, in our context,means that PN can be taken to be a singleton equipped with the trivial action of SN . A popular typeof evolutionary algorithm, known as a genetic algorithm, involves representing the elements of the searchspace, i.e. the genomes in S, as sequences of binary digits. Each genome is then assigned a fitness based onits viability as a solution to the problem at hand. (Unlike in biological populations, the fitness landscape,although complex, is inherently static and does not depend on the other members of the population.) Theupdate step, which is commonly designed to mimic sexual reproduction in nature, involves a combination ofselection, crossover, and mutation. A population of genomes is then repeatedly updated until a sufficiently fitgenome appears. Despite the fact that biological reproduction generally involves either one (asexual) or two(sexual) parents, evolutionary algorithms have been simulated using many parents (Chambers, 1998). Othercomponents of the update step in some algorithms, such as stochastic universal sampling (Baker, 1987),elitism (Baluja and Caruana, 1995), and tournament selection (Poli, 2005), are all readily incorporated intoour model of stochastic selection processes.

In Example 12, we saw an evolutionary process with an uncountably infinite state space. A state spaceof this sort arises naturally in the public goods game, for instance, where there is a continuous range ofinvestment levels. This example also illustrates the more complicated ways in which strategy mutationscan be incorporated into an evolutionary process. If a player has a strategy, x P r0,Ks for some K ą 0,then it may be the case that this player is more likely to “explore” strategies close to x than he or she isto switch to strategies farther away. A truncated Gaussian random variable on r0,Ks, whose variance is ameasure of how venturesome a player is, captures this type of strategy exploration and is easily incorporated

21

into the update rule of an evolutionary process. This type of mutation has appeared in the context ofadaptive dynamics (Doebeli et al., 2004) and, more recently, in a study of stochastic evolutionary branching(Wakano and Iwasa, 2012), but it has been largely ignored elsewhere in the literature on evolutionary gametheory, where strategy mutations typically involve switching between two strategies or else are governed bya uniform random variable over the strategy space. Further studies of the dynamics of processes with thesebiologically-relevant mutations are certainly warranted.

Our framework makes no assumptions on the cardinality of S and PN ; all that is required is that thesespaces be measurable (and that PN be equipped with an action of SN ). Markov chains on continuousstate spaces have unique stationary distributions under certain circumstances (see Durrett, 2009), but, inthe generality of this framework, it need not be the case that a stationary distribution is unique. Evenif S is finite and there are nonzero strategy-mutation rates, the spatial structure of the population mightbe disconnected, resulting in multiple stationary distributions. Particular instances of stochastic selectionprocesses may have the property that nonzero strategy-mutation rates imply that the Markov chain defined bythe process is irreducible (Fudenberg and Imhof, 2006, 2008; Allen and Tarnita, 2012), but this phenomenonneed not hold in general. Our goal here was not to study the dynamics of any particular subclass of stochasticselection processes, but rather to formalize what these processes are.

Our general theory of stochastic selection processes provides a mathematical foundation for a broad classof processes used to describe evolution by means of natural selection in finite populations. Stochastic selectionprocesses also provide a mathematical framework for processes with variable population size, a topic thathas received surprisingly little attention in the literature. Although many biological interactions have beenmodeled using classical games, the differences between stochastic games and stochastic selection processesillustrate a fundamental distinction between classical and evolutionary game theory. There is still a lotto discover about the dynamics of selection processes in finite populations (especially those with variablepopulation size), and our hope is that this framework elucidates the roles of the components of processesbased on natural selection and advances the effort to transform evolution into a mathematical theory.

Acknowledgments

The author thanks Christoph Hauert for many helpful discussions and for carefully reading earlier versionsof this manuscript. Financial support from the Natural Sciences and Engineering Research Council of Canada(NSERC) is gratefully acknowledged.

References

B. Allen and C. E. Tarnita. Measures of success in a class of evolutionary models with fixed popula-tion size and structure. Journal of Mathematical Biology, 68(1-2):109–143, Nov 2012. doi: 10.1007/s00285-012-0622-x.

T. Antal, H. Ohtsuki, J. Wakeley, P. D. Taylor, and M. A. Nowak. Evolution of cooperation by phenotypicsimilarity. Proceedings of the National Academy of Sciences, 106(21):8597–8600, Apr 2009. doi: 10.1073/pnas.0902528106.

R. J. Aumann. Correlated equilibrium as an expression of bayesian rationality. Econometrica, 55(1):1, Jan1987. doi: 10.2307/1911154.

T. Back. Evolutionary Algorithms in Theory and Practice: Evolution Strategies, Evolutionary Programming,

Genetic Algorithms. Oxford University Press, Oxford, UK, 1996. ISBN 0-19-509971-0.J. E. Baker. Reducing bias and inefficiency in the selection algorithm. In Proceedings of the Second Inter-

national Conference on Genetic Algorithms on Genetic Algorithms and Their Application, pages 14–21,Hillsdale, NJ, USA, 1987. L. Erlbaum Associates Inc. ISBN 0-8058-0158-8.

S. Baluja and R. Caruana. Removing the genetics from the standard genetic algorithm. In Machine Learning

Proceedings 1995, pages 38–46. Elsevier, 1995. doi: 10.1016/b978-1-55860-377-6.50014-1.L. Chambers, editor. Practical Handbook of Genetic Algorithms. CRC Press, Dec 1998. doi: 10.1201/9781420050080.

F. Debarre, C. Hauert, and M. Doebeli. Social evolution in structured populations. Nature Communications,5, Mar 2014. doi: 10.1038/ncomms4409.

22

R. Der, C. L. Epstein, and J. B. Plotkin. Generalized population models and the nature of genetic drift.Theoretical Population Biology, 80(2):80–99, Sep 2011. doi: 10.1016/j.tpb.2011.06.004.

M. Doebeli, C. Hauert, and T. Killingback. The Evolutionary Origin of Cooperators and Defectors. Science,306(5697):859–862, Oct 2004. doi: 10.1126/science.1101456.

L. A. Dugatkin. Game Theory and Animal Behavior. Oxford University Press, 2000.R. Durrett. Probability Models for DNA Sequence Evolution. Springer New York, 2002. doi: 10.1007/978-1-4757-6285-3.

R. Durrett. Probability: Theory and Examples. Cambridge University Press, 2009. doi: 10.1017/cbo9780511779398.

G. Ellison. Learning, local interaction, and coordination. Econometrica, 61(5):1047, Sep 1993. doi: 10.2307/2951493.

W. J. Ewens. Mathematical Population Genetics. Springer New York, 2004. doi: 10.1007/978-0-387-21822-9.R. Frucht. Herstellung von graphen mit vorgegebener abstrakter gruppe. Compositio Mathematica, 6:239–250, 1939.

D. Fudenberg and L. A. Imhof. Imitation processes with small mutations. Journal of Economic Theory, 131(1):251–262, Nov 2006. doi: 10.1016/j.jet.2005.04.006.

D. Fudenberg and L. A. Imhof. Monotone imitation dynamics in large populations. Journal of Economic

Theory, 140(1):229–245, May 2008. doi: 10.1016/j.jet.2007.08.002.D. Fudenberg and J. Tirole. Game Theory. The MIT Press, 1991.S. Hart and A. Mas-Colell. A simple adaptive procedure leading to correlated equilibrium. Econometrica,68(5):1127–1150, Sep 2000. doi: 10.1111/1468-0262.00153.

C. Hauert and L. Imhof. Evolutionary games in deme structured, finite populations. Journal of TheoreticalBiology, 299:106–112, Apr 2012. doi: 10.1016/j.jtbi.2011.06.010.

C. Hauert and H. G. Schuster. Effects of increasing the number of players and memory size in the iteratedprisoner’s dilemma: a numerical approach. Proceedings of the Royal Society of London B: Biological

Sciences, 264(1381):513–519, 1997. doi: 10.1098/rspb.1997.0073.J. Hofbauer and K. Sigmund. Evolutionary Games and Population Dynamics. Cambridge University Press,1998. doi: 10.1017/cbo9781139173179.

L. A. Imhof and M. A. Nowak. Evolutionary game dynamics in a wright-fisher process. Journal of Mathe-

matical Biology, 52(5):667–681, Feb 2006. doi: 10.1007/s00285-005-0369-8.J. F. C. Kingman. Mathematics of Genetic Diversity. Society for Industrial and Applied Mathematics, Jan1980. doi: 10.1137/1.9781611970357.

A. W. Knapp. Basic Algebra. Birkhuser Boston, 2006. doi: 10.1007/978-0-8176-4529-8.E. Lieberman, C. Hauert, and M. A. Nowak. Evolutionary dynamics on graphs. Nature, 433(7023):312–316,Jan 2005. doi: 10.1038/nature03204.

W. Maciejewski, F. Fu, and C. Hauert. Evolutionary game dynamics in populations with heterogenousstructures. PLoS Computational Biology, 10(4):e1003567, Apr 2014. doi: 10.1371/journal.pcbi.1003567.

J. Maynard Smith. Evolution and the Theory of Games. Cambridge University Press, 1982. doi: 10.1017/cbo9780511806292.

A. McAvoy and C. Hauert. Structure coefficients and strategy selection in multiplayer games. Journal of

Mathematical Biology, Apr 2015a. doi: 10.1007/s00285-015-0882-3.A. McAvoy and C. Hauert. Structural symmetry in evolutionary games. Journal of The Royal Society

Interface, 12(111):20150420, Sep 2015b. doi: 10.1098/rsif.2015.0420.A. McAvoy and C. Hauert. Asymmetric evolutionary games. PLOS Computational Biology, 11(8):e1004349,Aug 2015c. doi: 10.1371/journal.pcbi.1004349.

J. McMillan. Game Theory in International Economics. Routledge, Dec 2001. doi: 10.4324/9781315014562.J.-F. Mertens, S. Sorin, and S. Zamir. Repeated Games. Cambridge University Press, 2015. doi: 10.1017/cbo9781139343275.

P. A. P. Moran. Random processes in genetics. Mathematical Proceedings of the Cambridge Philosophical

Society, 54(01):60, Jan 1958. doi: 10.1017/s0305004100033193.A. Neyman. From markov chains to stochastic games. In Stochastic Games and Applications, pages 9–25.Springer Netherlands, 2003. doi: 10.1007/978-94-010-0189-2$z $2.

23

A. Neyman and S. Sorin, editors. Stochastic Games and Applications. Springer Netherlands, 2003. doi:10.1007/978-94-010-0189-2.

M. A. Nowak. Evolutionary Dynamics: Exploring the Equations of Life. Belknap Press, 2006.M. A. Nowak, A. Sasaki, C. Taylor, and D. Fudenberg. Emergence of cooperation and evolutionary stabilityin finite populations. Nature, 428(6983):646–650, Apr 2004. doi: 10.1038/nature02414.

M. A. Nowak, C. E. Tarnita, and T. Antal. Evolutionary dynamics in structured populations. PhilosophicalTransactions of the Royal Society B: Biological Sciences, 365(1537):19–30, Nov 2009. doi: 10.1098/rstb.2009.0215.

H. Ohtsuki and M. A. Nowak. The replicator equation on graphs. Journal of Theoretical Biology, 243(1):86–97, Nov 2006. doi: 10.1016/j.jtbi.2006.06.004.

H. Ohtsuki, C. Hauert, E. Lieberman, and M. A. Nowak. A simple rule for the evolution of cooperation ongraphs and social networks. Nature, 441(7092):502–505, May 2006. doi: 10.1038/nature04605.

H. Ohtsuki, M. A. Nowak, and J. M. Pacheco. Breaking the symmetry between interaction and replacementin evolutionary dynamics on graphs. Physical Review Letters, 98(10), Mar 2007a. doi: 10.1103/physrevlett.98.108106.

H. Ohtsuki, J. M. Pacheco, and M. A. Nowak. Evolutionary graph theory: Breaking the symmetry betweeninteraction and replacement. Journal of Theoretical Biology, 246(4):681–694, Jun 2007b. doi: 10.1016/j.jtbi.2007.01.024.

J. M. Pacheco, F. L. Pinheiro, and F. C. Santos. Population structure induces a symmetry breaking favoringthe emergence of cooperation. PLoS Computational Biology, 5(12):e1000596, Dec 2009. doi: 10.1371/journal.pcbi.1000596.

R. Poli. Tournament selection, iterated coupon-collection problem, and backward-chaining evolutionaryalgorithms. In Foundations of Genetic Algorithms, pages 132–155. Springer Science + Business Media,2005. doi: 10.1007/11513575$z $8.

W. H. Press and F. J. Dyson. Iterated prisoner’s dilemma contains strategies that dominate any evolutionaryopponent. Proceedings of the National Academy of Sciences, 109(26):10409–10413, May 2012. doi: 10.1073/pnas.1206569109.

M. L. Puterman, editor. Markov Decision Processes. John Wiley & Sons, Inc., Apr 1994. doi: 10.1002/9780470316887.

D. G. Rand, M. A. Nowak, J. H. Fowler, and N. A. Christakis. Static network structure can stabilizehuman cooperation. Proceedings of the National Academy of Sciences, 111(48):17093–17098, Nov 2014.doi: 10.1073/pnas.1400406111.

L. S. Shapley. Stochastic games. Proceedings of the National Academy of Sciences, 39(10):1095–1100, Oct1953. doi: 10.1073/pnas.39.10.1095.

G. Szabo and G. Fath. Evolutionary games on graphs. Physics Reports, 446(4-6):97–216, Jul 2007. doi:10.1016/j.physrep.2007.04.004.

G. Szabo and C. Toke. Evolutionary prisoner’s dilemma game on a square lattice. Physical Review E, 58(1):69–73, Jul 1998. doi: 10.1103/physreve.58.69.

C. E. Tarnita, T. Antal, H. Ohtsuki, and M. A. Nowak. Evolutionary dynamics in set structured populations.Proceedings of the National Academy of Sciences, 106(21):8601–8604, May 2009. doi: 10.1073/pnas.0903019106.

C. Taylor, D. Fudenberg, A. Sasaki, and M. A. Nowak. Evolutionary game dynamics in finite populations.Bulletin of Mathematical Biology, 66(6):1621–1644, Nov 2004. doi: 10.1016/j.bulm.2004.03.004.

P. D. Taylor and L. B. Jonker. Evolutionary stable strategies and game dynamics. Mathematical Biosciences,40(1-2):145–156, jul 1978. doi: 10.1016/0025-5564(78)90077-9.

P. D. Taylor, A. J. Irwin, and T. Day. Inclusive fitness in finite deme-structured and stepping-stone popu-lations. Selection, 1(1):153–164, Jan 2001. doi: 10.1556/select.1.2000.1-3.15.

P. D. Taylor, T. Day, and G. Wild. Evolution of cooperation in a finite homogeneous graph. Nature, 447(7143):469–472, May 2007. doi: 10.1038/nature05784.

A. Traulsen, J. M. Pacheco, and M. A. Nowak. Pairwise comparison and selection temperature in evolutionarygame dynamics. Journal of Theoretical Biology, 246(3):522–529, Jun 2007. doi: 10.1016/j.jtbi.2007.01.002.

A. Traulsen, C. Hauert, H. De Silva, M. A. Nowak, and K. Sigmund. Exploration dynamics in evolutionarygames. Proceedings of the National Academy of Sciences, 106(3):709–712, Jan 2009. doi: 10.1073/pnas.

24

0808450106.Nicholas Wage. Evolutionary games on structured populations under weak selection. Harvard senior thesis,2010.

J. Y. Wakano and Y. Iwasa. Evolutionary branching in a finite population: Deterministic branching vs.stochastic branching. Genetics, 193(1):229–241, Oct 2012. doi: 10.1534/genetics.112.144980.

J. Wakeley and T. Takahashi. The many-demes limit for selection and drift in a subdivided population.Theoretical Population Biology, 66(2):83–91, Sep 2004. doi: 10.1016/j.tpb.2004.04.005.

L. Wardil and C. Hauert. Origin and structure of dynamic cooperative networks. Scientific Reports, 4, Jul2014. doi: 10.1038/srep05725.

25