POLITECNICO DI MILANO Corso di Laurea in Ingegneria Informatica Dipartimento di Elettronica e Informazione Distributed Algorithms for Learning Balanced Partitions in Heterogeneous Multiagent Systems AI & R Lab Artificial Intelligence and Robotics Laboratory of the Politecnico di Milano Advisor: Prof. Andrea Bonarini Co-Advisors: Eng. Marcello Restelli Eng. Enrique Mu˜ noz de Cote Thesis Dissertation of: Maurizio Lattuada, matricola 666971 Academic Year 2005-2006
Transcript
POLITECNICO DI MILANO
Corso di Laurea in Ingegneria Informatica
Dipartimento di Elettronica e Informazione
Distributed Algorithms for Learning
Balanced Partitions in
Heterogeneous Multiagent Systems
AI & R Lab
Artificial Intelligence
and Robotics Laboratory of the Politecnico di Milano
Advisor:
Prof. Andrea Bonarini
Co-Advisors:
Eng. Marcello Restelli
Eng. Enrique Munoz de Cote
Thesis Dissertation of:
Maurizio Lattuada, matricola 666971
Academic Year 2005-2006
POLITECNICO DI MILANO
Corso di Laurea in Ingegneria Informatica
Dipartimento di Elettronica e Informazione
Algoritmi Distribuiti per
Apprendere Partizioni Bilanciate in
un Sistema a piu Agenti Eterogenei
AI & R Lab
Laboratorio di Intelligenza Artificiale e Robotica
del Politecnico di Milano
Relatore:
Prof. Andrea Bonarini
Correlatori:
Ing. Marcello Restelli
Ing. Enrique Munoz de Cote
Tesi di Laurea di:
Maurizio Lattuada, matricola 666971
Anno Accademico 2005-2006
A mamma, per tutto quel che mi ha dato e insegnato.
A papa, perche sarebbe stato molto orgoglioso di me e fiero di vedere “ul me fioeu
diventa ingegne”.
Summary
Nowadays, Reinforcement Learning techniques (RL for short) are widely applied
in many real environments as a plain trial-and-error framework. This peculiar
characteristic makes RL techniques very tricky to apply without having any dy-
namic environment model. RL has been also applied in multiagent systems. As a
consequence, we can formalize different behaviors of players as well as interactions
among them.
The aim of this thesis is to study how and why different types of agents form
coalitions (thus cooperate) in order to satisfy a goal that has certain characteristics.
In this dissertation we have formalized a new type of games in order to study such
social behaviors. We have studied how to formalize an environment when we have
such agents behaving in order to reach an optimal configuration. We have obtained
encouraging results with this framework. We have also demonstrated the existence
of a social cooperation among coalitions of different types of agents and that this
kind of cooperation is needed to reach a goal.
I
Sommario
In questa tesi abbiamo studiato le tecniche di apprendimento per rinforzo (Reinfor-
cement Learning, RL) in ambito multiagente. RL punta a imitare gli esseri viventi,
in modo particolare per quel che riguarda come essi apprendono delle abilita in un
ambiente a loro sconosciuto. Tramite l’apprendimento, un organismo impara a
essere autonomo e a interagire in maniera ottimale con l’ambiente in cui opera
(si veda [17]). Questo campo dell’intelligenza artificiale (AI) modella questa atti-
vita di apprendimento attraverso un agente che opera in un ambiente, e da questa
interazione, formalizzata con un segnale di rinforzo, affina la propria politica di
azione.
Prima si e citato il termine “agente”: ma cos’e un agente? Possiamo modelliz-
zare un agente come una entita che percepisce l’ambiente in cui opera attraverso
dei sensori e agisce in tale ambiente attraverso degli attuatori. Di conseguenza,
possiamo identificare come agente un robot che si muove in una stanza oppure un
programma che bilancia opportunamente il carico in una rete di calcolatori.
Un’altra fondamentale caratteristica degli agenti che deve essere presente in
RL riguarda la razionalita delle proprie azioni. Un agente e detto razionale se le
proprie azioni possono permettere a tale agente di raggiungere un obiettivo, che
solitamente e formalizzato come l’estremizzazione di una funzione di utilita.
RL e stato ampiamente studiato nel caso singolo agente e al giorno d’oggi puo
essere considerata una disciplina matura. La fase di apprendimento e completa-
mente autonoma e non e soggetta in alcun modo a una supervisione iniziale in cui
si istruirebbe l’agente con degli esempi noti. Invece, in questo caso si ha a che
III
fare con un puro paradigma “impara dagli errori” (trial-and-error) in cui l’agente
impara come comportarsi basandosi sulla propria esperienza passata.
Una chiara e immediata estensione di questo ambiente vede la presenza di
piu agenti che operano nello stesso ambiente (sistemi a molti agenti, RL-MAS).
Le conoscenze maturate nel caso a singolo agente sono state impiegate anche in
questa estensione con delle opportune modifiche. Ovviamente questa nuova forma-
lizzazione presenta maggiori difficolta che il caso a singolo agente, infatti bisogna
comprendere le interazioni tra gli agenti e l’ambiente e tra gli agenti stessi. Di con-
seguenza, si ha a che fare con un ambiente non stazionario, perche e influenzato
dalla politica di ogni agente.
Inoltre, le interazioni tra i diversi agenti possono essere modellizzate in modo
cooperativo o competitivo per soddisfare l’obiettivo da raggiungere (di solito e un
equilibrio di Nash, NE). In questa tesi ci focalizzeremo su comportamenti coope-
rativi, dove si ha un obiettivo che, per essere soddisfatto, necessita una forma di
cooperazione tra gli agenti. Questo comportamento cooperativo puo essere indotto
attraverso opportuni segnali di rinforzo immediato (reward) assegnati agli agenti.
A questo punto e naturale prevedere un’ulteriore estensione del caso a molti
agenti introducendo la possibiltia di formare coalizioni tra agenti. Chiaramente, in
questo caso si dovra tener conto anche delle interazioni tra le coalizioni di agenti.
Il rinforzo ora assume un carattere globale, dato che e assegnato a ogni coalizione.
Si necessitano dunque dei meccanismi per suddividere tale rinforzo tra tutti gli
agenti della coalizione (i piu noti sono il core e lo Shapley’s value, si veda [16]);
con questi meccanismi si puo dare diversa importanza e/o priorita a determinati
agenti.
Il contributo dato da questa tesi e duplice: dapprima sono stati analizzati diversi
metodi per coordinare un insieme di agenti individuando i pro e i contro in parti-
colari ambienti conosciuti in letteratura. In seguito, abbiamo proposto una nuova
tipologia di giochi (task allocation via coalition formation games) che riguardano i
problemi di allocazione delle risorse con la presenza di agenti eterogenei.
La coordinazione tra agenti analizzata si basa sull’approccio COIN (COllective
INtelligence, [23]) di Wolpert et al. e punta a indurre forme di cooperazione tra
IV
agenti senza bisogno di avere un modello della dinamica del mondo e senza che
questi possano comunicare tra loro, ma solo attraverso l’interazione con l’ambiente
(naturalmente attraverso i segnali di rinforzo immediato). Questa metodologia e
stata analizzata e verificata in problemi noti in letteratura che prevedono la coo-
perazione tra agenti per poter raggiungere un obiettivo (in particolare, il classico
mondo a griglia e il Bar Problem di Brian Arthur, [24] e [6]). Sono state eviden-
ziate le peculiaria di questo approccio e nel contempo sono state analizzate anche
le carenze in particolari ambienti di tipo non stazionario. Un grosso punto a fa-
vore di questo approccio prevede, oltre al fatto di non avere alcun modello della
dinamica, la disponibilita di una funzione di utilita globale (world utility) che e
impiegata per valutare il comportamento globale del mondo che emerge dai sin-
goli comportamenti degli agenti. Con questa funzione di utilita globale, vengono
opportunamente calcolati i segnali di rinforzo immediato da distribuire agli agenti
in modo tale da indurre una forma di cooperazione implicita. Analizzando questo
approccio sono state evidenziate delle lacune: alcune riguardano le prestazioni in
ambienti non stazionari (che sono comunque buone, ma non ottime come presenta-
to in letteratura), altre riguardano il modo con cui i segnali di rinforzo immediato
sono calcolati (questi possono essere simmetrici, dunque si potrebbero avere delle
velocita di convergenza non ottimali).
A questo punto si e deciso di complicare ulteriormente il problema introducendo
una diversificazione tra gli agenti, vale a dire la presenza di piu agenti, ma con
ruolo diverso. Queste diverse tipologie di agenti hanno un duplice scopo: devono
trovare una forma di coalizione e con questa devono cercare di raggiungere un
obiettivo prefissato. In letteratura, il campo che studia la creazione di coalizioni
(coalition formation) e stato studiato e analizzato nei suoi diversi aspetti ([16]),
ma quel che noi proponiamo e una ulteriore estensione che vede la presenza di
coalizioni nei giochi che si occupano di allocazione di risorse. E stata definita una
metodologia di assegnamento dei rinforzi che pone particolare riguardo alle richieste
computazionali per fare in modo che ogni agente sia in grado di discernere la
migliore struttura di coalizione da formare per poter poi raggiungere l’obiettivo; se
poi questo obiettivo non e stato raggiunto, gli agenti sono in grado autonomamente
V
(cioe senza avere alcuna possibilita di comunicazione, dunque senza una forma
di coordinamento esplicito) di cambiare la struttura della coalizione per tentare
nuovamente di raggiungere l’obiettivo prefissato.
I risultati ottenuti con questo approccio sono particolarmente incoraggianti,
soprattutto nel problema analizzato (si tratta di una estensione del Bar Problem,
[25]) in cui sono state prese in considerazione diverse configurazioni per poter
verificare l’estensione del problema da noi proposta.
Questo studio apre nuovi campi di ricerca futuri in diverse direzioni. E interes-
sante studiare l’influenza che ha lo spazio di stato su queste particolari tipologie di
problemi: a questo punto potrebbe tornare utile un approccio simile a LEAP ([2])
in cui si passa da uno spazio a uno spazio di caratteristiche piu compatto, oppure
una diversa caratterizzazione dello spazio di stato attuata da ciascun agente in
relazione alle coalizioni formate finora.
Un altro aspetto da studiare riguarda la definizione di contributo marginale da
noi proposta. Dato che questa funzione e strettamente legata alla definizione di
funzione caratteristica di una coalizione, si possono formalizzare diversi comporta-
menti con quest’ultima e dunque si rende necessario studiare le differenti prestazioni
ottenute. Inoltre, la funzione di contributo marginale si preoccupa principalmente
di assegnare un rinforzo alla coalizione di agenti, ma non di come questo sia poi
suddiviso tra essi. Il contributo marginale e il nocciolo del valore di Shapley. Sicco-
me quest’ultimo e computazionalmente pesante da calcolare, sarebbe interessante
trovare una relazione tra tale valore e la funzione di contributo marginale in modo
tale da ricostruire una approssimazione o un valore di Shapley atteso futuro in
modo da poter poi essere utilizzato per dividere effettivamente il rinforzo tra tutti
gli agenti di una coalizione.
VI
Ringraziamenti
Prima di tutto vorrei ringraziare il Relatore, Prof. Andrea Bonarini, per avermi
introdotto e motivato in questo lavoro e per la grande disponibilita accordatami.
Un gigantesco grazie va a Marcello, Enrique e Alessandro per tutto il tempo
che mi hanno dedicato, per le innumerevoli discussioni (e risate) e soprattutto per
la vicinanza dimostratami in particolari periodi “extra-tesi”. Senza il loro aiuto
e soprattutto senza la loro amicizia non sarei arrivato fino a questo traguardo.
Grazie di cuore.
Ringrazio anche gli amici che mi hanno accompagnato in questa avventura
chiamata “Poli” (Gabriele, Emanuele, Massimo, Bedda, il Guso, il Vince, Ciusipp,
. . .) e tutti quelli con cui ho diviso il tempo in AIRLab (Simone, Daniel e la sua
sangria, Mario, il Lazza, Matteo, . . .) scambiandoci consulenze tecniche su come
smontare il PRLT (per la gioia di Alessandro). Ovviamente non dimentico anche
altri amici: Homer, Centu, Mini, Lindi, Monfro, Passe, quelli de “LZD”, Lele,
Dino, . . .
Un grazie e piu dovuto anche per Jessica che ha avuto la pazienza e la forza di
sopportarmi in quest’anno di tesi, e anche per tante altre cose che non sto qui a
scrivere.
Infine un ringraziamento enorme va a mamma e papa per . . . per . . . per tutto!
A mamma, perche possa essere orgogliosa di me. A papa, perche ha fatto in tempo
a leggere un primissimo abbozzo di questa tesi e perche tanto desiderava esserci
5.1 Packet Routing in a Network 5. Experimental Applications
It is possible to formalize many real systems like a COIN (see Section 3.2).
This framework is widely applied, e.g. vehicular traffic control, learning sequences
of actions, packet routing in a network. Across-the-board it is useful to use this
distributed control methodology where we have many processes (packets, rovers,
agents) in such a way hard to be formalized, where centralized controller has plan-
ning (how “make” the controller) and computational (best actions to be chosen
may take a long time according to the domain to be controlled) difficulties.
In the following sections we explain real cases in literature where a world where
many agents act has been formalized and controlled by a COIN:
packet routing in a network : we need to route packets in a network com-
posed by routers and computers with the well known SPA (Shortest Path
Algorithm) and with the new approach introduced by COIN (see [22]);
learning sequences of actions : we have a grid world where agents act trying
to take as many tokens as possible (see [24]);
attendance model of El Farol Bar Problem : in this well known case we
have many agents that must attend the bar in a week both avoiding over-
crowed days and days with few agents (see [6]).
The environments above mentioned are extremely different from each other: if
we try to formalize them just with a simple RL algorithm we may face the problem
of how we can assign rewards to all agents acting in these environments; this may
lead to introduce important approximations that necessarily impact on the system
behavior at runtime. As a consequence, the authors explain how COIN was applied
in known cases in literature in order to induce a cooperative behavior among a set
of agents, and they compare the COIN performance with other utility function
ones in order to verify it usefulness.
5.1 Packet Routing in a Network
We are facing the problem of how can we route information packets in a network
to make them reaching their destinations according to an opportune metrics about
42
5. Experimental Applications 5.1 Packet Routing in a Network
the path chosen. A well known metrics is SPA where each packet is routed on a
link such that it can reach its destination with as less steps as possible. In this
situation, the microlearner sets the internal parameters of its router running the
SPA. Microlearners do not completely address the problem of guaranteeing that
private utilities do not induce the learning algorithm to work at cross-purposes to
satisfy the main goal of the routing.
In this case, we use 3 algorithms: SPA and COIN both with full knowledge
(FK) of the true rewards obtained following a specified path (with reward being
the time taken by a packet to be routed) and COIN memory based (MB – it has
just local knowledge).
5.1.1 COIN for Network Routing
In this example we concentrate on the two networks depicted in Figure 5.1 where
traffic originated by routers represented by white boxes had only the routers rep-
resented by dark boxes as ultimate destinations (point out that in both networks
router 2 is a bottleneck). As standard definition traffic at router is a pair 〈r, d〉,
(a) Network A (b) Network B
Figure 5.1: Network architectures (from [22])
where r is a real number (source router) and d is the destination tag (e.g.: a com-
puter) to be reached. At each timestep each router sums all traffic received from
upstream routers to obtain the amount of traffic, then it chooses where to send
that load.
43
5.1 Packet Routing in a Network 5. Experimental Applications
We keep a running average of the total value of the load of each router in L
previous timesteps; this average is used to compute W (x) to get the sum of the
total delay accumulated at this timestep by all the packets traversing this router at
this timestep. Each router has a different definition of such a W (x) (according to
the hardware, queue length, . . . ): in this testbed routers 1 and 3 have W (x) = x3,
while router 2 (bottleneck) has W (x) = log(x + 1). Obviously, the overall goal is
to minimize the total delay encountered by all traffic.
In COIN, with η we identify the pair 〈router, destination〉, so ζη,t
is the vector
of traffic sent along all link exiting from the router η (with the destination of packets
traversing this router) at time t. As subworld we identify each set of routers whose
packets share the same ultimate destination.
In the classic SPA each node η tries to set ζη,t
to minimize the sum of delays to
be accumulated by traffic on the way to its ultimate destination. In COIN we use
a complementary approach, i.e., η tries to set ζη,t
in such a way to optimize gω for
subworld ω containing η. With the term “full knowledge” we mean that at time t
all routers know the average loads of all routers at time t−1 and assume that those
values will be the same at time t (this can be a good assumption for large values
of L), so we can make accurate estimates of how best route their traffic according
to their respective criteria.
Having limited knowledge, COIN routing can only predict the WLU value re-
sulting from each routing decision. More precisely, for each pair 〈router, destination〉the microlearner estimates the mapping between the load on all outgoing links to
WLU-based reward; then each router sends packets along the path resulting with
the best estimated reward. In this case, we use a more conservative method, i.e.
we randomly choose between the path with the best estimated reward and the path
chosen by FK SPA.
The load of a router r at time t is given by ζ, while Wr,t(ζ) is the function
W (x) of node r at time t. The world utility function is given by the total delay,
i.e. G(ζ) =∑
r,t Wr,t(ζ). Using WLU to set the local utility of each microlearner
we have g(ζ) =∑
r,t ∆ω,r,t(ζ), where ∆ω,r,t =[Wr,t(ζ) − Wr,t(CLw(ζ))
].
At each time step the MB COIN uses∑
r ∆ω,r,t(ζ) as reward signal for trying
44
5. Experimental Applications 5.1 Packet Routing in a Network
to optimize this full WLU. This reward is computed in a decentralized way: all
packets have a header containing a running sum of the ∆s encountered in all routers
it has traversed so far. Each destination node sums the values of the headers and
sends this sum back to all routers that had routed to it.
5.1.2 Experimental Results
The networks discussed above (Figure 5.1) were tested under light, medium and
heavy traffic loads as depicted in Table 5.1; moreover, from each source router a
Table 5.1: Source–destination pairings for the three traffic loads
(a) Network A (b) Network B
Figure 5.2: Overall delay of the networks (from [22])
As depicted in Figure 5.2 (these results are averaged on 50 different executions
with a window-size L = 50) we see that FK COIN outperforms the FK SPA. So
with COIN we have a system operating in a way that reduces the average total
45
5.2 Learning Sequences Of Actions 5. Experimental Applications
delay for all packets, not in a greedy fashion like SPA. Moreover MB COIN has
better performance than FK SPA: so we deduce that COIN will always outperform
any algorithm that estimates the shortest path.
5.2 Learning Sequences Of Actions
Another typical application of RL is the Grid world, where an agent navigates in
a two-dimensional grid and at each time step it receives a reward related to the
action chosen. In the episodic version of the Grid world an agent moves for a
certain amount of time steps and then it is returned to its initial position; in this
situation we need a learner optimizing the sum of rewards obtained (Q-learning
and SARSA are typically used for this problem).
In this problem there are many agents navigating in the grid simultaneously
and interacting with the reward of each others. These interactions are modeled
through tokens with different values laid on the grid: each one has values between
0 and 1 and each cell may have at most one token. When an agent moves into a
cell with a token it receives a reward equal to the value of the token and then it
removes that token (so that reward will no longer be available if an agent enters
in that cell). At the end of the episode all the tokens are reset and each agent is
returned to its initial position. The main goal is to collect the highest amount of
tokens in a fixed number of time steps.
Interactions among agents are a useful formalization to examine coordination
and selfish behavior, so considering TOC and the likes.
5.2.1 COIN Solution
Here we pose this problem in the form of COIN and we define:
• Lη,t is the matrix representing the location of agent η at time t. If it is in
location (x, y) then Lη,t,x,y = 1, otherwise Lη,t,x,y = 0. With {Lη,t} we denote
the set of location matrices.
• Laη,t is the location of agent η would have had at time t had it taken action
46
5. Experimental Applications 5.2 Learning Sequences Of Actions
a at time step t − 1.
• Lη is the location matrix of agent η across all time (Lη =∑
t Lη,t).
• Lη,<t is the location matrix of agent η across all time before t (Lη =∑
t′<t Lη,t′).
• L is the location matrix of all agents across all time (L =∑
t
∑η Lη,t).
• L<t is the location matrix of all agents across all time less than t (L =∑
t′<t
∑η Lη,t′).
• L−η is the location matrix of all agents, but η, across all time (L−η = L−Lη).
• L−η,<t is the location matrix of all agents other than η across all time before
t (L−η,<t = L<t − Lη,<t).
• Θ stores the initial values and locations of all tokens.
The space Z is composed by Θ and the set {Lη,t} of all location matrices, while
a worldline ζ is a point in that space. We define the function V (L, Θ) which returns
the value of a token received from a location matrix as follows:
V (L, Θ) =∑
x,y
Θx,y · min(1, Lx,y) (5.1)
The world utility function G(ζ) is given by the sum of all the tokens taken
during an episode:
G(ζ) = V (L, Θ) (5.2)
To formulate the WLU in this problem let us suppose that the operator CLη
sets the state of η to the null vector, so we have the WLU where the agent is
removed from the worldline:
WLU−→0
η (ζ) = G(ζ) − V (L−η, Θ) (5.3)
The utility stated in Equation (5.3) is different from one where the values of the
tokens present in locations visited by agents are summed (rather such a function
is known as Selfish Utility, SU for short). WLU−→0 returns the values of the tokens
in locations not visited by other agents, i.e. the values of the tokens that would
not have been taken, if agent η had not been in the system.
47
5.2 Learning Sequences Of Actions 5. Experimental Applications
These utility functions are based on the performance on a full episode: to learn
an optimal sequence of actions we introduce a reward related to a single time step.
To that end, let us decompose an arbitrary utility function U as follows:
U(L) =∑
t
U(L<t+1) − U(L<t)
The reward for a single time step is given by:
Rt(L) = U(L<t+1) − U(L<t) (5.4)
As a consequence for the two utilities depicted above (global and WL ones) we
introduce the associated single time step utilities:
GRt(ζ) = V (L<t+1, Θ) − V (L<t, Θ) (5.5)
WLUR−→0η (ζ) = GRt(ζ) − (V (L−η,<t+1, Θ) − V (L−η,<t, Θ)) (5.6)
5.2.2 Results
In the experiments we used 3 different utility functions (they were opportunely
changed as stated in Equation (5.4)):
• SU (Selfish Utility): each agent receives the discounted sum of the values of
tokens that it alone collected;
• TG (Team Game utility): each agent receives the full world utility;
• WLU (Wonderful Life Utility): is the contribute given by an agent to the
token collection, i.e. it is the difference in the total token collection with and
without that agent.
Each agent was controlled by a Q-learner: the input space for each one consists
of its location in the grid, while the action space is given by the 4 directions an
agent can choose. The discount parameter γ is set to 0.95 and actions are chosen
stochastically based on Q-values, so the probability an agent can choose action ai
in state s is given by
P sai
=kQ(s,ai)
∑j kQ(s,aj)
, k = 50
48
5. Experimental Applications 5.3 Bar Problem
Figure 5.3 depicts clearly that the SU function produces poor results, worse
than random actions, because each agent tries to collect as token as possible, so
competing with the others. TG utility seem to be quite good with respect to SU,
but the learning time is extremely large, because each agent receives a noisy reward
(it does not perceive clearly the consequences related only to its actions). Instead,
this problem does not occur with WLU (there is also the Aristocrat Utility in [24],
but here is not examined), because an agent can discern clearly how its actions
affected the world reward.
Figure 5.3: System performance with 10 agents on a 10×10 grid (from [24])
In Figure 5.4 we have qualitatively similar results depicted in Figure 5.3. TG
utility has harder time learning respect to the small grid, because in this case the
payoff is even more noisy. Instead, if agents use WLU they were able to cooperate
collectively because they discern the consequence of their actions from the obtained
rewards.
5.3 Bar Problem
In this well known problem in literature (see [6, Section 4]) we try to apply COIN
in a problem widely examined (this is known as dispersion game, where we have
n agents and k tasks to be assigned to those agents). Here we have n agent, each
of whom picks one of seven nights to attend a bar the following week (this process
is repeated every week) to avoid both overcrowed nights and boring ones (nights
49
5.3 Bar Problem 5. Experimental Applications
Figure 5.4: System performance with 100 agents on a 32×32 grid (from [24])
where there are few agents attending the bar). In each week each agent uses its
own RL algorithm to choose which night to attend the bar to maximize its utility.
The world utility function is given by:
G(ζ) =∑
t
7∑
k=1
γk(xk(ζ, t)) (5.7)
where xj(ζ, t) is the j-th component of the vector x(ζ, t), i.e. the number of agents
attending night j at week t, while γk(y) ≡ αk · y · exp(−y/c) (c and {αk} are real
values, where c is the given optimal number of agents attending the bar and {αk}are weight factors). This world utility is the sum of rewards for each night in each
week.
This G is chosen to reflect the attendance at the bar in different night config-
urations: with few and many agents G returns a small (or even negative) reward
(note that γk(u) has a maximum in u = c).
The vector α weights different nights to give them more or less importance. In
[6] the author chooses two different vectors to guarantee the effectiveness of COIN:
α1 = [1 1 1 1 1 1 1] and α2 = [0 0 0 7 0 0 0].
In these experiments, the authors set c = 6 and a number n of agents 4 times
than the number of agents necessary to have c agents attend the bar on each of
the seven nights (n = 7 · 6 · 4 = 168 agents).
Each agent is configured with different reward functions1:
1Pay attention to not confuse reward functions and utility functions, since the former reflect
50
5. Experimental Applications 5.3 Bar Problem
• Uniform Division reward (UD for short):
UD(dω(t), ζ, t) ≡ γdω(xdω
(ζ, t))/xdω(ζ, t)
• Global Reward (GR for short):
GR(dω(t), ζ, t) ≡7∑
k=1
γk(xk(ζ, t))
• Wonderful Life reward (WL for short):
WL(dω(t), ζ, t) ≡7∑
k=1
γk(xk(ζ, t)) −7∑
k=1
γk(xk(CLω(ζ), t)) =
= γdω(xdω
(ζ, t)) − γdω(xdω
(CLω(ζ), t))
where dω is the night chosen by subworld ω (to keep the problem simple we
assume that each subworld ω is composed by only one agent).
With the GR utility each agent receives the same reward of others: obviously
the system is factored, but evaluating this function requires a centralized commu-
nication. This characteristic (to be avoided) is not present in WL, since each agent
only needs to know the total attendance on the night it attended.
(a) Performance with α1 (b) Performance with α2
Figure 5.5: Average performance of the Bar Problem (from [6])
the state of the system in a given time step (is is potentially observable) and the latter are a
formalization of the main goal of each agents (there depend by the past behavior of each agent).
51
5.3 Bar Problem 5. Experimental Applications
5.3.1 Results
Figure 5.5 depicts the average performance averaged over 50 separate runs; for
both Figure 5.5(a) and Figure 5.5(b) the top curve is WL, the middle is GR and
the bottom is UD. Using WL we have convergence to near optimal performance,
so we deduce that the Bar Problem is enough suited to have cooperation.
Note the convergence time of WL as compared with GR in Figure 5.5(b), about
4 times faster. With α1 (Figure 5.5(a)) the convergence time of WL is 30 times
lower than the GR. In both cases UD utility has awful performance worsening in
future weeks, behavior due typically to a low signal-to-noise ratio (see team games
in Section 4.2.3).
52
Part II
Innovation
53
Chapter 6Theoretical Considerations
In mathematics you don’t understand things. You just get
In this chapter we discuss coordination problems in dispersion games. Coordi-
nation problems are a key factor of RL if we have many agents and our aim is to
induce them to cooperate. At first, we introduce the bothersome problem of the
delayed reward that is often present in RL giving an intuitive example, then we
introduce the theoretical differences between Grid world and Bar Problem; then
we analyze the Bar Problem, in particular its reward function and its behavior (the
latter using the Q-learning dynamics following the approach proposed in [20]).
Next, we discuss about the Q-learning dynamics in order to understand how
an agent policy evolves during the learning phase using different reward functions.
Due to this analysis, we can understand how the policy of each agent evolves during
the learning phase using different reward functions.
Finally, we introduce the theoretical grounds of the Coalition Formation theory,
that will be extended in Chapter 7.
6.1 Class of Games
In the following sections, we discuss about two main classes of games, how their
structures influence the trend of an environment formalized with such games and
how they can avoid or emerge some bothersome aspects of RL. After a brief the-
oretical introduction, we analyze the Bar Problem and the Grid world in the eyes
of these definitions.
6.1.1 Matrix Games
Definition 6.1. A matrix game is defined as a tuple 〈N, A1...N , R1...N〉, where N
is a collection of n agents, Ai is the set of actions available to agent i (let A be the
joint action space, i.e. A1 × A2 × . . . × An) and Ri is the reward function A → R
for each agent i.
In such a game, each agent chooses actions maximizing its own reward function
(it can be viewed as a n-dimensional matrix) which depends on the actions chosen
by other agents.
56
6. Theoretical Considerations 6.1 Class of Games
What does it mean “solve such a game”? Solving the game means to find
the agent’s best response policy, which allows the agent to collect the highest
reward given the other agents’ policies. A stationary strategy can be evaluated
only if strategies of other agents are known (for instance the prisoner dilemma,
the matching pennies game, . . . ). Agents can also play mixed strategies, so they
select actions according to a probability distribution. The latter lead us to define
an opponent-dependant solution (i.e., the best response to the joint action of other
players), thus the definition on Nash equilibrium.
These games can be purely collaborative (agents share the same reward func-
tion, so these games belong to the class of zero-sum games) or purely competitive
(each agent has a reward function counteracting with that of others, these games
belong to the class of general-sum games), and they are usually played for an un-
defined number of iterations. In particular, games each agent can perceive actions
of others, but they know neither the intentions of others nor the reward functions
(for such a game see [1, Section 3]).
6.1.2 Stochastic Games
Definition 6.2. A stochastic game is defined as a tuple 〈N, S, A1...N , T, R1...N〉,where N is a collection of n agents, S is the set of all the possible states of the
game, Ai is the set of actions available to agent i (let A be the joint action space,
i.e. A1 × A2 × . . . × An), T is a transition function S × A × S → [0, 1] and Ri is
the reward function of agent i (Ri : S × A → R).
Definition 6.2 looks very similar to Definition 6.1, in fact each state of a stochas-
tic game can be viewed as a matrix game with the payoff for each joint action deter-
mined by Ri(s, a): after playing the matrix game and receiving payoffs, agents are
transitioned to another state determined by their joint action (that is associated
to a new matrix game).
57
6.1 Class of Games 6. Theoretical Considerations
6.1.3 Differences between Grid world and Bar Problem
The Grid world and the Bar problem are formalized as different typology of games:
the former belong to the class of stochastic games (Section 6.1.2), the latter to the
matrix games ones (Section 6.1.1). This is a classification based upon the structure
of a game. Moreover, we can have episodic games (i.e., games played a certain
amount of times) as well as games where there is an environment dynamically
changed by agents’ behavior.
Following the notation introduced in Section 6.1.1 and in Section 6.1.2, in the
Bar Problem (Section 5.3) Ai are the nights available to each agent i and Ri
depends only on a function of the joint action (see function xk(·, ·) of Equation
(5.7)). Furthermore, the state space is drastically reduced since each agent knows
only the number of agents attending the bar on the same night it has chosen and
not their identity1, so it can be viewed as a single-agent problem (with a reduced
state space size too). With this particular configuration and, above all, since the
Bar Problem belongs to the class of matrix games, we avoid the delayed reward
issue: the reward of a joint action (i.e. in which night each agent attends the bar)
is immediately available at the next time step, so agents have a factual evaluation
of their own actions.
The Grid world environment (Section 5.2) is a stochastic non-stationary game,
since the environment where agents act is dynamically changed accordingly on
actions chosen by agents (i.e. tokens collected are no longer present), and mainly
because each single reward depends upon the joint action and the system’s state.
In this kind of games we could face the delayed reward problem. As we will see in
Section 6.2, TG, SU and UD utility functions are immune to this problem, since the
reward returned to each agent relies only on the joint action executed at previous
time step (it obviously depends on the environment configuration, but this one
does not involve a delayed reward). With the WLU function these considerations
do not hold, as shown by the example proposed in Section 6.2.
1So having agents η1 and η2 or agents η41 and η79 does not make difference.
58
6. Theoretical Considerations 6.2 Delayed Reward
6.2 Delayed Reward
RL is based upon the concept of reward: an agent (or more than one) perceives a
state st of the environment at time t, it chooses an action at complying with its
policy π (so it reaches to another state s′t) and then it receives a reward signal rt+1
(which usually ranges on R). Given that reward, an agent can change its policy
following, for instance, Q-learning or SARSA (or any other RL algorithm). It is
desirable that this reward signal will be immediately available, in particular if it
is fundamental to evaluate the action at chosen in state st: in this case, an agent
can understand the consequences of its action (“it is a good action” or “it is not a
good action”) because the reward immediately obtained is directly related to that
action. If this reward signal is given to that agent with a certain delay τ and that
agent is not aware of this reward, it could not understand that such a reward is
related to an action at rather than the current action at+τ (τ ∈ N, τ > 0). In
this case the agent changes its policy accordingly to that reward, but it refers to a
wrong situation (i.e. to the pair 〈st+τ , at+τ 〉 rather than 〈st, ar〉).This is a situation known as delayed reward in literature (see [10] and [17,
Chapter 7]), and next we see a simple example.
Recall the Grid world presented in Section 5.2, where two or more agents move
in a grid collecting tokens in such a way to avoid gathering tokens that will be taken
by other agents (this is a modelization to induce cooperation among them). Let
us consider Equation (5.6): the first term computes the sum of the token values
collected at time t, while the second one computes the sum of the token values
collected at time t without agent η. Let us take that this sequence of events holds:
• agent η1 takes the token k1 with value 2.5 at time t;
• since time t to time t + τ − 1 (τ ∈ N, τ > 1) no tokens will be collected;
• at time t + τ agent η2 is on the cell where at time t there was the token k1,
while agent η1 is far away from agent η2.
In such a situation the first term of Equation (5.3) is zero, since in this time step
t + τ no tokens were found, while the second one for agent η1 equal to the sum
59
6.3 Reward Function of the Bar Problem 6. Theoretical Considerations
of the token values collected in this time step as if it would never be existed, that
is k1. So agent η1 receives -2.5 as reward that penalizes the fact it has taken the
token k1 despite it is away from such token at time t + τ .
Note that this problem only affects WLU function in the Grid world, but not
TG, SU and UD utility functions. This is obvious, in fact this consideration can
be derived by looking at the definition of such utility functions (see Section 5.2.2):
TG, SU and UD utility functions are only related to tokens collected in the actual
time step, without considering tokens taken in the past by other agents.
6.3 Reward Function of the Bar Problem
With a deeper observation of the Bar Problem we can discuss about its exponential
world utility function γ(·) (Equation (5.7)). It is not symmetrical and it has a large
right tail that increases when c raises as depicted in Figure 6.1.
Finally, for the UD utility function with the same strategy as above we have:
(Ay)i = Ψ (1 + agents−i (Monday) , c) (6.12)
x · Ay = Pi [Monday] · Ψ (1 + agents−i (Monday) , c)+
Pi [Tuesday] · Ψ (1 + agents−i (Tuesday) , c)(6.13)
For instance, P1 [Monday] = a(t) and agents−1 (Monday) = b(t) + c(t) + d(t) +
e(t) + f(t) + g(t) + h(t) (see Equation (6.7)).
In Section 8.2.3 we will use this approach and we will show the results obtained.
6.5 Introduction to Coalition Formation
In many domains we may need to work with a large amount of agents in order to
reach a goal. In such cases, we can try to model their interactions with normal
form games, but this model might be neither accurate nor useful, because we have
a huge and unreliable model. Instead, another way to consider this problem is
to study such games in a more abstract setting from a cooperative game theory
point of view called characteristic function game. In such games, the value of
each coalition of agents S is given by a characteristic function v(S). It may be
interpreted as the value created when the members of S come together and interact.
As a consequence, a cooperative game is a pair 〈N, v(·)〉, where N is the finite set
of players and v(·) is a function mapping subsets of N to numbers.
Coalition formation involves three main activities:
66
6. Theoretical Considerations 6.5 Introduction to Coalition Formation
Coalition structure generation : it is the coalition formation phase done by
agents such that agents within each coalition coordinate their activities, but
not between coalitions. This means partitioning the set of agents N into
exhaustive and disjoint coalitions called coalition structure (CS). Notice the
difference between coalition and CS: the former is the powerset of the set
N (also indicated as 2n, where |N | = n), while the latter involves a set of
constraints (i.e.: there is not a coalition structure where a generic agent η
belongs to two or more disjoint coalitions C1 and C2).
Solving the optimization problem of each coalition : this means polling the
task and resources of agents in the coalition maximizing monetary value, that
is money received from outside the system for accomplishing tasks minus the
cost of using resources.
Dividing the value of the general solution among agents : the reward is
assigned to each coalition, but agents need that reward in order to update
their policies. As a consequence, that reward must be divided up among
them so they can understand the goodness of their actions.
These activities may overlap and they are not independent, e.g.: the coalition that
an agent wants to join depends on the portion of the value that the agent would
be allocated in each potential coalition.
6.5.1 Coalition Structure Generation
Let N be the set of all agents, and n = |N |, while S is a generic coalition. Here we
assume that each coalition’s value v = v(S) is nonnegative. In a coalition structure
CS each agent belongs to exactly one coalition and some agents may be alone in
their coalitions. We will call this set of coalition structures M . The value of a
coalition structure is given by:
V (CS) =∑
S∈CS
v(S) (6.14)
67
6.5 Introduction to Coalition Formation 6. Theoretical Considerations
Usually, the goal is to maximize the social welfare of agents by finding a coalition
structure CS∗ such that:
CS∗ = arg maxCS∈M
V (CS) (6.15)
It is easy to note that the number of coalition structures is large (Θ (nn)), so not
all the coalition structures can be enumerated unless the number of agents is small.
The exact number of coalition structures is:n∑
i=1
Z(n, i), (6.16)
where Z(n, i) is the number of coalition structures with i coalitions. This quantity
is also known as the Stirling number of the second kind and it is captured by the
following recurrence:
Z(n, i) = i · Z(n − i, i) + Z(n − 1, i − 1), (6.17)
where Z(n, n) = Z(n, 1) = 1. The first term counts the number of coalition
structures formed by adding the new agent to one of the existing coalitions (there
are i choices because the existing coalition structure has i coalitions). The second
term considers adding the new agent to a coalition of its own, and therefore existing
coalition structures with only i − 1 agents are counted.
Recall that if we have n agents, all the possible coalitions (that is the powerset of
N) is 2n−1 (not counting the empty set). If we decide to exclude some coalitions a
priori, this exclusion might cause the value of the best remaining coalition structure
to be arbitrarily far from the optimum.
In literature many researchers have mostly focused on superadditive games
(v(S ∪ T) > v(S) + v(T) for all disjoint coalitions S,T ⊆ N): in such games
coalition structure generation is trivial, in fact all agents form the grand coalition
so they operate together. However, many games are not superadditive, because
a cost to form a coalition or some constraints may exist (coordination overhead,
anti-trust penalties, limited amount of time to carry out the communications and
computations). This class of games may be subadditive (v(S ∪ T) < v(S) + v(T)
for all disjoint coalitions S,T ⊆ N), where agents are best off by operating alone,
or it may be neither superadditive nor subadditive, where some coalitions are best
off merging while others are not.
68
6. Theoretical Considerations 6.5 Introduction to Coalition Formation
6.5.2 Optimization within a Coalition
Under limited and costless computation, each coalition would solve its optimization
problem. However, in many domains we can’t solve the problem from a combi-
natorial viewpoint, so an approximate solution must be found. In such a case,
selfish interested agents would want to strike the optimal tradeoff between solution
quality and the associated computation. This will affect the values of coalitions,
which in turn will affect which coalition structure gives the highest welfare.
6.5.3 Payoff Division
Payoff division strives to divide the value gained by a coalition structure among
agents in a fair and stable way so that agents are motivated to stay with the
coalition structure rather than move out of it. Many payoff division methods have
been proposed in literature. Here we discuss about two of them: the core and the
Shapley’s value.
The Core
The core of a coalition formation game is a set of payoff configurations (−→x , CS),
where −→x is a vector of payoffs given to agents in such a way that no subgroup is
motivated to depart from the coalition structure CS (it is like the Nash equilib-
rium):
Core =
{(−→x , CS) | ∀S ⊂ N,
∑
i∈S
xi > v(S) ∧∑
i∈N
xi =∑
S∈CS
v(S)
}(6.18)
The core is the strongest of the classical solution concepts in coalition formation.
In many cases, it may be empty, because there is no way to divide the social good
so that the coalition structure becomes stable, so there will be an infinite sequence
of steps from one payoff configuration to another. To avoid such problems, explicit
mechanism were proposed, like limits on negotiation rounds, contract costs or some
social norms to limit the negotiation.
Another opposite problem is to have multiple payoff vectors in the core, so all
agents have to agree on one of them (such vector is usually called the nucleolus,
69
6.5 Introduction to Coalition Formation 6. Theoretical Considerations
that is the payoff vector that is in the center of the set of payoff vectors in the
core).
A further problem related to the core is that the constraints in the definition
become numerous as the number of agents increases (point out the term ∀S ⊂ N
in Equation (6.18)).
The Shapley’s Value
The Shapley’s value is another policy for dividing payoffs in coalition formation
games and it will be defined axiomatically. Agent i is called dummy if v(S∪{i})−v(S) = v({i}) for every coalition S that does not include agent i. Agents i and j
are called interchangeable if v((S \ {i}) ∪ {j}) = v(S) for every coalition S that
includes agent i but not agent j. The three axioms of the Shapley’s value are:
Symmetry : if agents i and j are interchangeable then xi = xj .
Dummies : if agent i is a dummy then xi = v({i})
Additivity : for any two games v and w, xi in v + w equals xi on v plus xi in w,
where the game v + w is defined by (v(S) + w(S)) = v(S) + w(S).
The Shapley’s value is the only payoff division scheme that satisfies the previous
three axioms and it is defined as follow:
xi =∑
S⊆N
(|N | − |S|)! − (|S| − 1)!
|N |! · (v(S) − v(S− {i})) (6.19)
This payoff can be interpreted as the marginal contribution of agent i to the
coalition structure averaged over all the possible joining orders (it recalls the ground
idea of COIN). Notice that the payoff must be computed over all the possible |N |!joining orders, thus it is computationally hard.
It is interesting to note that the Shapley’s value, like the core, guarantees that
individual agents and the grand coalition are motivated to stay with the coalition
structure. However, unlike the core, it does not guarantee that all subgroups of
agents are better off in the coalition structure than by breaking off into a coalition
of their own.
70
6. Theoretical Considerations 6.5 Introduction to Coalition Formation
In such games the joining order of agents matters, but in the real world there
may exist situations that can be formalized with games where that joining order is
irrelevant. In this case, the core and the Shapley’s value are unnecessarily hard to
be computed. In Chapter 7 we will analyze these games and we will propose new
solutions of the reward distribution problem.
71
Chapter 7Task Allocation via Coalition Formation
Computers are useless. They can only give you answers.
In this situation we have C coalitions (Equation (7.1)), but in such games we
must consider also all the coalition structures, thus we have a vector Υ such that
76
7. Task Allocation via Coalition Formation 7.1 Game Outline
|Υ| = 2C . As a consequence, we have a representation state used to represent the
perceived environment of any players of size is 2C ·C. The representation state size
may become very large, but we must consider that not all the possible coalition
structures are admissible, thus an agent needs not to consider all the 2C ·C coalition
structures.
It is easy to understand that even with a small amount of types of agents we can
have a huge state space and the problem may become intractable. For example, let
us suppose to have a population Q = {3, 2, 4, 5} thus having C = 359 coalitions.
The representation space is 2C · C ≃ 4.2156 · 10110 and it is obviously intractable.
Furthermore, if we are dealing with a non-stationary environment the state space
will get increasingly huge, and we must not ignore the well known problems of such
games as depicted in Section 6.1.2 and in Section 6.2.
As a consequence of the previous considerations, we do not use any state space
information. In addition to its size, we must deal with the fact that if we use
the state space, each state goodness depends on all types of agents belonging to
the coalition visiting such state. Let us suppose to have a coalition S1 = {0, 4}visiting the state s1 and the characteristic function v(·) that rates as good this
coalition (thus the related state). The four agents belonging to S1 believe that
they impliedly1 formed a good coalition in this state s1 related to a particular task
t, thus they tend to choose this task in the future (let us suppose to ignore the
exploration policy, if exists). If at next time step one agent of the first type joins
S1 (thus we have a new coalition S2 = {1, 4}), all of them choose the same task t
and the characteristic function rates as bad this new coalition, then the four agents
of the second type do not understand that the bad payoff obtained is not due to
themselves, but instead it is due to the agent of the first type. As a consequence,
now they rate as not good the task t (thus they choose another task), while it was
previously rated as good. In this situation, all agents tend to pick other tasks with
respect to those picked in the past and to rate in a wrong way different states of
environment.
In such a situation, the state space induces a further non-stationary extension
1In these games we do not allow any centralized communication among agents.
77
7.1 Game Outline 7. Task Allocation via Coalition Formation
to the problem. To reach a (near) optimal state in this huge space we must ensure
coordination among different agents belonging to a coalition, and these coalitions
must be coordinated by themselves. In the simple example described above we
have two coalitions not coordinated, thus agents do not reach any optimal state.
7.1.2 Fuzzy Games and Groups of Agents
Let P(S) be the set of all possible coalitions. Any population defined by the vector
Q ∈ Rk+ generates a characteristic function called fuzzy game:
Definition 7.3 (Fuzzy game). A Fuzzy game is a pair (Q, v) such that:
• v is the characteristic function and is a mapping function v : P \ {∅} 7→ R;
• Q ∈ Rk+.
The number of possible coalition structures is limited by the number of tasks
t. Given such a number of tasks, no more than t coalitions can be formed, thus we
introduce the following
Definition 7.4 (Coalition structure). A coalition structure CS = (S1,S2, . . . ,SH}(where 0 6 H 6 |Q|) is a partition of Q, that is Sh 6= ∅ for any h ∈ {1, 2, . . . , H},⋃H
h=1 Sh = Q and Si ∩ Sj = ∅ for any i, j ∈ {1, 2. . . .H} with i 6= j.
With this model we allow agents belonging to a set to organize themselves into
a precise coalition in order to achieve much efficient individual payoffs and possibly
to obtain large world utility values. As mentioned in Section 6.5.1, not all games
are superadditive, thus large organizations could operate less efficiently than the
sum of their constituent parts, thus a grand coalition will not form.
Given such games, in this section we describe economies with a small group
of agents belonging to a finite number of types. The main goal is to build stable
coalitions that will end up in a stable and possibly meaningful coalition structure.
Here we need to redefine the concept of core used in Equation (6.18) because we
are dealing with a priori non superadditive characteristic functions.
Definition 7.5 (Core of a fuzzy game). The core of the fuzzy game (v,Q), that is
Fcore(v,Q), is the set of vectors x = {x1,x2, . . . ,xn} such that:
78
7. Task Allocation via Coalition Formation 7.2 Utility Functions of the Game
1. xQ = maxCS∈P(Q)
∑S∈C vS, where P(Q) denotes the set of all possible coali-
tion structures and C is a set of coalitions;
2. xS > v(S) for any coalition S ∈ C.
7.2 Utility Functions of the Game
In the following sections we discuss about different utility functions of this type
of game as well as how to find a useful way to deal the reward obtained. These
functions are used both to evaluate a coalition S and the resulting task allocation,
so they formalize different aspects of this type of game.
7.2.1 Reward Distribution among Agents
One of the main interesting problems consists of finding a suitable way to distribute
the reward among agents of a coalition, so we can apply the Shapley’s value or the
core as stated in Section 6.5.3.
As discussed in Section 7.1 now we are facing with a new kind of game where we
do not deal with the joining order of agents in a coalition, but we must consider task
allocation. In order to find a right method to split rewards, we should consider its
usefulness as well as its computational cost. The Shapley’s value shares the ground
idea of COIN (notice the similarity between the meaning of the difference terms
of Equation (4.7) and Equation (6.19)) and it has some useful properties as stated
in Section 6.5.3. Unfortunately, this method suffers of possibly bad performance,
since in Equation (6.19) we compute the reward to be assigned to each agent by
examining all possible coalitions having a set N of n agents. In RL this means that
at each time step, a priori, any agent can change the resulting coalition structure
(thus coalitions) according to their policy (it obviously depends on the reward
received). All that means we must compute the Shapley’s value for each agent,
and this process repeats over time. Furthermore the Shapley’s value considers the
joining order of agents in each coalition, but in this kind of game this is useless
(besides its computational cost).
79
7.2 Utility Functions of the Game 7. Task Allocation via Coalition Formation
In order to avoid such undesirable features we propose marginal contribution,
a new payoff assignment method that shares the interesting characteristics of the
Shapley’s value, but avoids such a heavy computational burden.
Definition 7.6. Given a coalition S = {n1, n2, . . . , nk} we define marginal contri-
bution for agent type i the following function:
mi = v (S) − v (S−i) , (7.2)
where:
• v (S) is the characteristic function of the task allocation via coalition forma-
tion problem;
• S−i is the coalition S without one agent of type i;
• ni is the number of agents of type i.
Example 1. Let us take to have S = {5, 1}. We can compute the marginal
contribution for the first agent type (m1) and for the second one (m2) obtaining:
• m1 = v (5, 1) − v (4, 1);
• m2 = v (5, 1) − v (5, 0).
Recall the definition of marginal contribution in Equation (7.2): we can see the
marginal contribution shares the ground idea of COIN (in particular the clamping
function, that is CLσ(·) in Equation (4.7)), since it evaluates the goodness of
each agent belonging to a given coalition by computing the difference between
the characteristic function value obtained with full coalition and the characteristic
function value obtained without one agent of a certain type of that coalition.
It is interesting to note that the marginal contribution described in Definition
7.6 computes the payoff to be assigned to each coalition S in a symmetric way
with respect to each agent of the same type. The second term of Equation (7.2)
is computed on the clamped coalition S−i, and this is done by each agent of such
coalition. Hence we consider fair important all agents belonging to S, because in
this framework we are interested about each agent’s type rather than an agent
itself in order to satisfy a task as described in Definition 7.2.
80
7. Task Allocation via Coalition Formation 7.2 Utility Functions of the Game
7.2.2 Characteristic and Reward Functions
As stated in Section 6.5, the characteristic function of a game affects the game
itself, since it evaluates the goodness of a coalition, thus the structure of game.
In literature, many researchers have been focused on superadditive games (Section
6.5.1) where coalition structure generation is trivial. However, many games are
not superadditive, so we aim to have a characteristic function not superadditive:
this will avoid the grand coalition formation thus coalitions will obtain meaningful
rewards.
It is useful to note that, during the formalization of a problem, we must use a
feasible (and possibly robust) characteristic function to model a desired behavior. If
we introduce a strict characteristic function (i.e., a characteristic function allowing
one feasible coalition structure to have a large amount of tasks and agents) we
might not reach an optimal (or a near one) coalition structure. This problem
becomes more evident if we consider the state space, since in this case, besides
the characteristic function, we must deal with a huge state space in order to avoid
negative effects on the learning process.
We draw on Example 1 a useful feature of the characteristic function and its
association with the game. In this example we compute the difference between
two marginal contribution values, but it may happen that one of them might be
unfeasible. In such situation, the characteristic function should be robust and it
must return a value indicating that such coalition is unfeasible in order to make
possible to compute the marginal contribution, e.g.: in Example 1 if the coalition
S = {5, 0} is unfeasible, then we could impose v(S) = 0.
RL considers that each agent, after the execution of an action selected with
respect to its policy, receives a reward based on a reward function. Until now,
here we have considered only characteristic functions which evaluate only coalition
goodness, but we need to introduce a reward function such that it can parcel out
the coalition value given by the characteristic function.
We can define the reward value gi for each agent of type i in task allocation via
coalition formation games.
81
7.2 Utility Functions of the Game 7. Task Allocation via Coalition Formation
Definition 7.7. The reward value gi for each agent of type i is given by
gi = g(mi), (7.3)
where g(mi) is the reward function that evaluates the marginal contribution mi
computed according to Equation (7.2).
With these two functions we aim to induce an implicit coordination mechanism
among different types of agents belonging to a coalition. The marginal contribu-
tion (Equation (7.2)) provides a measure to coordinate different agents within a
coalition, while the reward function (Equation (7.7)) is used to coordinate different
coalitions in order to reach an optimal configuration (that is a equilibrium).
In these games we are dealing also with task allocation, so we need a world
utility function that evaluates the overall behavior of environment.
Definition 7.8. The world utility value G is given by
G = G (g (m1) , g (m2) , . . . , g (mk)) , (7.4)
where G(·) is the given world utility function of environment.
According to Definition 7.7 we can use the reward functions stated in Section 5.2
and in Section 5.3, so we can evaluate the goodness of an environment formalized
with RL.
As stated in Section 4.3.3, the COIN clamping function suggests to clamp the
elements of state ζ pertaining to agent η to a prefixed arbitrary value. In that
situation we are free to prefer one specific clamping value rather than another one
(i.e.: in the Bar Problem we can clamp to the null action or to a random one),
but anyway the clamping function acts on the state space. In this new typology
of game we are dealing with a huge state space, thus we proposed in Equation
(7.2) a clamping function acting on a given coalition S clamping to a fixed value
related to agents’ type in that coalition, that is it removes one agent of type i then
it evaluates the resulting coalition S−i with the characteristic function v(·) of the
game.
Let us focus on the WLU function of Equation (4.7): as said above, this utility
function is mainly based on the state, but here we must deal with a huge state
82
7. Task Allocation via Coalition Formation 7.3 Testbed Problem: Cooking Teams
space as mentioned in Section 7.1.1. As a consequence we must pay attention on
which clamping function we can use, in fact if we clamp to a different state and not
to the null state, each agent will receive a meaningful reward, but useless, in fact
it can’t reuse this precious information in the future in order to avoid or to choose
a particular action because it has not any state information (recall that as stated
in Section 7.1.1 we do not use any state information). Anyway, if we consider the
state space, this reward might be still useless, in fact the state space is huge and
each agent will have difficulties to visit another time the same state (recall that
such a visit depends on the joint action and not only to the agent’s action).
This particular feature does not affect other reward functions (selfish utility
function, team game utility function and uniform division reward), because with
such functions we do not compute any difference involving whatsover clamping
value, thus involving any state information. This may be useful with this kind of
games if we do no consider the state space. However this feature might be affected
by whether the state space exists. For example, following the definition of the TG
utility function (see Section 4.2.3 and Section 5.2.2), in general we must compute
the sum over all tasks of the reward function evaluated in such tasks. This could
lead the learning process towards to non optimal values, since the reward value
assigned to each agent belonging to a coalition can be influenced by other agents
and coalitions related to other tasks.
7.3 Testbed Problem: Cooking Teams
Here, we introduce the cooks and helpers problem (see [25]), where we have 2 types
of players k = {cook, helper} and 3 different cooking teams:
• 1 cook and 2 helpers cook one cake;
• 4 cooks alone cook one cake (too many cooks encounter difficulties reaching
an agreement);
• 1 helper alone cooks one cookie.
83
7.3 Testbed Problem: Cooking Teams 7. Task Allocation via Coalition Formation
Each cake is worth 10 and each cookie 1. Moreover we have some constraints about
the number of possible kitchens available, in fact we have U = 7 different kitchens
(tasks) and each one can be selected by any coalition S = {fc, fh} of fc cooks and
fh helpers.
Let us suppose that Su describes the coalitions cooking in kitchen u (u ∈ U =
{1, . . . , 7}). We define the following characteristic function (n ∈ N):
v(S) =
n · 10 if we have a coalition S = {n, 2 · n} ,
n · 10 if we have a coalition S = {4 · n, 0} ,
n if we have a coalition S = {0, n} ,
0 elsewhere.
(7.5)
The world utility is defined as:
G (Q) =
U∑
u=1
v(Su) (7.6)
where Su is the coalition undertaking the u-th task.
The main goal of this game is to maximize Equation (7.6) with respect to the
U = 7 tasks, without communication among agents acting in this environment.
7.3.1 Configurations
In this problem we will analyze five different possible cases in order to find an
admissible coalition structure CS∗ such that maximizes G (Q).
Case 1 : the population Q = {fc, fh} consists of fc > 0 cooks and fh = 0 helpers.
If fc ≤ 4 the core is nonempty (recall that the core focuses on the stability of
an admissible coalition, and in this case any solution is stable because there
are not leftover cooks creating instability). If fc > 4, the core is nonempty if
and only if fc is an integer multiple of 4 (so they cook fc mod 4 cakes). In any
other case, in the population there are leftover cooks who create instability.
Case 2 : the population Q = {fc, fh} consists of fh > 0 helpers and fc = 0 cooks,
thus the core is always nonempty and it assigns 1 to each helper.
84
7. Task Allocation via Coalition Formation 7.3 Testbed Problem: Cooking Teams
Case 3 : the population Q = {fc, fh} is composed by r1 + r2 coalitions (both of
them greater than 0), where r1 is the number of coalitions (1, 2) (1 cook, 2
helpers) and r2 is the number of coalitions (0, 1). In this case there are many
helpers relative to the number required for teams with composition (1, 2).
Competition among helpers exists and it keeps the price of a helper down.
The core assigns 1 to each helper and 8 to each cook.
Case 4 : the population Q = {fc, fh} is composed by r1 + r2 coalitions, where
r1 is the number of coalitions (1, 2) and r2 is the number of coalitions (4, 0)
(both of them greater than 0). This is the opposite of the previous case:
here we have many cooks with respect to the number of helpers. Now the
competition exists among cooks in order to be in a coalition with 2 helpers.
In this case the core assigns 154
to each helper and only 52
to each cook.
Case 5 : the population Q = {fc, fh} is composed by r1 > 0 coalitions of type
(1, 2). Here the core contains a continuum of points and the extremes are
described by the core of Case 3 and Case 4.
Let us consider the characteristic function of Equation (7.5): at first sight
it is a strict function that rates only known coalitions. Obviously, if we change
the characteristic function, the problem is changed too. For example, with the
characteristic function of Equation (7.5) we do not formalize the fact that there
may exist only one oven. In order to allow the cook of only one cake or only one
cookie we must impose n = 1 in Equation (7.5): i.e., both coalitions S1 = {1, 2} and
S2 = {4, 8} can cook only one cake. Moreover, we can formalize a moderator that
organizes agents belonging to a coalition is such a way to maximize the welfare, e.g.:
if we have the coalition S = {2, 5}, players can cook two cakes and one cookie,
while with Equation (7.5) they can cook anything. Anyway, with these simple
considerations we show the importance to have a good and a robust characteristic
function with respect to the problem we aim to formalize.
To test the goodness of the characteristic function proposed in Equation (7.5)
we propose four different configurations Q = {cooks, helpers} according to the five
different cases discussed above:
85
7.3 Testbed Problem: Cooking Teams 7. Task Allocation via Coalition Formation
Bar-1 : we have 5 cooks and 20 helpers;
Bar-2 : we have 11 cooks and 14 helpers;
Bar-3 : we have 21 cooks and 14 helpers;
Bar-4 : we have 21 cooks and 5 helpers.
Using Equation (7.5) we can obtain the optimal coalition structures described
in Table 7.1. Obviously, according to Equation (7.5), it is considered optimal any
admissible permutation of the coalition structures of Table 7.1, e.g.: an admissible
optimal coalition structure for the Bar-2 is reported in Table 7.2.
Tasks Bar-1 Bar-2 Bar-3 Bar-4
1 (Monday) S = {1, 2} S = {7, 14} S = {7, 14} S = {12, 0}payoff = 10 payoff = 70 payoff = 70 payoff = 30
2 (Tuesday) S = {1, 2} S = {4, 0} S = {4, 0} S = {1, 2}payoff = 10 payoff = 10 payoff = 10 payoff = 10
3 (Wednesday) S = {1, 2} S = {0, 0} S = {4, 0} S = {0, 0}payoff = 10 payoff = 0 payoff = 10 payoff = 0
4 (Thursday) S = {1, 2} S = {0, 0} S = {4, 0} S = {0, 0}payoff = 10 payoff = 0 payoff = 10 payoff = 0
5 (Friday) S = {1, 2} S = {0, 0} S = {2, 0} S = {4, 0}payoff = 10 payoff = 0 payoff = 0 payoff = 10
6 (Saturday) S = {0, 10} S = {0, 0} S = {0, 0} S = {4, 0}payoff = 10 payoff = 0 payoff = 0 payoff = 10
7 (Sunday) S = {0, 0} S = {0, 0} S = {0, 0} S = {0, 3}payoff = 0 payoff = 0 payoff = 0 payoff = 3
World utility value 60 80 100 63
Table 7.1: Optimal coalition structures for the four bar problems
86
7. Task Allocation via Coalition Formation 7.3 Testbed Problem: Cooking Teams
Tasks Coalitions Payoffs
1 (Monday) S = {1, 2} 10
2 (Tuesday) S = {1, 2} 10
3 (Wednesday) S = {1, 2} 10
4 (Thursday) S = {1, 2} 10
5 (Friday) S = {2, 4} 20
6 (Saturday) S = {1, 2} 10
7 (Sunday) S = {4, 0} 10
Table 7.2: Another admissible coalition structure for the Bar-2
7.3.2 Reward Functions
In order to worth different coalitions, we evaluate them with the marginal contri-
bution proposed in Section 7.2.1. Furthermore, in order to split the payoff obtained
with the marginal contribution among agents of a coalition, we propose four dif-
ferent reward functions.
Selfish utility function Each agent η of type kη = i (where
i ∈ K = {cook, helper}) obtains a payoff given by
gi = mi (7.7)
where mi is the marginal contribution computed according to Equation (7.2).
COIN utility function Each agent η of type kη = i (where
i ∈ K = {cook, helper}) obtains a payoff given by
gc = fc ·[v (fc, fh) − v (fc − 1, fh)
]+ fh ·
[v (fc, fh) − v (fc, fh − 1)
]−
[(f r
c + 1) ·(v (f r
c + 1, f rh) − v (f r
c , f rh))+
f rh ·(v (f r
c + 1, f rh) − v (f r
c + 1, f rh − 1)
)](7.8)
gh = fc ·[v (fc, fh) − v (fc − 1, fh)
]+ fh ·
[v (fc, fh) − v (fc, fh − 1)
]−
[f r
c ·(v (f r
c , f rh + 1) − v (f r
c − 1, f rh + 1)
)+
(f r
h + 1)·(v (f r
c , f rh + 1) − v (f r
c , f rh))]
(7.9)
87
7.3 Testbed Problem: Cooking Teams 7. Task Allocation via Coalition Formation
where as usual v(·, ·) is the characteristic function, fc and fh are the number
of cooks and helpers in the day attended by agent η, while f rc and f r
h are the
number of cooks and helpers in a different random day.
Team Game utility function Each agent η of type kη = i (where i ∈ K =
{cook, helper}) obtains a payoff given by
gi =1
n·
D∑
d=1
v(fdc , fd
h) (7.10)
where as in the previous case f dk is the number of agents of type k on day d
(obviously D is the number of days per week) and n is the number of agents.
Uniform Division utility function Each agent η of type kη = i (where i ∈K = {cook, helper}) obtains a payoff given by
gi =1
fc + fh
·∑
i∈k
fi · mi (7.11)
where fi is the number of agents of type i in the day attended by agent η
and mi is the agent’s marginal contribution of type i.
7.3.3 State Space
We will analyze this problem with and without any state information in order to
see how agents behave in this environment. If we use the state space, it will be
huge, so we expect to have worse performance than without any state information.
As a consequence, we introduce the concept of difficulty of a problem that is
related to the number of optimal states in the joint state space. In the four bar
configurations described above we formalized this idea. Bar-1 looks “easiest” than
Bar-4: the former has many helpers with respect to the number of cooks (thus each
cook tends to cook one cake with two helpers and they will not find any difficulty),
while in the latter all cooks tend to take two helpers in order to obtain better payoffs
(helpers are shared resources). In these situations we have a joint state space with
a different amount of optimal states and they depend on the characteristic function
used. The proposed characteristic function rates only coalitions with a predefined
number of agents as described in Equation (7.5), thus the joint state space will be
88
7. Task Allocation via Coalition Formation 7.3 Testbed Problem: Cooking Teams
formed only by single states where an admissible coalition is. For example, let us
analyze the Bar-4: in this case the characteristic function proposed identifies only
few optimal states, so we have a huge state space with a very small amount of good
states.
An easy way to deal with this a priori huge state space is to reduce it using an
appropriate state space representation. Following the idea proposed in Section 5.3,
we analyze this problem with a reduced state space. Each state s represents the
number of cooks and helpers attending the bar in a day chosen by the agent that is
evaluating its policy. Hence, the state space for each bar is drastically reduced as
follow (enclosed in parenthesis is reported the original state space size computed
according to Equation (7.1)):
Bar-1 : the new state space has 126 states (2125 · 125 ≃ 5.3169 · 1039).
Bar-2 : the new state space has 180 states (2179 · 179 ≃ 1.3716 · 1056).
Bar-3 : the new state space has 330 states (2329 · 329 ≃ 3.5980 · 10101).
Bar-4 : the new state space has 132 states (2131 · 131 ≃ 3.5662 · 1041).
As described above, the state space size is heavily decreased, but we must bear in
mind that the goodness of each state depends on the number of cooks and helpers
attending the bar. Hence a state can be rated in a different way by cooks and
helpers, thus if a cook rates as bad a state s, a helper can rate it as good. As
a consequence, they do not agree on the evaluation of that state, so cooks and
helpers tend to visit the same set of states and the world utility will assume lower
values.
In Section 8.3 we propose the result obtained with these different bar configu-
rations, with and without state in order to analyze the overall coalitions’ behavior.
89
Chapter 8Results
The only real valuable thing is intuition. The intellect has
