Sisteme de programeSisteme de programepentru timp realpentru timp real

Universitatea “Politehnica” din Bucuresti

2003-2004

Adina Magda Floreahttp://turing.cs.pub.ro/sptr_2004

Curs Nr. 7 si 8

Genetic Algorithms• Introduction• Basic schema• GA Functioning• An example• Selection• Recombination• Mutation• Using GAs to solve the TSP• Parallel implementations of Gas• Co-evolution

2

1. Introduction

Genetic Algorithms (GAs) are adaptive heuristic search algorithms

Basic concept = simulate processes in natural systems necessary for evolution - principles first laid down by Charles Darwin of survival of the fittest.

GAs provide an alternative method to solving optimization problems (but not only)

GAs are useful and efficient when:• The search space is large, complex or poorly understood. • Domain knowledge is scarce or expert knowledge is difficult

to encode to narrow the search space. • No mathematical analysis is available.• Traditional search methods fail

3

Introduction - contMain schools of evolutionary algorithms: genetic algorithms, mainly developed in the USA by J. H.

Holland (1975) evolutionary strategies, developed in Germany by I. Rechenberg

and H.-P. Schwefel (1975-1980) evolutionary programming (1960-1980)

There are many ways to view genetic algorithms.• Optimisation tasks: numerical optimisation, and

combinatorial optimisation problems such as traveling salesman problem (TSP), circuit design, job shop scheduling and video & sound quality optimisation.

• Automatic Programming: evolve computer programs for specific tasks or design other computational structures, for example cellular automata.

4

Introduction - cont• Machine and robot learning: classification, prediction,

protein structure prediction, evolve rules for classifier systems or symbolic production systems, and to design and control robots.

• Economic models: model processes of innovation, the development of bidding strategies, and the emergence of economic markets.

• Ecological models: biological arms races, host-parasite co-evolutions, symbiosis and resource flow in ecologies.

• Population genetics models: study questions in population genetics, such as “under what conditions will a gene for recombination be evolutionarily viable?”

• Models of social systems: study evolutionary aspects of social systems, such as the evolution of cooperation, the evolution of communication, and trail-following behaviour in ants.

5

Introduction - cont• GAs operate on a population of potential solutions applying the

principle of survival of the fittest to produce better and better approximations to a solution.

• At each generation, a new set of approximations is created by the process of selecting individuals according to their level of fitness in the problem domain and breeding them together using operators borrowed from natural genetics.

• This process leads to the evolution of populations of individuals that are better suited to their environment than the individuals that they were created from, just as in natural adaptation.

• GAs model natural processes, such as selection, recombination, mutation, migration, locality and neighbourhood.

• GAs work on populations of individuals instead of single solutions.

• In this way the search is performed in a parallel manner.

6

2. Basic schema

7

Generateinitialpopulationof individuals(genes)

start

Problemrepresentation

Evaluateobjectivefunction

Are optimizationcriteria met?

Selection

Crossover/Mate

Mutation

yes

bestindividuals

resultGeneratenewpopulation

no

Fitness function

generations

offsprings

2.1 Problem solving using GAs

8

Problem solving using GAs - cont

9

Better results can be obtained by introducing many populations, called subpopulations.

Every subpopulation evolves for a few generations isolated (like the single population evolutionary algorithm).

Then one or more individuals are exchanged between the subpopulations.

The Multipopulation GAs models the evolution of a species in a way more similar to nature than the single population evolutionary algorithm.

3. GA Functioning

10

Initial Population

• A gene (individual) is a string of bits; some other representations can be used

• The initial population of genes (bitstrings) is usually created randomly.

• The length of the individual depends on the problem to be solved

Selection (1)

• Selection means to extract a subset of genes from an existing population, according to any definition of quality.

• Selection determines, which individuals are chosen for mating (recombination) and how many offsprings each selected individual produces.

3.1 Selection

11

The first step is fitness assignment by: • proportional fitness assignment or • rank-based fitness assignment• Actual selection: parents are selected according to

their fitness by means of one of the following algorithms:

• roulette-wheel selection• stochastic universal sampling• local selection• truncation selection• tournament selection [BT95].

Selection - cont

12

Proportional fitness assignment Consider the population being rated, that means: each gene

has a related fitness. The higher the value of the fitness, the better.

The mean-fitness of the population is calculated. Every individual will be copied as often to the new

population, the better it fitness is, compared to the average fitness. E.g.: the average fitness is 5.76, the fitness of one individuum is 20.21. This individuum will be copied 3 times. All genes with a fitness at the average and below will be removed.

Following this steps, in many cases the new population will be a little smaller than the old one. So the new population will be filled up with randomly chosen individuals from the old population to the size of the old one.

3.2 Reinsertion

13

If less offsprings are produced than the size of the original population, the offsprings have to be reinserted into the old population.

Similarly, if not all offsprings are to be used at each generation or if more offsprings are generated than needed then a reinsertion scheme must be used to determine which individuals should be inserted into the new population.

The selection algorithm used determines the reinsertion scheme:

• global reinsertion for all population based on the selection algorithm (roulette-wheel selection, stochastic universal sampling, truncation selection),

• local reinsertion for local selection.

3.3 Crossover/Recombination

14

• Recombination produces new individuals in combining the information contained in the parents (parents - mating population).

• Several recombination schemes exist

• The b_nX crossover - random mating with a defined probability

This type is described most often as the parallel to the Crossing Over in genetics

1. PM percent of the individuals of the new population will be

selected randomly and mated in pairs.

2. A crossover point will be chosen for each pair

3. The information after the crossover point will be exchanged between the two individuals of each pair.

Crossover

15

3.4 Mutation

16

• After recombination every offspring undergoes mutation.

• Offspring variables are mutated by small perturbations (size of the mutation step), with low probability.

• The representation of the variables determines the used algorithm.

• Mutation - add some effect of exploration to the algorithm.

• One simple mutation scheme:– Each bit in every gene has a defined probability P to

get inverted

Mutation

17

The effect of mutation is in some way antagonist to selection

18

4 An example

19

Compute the maximum of a function• f(x1, x2, ... xn)

• The task is to calculate x1, x2, ... xn for which

the function is maximum• Using GA´s can be a good solution.

Example- Representation

20

1. Scale the variables to integer variables by multiplying them with 10n, where n is the the desired precision New Variable = integer(Old Variable ×10 n)

2. Represent the variables in binary form 3. Connect all binary representations of these variables to form

an individual of the population 4. If the sign is necessary two ways are possible: • Add the lowest allowed value to each variable and transform

the variables to positive ones • Reserve one bit per variable for the sign

Usually not the binary representation is used, but the Gray-code representation

Example- Representation

21

Example- Computation

22

1. Make random initial population 2. Perform selection3. Perform crossover4. Perform mutation5. Transform the bitstring of each individual back to

the model-variables: x1, x2, ... xn 6. Test the quality of fit for each individual, e.g., the

f(x1, x2, ... xn)7. Check if the quality of the best individual is good

enough (not significantly improved over last steps)8. If yes then stop iterations else go to 2

5. Selection

The first step is fitness assignment. Each individual in the selection pool receives a

reproduction probability depending on the own objective value and the objective value of all other individuals in the selection pool.

This fitness is used for the actual selection step afterwards.

23

Terms

• selective pressure: probability of the best individual being selected compared to the average probability of selection of all individuals

• bias: absolute difference between an individual's normalized fitness and its expected probability of reproduction

• spread: range of possible values for the number of offspring of an individual

• loss of diversity: proportion of individuals of a population that is not selected during the selection phase

• selection intensity: expected average fitness value of the population after applying a selection method to the normalized Gaussian distribution

• selection variance: expected variance of the fitness distribution of the population after applying a selection method to the normalized Gaussian distribution

24

5.1 Rank-based fitness assignment The population is sorted according to the objective values (fitness). The fitness assigned to each individual depends only on its position

in the individuals rank and not on the actual objective value. Rank-based fitness assignment overcomes the scaling problems of

the proportional fitness assignment = Stagnation in the case where the selective pressure is too small or premature convergence where selection has caused the search to narrow down too quickly.

The reproductive range is limited, so that no individuals generate an excessive number of offsprings.

Ranking introduces a uniform scaling across the population and provides a simple and effective way of controlling selective pressure.

Rank-based fitness assignment behaves in a more robust manner than proportional fitness assignment.

25

Rank-based fitness assignment - cont

• Nind - the number of individuals in the population• Pos - the position of an individual in this population

(least fit individual has Pos=1, the fittest individual Pos=Nind)

• SP - the selective pressure.• The fitness value for an individual is calculated as:

Linear ranking

Fitness(Pos) = 2 - SP + 2*(SP - 1)*(Pos - 1) / (Nind - 1) • Linear ranking allows values of SP in [1.0, 2.0].

26

Rank-based fitness assignment - cont

Non-linear ranking: Fitness(Pos) =

Nind*X (Pos - 1) / i = 1,Nind (X(i - 1))• X is computed as the root of the polynomial:

0 = (SP - 1)*X (Nind - 1) + SP*X (Nind - 2) + ... + SP*X + SP • Non-linear ranking allows values of SP in

[1.0, Nind - 2.0]• The use of non-linear ranking permits higher selective

pressures than the linear ranking method.• The probability of each individual being selected for

mating is its fitness normalized by the total fitness of the population.

27

Rank-based fitness assignment - cont

28

Fitness assignment for linear and non-linear ranking

Rank-based fitness assignment - cont

Properties of linear ranking Selection intensity: SelIntRank(SP) = (SP-1)*(1/sqrt()).

Loss of diversity: LossDivRank(SP) = (SP-1)/4.

Selection variance: SelVarRank(SP) = 1-((SP-1)2/ ) = 1-SelIntRank(SP)2.

29

5.2 Roulette wheel selection

The simplest selection scheme is roulette-wheel selection, also called stochastic sampling with replacement

11 individuals, linear ranking and selective pressure of 2

sample of 6 random numbers (uniformly distributed between 0.0 and 1.0):

• 0.81, 0.32, 0.96, 0.01, 0.65, 0.42.

30

Numberof individual

1 2 3 4 5 6 7 8 9 10 11

Fitnessvalue

2.0 1.8 1.6 1.4 1.2 1.0 0.8 0.6 0.4 0.2 0.0

Selectionprobability

0.18 0.16 0.15 0.13 0.11 0.09 0.07 0.06 0.03 0.02 0.0

Roulette wheel selection - cont

• After selection the mating population consists of the individuals:

1, 2, 3, 5, 6, 9.

• The roulette-wheel selection algorithm provides a zero bias but does not guarantee minimum spread

31

5.3 Stochastic universal sampling

• Equally spaced pointers are placed over the line as many as there are individuals to be selected.

• NPointer - the number of individuals to be selected

• The distance between the pointers are 1/Npointer

• The position of the first pointer is given by a randomly generated number in the range [0, 1/NPointer].

• For 6 individuals to be selected, the distance between the pointers is 1/6=0.167.

• sample of 1 random number in the range [0, 0.167]: 0.1.

32

Stochastic universal sampling - cont

• After selection the mating population consists of the individuals:

1, 2, 3, 4, 6, 8.

• Stochastic universal sampling ensures a selection of offspring which is closer to what is deserved then roulette wheel selection

33

5.4 Local selection

• Every individual resides inside a constrained environment called the local neighbourhood

• The neighbourhood is defined by the structure in which the population is distributed.

• The neighbourhood can be seen as the group of potential mating partners.

• Linear neighbourhood: full and half ring

34

Local selection - contThe structure of the neighbourhood can be:

• Linear: full ring, half ring

• Two-dimensional:

– full cross, half cross

– full star, half star

• Three-dimensional and more complex with any combination of the above

structures.

35

Local selection - cont• The distance between possible neighbours together with the structure

determines the size of the neighbourhood.

The first step is the selection of the first half of the mating population uniform at random (or using one of the other mentioned selection algorithms, for example, stochastic universal sampling or truncation selection).

Then a local neighbourhood is defined for every selected individual. Inside this neighbourhood the mating partner is selected (best, fitness proportional, or uniform at random).

36

Local selection - cont

Between individuals of a population an 'isolation by distance' exists.

The smaller the neighbourhood, the bigger the isolation distance.

However, because of overlapping neighbourhoods, propagation of new variants takes place.

This assures the exchange of information between all individuals.

The size of the neighbourhood determines the speed of propagation of information between the individuals of a population, thus deciding between rapid propagation or maintenance of a high diversity/variability in the population.

A higher variability is often desired, thus preventing problems such as premature convergence to a local minimum.

37

5.5 Tournament selection

A number Tour of individuals is chosen randomly from the population and the best individual from this group is selected as parent.

This process is repeated as often as individuals to choose. The parameter for tournament selection is the tournament size

Tour. Tour takes values ranging from 2 - Nind (number of individuals

in population). Relation between tournament size and selection intensity

38

tournament size 1 2 3 5 10 30

selection intensity 0 0.56 0.85 1.15 1.53 2.04

Tournament selection - cont Selection intensity:

SelIntTour(Tour) = sqrt(2*(log(Tour)-log(sqrt(4.14*log(Tour)))))

Loss of diversity:

LossDivTour(Tour) = Tour -(1/(Tour-1)) - Tour -(Tour/(Tour-1))

(About 50% of the population are lost at tournament size Tour=5). Selection variance:

SelVarTour(Tour) = 1- 0.096*log(1+7.11*(Tour-1)), SelVarTour(2) = 1-1/

39

6. Crossover / recombination

• Recombination produces new individuals in combining the information contained in the parents

40

6.1 Binary valued recombination (crossover)

One crossover position k is selected uniformly at random and the corrsponding parts (or variables) are exchanged between the individuals about this point two new offsprings are produced.

41

Binary valued recombination - cont

Consider the following two individuals with 11 bits each: individual 1: 0 1 1 1 0 0 1 1 0 1 0 individual 2: 1 0 1 0 1 1 0 0 1 0 1

crossover position is 5 After crossover the new individuals are created: offspring 1: 0 1 1 1 0| 1 0 0 1 0 1 offspring 2: 1 0 1 0 1| 0 1 1 0 1 0

42

2.2 Multi-point crossover

m crossover positions ki

individual 1: 0 1 1 1 0 0 1 1 0 1 0 individual 2: 1 0 1 0 1 1 0 0 1 0 1 cross pos. (m=3) 2 6 10 offspring 1: 0 1| 1 0 1 1| 0 1 1 1| 1 offspring 2: 1 0| 1 1 0 0| 0 0 1 0| 0

43

2.3 Uniform crossover

Uniform crossover generalizes single and multi-point crossover to make every locus a potential crossover point.

A crossover mask, the same length as the individual structure, is created at random and the parity of the bits in the mask indicate which parent will supply the offspring with which bits.

individual 1: 0 1 1 1 0 0 1 1 0 1 0 individual 2: 1 0 1 0 1 1 0 0 1 0 1

mask 1: 0 1 1 0 0 0 1 1 0 1 0mask 2: 1 0 0 1 1 1 0 0 1 0 1 (inverse of

mask 1) offspring 1: 1 1 1 0 1 1 1 1 1 1 1 offspring 2: 0 0 1 1 0 0 0 0 0 0 0 Spears and De Jong (1991) have poposed that

uniform crossover may be parameterised by applying a probability to the swapping of bits.

44

2.4 Real valued recombination

2.4.1 Discrete recombination Performs an exchange of variable values between

the individuals. individual 1: 12 25 5 individual 2: 123 4 34 For each variable the parent who contributes its

variable to the offspring is chosen randomly with equal probability.

sample 1: 2 2 1 sample 2: 1 2 1 After recombination the new individuals are created: offspring 1: 123 4 5 offspring 2: 12 4 5

45

Real valued recombination - cont

Discrete recombination - cont Possible positions of the offspring after discrete

recombination

Discrete recombination can be used with any kind of variables (binary, real or symbols).

46

Real valued recombination - cont

2.4.2 Intermediate recombination A method only applicable to real variables (and not

binary variables). The variable values of the offsprings are chosen

somewhere around and between the variable values of the parents.

Offspring are produced according to the rule: offspring = parent 1 + Alpha (parent 2 - parent

1) where Alpha is a scaling factor chosen uniformly at random over an interval [-d, 1 + d].

In intermediate recombination d = 0, for extended intermediate recombination d > 0.

A good choice is d = 0.25. Each variable in the offspring is the result of combining

the variables according to the above expression with a new Alpha chosen for each variable. 47

Real valued recombination - cont

Intermediate recombination - cont Area for variable value of offspring

compared to parents in intermediate recombination

48

Real valued recombination - cont

Intermediate recombination - cont individual 1: 12 25 5 individual 2: 123 4 34Alpha are: sample 1: 0.5 1.1 -0.1 sample 2: 0.1 0.8 0.5 The new individuals are calculated

(offspring = parent 1 + Alpha (parent 2 -

parent 1) offspring 1: 67.5 1.9 2.1 offspring 2: 23.1 8.2 19.5

49

Real valued recombination - cont

Intermediate recombination is capable of producing any point within a hypercube slightly larger than that defined by the parents

Possible area of the offspring after intermediate recombination

50

Real valued recombination - cont

2.4.3 Line recombination Line recombination is similar to intermediate recombination,

except that only one value of Alpha for all variables is used. individual 1: 12 25 5 individual 2: 123 4 34Alpha are: sample 1: 0.5 sample 2: 0.1The new individuals are calculated as: offspring 1: 67.5 14.5 19.5

offspring 2: 23.1 22.9 7.9

51

Real valued recombination - cont

Line recombination - cont Line recombination can generate any point on the line defined

by the parents

52

7. Mutation

• After recombination offsprings undergo mutation.

• Offspring bits or variables are mutated by inversion or the addition of small random values (size of the mutation step), with low probability.

• The probability of mutating a variable is set to be inversely proportional to the number of variables (dimensions).

• The more dimensions one individual has as smaller is the mutation probability.

53

7.1 Binary mutation

For binary valued individuals mutation means flipping of variable values.

For every individual the variable value to change is chosen uniform at random.

A binary mutation for an individual with 11 variables, variable 4 is mutated.

Individual before and after binary mutation

54

before mutation 0 1 1 1 0 0 1 1 0 1 0

after mutation 0 1 1 0 0 0 1 1 0 1 0

7.2 Real valued mutation

Effectof mutation

The size of the mutation step is usually difficult to choose. The optimal step size depends on the problem considered and may even vary during the optimization process. Small steps are often successful, but sometimes bigger steps are quicker.

A mutation operator (the mutation operator of the Breeder Genetic Algorithm) : mutated variable = variable ± range·delta (+ or - with equal probability) range = 0.5*domain of variabledelta = sum(a(i) 2-i), a(i) = 1 with probability 1/m, else a(i) = 0; m = 20. 55

Real valued mutation - cont

This mutation algorithm is able to generate most points in the hypercube defined by the variables of the individual and range of the mutation.

With m=20, the mutation algorithm is able to locate the optimum up to a precision of (range*2-19)

Probability and size of mutation steps (compared to range)

56

8. Using GAs to solve the TSP

57

8.1 Problem representation

Evaluation function• The evaluation function for the N cities is the sum of the

Euclidean distances

Representation• a path representation where the cities are listed in the order in

which they are visited

58

N

iiiii yyxxFitness

1

21

21 )()(

3 0 1 4 2 5

0 5 1 4 2 3

5 1 0 3 4 2

8.2 TSP Genetic operators

Crossover• Traditional crossover is not suitable for TSPs

• Before crossover1 2 3 4 5 0 (parent 1)2 0 5 3 1 4 (parent 2)

• After crossover1 2 3 3 1 4 (child 1)2 0 5 4 5 0 (child 2)

• Greedy Crossover which was invented by Grefenstette in 1985

59

TSP Genetic operators - contTwo parents 1 2 3 4 5 0 and 4 1 3 2 0 5 • To generate a child using the second parent as the template, we select city 4

(the first city of our template) as the first city of the child. 4 x x x x x

• Then we find the edges after city 4 in both parents: (4, 5) and (4, 1) and compare the distance of these two edges. If the distance between city 4 and city 1 is shorter, we select city 1 as the next city of child 2. 4 1 x x x x

• Then we find the edges after city 1 (1, 2) and (1, 3). If the distance between city 1 and city 2 is shorter, we select city 2 as the next city. 4 1 2 x x x

• Then we find the edges after city 2 (2, 3) and (2, 0). If the distance between city 2 and city 0 is shorter, we select city 0. 4 1 2 0 x x

• The edges after city 0 are (0, 1) and (0, 5). Since city 1 already appears in child 2, we select city 5 as the next city. 4 1 2 0 5 x

• The edges after city 5 are (5, 0) and (5, 4), but city 4 and city 0 both appear in child 2. We select a non-selected city, which is city 3. and thus produce a legal child 4 1 2 0 5 3

We can use the same method to generate the other child. 1 2 0 5 4 3

60

TSP Genetic operators - cont

Mutation• We can not use the traditional mutation operator. For example if

we have a legal tour before mutation 1 2 3 4 5 0, assuming the mutation site is 3, we randomly change 3 to 5 and generate a new tour 1 2 5 4 5 0

• Instead, we randomly select two bits in one chromosome and swap their values.

• Swap mutation: 1 2 3 4 5 0 1 5 3 4 2 0

Selection• When using traditional roulette wheel selection, the best

individual has the highest probability of survival but does not necessarily survive.

• We use CHC selection to guarantee that the best individual will always survive in the next generation (Eshelman 1991).

61

8.3 TSP Behavior• In CHC selection if the population size is N, we generate N

children by using roulette wheel selection, then combine the N parents with the N children, sort these 2N individuals according to their fitness value and choose the best N individuals to propagate to the next generation.

62

Comparison ofRoulette and CHC selection

With CHC selection the populationconverges quickly comparedto roulette wheel selectionand the performance is also better.

TSP Behavior - cont• But fast convergence may mean less exploration.

• To prevent convergence to a local optimum, when the population has converged we save the best of the individuals and re-initialize the rest of the population randomly.

• We call this modified CHC selection R-CHC.

63

Comparison of CHC selectionwith or without re-initialization

9 Parallel implementations of GAs

64

• Migration

• Global model - worker/farmer

• Diffusion model

9.1 Migration

• The migration model divides the population in multiple subpopulations.

• These subpopulations evolve independently from each other for a certain number of generations (isolation time).

• After the isolation time a number of individuals is distributed between the subpopulations (migration).

• The number of exchanged individuals (migration rate), the selection method of the individuals for migration and the scheme of migration determines how much genetic diversity can occur in the subpopulations and the exchange of information between subpopulations.

65

Migration - cont

• The parallel implementation of the migration model showed not only a speedup in computation time, but it also needed less objective function evaluations when compared to a single population algorithm.

• So, even for a single processor computer, implementing the parallel algorithm in a serial manner (pseudo-parallel) delivers better results (the algorithm finds the global optimum more often or with less function evaluations).

66

Migration - cont

• The selection of the individuals for migration can take place: – uniform at random (pick individuals for migration in a random

manner),

– fitness-based (select the best individuals for migration).

• Many possibilities exist for the structure of the migration of individuals between subpopulations. For example, migration may take place: – between all subpopulations (complete net topology -

unrestricted)

– in a ring topology

– in a neighbourhood topology

67

Migration - cont

Unrestricted migration topology (Complete net topology)

68

Migration - cont

Scheme for migration of individuals between subpopulations

69

Migration - cont

Ring migration topology

70

Neighbourhood migration topology (2-D

implementation)

9.2 Global model - worker/farmer

• The global model employs the inherent parallelism of genetic algorithms (population of individuals). The Worker/Farmer algorithm is a possible implementation.

71

9.3 Diffusion model• The diffusion model handles every individual separately and selects the mating partner in a local neighbourhood similar to local selection. Thus, a diffusion of information through the population takes place. During the search virtual islands will evolve.

72

9.2 Global model - worker/farmer

• The selection of the individuals for migration can take place: – uniform at random (pick individuals for migration in a random

manner),

– fitness-based (select the best individuals for migration).

• Many possibilities exist for the structure of the migration of individuals between subpopulations. For example, migration may take place: – between all subpopulations (complete net topology -

unrestricted)

– in a ring topology

– in a neighbourhood topology

73

A multi-agent system with competitive agents that tries to solve the “tragedy of commons” problem: the rational exploitation of natural

renewable resources by self-interested agents (human or artificial) that have to achieve a certain degree of cooperation in order to avoid the overexploitation of resources.

1010 CoevolutionCoevolution

74

“Tragedy of commons” instances: use of common pastures by sheep farmers fish and whales catching environmental pollution urban transportation problems preservation of tropical forests common computer resources used by different processors

The problem instance considered:The problem instance considered: Fishing companies (Ci) owing several ships (NSi) and having the

possibility to fish in several fishing banks (Rp), during different seasons (Tj).

Each company aims at collecting maximum assets expressed by the amount of money they earn (and the number of ships). The money is generated by fish catching at fish banks.

A fish bank is a renewable resource and will not be regenerated if the companies will be catching too much fish, leading to the exhaustion of the resource, ecological unbalance, and profit loss.

10.110.1 Problem DescriptionProblem Description

75

Company agents (CompA) A company is represented by a cognitive agent following the BDIG

model A CompA may be ecologically oriented or profit oriented (fix the

goals) A CompA have a planning component to develop plans for sending

ships to fish banks (in shore or deep sea), under the uncertainty induced by the fishing actions of the other company agents.

Environment agent (EnvA) The attributes of the environment The evolution of the environment status Each company in the system has to register at the EnvA (EnvA is

also a facilitator in the MAS)

10.210.2 The MAS modelThe MAS model

76

Request

Request

Accept

ModifyRequest

Inquiry

Inform

Environment Agent Companies Fish banks Environment state

Company 1 Agent

Self representation World representation

Company 2 Agent

Company 3 Agent

Declare Result

Multi-agent system to model the “tragedy of commons”

Beliefs, Goals, Plans,Negotiation Strategy,

Ships, Profile

Fish banks, Seasons,Regeneration limit/rate,Names of other agents,

Beliefs about other agents

77

Agent interactionsAgent interactions

Communication between CopmAs and EnvA is aimed at interacting with and acquiring knowledge on the environment. Declare(Fishbank, NoShips, Fc, Time) Result(Fishbank, Amount, Time) Inquiry(Fishbank, Time) Inform(Fishbank, Amount, Time, List)

Communication between one CompA and another CompA is dedicated to negotiate the fishing amount that will be caught at a particular time moment by a company agent. Request(Fishbank , Amount, Time) Accept(Fishbank, Amount, Time) Reject(Fishbank , Amount, Time) ModifyRequest(Fishbank , Amount, Time) Declare(Fishbank, NoShips, Fc, Time)

78

10.310.3 Genetic representation Genetic representation of plansof plans

The generation of plans of a company is achieved by means of a GA with genetic coadapted components.

Cooperative Coadapted Species Cooperative Coadapted Species (Potter and De Jong, 2000): A subcomponent evolves separately by a genetic algorithm, but the

evolution is influenced by the existence of the other subcomponents in the ecosystem.

Each species is evolved in its own population and adapts to the environment through the repeated application of a genetic algorithm. The influence of the environment on the evolution, namely the existence of the other species, is handled by the evaluation function of the individuals, which takes into account representatives from other species. Thus, the interdependency among subcomponents is achieved by evolving the species in parallel and evaluating them within the context of each other.

79

Evaluate individualsof Company i

Best of C1

Population ofCompany 1

Best of CM

Population ofCompany M

Genetic coadapted components to model the companies

Population ofCompany i

Individual1 of Ci

Individualj of Ci

Individualn of Ci

...

...

80

The fitness of an individual in a population is computed based on its configuration and on representatives chosen from the other populations in the ecosystem.

From each population the representative is chosen as the “bestbest” individual of the population, with two possible interpretations for what “best” means in the context of a subspecies.

If the company is profit oriented, the best individual is the one that will bring the maximum profit.

For ecologically oriented companies, the representative is the individual that will bring maximum profit, while keeping the minimum amount of fish that allows regeneration. If the profile of the company is not known, an ecologically profile is assumed by default.

The fitness of the individual is evaluated in the context of the selected representative and is based on the profit, taking care of the ecological balance (individuals that do not ensure the minimum regeneration amount are assigned 0 fitness) or not, depending on the company profile.

81

Season: T1, T2, T3, T4

Place at Ti: H, R1, R2, R3

Ship 1 Ship 2 Ship NSi

Company i

Place of Ship NSi: H, R1, R2, R3

Place of Ship 1: H, R1, R2, R3

Company i

T1 TnT2

First representationFirst representation(NoBPlace + NoBSeason) * NoShips

Second representationSecond representationNoBPlace * NoShips * NoSeasons

82

The genetic model of the ecosystem may be used in different ways.

A CompA builds its own plan by genetic evolutionplan by genetic evolution while keeping the ecological balance: it models himself and the way it believes the other CompAs will act.

ReplanningReplanning, after seasons T1..Ti have passed, by the same approach as above.Considering the difficult problem of distributed planning and dynamical replanning, the genetic model offers a simple solution to develop a plan for sending ships to different places, or, as more knowledge about the evolution of

the world is gathered, to rebuild the plan adequately.

10.410.4 Genetic Model Utilization in Genetic Model Utilization in MASMAS

83

A CompA may keep alternate “good plansgood plans” by selecting the first x “best plans” from genetic evolution, for negotiation with other CompAs.

A CompA may model the evolution of the worldmodel the evolution of the world (the other CompAs action plans) in case it performs symbolic distributed planning.

In this case, the criteria for selecting the preferred solution from several runs are somehow different. When selecting the presumed course of actions of the other agents from the several runs of the algorithm, the agent will prefer the one which brings maximum profit to the society and, if several such solutions exit, the one that will bring maximum gain to the profit oriented companies.

84

For negotiationnegotiation, when a CompA has a plan: Once the CompA has developed a plan, it will use this plan as the fixed description of the self subspecies in the population and run the genetic algorithm with this description. The evolved model of the world and its own strategy will lead the negotiation. The agent will also prefer the other agents plans which brings maximum profit to the society (and maximum gain to the profit oriented companies). The negotiation starts either in case the CompA detects that the ecological balance is threaten by the actions of the other agents, or when it would like to fish more in a particular fishing bank and it detects there will be not enough fish left at a particular time moment.

85

More on negotiationMore on negotiation

A cooperation profile of the other agents in the system is built to guide the negotiation strategy.

The cooperation profile is defined by a set of attributes and is updated constantly while interacting with the other company agents. The cooperation profile of another agent x as developed by the current agent A contains the following attributes:

No of requests accepted by x No of requests rejected by x No of requests modified by x No of x’s requests accepted by A No of x’s requests rejected by A

86

The genetic model may be used by the EnvA also If we endow the EnvA with more cognitive capabilities, then it will evolve a model of the world for which it acts as the environment and will start negotiating with the company agents in the same way these agents are negotiating among them.

Therefore the agents in the MAS are hybrid: cognitive as they have mental attitudes, negotiation strategies, and symbolic planning, and genetic, as they use the coadapted species genetic model in different ways.

87

10.5 10.5 Experimental resultsExperimental results

The planning process was tested for several situations, with different GA and EA parameters.

We present results for 5 fishing seasons, three fishing banks (R1, R2, R3) and in-port (H), and 3 companies.

The fitness of an individual was computed as 95% profit and 5% preservation of resources, with a 0 fitness value if at least one resource goes beyond the regeneration level.

88

Genetic Algorithm's parameters: Two-point crossover rate: 0.6 Probability of mutation in every individual: 0.1 Population size: 100 Length of an individual: 100 The selection is based on the stochastic remainder technique Number of generations: 20..200

Evolutionary Algorithm’s parameters: Crossover rate: 0 Probability of mutation in every individual: 0.05..0.50 Population size: 100 Length of an individual: 100 Number of generations:150

89

RUN 1 RUN 2Company 1 H R1 R2 R3 Company1 H R1 R2 R3

Season 1 3 1 5 1 Season 1 2 2 1 5Season 2 2 3 3 2 Season 2 1 3 3 3Season 3 1 3 2 4 Season 3 0 3 3 4Season 4 2 0 3 5 Season 4 3 4 1 2Season 5 0 3 3 4 Season 5 1 3 0 6

Fitness value 0.836166666666 Fitness value 0.856229166666Company 2 H R1 R2 R3 Company2 H R1 R2 R3

Season 1 2 4 2 2 Season 1 2 3 1 4Season 2 1 4 2 3 Season 2 3 3 1 3Season 3 1 1 2 6 Season 3 4 2 1 3Season 4 3 2 2 3 Season 4 2 3 4 1Season 5 0 2 3 5 Season 5 3 2 3 2

Fitness value 0.855166666666 Fitness value 0.730333333333Company 3 H R1 R2 R3 Company 3 H R1 R2 R3

Season 1 1 4 3 2 Season 1 2 4 1 3Season 2 3 4 0 3 Season 2 3 2 2 3Season 3 2 5 3 0 Season 3 1 2 3 4Season 4 1 1 5 3 Season 4 2 0 5 3Season 5 2 3 2 3 Season 5 1 0 5 4

Fitness value 0.817166666666 Fitness value 0.826041666666

Results of genetic plan evolution for3 companies and 2 runs

90

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170 180 190

Generations No.

Fitn

ess

Run 1 Run 2 Run 3 Average

Average best fitness of genetic plansdepending on the number of generations

(the average is for 5 runs while only the results of the first three are shown)

91

Evolutionary Algorithm (150 Generations)

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50

Mutation Rate

Fit

ne

ssRun 1 Run 2 Run 3 Average

Average best fitness of evolutionary plansdepending on mutation rate

(the average is for 5 runs while only the results of the first three are shown)

92

10.610.6 ConclusionsConclusions We have presented a multi-agent system with competitive

agents that tries to solve the problem of the “tragedy of commons”

The self-interested agents in the system have to achieve a certain degree of cooperation in order to avoid the overexploitation of resources and, in this way, to ensure individual and global welfare.

The multi-agent system proposed is based on agents which are capable of developing plans, of interactions with the other agents in the system and with the environment. The agents are hybrid, containing cognitive components and a genetic representation of their planned actions and of those of the other agents in the society.

93

ConclusionsConclusions We have shown that the model of coadapted subspecies is

also suited to represent competitive subcomponents that should collaborate as they are living in a common environment and they are sharing common resources.

As opposed to other multi-agent approaches for the “tragedy of commons” problem, in which the ecological balance is kept by a centralised authority, the MAS  solution we propose is inherently distributed.

The CompAs are the active, distributed elements that are trying to achieve their goals and to keep the adequate level of renewable resources, while the EnvA is an agent which updates the environment status and informs about the changes in the world around.

94