Cognitive Bodyspaces:

Learning and Behavior

GECCO 2011Competitions

Martin V. Butz

Department of Psychology IIIUniversity of WürzburgRöntgenring 11, 97070

Würzburg, Germany

butz@psychologie.uni-wuerzburg.de

Demolition Derbyhttp://www.coboslab.psychologie.uni-wuerzburg.de/competitions

• Martin V. Butz, University of Würzburg, Germany

• Matthias J. Linhardt, University of Würzburg, Germany

• Daniele Loiacono, Politecnico di Milano, Italy

• Luigi Cardamone, Politecnico di Milano, Italy

• Pier Luca Lanzi, Politecnico di Milano, Italy

Demolition Derby: Purpose

• Optimize opponent interactions

– Avoid being hit – run away when necessary

– Try to hit others at the right moment

• Enables (co-) optimization of interaction behavior.

– Fitness may be based on damage caused to other cars.

– Co-development of two or more competitors is possible (possibly with different approaches).

3

Goal & Setup• Goal: Wreck all opponent cars by

crashing into them without gettingwrecked yourself.

• Setup: Local sensor information as in the Simulated Car Racing Competition.

• Modifications:– Sensors:

• Same simulated sensors – but without noise.

• The range of the 36 opponent sensors has been increased to 300m.

– Damage model:• Cars do not take any damage when colliding with walls.

• Cars do not take any damage in the front when colliding with each other.

• Cars only take damage when their rear is hit by another car.4

Winner Determination• Arena: Large circular track (surface: asphalt; length: 640m,

width: 90m)• Qualifying

– 1-vs-1 matches evaluating all against all (winner = 1 point)– Eight best controllers qualify for the final showdown.

• Final match:– The best eight controllers fight each other.– The last car standing in the final match is declared the winner.

5

Additional Goodies for a Quick Start

• Basic controller clients for Java and C++, to easily add additional functionality to.

• COBOSTAR client in Java– With opponent monitor that tracks opponents over time.

– With simple crashing strategy that targets closest car in range.

• Evolvable client setup that – receives caused damage signal,

– applies CMA evolution strategy-based optimization,

– runs continuously with or (even faster) without visualization for as many generations as desired.

Entries

• Thies Lönneker– Dep. of Computer Science

– University of Würzburg

– GERMANY

– Controller: DemoStar

• Zygmunt Horodyski– Piłsudskiego 39/1

– 66-530 Drezdenko

– POLAND

– Controller: Spartiat

Entry Information

• Thies Lönneker

– Dep. of Computer Science

– University of Würzburg

– GERMANY

– Controller: DemoStar

DemoStar: Orientation

Task:

Get an idea of the

car's position

Approach 2 (DemoStar):

gather global information

sensors:

distance from start line

track position

result: 3rd person perspective

requires known trade geometry

Approach 1:

gather local information

sensors:

track edge sensors

focus sensors

result: 1st person perspective

works in unknown environment

DemoStar: Opponents

• Task:

Locate opponents

Opponent sensors provide information about

sector

distance

of an opponent from 1st-person point of view

Accuracy is restricted by the sector's central angle

Sector transitions lead to the most exact sensor readings

DemoStar: Opponent Tracking

• Task:

Get more information about the opponents

Record the last opponent sensor readings

Remember information about sector transitions

Get opponent's moving direction from last sector transitions

Extrapolate opponent's current position

In case of multiple opponents:

Map sensor readings to extrapolated positions

DemoStar: Agility• Task:

Find the best way to quickly head for an opponent

Parameter optimization for best steering efficiency

Method: Covariance Matrix Adaption evolutionary

strategy (CMA-ES)

Resulting parameter set makes use of drift effects at

the limit of the car's controllability

0 23 46 69 92 115138161184207230253276299

0,000

0,020

0,040

0,060

0,080

0,100

0,120

0,140

0,160

0,180

angluar speed [rad/tick]

steering eff iciency [rad/m]

game ticks

DemoStar: Recovery

• Task:

Escape stuck situations

Stuck face-to-face

Indications: opponent close to the car, combined with involuntary negative velocity

Resolution: go into reverse with maximum steering

Stuck at a wall

Indications: wall close to the car, combined with involuntary zero-velocity

Resolution: go into reverse for a short time

Entry Information

• Zygmunt Horodyski

– Piłsudskiego 39/1

– 66-530 Drezdenko

– POLAND

– Controller: Spartiat

M

Algoritm

Main class

Controller

ModelB

eh

avior

GetA

ction

SensorModel(SM)

Action

Up

date

mo

de

l every1

min

(Ru

n A

lgoritm

every1

min

)

Actio

n

SM

Up

date

Co

llectd

ata Sen

dactio

n

Selecto

ne

of b

eh

aviors

Zygmunt Horodyski

Overview of how the controller works

Model

Zygmunt Horodyski

The model is a group of elements, which determine how the controller moves on the track.The model is also an individual solution selected by the algorithm.

Components of model are:•Maximum speed/speed when we turn back[depends on strategy]•[A] Average distance from the edge when we move back [outer edge]•[B] Average distance from the edge when we move back [inner edge]•[C] Average distance from the edge when we slow down•[D] Distance from the edge when we slow down•[E] Distance from the edge when we use brake• Time after controller will act like it stuck• Strategy

CA

R

Legend:• Track edge:• Sensor:• Car:

B, CA, C

D

D, E

Algorithm

Zygmunt Horodyski

The Algorithm is inspired by “life” observations – and the solution is some sort of model.Referring to the reality, we don’t always think about every single move. From time to time wehave to update factors that determine our behavior, in other words we’re changing ourmodel.

The select is my interpretation of tournament. Each winner of each tournament (10 in total)has the possibility to crossbreed with the best one from the previous population. Also thetwo worst solutions form each tournament are crossing with each other (to eliminate uselesssolutions).

Each solution has it’s own probability of mutation based on it’s rating. Probability of mutationis checked for each parameter of solution, adding a Gaussian-distributed parameter value ifapplicable.

Evaluation function

Zygmunt Horodyski

The evaluation function is also related to life. In many areas of life, especially in cars andracing we know the best way, the best solution. But not always we follow that best solution,although we can calculate it. And that make as unpredictable.

In my evaluation function firstly we check which one of strategies suits the situation best.Select of it is based on time to the end, current damage (ours and others), current fuel leveland others.

Two different strategies determine two different best solutions. Each judged solution iscompared to the best one and rated. The perfect score that can be granted is for solutions isthe one with 10-20% difference from the best solution.

Controller

Zygmunt Horodyski

M

Co

llectd

ata Sen

dactio

n

Selecto

ne

of b

eh

aviors

Update Action

SensorModel

Be

havio

r

Actio

n

Controller is part of program that, based on model and current state of environment, sets theaction of our car.

There are two different controllers, each specialized to different tasks. One that avoidsdamage and safes fuel (Dumbass) and the other one that tries to cause damage (Pro). Twostrategies protect us from programs that learn the opponent.

Firstly we select one behavior, then we calculate each Action parameter (based on theselected behavior) and then return that one to the main class.

Behavior

Zygmunt Horodyski

Selection of behavior depends on current model and environment. In strategy Dumbass (sd)and Pro(sp) there are five behaviors that can be selected:• Search (sd+sp)

o Controller will move around the map searching for the opponent• [C] Charge (sd+sp)

o If we see an opponent and it is possible controller will attack• [M] MoveBack (sd+sp)

o If there is no way we can move, controller will move back• [B] Brake (sd+sp)

o If it is necessary, controller will use brake to stop the car• [T] TurnBack (sd+sp)

o If there is an opponent behind us and it is possible, the controller will turn back• [S] Strike (sp)

o If opponent is directly in front of us, controller will attack• [E] Evade (sp)

o If controller is near edge, it will evade it.EXAMPLES ON THE NEXT SLIDE

(The way how it chooses can be checked in source code in DD_D_Pro and DD_D_Dumbass)

Behavior examples

CAR

CA

RC

AR

Legend:• Track edge:•Car:

• Enemy:T

SC

M

E

B

Results………

• 8 (2xDemoStar, 2xSpartiat, 4xBase-Client)

• 3 (DemoStar, Spartiat, Base-Client)

• 1 vs. 1

Results with 8 cars(2xDemoStar, 2xSpartiat, 4xBase-Client)

DemoStar Spartiat BaseClient DemoStar Spartiat BaseClient BaseClient BaseClient

Wins 0 1 3 2 1 0 3 0

Sum Of Points 22 59 59 59 52 21 60 28

Reversed Order:

Wins 0 0 0 1 1 2 4 2

Sum Of Points 26 50 27 44 59 52 52 50

On W7 (14 runs):

Wins 1 1 3 0 2 2 3 2

Sum Of Points 52 59 48 47 50 49 47 68

Results with Three Cars(DemoStar, Spartiat, Base-Client)

3 runs eachPoints

(the more the better) Number of Times First Place

order DemoStar Spartiat BaseClient DemoStar Spartiat BaseClient

123 3 6.5 6.5 1 1.5 0.5

132 9 3 6 3 0 0

213 7.5 3 7.5 1.5 0 1.5

231 6 9 3 0 3 0

312 8 7 3 2 1 0

321 7.5 3 7.5 1.5 0 1.5

total 41 31.5 33.5 9 5.5 3.5

… DemoStar wins… Spartiat similar to BaseClient.

Results 1 vs. 1(DemoStar, Spartiat, Base-Client)

… DemoStar wins… Spartiat better than BaseClient.

DemoStar Spartiat Base Client

DemoStar-Spartiat 9 1 *

Order Spartiat - DemoStar 5 5 *

DemoStar vs- Base Client 10 * 0

Spartiat vs. Base Client * 6 4

Cognitive Bodyspaces:

Learning and Behavior

Thies Lönnekerwins with

DemoStar

Check out the competition video online: http://www.youtube.com/watch?v=YZ1Pi7V83Ao