Evolving Neural Networks Through Augmenting Topologies

7/21/2019 Evolving Neural Networks Through Augmenting Topologies

1/37

The MIT Press Journals

http://mitpress.mit.edu/journals

This article is provided courtesy of The MIT Press.

To join an e-mail alert list and receive the latest news on our publications, please visit:

http://mitpress.mit.edu/e-mail
http://mitpress.mit.edu/journalshttp://mitpress.mit.edu/e-mailhttp://mitpress.mit.edu/e-mailhttp://mitpress.mit.edu/journals


2/37

Evolving Neural Networks through

Augmenting Topologies

Kenneth O. StanleyDepartment of Computer Sciences, The University of Texas at Austin, Austin, TX

78712, USA

isto MiikkulainenDepartment of Computer Sciences, The University of Texas at Austin, Austin, TX

78712, USA

A!stra"t

An important question in neuroevoution is ho! to "ain an a#vanta"e from evovin"neura net!or$ topoo"ies aon" !ith !ei"hts% &e present a metho#, 'euro(vou)tionof Au"mentin" Topoo"ies *'(AT+, !hich outperforms the est fixe#)topoo"y metho#on a chaen"in" enchmar$ reinforcement earnin" tas$% &e caim that the increase#efficiency is #ue to *1+ empoyin" a principe# metho# of crossover of #iffer)enttopoo"ies, *2+ protectin" structura innovation usin" speciation, an# *-+ incremen)tay "ro!in" from minima structure% &e test this caim throu"h a series of aationstu#ies that #emonstrate that each component is necessary to the system as a !hoean# to each other% &hat resuts is si"nificanty faster earnin"% '(AT is aso an im)portant contriution to .As ecause it sho!s ho! it is possie for evoution to othoptimi/e and complexifysoutions simutaneousy, offerin" the possiiity of evovin"increasin"y compex soutions over "enerations, an# stren"thenin" the anao"y !ithioo"ica evoution%

Keywor#s.enetic a"orithms, neura net!or$s, neuroevoution, net!or$ topoo"ies, speciation,competin" conventions%

$ Intro#u"tion

'euroevoution *'(+, the artificia evoution of neura net!or$s usin" "enetic a"o)rithms,

has sho!n "reat promise in compex reinforcement earnin" tas$s *.ome/ an#

0ii$$uainen, 1 .ruau et a%, 13 0oriarty an# 0ii$$uainen, 17 4otter et a%,

15 &hitey et a%, 1-+% 'euroevoution searches throu"h the space of ehaviors for a

net!or$ that performs !e at a "iven tas$% This approach to sovin" compex contro

proems represents an aternative to statistica techniques that attempt to estimate the

utiity of particuar actions in particuar states of the !or# *6aein" et a%, 13+% '( is a

promisin" approach to sovin" reinforcement earnin" proems for severa reasons% 4ast

stu#ies have sho!n '( to e faster an# more efficient than reinforcement earn)in"

metho#s such as A#aptive euristic Critic an# )9earnin" on sin"e poe aanc)in" an#

root arm contro *0oriarty an# 0ii$$uainen, 13 0oriarty, 17+% :ecause '( searches

for a ehavior instea# of a vaue function, it is effective in proems !ith continuous an#

hi"h)#imensiona state spaces% ;n a##ition, memory is easiy repre)sente# throu"h

recurrent connections in neura net!or$s, ma$in" '( a natura choice for earnin" non)

0ar$ovian tas$s *.ome/ an# 0ii$$uainen, 1, 2


3/37

6% >% Staney an# ?% 0ii$$uainen

;n tra#itiona '( approaches, a topoo"y is chosen for the evovin" net!or$s e)

fore the experiment e"ins% Usuay, the net!or$ topoo"y is a sin"e hi##en ayer of

neurons, !ith each hi##en neuron connecte# to every net!or$ input an# every

net!or$ output% (voution searches the space of connection !ei"hts of this fuy)connecte# topoo"y y ao!in" hi"h)performin" net!or$s to repro#uce% The !ei"ht

space is expore# throu"h the crossover of net!or$ !ei"ht vectors an# throu"h the

mu)tation of sin"e net!or$s@ !ei"hts% Thus, the "oa of fixe#)topoo"y '( is to

optimi/e the connection !ei"hts that #etermine the functionaity of a net!or$%

o!ever, connection !ei"hts are not the ony aspect of neura net!or$s that con)triute to their ehavior% The topoo"y, or structure, of neura net!or$s aso affectstheir functionaity% 0o#ifyin" the net!or$ structure has een sho!n effective as part ofsupervise# trainin" *Chen et a%, 1-+% There has aso een a "reat #ea of inter)estin evovin" net!or$ topoo"ies as !e as !ei"hts over the ast #eca#e *An"eine eta%, 1- :ran$e, 15 .ruau et a%, 13 ao, 1+% The asic question, ho!)ever, remains= Can evovin" topoo"ies aon" !ith !ei"hts provi#e an a#vanta"e overevovin" !ei"hts on a fixe#)topoo"yB A fuy connecte# net!or$ can in principe

approximate any continuous function *Cyen$o, 18+% So !hy !aste vauae effortpermutin" over #ifferent topoo"iesB

The ans!ers provi#e# thus far are inconcusive% Some have ar"ue# that net!or$compexity can affect the spee# an# accuracy of earnin" *han" an# 0uhenein,1-+% Athou"h this assertion is true for the ac$propa"ation a"orithm, it is not cear!hether it appies !hen !ei"hts are ein" optimi/e# y evoution an# notac$propa)"ation%

A persuasive ar"ument for the evoution of oth topoo"y an# !ei"hts !as put

for!ar# y .ruau et a% *13+, !ho caime# that evovin" structure saves the time !aste#

y humans tryin" to #eci#e on the topoo"y of net!or$s for a particuar '( proem%

Athou"h amost a fixe#)topoo"y '( systems use a fuy connecte# hi#)#en ayer,

#eci#in" ho! many hi##en no#es are nee#e# is a tria)an#)error process% .ruau et a%

supporte# their ar"ument y evovin" the topoo"y an# !ei"hts of an ar)tificia neura

net!or$ that sove# the har#est poe)aancin" enchmar$ proem to #ate% o!ever,ater resuts su""este# that structure !as not necessary to sove the #if)ficut proem% A

fixe#)topoo"y metho# cae# Enforced Subpopulations *(S4+ *.ome/ an# 0ii$$uainen,1+ !as ae to sove the same proem 5 times faster simpy y restartin" !ith a

ran#om numer of hi##en neurons !henever it ecame stuc$ %

This artice aims to #emonstrate the opposite concusion= if #one ri"ht, evovin"structure aon" !ith connection !ei"hts can si"nificanty enhance the performance of'(% &e present a nove '( metho# cae# NeuroEvolution of Augmenting Topolo-gies *'(AT+ that is #esi"ne# to ta$e a#vanta"e of structure as a !ay of minimi/in"the#imensionaity of the search space of connection !ei"hts% ;f structure is evove#such that topoo"ies are minimi/e# an# "ro!n incrementay, si"nificant "ains inearnin" spee# resut% ;mprove# efficiency resuts from topoo"ies ein" minimi/e#throughoutevoution, rather than ony at the very en#%

(vovin" structure incrementay presents severa technica chaen"es= *1+ ;sthere a "enetic representation that ao!s #isparate topoo"ies to cross over in a

meanin"fu !ayB *2+ o! can topoo"ica innovation that nee#s a fe! "enerations to

e optimi/e# e protecte# so that it #oes not #isappear from the popuation

prematureyB *-+ o! can topoo"ies e minimi/e# throughout evolution!ithout thenee# for a speciay con)trive# fitness function that measures compexityB

1


4/37

(vovin" ''@s throu"h Au"mentin" Topoo"ies

The '(AT metho# consists of soutions to each of these proems as !i e #e)

scrie# eo!% The metho# is vai#ate# on poe aancin" tas$s, !here '(AT per)

forms 25 times faster than Ceuar (nco#in" an# 5 times faster than (S4% The resuts

sho! that structure is a po!erfu resource in '( !hen appropriatey utii/e#% '(AT isunique ecause structures ecome increasin"y more compex as they ecome more

optima, stren"thenin" the anao"y et!een .As an# natura evoution%

% &a"kgroun#

0any systems have een #eveope# over the ast #eca#e that evove oth neura net)

!or$ topoo"ies an# !ei"hts *An"eine et a%, 1- :raun an# &eisro#, 1- Das)"upta

an# 0c.re"or, 12 Fumer an# 0ii$$uainen, 12 .ruau et a%, 13 6rish)nan an#

Ciesies$i, 1G 9ee an# 6im, 13 0an#ischer, 1- 0anie//o, 1G >pit/ an#

Shavi$, 17 4uHo an# 4oi, 18 ao an# 9iu, 13 han" an# 0uhenein, 1-+%

These metho#s encompass a ran"e of i#eas aout ho! Topology and Weight Evolv-ingArtificial Neural Networs *T&(A''s+ shou# e impemente#% ;n this section, !ea##resssome of the i#eas an# assumptions aout the #esi"n of T&(A''s, an# of)fer soutions to

some unsove# proems% >ur "oa is to fin# ho! a neuroevoution metho# can use the

evoution of topoo"y to increase its efficiency%

%.$ T'EANN En"o#ing

The question of ho! to enco#e net!or$s usin" an efficient "enetic representation

must e a##resse# y a T&(A''s% &e !i #iscuss severa prototypicarepresentationa schemes%

T&(A''s can e #ivi#e# et!een those that use a #irect enco#in", an# those that

use an in#irect one% Direct enco#in" schemes, empoye# y most T&(A''s, specify in

the "enome every connection an# no#e that !i appear in the phenotype *An"e)ine et a%,

1- :raun an# &eisro#, 1- Das"upta an# 0c.re"or, 12 Fumer an#

0ii$$uainen, 12 6rishnan an# Ciesies$i, 1G 9ee an# 6im, 13 0anie//o, 1G

>pit/ an# Shavi$, 17 4uHo an# 4oi, 18 ao an# 9iu, 13 han" an# 0uhenein,1-+% ;n contrast, in#irect enco#in"s usuay ony specify rues for con)structin" a

phenotype *.ruau, 1- 0an#ischer, 1-+% These rues can e ayer specifi)cations or

"ro!th rues throu"h ce #ivision% ;n#irect enco#in" ao!s a more compact representation

than #irect enco#in", ecause every connection an# no#e are not speci)fie# in the

"enome, athou"h they can e #erive# from it%

%.$.$ &inary En"o#ing

Direct enco#in"s usuay require simper impementations than in#irect enco#in"s%

The simpest impementation is ase# on the tra#itiona it strin" representation use#

y .As% For exampe, Das"upta an# 0c.re"or *12+ use such an enco#in" in their

metho#, cae# Structured !enetic Algorithm*s.A+, !here a it strin" represents theconnection matrix of a net!or$% s.A is notae for its simpicity, ao!in" it to operate

amost i$e a stan#ar# .A% o!ever, there are severa imitations as !e% First, thesi/e of the connectivity matrix is the square of the numer of no#es% Thus, the

representa)tion o!s up for a ar"e numer of no#es% Secon#, ecause the si/e of

the it strin" must e the same for a or"anisms, the maximum numer of no#es *an#

hence con)nections as !e+ must e chosen y a human runnin" the system, an# ifthe maximum is not sufficient, the experiment must e repeate#% Thir#, usin" a inear

strin" of its to represent a "raph structure ma$es it #ifficut to ensure that crossover!i yie# usefu cominations%

(voutionary Computation Eoume 1


5/37


%.$.% (raph En"o#ing

:ecause it strin"s are not the most natura representation for net!or$s, most

T&(A''s use enco#in"s that represent "raph structures more expicity% 4uHo an#

4oi *17+ use a #ua representation scheme to ao! #ifferent $in#s of crossover intheir "arallel #istributed !enetic "rogramming *4D.4+ system% The firstrepresentation is a "raph structure% The secon# is a inear "enome of no#e #efinitions

specifyin" in)comin" an# out"oin" connections% The i#ea is that #ifferentrepresentations are appro)priate for #ifferent $in#s of operators% Su"raph)s!appin"

crossovers an# topoo"ica mutations use the "ri#, !hie point crossovers an#connection parameter mutations use the inear representation%

As in s.A, 4D.4 has a finite imit on the numer of no#es in the net!or$, cor)

respon#in" to the numer of no#es in the t!o)#imensiona "ri# that represents the

"raph version of the "enome% 4D.4 uses "raph enco#in" so that su"raphs can e

s!appe# in crossover% Su"raph s!appin" is representative of a prevaiin" phioso)

phy in T&(A''s that su"raphs are functiona units an# therefore s!appin" them

ma$es sense ecause it preserves the structure of functiona components% o!ever,

!e cannot e sure !hether the particuar su"raphs ein" comine# in 4D.4 are theri"ht ones to create a functiona offsprin"%

%.$.) Nonmating

:ecause crossover of net!or$s !ith #ifferent topoo"ies can frequenty ea# to a oss

of functionaity, some researchers have "iven up on crossover ato"ether in !hat is

cae# (voutionary 4ro"rammin" *ao an# 9iu, 13+% An"eine et a% *1-+

impemente# a system cae# !eNerali$ed Ac%uisition of &ecurrent 'ins *.'A?9+,commentin" that Ithe prospect of evovin" connectionist net!or$s !ith crossover

appears imite# in "enera%J Athou"h .'A?9 uses a "raph enco#in", it is

fun#amentay #ifferent from 4D.4 in that it si#esteps the issue of crossover entirey%

.'A?9 #emonstrates that a T&(A'' #oes not nee# crossover to !or$, eavin" the

proem of #emonstratin" the a#vanta"es of crossover to other metho#s%

%.$.* In#ire"t En"o#ing

.ruau@s *1-+ (ellular Encoding*C(+ metho# is an exampe of a system that utii/esin#irect enco#in" of net!or$ structures% ;n C(, "enomes are pro"rams !ritten in a

speciai/e# "raph transformation an"ua"e% The transformations are motivate# y na)

ture in that they specify cell divisions% Different $in#s of connectivities can resut froma #ivision, so there are severa $in#s of ce #ivisions possie% A maHor a#vanta"e of

C( is that its "enetic representations are compact% .enes in C( can e reuse#

mutipe times #urin" the #eveopment of a net!or$, each time requestin" a ce#ivision at a #if)ferent ocation% C( sho!s that ce #ivisions can enco#e the

#eveopment of net!or$s from a sin"e ce, much as or"anisms in nature e"in as asin"e ce that #ifferentiates as it spits into more ces%

Athou"h C( #emonstrates that it is possie to evove #eveopmenta systems, !echose #irect enco#in" for '(AT ecause, as :raun an# &eisro# *1-+ ar"ue, in#irect

enco#in" requires Imore #etaie# $no!e#"e of "enetic an# neura mechanisms%J ;n other

!or#s, ecause in#irect enco#in"s #o not map #irecty to their phenotypes, they can ias

the search in unpre#ictae !ays% To ma$e "oo# use of in#irect enco#in"s, !e nee# to

first un#erstan# them !e enou"h to ma$e sure that they #o not focus the search on some

suoptima cass of topoo"ies% Further, experimenta resuts su""est that C( is not

necessariy more efficient than #irect enco#in" metho#s *Section G%-%-+%

1


6/37


3 3

A B C C B A

1 2 1 2

[A,B,C]x[C,B,A]

Crossovers: [A,B,A] [C,B,C](both are missing information)

Fi"ure 1= The competin" conventions proem% The t!o net!or$s compute the same

exact function even thou"h their hi##en units appear in a #ifferent or#er an# are

repre)sente# y #ifferent chromosomes, ma$in" them incompatie for crossover% The

fi"ure sho!s that the t!o sin"e)point recominations are oth missin" one of the -

main components of each soution% The #epicte# net!or$s are ony 2 of the 3 possie

per)mutations of hi##en unit or#erin"s%

&e no! turn to severa specific proems !ith T&(A''s an# a##ress each in

turn%

%.% +ompeting +onventions

>ne of the main proems for '( is the (ompeting (onventions "roblem*0ontana

an# Davis, 18 Schaffer et a%, 12+, aso $no!n as the "ermutations "roblem*?a#ciffe, 1-+% Competin" conventions means havin" more than one !ay to

express a soution to a !ei"ht optimi/ation proem !ith a neura net!or$% &hen

"enomes represent)in" the same soution #o not have the same enco#in", crossoveris i$ey to pro#uce #ama"e# offsprin"%

Fi"ure 1 #epicts the proem for a simpe -)hi##en)unit net!or$% The three hi#)#en neuronsA, ), an# (, can represent the same "enera soution in -K L 3 #ifferentpermutations% &hen one of these permutations crosses over !ith another, critica in)formation is i$ey to e ost% For exampe, crossin" MA* )* (N an# M(* )* AN can resut inM(* )* (N, a representation that has ost one thir# of the information that oth of theparents ha#% ;n "enera, for nhi##en units, there are nK functionay equivaent sou)tions% The proem can e further compicate# !ith differingconventions, i%e%, MA* )* (Nan# M#* )* EN, !hich share functiona inter#epen#ence on )%

An even more #ifficut form of competin" conventions is present in T&(A''s,ecause T&(A'' net!or$s can represent simiar soutions usin" entirey #ifferent

topoo"ies, or even "enomes of #ifferent si/es% :ecause T&(A''s #o not satisfy strict

constraints on the $in#s of topoo"ies they pro#uce, propose# soutions to the com)petin"

conventions proem for fixe# or constraine# topoo"y net!or$s such as nonre)#un#ant

"enetic enco#in" *Thierens, 13+ #o not appy% ?a#ciffe *1-+ "oes as far as cain" an

inte"rate# scheme cominin" connectivity an# !ei"hts the Ioy .rai in



7/37


this area%J Athou"h some T&(A''s such as 4D.4 have attempte# to a##ress the

proem y assumin" that sunet!or$s represent functiona units that can e recom)

ine#, #ifferent topoo"ies may not e ase# on the same sunet!or$s at a, in !hich

case no meanin"fu comination of sustructures exists%The main intuition ehin# '(AT ori"inates from the fun#amenta proem !ith

representin" #ifferent structures= their representations !i not necessariy match up%Sometimes, the "enomes can have #ifferent si/es% >ther times, "enes in the exactsame position on #ifferent chromosomes may e expressin" competey #ifferenttraits% ;n a##ition, "enes expressin" the same trait may appear at #ifferent positionson #ifferent chromosomes% o! can these compications e resove#B

'ature faces a simiar proem !ith "ene ai"nment in sexua repro#uction%.enomes in nature are not of fixe#)en"th either% Some!here aon" the evoutionfrom sin"e ces to more compex or"anisms, ne! "enes !ere a##e# to the "enomesin a process cae# gene amplification*Darne an# Dooitte, 183 &atson et a%,187+% ;f ne! "enes cou# Hust ran#omy insert themseves in positions on the "enome!ithout any in#ication of !hich "ene is !hich, ife proay !ou# not have

succee#e#, e)cause the competin" conventions proem !ou# #ecimate a hu"echun$ of offsprin"% There nee#e# to e some !ay to $eep crossover or#ery, so thatthe right"enes cou# e crosse# !ith the ri"ht "enes%

'ature@s soution utii/es homology= t!o "enes are homoo"ous if they are aees ofthe same trait% For exampe, in E+ coli, in a process cae# synapsis, a specia proteincae# ?ecA "oes throu"h an# ines up homoo"ous "enes et!een t!o "enomes efore

crossover occurs *?a##in", 182 Si"a an# Aerts, 172+% Actua homoo"y et!een

neura net!or$s cannot e easiy ascertaine# y #irect structura anaysis *hence, the

competin" conventions proem+% The main insi"ht in '(AT is that the historical originoft!o "enes is #irect evi#ence of homoo"y if the "enes share the same ori"in% Thus, '(AT

performs artificial synapsisase# on historica mar$in"s, ao!in" it to a## ne! structure!ithout osin" trac$ of !hich "ene is !hich over the course of a simuation%

%.) Prote"ting Innovation with Spe"iation;n T&(A''s, innovation ta$es pace y a##in" ne! structure to net!or$s throu"h

mutation% Frequenty, a##in" ne! structure initiay causes the fitness of a net!or$ to

#ecrease% For exampe, a##in" a ne! no#e intro#uces a noninearity !here there !as

none efore a##in" a ne! connection can re#uce fitness efore its !ei"ht has a chance

to optimi/e% ;t is uni$ey that a ne! no#e or connection Hust happens to express a usefu

function as soon as it is intro#uce#% Some "enerations are require# to optimi/e the ne!

structure an# ma$e use of it% Unfortunatey, ecause of the initia oss of fitness cause# y

the ne! structure, the innovation is uni$ey to survive in the popuation on" enou"h to e

optimi/e#% Thus, it is necessary to someho! protect net!or$s !ith structura innovations

so they have a chance to ma$e use of their ne! structure%

The .'A?9 system a##resses the proem of protectin" innovation y a##in"nonfunctiona structure% A no#e is a##e# to a "enome !ithout any connections, in the

hopes that in the future some usefu connections !i #eveop% o!ever, nonfunctionastructures may never en# up connectin" to the functiona net!or$, a##in" extraneousparameters to the search%

;n nature, #ifferent structures ten# to e in #ifferent species that compete in #if)

ferent niches% Thus, innovation is impicity protecte# !ithin a niche% Simiary, if net)

!or$s !ith innovative structures cou# e isoate# into their o!n species, they !ou#

1


8/37


have a chance to optimi/e their structures efore havin" to compete !ith the popua)

tion at ar"e%

Speciation, aso $no!n as niching, has een stu#ie# in .As, ut is not usuay

ap)pie# to neuroevoution% Speciation is most commony appie# to mutimo#afunction optimi/ation *0ahfou#, 15+, !here a function has mutipe optima, an# a.A !ith severa species is use# to fin# those optima% Speciation has aso eenappie# in the cooperative coevoution of mo#uar systems of mutipe soutions*Dar!en an# ao, 13 4otter an# De Oon", 15+%

Speciation requires a compatiiity function to te !hether t!o "enomes shou#e in the same species or not% ;t is #ifficut to formuate such a compatiiity functionet!een net!or$s of #ifferent topoo"ies, !hich may e the reason !hy speciationhas not een rou"ht into T&(A''s% The competin" conventions proem ma$esmeasurin" compatiiity particuary proematic ecause net!or$s that compute thesame function can appear very #ifferent%

o!ever, ecause '(AT has a soution to the competin" conventions proem,

usin" historica information aout "enes, the popuation in '(AT can easiy e spe)

ciate#% &e use explicit fitness sharing, !hich forces in#ivi#uas !ith simiar "enomesto share their fitness payoff *.o#er" an# ?ichar#son, 187+% The ori"ina impicit

version of fitness sharin" intro#uce# y oan# *175+ "roupe# in#ivi#uas y perfor)

mance simiarity rather than "enetic simiarity% The expicit version is appropriate for

T&(A''s ecause it ao!s "roupin" net!or$s ase# on topoo"y an# !ei"ht con)fi"urations% The resut of sharin" fitness is that the numer of net!or$s that can exist

in the popuation on a sin"e fitness pea$ is imite# y the si/e of the pea$% Therefore,the popuation #ivi#es into a numer of species, each on a #ifferent pea$, !ithout the

threat of any one species ta$in" over% (xpicit fitness sharin" is !e)suite# for '(AT,

ecause simiarity can easiy e measure# ase# on the historica information in the

"enes% Thus, innovations in '(AT are protecte# in their o!n species%

%.* Initial Populations an# Topologi"al Innovation

;n many T&(A'' systems, the initia popuation is a coection of ran#om topoo)"ies%

Such a popuation ensures topoo"ica #iversity from the start% o!ever, ran#om initia

popuations turn out to pro#uce many proems for T&(A''s% For exampe, un#er

many of the #irect enco#in" schemes, there is a chance that a net!or$ !i have no

path from each of its inputs to its outputs% Such infeasie net!or$s ta$e time to !ee#

out of the popuation%o!ever, there is a more sute an# more serious proem !ith startin" ran#omy% As

our experiments !i confirm *Section 5%G+, it is #esirae to evove minima sou)tions that

!ay, the numer of parameters that have to e searche# is re#uce#% Startin" out !ith

ran#om topoo"ies #oes not ea# to fin#in" minima soutions, since the popu)ation starts

out !ith many unnecessary no#es an# connections area#y present% 'one of these no#es

or connections have ha# to !ithstan# a sin"e evauation, meanin" there is no Hustification

for their confi"uration% Any minimi/ation of net!or$s !ou# have to e spent "ettin" ri# ofapparatus that shou# not have een there in the first pace, an# nothin" in the process of

recominin" #ifferent topoo"ies pushes to!ar#s such mini)mi/ation% Since there is no

fitness cost in creatin" ar"er net!or$s, they !i #ominate as on" as they have hi"h

fitness%

>ne !ay to force minima topoo"ies is to incorporate net!or$ si/e into the fitness

function, an# some T&(A''s actuay #o this *han" an# 0uhenein, 1-+% ;n such

metho#s, ar"er net!or$s have their fitnesses penai/e#% Athou"h aterin" the



9/37


Genome (Genotype)NodeGenes

Node 1

Sensor

Connect. In 1

Genes Out 4Weight 0.7

Enabled

Innov 1

Network (Phenotype)

4

5

123

Fi"ure 2= A "enotype to

phenotype mappin" exampe%

A "enotype is #epicte# that

pro#uces the sho!n

phenotype% There are - input

no#es, one hi##en, an# one

output no#e, an# seven

connection #efinitions, one of

!hich is recurrent% Thesecon# "ene is #isae#, so

the connection that it specifies

*et!een no#es 2 an# G+ is

not expresse# in the

phenotype%

fitness function in this !ay can

encoura"e smaer net!or$s,

it is #ifficut to $no! ho! ar"e

the penaty shou# e for any

particuar net!or$ si/e,

particuary ecause #ifferentproems may have

si"nificanty #ifferent

topoo"ica requirements%

Aterin" the fitness function is

a# hoc an# may cause

evoution to perform #ifferenty

than the #esi"ner of the

ori"ina unmo#ifie# fitnessfunction inten#e#%


10/37

An aternative soution isfor the neuroevoution metho#itsef to ten# to!ar#sminimaity% ;f the popuatione"ins !ith no hi##en no#esan# "ro!s structure ony as itenefits the soution, there is

no nee# for a# hoc fitnessmo#ification to minimi/enet!or$s% Therefore, startin"out !ith a minima popuationan# "ro!in" structure fromthere is a #esi"n principe in'(AT%

:y startin" out minimay,

'(AT ensures that the systemsearches for the sou)tion in

the o!est)#imensiona !ei"htspace possie over the

course of all "enerations%

Thus, the "oa is not tominimi/e ony the fina

pro#uct, ut a intermediatenet!or$s aon" the !ay as

!e% This i#ea is they $ey to

"ainin" an a#vanta"e from

the evo)ution of topoo"y= it

ao!s us to minimi/e the

search space, resutin" in

#ramatic performance "ains%

>ne reason current

T&(A''S #o not start outminimay is that !ithout

topoo"ica #iversity present inthe initia popuation,

topoo"ica innovations !ou#

not survive% The proem of

protectin" innovation is not

a##resse# y these metho#s,

so net!or$s !ith maHor

structura a##itions are i$ey

not to repro#uce% Thus,

speciatin" the popuation

enaes startin" minimay in

'(AT%

) NeuroEvolution o,Augmenting Topologies-NEAT

The '(AT metho#, as

#escrie# in #etai in this

section, consists of puttin"

to"ether the i#eas #eveope#

in the previous section into


11/37


12/37


1 2 3 1 2 3

5

1 2 3


13/37

5 61 2 3

Fi"ure -= The t!o types of structura

mutation in '(AT% :oth types, a##in" a

connec)tion an# a##in" a no#e, are

iustrate# !ith the connection "enes of anet!or$ sho!n aove their phenotypes%

The top numer in each "enome is theinnovation number of that "ene% Theinnovation numers are historica mar$ersthat i#entify the ori"ina his)torica ancestor

of each "ene% 'e! "enes are assi"ne#

ne! increasin"y hi"her num)ers% ;n

a##in" a connection, a sin"e ne!

connection "ene is a##e# to the en# of the

"enome an# "iven the next avaiae

innovation numer% ;n a##in" a ne! no#e,

the connection "ene ein" spit is #isae#,

an# t!o ne! connection "enes are a##e#to the en# the "enome% The ne! no#e is

et!een the t!o ne! connections% A ne!

no#e "ene *not #epicte#+ representin" thisne! no#e is a##e# to the "enome as !e%

the "enetic enco#in" use# in '(AT an#

continue y #escriin" the components

that specificay a##ress each of the three

proems of T&(A''s%

).$ (eneti" En"o#ing

'(AT@s "enetic enco#in" scheme is

#esi"ne# to ao! correspon#in" "enes to

e easiy ine# up !hen t!o "enomescross over #urin" matin"% .enomes are

inear represen)tations of net!or$connectivity *Fi"ure 2+% (ach "enome

incu#es a ist of connectiongenes, each of!hich refers to t!o node genes ein"connecte#% 'o#e "enes provi#e a ist ofinputs, hi##en no#es, an# outputs that can

e connecte#% (ach connection "ene

specifies the in)no#e, the out)no#e, the

!ei"ht of the connection, !hether or notthe connection "ene is expresse# *an

enae it+, an# an innovation number,!hich ao!s fin#in" correspon#in" "enes

*as !i e expaine# eo!+%0utation in '(AT can chan"e oth

connection !ei"hts an# net!or$ structures%

Connection !ei"hts mutate as in any '(

system, !ith each connection either per)

ture# or not at each "eneration% Structura


14/37

mutations occur in t!o !ays *Fi"ure -+% (ach

mutation expan#s the si/e of the "enome y

a##in" "ene*s+% ;n the add connectionmutation, a sin"e ne! connection "ene !ith

a ran#om !ei"ht is a##e# connectin" t!o

previousy unconnecte# no#es% ;n the addnodemutation, an existin" connection is



15/37


spit an# the ne! no#e pace# !here the o# connection use# to e% The o#

connection is #isae# an# t!o ne! connections are a##e# to the "enome% The ne!

connection ea#in" into the ne! no#e receives a !ei"ht of 1, an# the ne! connection

ea#in" out receives the same !ei"ht as the o# connection% This metho# of a##in"no#es !as cho)sen in or#er to minimi/e the initia effect of the mutation% The ne!

noninearity in the connection chan"es the function si"hty, ut ne! no#es can e

imme#iatey inte"rate# into the net!or$, as oppose# to a##in" extraneous structure

that !ou# have to e evove# into the net!or$ ater% This !ay, ecause of speciation,

the net!or$ !i have time to optimi/e an# ma$e use of its ne! structure%

Throu"h mutation, the "enomes in '(AT !i "ra#uay "et ar"er% .enomes ofvaryin" si/es !i resut, sometimes !ith #ifferent connections at the same positions%

The most compex form of the competin" conventions proem, !ith numerous #iffer)in" topoo"ies an# !ei"ht cominations, is an inevitae resut of ao!in" "enomes to

"ro! unoun#e#% o! can '( cross over #ifferenty si/e# "enomes in a sensie

!ayB The next section expains ho! '(AT a##resses this proem%

).% Tra"king (enes through /istori"al Markings

There is unexpoite# information in evoution that tes us exacty !hich "enes match

up !ith !hich "enes et!een any in#ivi#uas in a topoo"icay #iverse popuation%That information is the historica ori"in of each "ene% T!o "enes !ith the same histori)

ca ori"in must represent the same structure *athou"h possiy !ith #ifferent

!ei"hts+, since they are oth #erive# from the same ancestra "ene of some point inthe past% Thus, a a system nee#s to #o to $no! !hich "enes ine up !ith !hich is to

$eep trac$ of the historica ori"in of every "ene in the system%Trac$in" the historica ori"ins requires very itte computation% &henever a ne! "ene

appears *throu"h structura mutation+, a global innovation number is incremente# an#assi"ne# to that "ene% The innovation numers thus represent a chronoo"y of the

appearance of every "ene in the system% As an exampe, et us say the t!o mutations in

Fi"ure - occurre# one after another in the system% The ne! connection "ene create# inthe first mutation is assi"ne# the numer 7, an# the t!o ne! connection "enes a##e#

#urin" the ne! no#e mutation are assi"ne# the numers 8 an# % ;n the future, !hen)ever

these "enomes mate, the offsprin" !i inherit the same innovation numers on each

"ene innovation numers are never chan"e#% Thus, the historica ori"in of every "ene in

the system is $no!n throu"hout evoution%

A possie proem is that the same structura innovation !i receive #ifferent in)

novation numers in the same "eneration if it occurs y chance more than once% o!)

ever, y $eepin" a ist of the innovations that occurre# in the current "eneration, it is

possie to ensure that !hen the same structure arises more than once throu"h in)

#epen#ent mutations in the same "eneration, each i#entica mutation is assi"ne# the

same innovation numer% Thus, there is no resutant exposion of innovation numers%

The historica mar$in"s "ive '(AT a po!erfu ne! capaiity% The system no! $no!s

exacty !hich "enes match up !ith !hich *Fi"ure G+% &hen crossin" over, the "enes inoth "enomes !ith the same innovation numers are ine# up% These "enes are cae#

matching"enes% .enes that #o not match are either dis,oint or excess, #epen#)in" on!hether they occur !ithin or outsi#e the ran"e of the other parent@s innovation numers%

They represent structure that is not present in the other "enome% ;n com)posin" the

offsprin", "enes are ran#omy chosen from either parent at matchin" "enes, !hereas a

excess or #isHoint "enes are a!ays incu#e# from the more fit parent% This !ay, historica

mar$in"s ao! '(AT to perform crossover usin" inear "enomes !ith)

1


16/37


Parent1

Parent2


17/37


18/37

topoo"ies% o!ever, it turns out that such

a popuation on its o!n cannot maintain

topoo"ica innovations% :ecause smaer

structures optimi/e faster than ar"erstructures, an# a##in" no#es an#

connections usuay initiay #ecreases thefitness of the net!or$, recenty au"mente#

structures have itte hope of survivin"more than one "eneration even thou"h the

in)novations they represent mi"ht e

crucia to!ar#s sovin" the tas$ in the on"

run% The soution is to protect innovation y

speciatin" the popuation, as expaine# in

the next section%

).) Prote"ting Innovation throughSpe"iation

Speciatin" the popuation ao!s or"anisms

to compete primariy !ithin their o!n

niches instea# of !ith the popuation at

ar"e% This !ay, topoo"ica innovations are



19/37


protecte# in a ne! niche !here they have time to optimi/e their structure throu"h

competition !ithin the niche% The i#ea is to #ivi#e the popuation into species such

that simiar topoo"ies are in the same species% This tas$ appears to e a topoo"y

matchin" proem% o!ever, it a"ain turns out that historica mar$in"s offer an effi)cient soution%

The numer of excess an# #isHoint "enes et!een a pair of "enomes is a natura

measure of their compatiiity #istance% The more #isHoint t!o "enomes are, the ess

evoutionary history they share, an# thus the ess compatie they are% Therefore, !e

can measure the compatiiity #istance of #ifferent structures in '(AT as a simpein)ear comination of the numer of excess Ean# #isHoint #"enes, as !e as theavera"e !ei"ht #ifferences of matchin" "enes W, incu#in" #isae# "enes=

L

The coefficients c1, c2, an# c-ao! us to a#Hust the importance of the three factors, an#

the factor N, the numer of "enes in the ar"er "enome, normai/es for "enome si/e * Ncan e set to 1 if oth "enomes are sma, i%e%, consist of fe!er than 2< "enes+%

The #istance measure ao!s us to speciate usin" a compatiiity thresho# t% Anor#ere# ist of species is maintaine#% ;n each "eneration, "enomes are sequentiay pace#into species% (ach existin" species is represente# y a ran#om "enome insi#e thespecies from the previous generation% A "iven "enome g in the current "eneration ispace# in the first species in !hich gis compatie !ith the representative "enome of that

species% This !ay, species #o not overap%1 ;f g is not compatie !ith any existin"

species, a ne! species is create# !ith gas its representative%As the repro#uction mechanism for '(AT, !e use explicit fitness sharing

*.o#er" an# ?ichar#son, 187+, !here or"anisms in the same species must sharethe fitness of their niche% Thus, a species cannot affor# to ecome too i" even ifmany of its or"anisms perform !e% Therefore, any one species is uni$ey to ta$eover the entire popuation, !hich is crucia for speciate# evoution to !or$% The

a#Huste# fitness fi.for or"anism iis cacuate# accor#in" to its #istance from everyother or"anism,in the popuation=

f.

i

The sharin" function sh is set to < !hen #istance *i* ,+ is aove the thresho# tother!ise, sh**i* ,++ is set to 1 *Spears, 15+% Thus,

4n sh**i* ,++ re#uces to the,L1

numer of or"anisms in the same species as or"anism i% This re#uction is naturasince species are area#y custere# y compatiiity usin" the thresho# t% (veryspecies is assi"ne# a potentiay #ifferent numer of offsprin" in proportion to the sumof a#)Huste# fitnesses fi

.of its memer or"anisms% Species then repro#uce y first

eiminatin" the o!est performin" memers from the popuation% The entire popuationis then repace# y the offsprin" of the remainin" or"anisms in each species%

2

The net #esire# effect of speciatin" the popuation is to protect topoo"ica inno)vation% The fina "oa of the system, then, is to perform the search for a soution asefficienty as possie% This "oa is achieve# throu"h minimi/in" the #imensionaity ofthe search space%

1;t is aso possie to #etermine the compatiiity of a "enome g!ith a species sy usin" the avera"e

compatiiity of g!ith every "enome in a species s, ut in practice, ony comparin" to the first "enome in sis sufficient an# ta$es constant time%

2;n rare cases !hen the fitness of the entire popuation #oes not improve for more than 2< "enerations,

ony the top t!o species are ao!e# to repro#uce, refocusin" the search into the most promisin" spaces%

11< (voutionary Computation Eoume 1


20/37


).* Minimi0ing 1imensionality through In"remental (rowth ,rom Minimal

Stru"ture

As #iscusse# in Section 2%G, T&(A''s typicay start !ith an initia popuation of

ran#om topoo"ies in or#er to intro#uce #iversity from the outset% ;n contrast, '(ATiases the search to!ar#s minima)#imensiona spaces y startin" out !ith a uniformpopuation of net!or$s !ith /ero hi##en no#es *i%e%, a inputs connect #irecty to out)

puts+% 'e! structure is intro#uce# incrementay as structura mutations occur, an#ony those structures survive that are foun# to e usefu throu"h fitness evauations%

;n other !or#s, the structura eaorations that occur in '(AT are a!ays Hustifie#%Since the popuation starts minimay, the #imensionaity of the search space is

minimi/e#, an# '(AT is a!ays searchin" throu"h fe!er #imensions than other

T&(A''s an# fixe#)topoo"y '( systems% 0inimi/in" #imensionaity "ives '(AT a

performance a#vanta"e compare# to other approaches, as !i e #iscusse# next%

* Per,orman"e Evaluations

&e evauate the system@s performance in or#er to ans!er t!o questions= *1+ Can'(AT evove the necessary structuresB *2+ Can '(AT fin# soutions more efficienty

than other neuroevoution systemsB The first question estaishes that topoo"yui#in" in#ee# happens in '(AT in a reiae !ay, meanin" that '(AT !i "ro! ne!

structure to cope !ith proems that require it% For this reason, '(AT is appie# to the

proem of ui#in" an X>? net!or$% Athou"h this tas$ is simpe, it requires "ro!in"

hi##en units, an# therefore serves as a simpe test for the metho#%

The secon# question is ans!ere# in the course of successivey more #ifficut poe

aancin" tas$s, !here the oHective is to aance t!o poes attache# to a cart y

movin" the cart in appropriate #irections to $eep the poe from fain"% 4oe aancin"is a "oo# enchmar$ tas$ ecause there are many #ifferent systems avaiae for

comparison% The most #ifficut proem of aancin" t!o poes !ithout veocityinformation, a non)0ar$ovian tas$, provi#es very stron" evi#ence that evovin"

au"mentin" topoo"ies is not ony interestin" for its capacity to fin# structures, ut isaso efficient in #ifficut contro tas$s%

*.$ Parameter Settings

The same experimenta settin"s are use# in a experiments they !ere not tune#

specif)icay for any particuar proem% The one exception is the har#est poeaancin" pro)em *Doue poe, no veocities, or D4'E+ !here a ar"er popuation

si/e !as use# to match those of other systems in this tas$% :ecause some of '(AT@ssystem parameters are sensitive to popuation si/e, !e atere# them accor#in"y%

A experiments except D4'E, !hich ha# a popuation of 1,


21/37


vi#es%+ There !as a 75P chance that an inherite# "ene !as #isae# if it !as #isae# in

either parent% ;n each "eneration, 25P of offsprin" resute# from mutation !ithout

crossover% The interspecies matin" rate !as ? in an avera"e of -2 "enerations *G,755 net!or$s evauate#, st# 2,55-+% >navera"e a soution net!or$ ha# 2%-5 hi##en no#es an# 7%G8 non#isae# connection

"enes% The numer of no#es an# connections is cose to optima consi#erin" that thesmaest pos)sie net!or$ has a sin"e hi##en unit *Fi"ure 5*++% '(AT is veryconsistent in fin#in" a soution% ;t #i# not fai once in 1? proem !ithout troue an# in#oin" so $eeps the topoo"y sma%

The X>? proem has een use# to #emonstrate performance of severa prior

T&(A'' a"orithms% Unfortunatey, quantitative performance comparisons are #if)

1Qe1G/x

2


22/37

112 (voutionary Computation Eoume 1


23/37


Out

(No hidden nodes)

Fi"ure 5= ;nitia phenotype an# optima X>?% Fi"ure *a+ sho!s the phenotype "iven to

the entire initia popuation% 'otice that there are no hi##en no#es% ;n '(AT, a ias is

a no#e that can connect to any no#e other than inputs% Fi"ure *+ sho!s an optima

soution !ith ony 1 hi##en no#e% *A net!or$ !ithout hi##en no#es cannot computeX>?%+ The ias connections are not a!ays nee#e# #epen#in" on the soution Aother connections are necessary% The optima *1 hi##en no#e+ soution !as foun# in

22 of 1


24/37


'( metho#s for t!o reasons= *1+ The focus is on #eveopin" an# #emonstratin" et)

ter performance on evovin" neura net!or$s an# *2+ '( metho#s in this comparison

have outperforme# reinforcement earnin" metho#s in prior comparisons on the poe

aancin" tas$ *0oriarty an# 0ii$$uainen, 13+% Thus, the question here is !hetherevovin" structure can ea# to "reater '( performance%

*.).$ Pole &alan"ing +omparisons

&e set up the poe aancin" experiments as #escrie# y &iean# *11+ an#.ome/ an# 0ii$$uainen *1+% The ?un"e)6utta fourth)or#er metho# !as use# toimpe)ment the #ynamics of the system, !ith a step si/e of


25/37


Tae 1= Doue poe aancin" !ith veocity information *D4E+% (voutionary pro)

"rammin" resuts !ere otaine# y Saravanan an# Fo"e *15+% Conventiona neu)

roevoution #ata !as reporte# y &iean# *11+% SA'( an# (S4 resuts !ere reporte#

y .ome/ an# 0ii$$uainen *1+% ;n a##ition, .ruau et a% *13+ reporte# -G,


26/37


Tae 2= Doue poe aancin" !ithout veocity information *D4'E+% C( is Ceuar

(nco#in" of .ruau et a% *13+% (S4 is (nforce# Supopuations of .ome/ an# 0i)

i$$uainen *1+% A resuts are avera"es over 2< simuations% The stan#ar# #eviation for

'(AT is 21,7< evauations% Assumin" simiar variances for C( an# (S4, a #iffer)ences

in numer of evauations are si"nificant *p 4


27/37


Tae -= '(AT aations summary% The tae compares the avera"e numer of evau)

ations for a soution in the #oue poe aancin" !ith veocities tas$% (ach aation

ea#s to a !ea$er a"orithm, sho!in" that each component is necessary%

Metho#

'o).ro!th '(AT *Fixe#)Topoo"ies+'onspeciate# '(AT;nitia ?an#om '(AT'onmatin" '(AT

Fu '(AT

han#, perhaps the situation is the opposite, an# speciation uys nothin"= protectin"in)novation mi"ht not e as important as !e have ar"ue#% This hypothesis !i e

chec$e# y aatin" speciation from the system% Finay, !e caime# that '(AT is aeto ma$e use of crossover even thou"h "enomes in '(AT have #ifferent si/es% This

point is more controversia than it mi"ht seem% For exampe, An"eine et a% *1-+

caime# that crossover in T&(A''s #oes more harm than "oo#% &e !i chec$ this

hypothesis y aatin" crossover from the system%

The reason !hy !e #o not aate historica mar$in"s #irecty is that !ithout his)

torica mar$in"s the system !ou# e a conventiona '( system% istorica mar$in"s

are the asis of every function in '(AT= Speciation uses a compatiiity operator thatis ase# on historica mar$in"s, an# crossover !ou# not e possie !ithout them% A

other system components can e aate# systematicay%

5.$ A!lations Setup

Aations can have a si"nificant #etrimenta effect on performance, potentiay to thepoint !here the system cannot sove the tas$ at a% Therefore, !e use# #oue poe

aancin" !ith veocities as the tas$ for aation stu#ies% The tas$ is compex enou"hto e interestin", yet sti not too har#, so that aate# systems !or$ as !e% Thus, it is

possie to compare the aate# versions of the system to the unaate# system%

A settin"s !ere the same as in the #oue poe aancin" !ith veocities experi)

ment% ?esuts are avera"es over 2< runs, except nonmatin" an# fu '(AT, !hich are

avera"es over 12< runs *nonmatin" '(AT !as fast enou"h to ao! many runs+%

5.% A!lations esults

Tae - sho!s the resuts of a the aations, in terms of avera"e evauations require#

to fin# a soution% Avera"es in this tae excu#e trias that faie# to fin# a soution in 1 *


28/37


Starting minimally

Historical Marking

Growth Speciation

Fi"ure 3= Depen#encies amon" '(AT components% Stron" inter#epen#encies can e

i#entifie# amon" the #ifferent components of '(AT%

hi##en units i$e the fixe# topoo"y metho#s in this tas$ *Saravanan an# Fo"e, 15

&iean#, 11+% &ithout "ro!th, '(AT !as sti ae to speciate, ut ony ase# on

!ei"ht #ifferences% .iven 1,n avera"e, this initiay



29/37


est)performin" topoo"y has aout 5 hi##en no#es% Thus, the popuation ten#s to

conver"e to a reativey hi"h)#imensiona search space, even thou"h the smaer net)

!or$s in the initia popuation !ou# have optimi/e# faster% The smaer net!or$s Hust #o

not "et a chance ecause ein" sma offers no imme#iate a#vanta"e in the initiayran#om !ei"ht space% >f course, once in a !hie sma net!or$s are foun# that per)form

!e, ao!in" a soution to e foun# more quic$y% &hether or not such net!or$s are foun#

eary on accounts for the ar"e stan#ar# #eviation of G1,7


30/37


Fi"ure 7= Eisuai/in" speciation #urin" a run of the #oue poe aancin" !ith veoc)

ity information tas$% T!o species e"in to cose in on a soution soon after the 2


31/37


position an# its area#y estaishe# metho# for comparin" short poe veocity to on"

poe an"e% This innovation !as enou"h to sove the proem !ithin one "eneration of

a##itiona !ei"ht mutations% ;n the fina "eneration, the !innin" species !as 11

"enerations o# an# incu#e# -8 neura net!or$s out of the popuation of 15


32/37


Fi"ure 8= A '(AT soution to the D4'E proem% This cever soution !or$s y ta$in"

the #erivative of the #ifference in poe an"es% Usin" the recurrent connection to itsef,the sin"e hi##en no#e #etermines !hether the poes are fain" a!ay or to!ar#s

each other% This soution ao!s controin" the system !ithout computin" theveocities of each poe separatey% &ithout evovin" structure, it !ou# e #ifficut to

#iscover such sute an# compact soutions% Startin" minimay ma$es #iscoverin"

such compact soutions more i$ey%

output no#e of a soution net!or$s% ;n


33/37


popuation to compete a"ainst each other% The hope is that an Iarms raceJ !i force

the opponents to continuay evove strate"ies etter than the strate"ies of other

net!or$s in the popuation% This metho# is usefu ecause it can pro#uce hi"h)eve

strate"ies an# tactics !ithout the nee# for an expert payer to teach the system%;#eay, strate"ies shou# ecome more sophisticate# as evoution pro"resses%

o!ever, evoution ten#s to fin# the simpest soutions that can !in, meanin" that

strate"ies osciate et!een #ifferent i#iosyncratic yet uninterestin" variations

*Dar!en, 13 ?osin an# :ee!, 17+%

&e hypothesi/e that once competitive coevoution conver"es onto a #ominantstrate"y, it cannot e improve# upon, ecause it ta$es the entire set of connection!ei"ht vaues to represent the strate"y% Aterin" the !ei"hts means aterin" the strat)e"y, rather than ui#in" upon an# compexifyin" the strate"y% Thus, if a ne! strate"yis to ta$e ho#, it must !in y ein" different than the previous #ominant strate"y,rather than y ein" more sophisticated% ;n contrast, '(AT can evove increasin"ymore sophisticate# strate"ies continuay, ecause as soon as the popuationconver"es on a ne! #ominant strate"y, ne! connections an# ne! no#es can e

a##e# to the current strate"y% 'e! structure means ne! expressive space forelaboratingon the existin" strat)e"y, rather than repacin" it% Thus, this approachao!s continual coevolution, i%e%, non)conver"ent innovation on #ominant strate"ies%;n a##ition, ecause #ifferent strate"ies in #ifferent species are protecte#, there !ie mutipe #ominant an# continuay more compex strate"ies%

;n a##ition to continua coevoution, the evoution of structure shou# ao! theinte"ration of separate expert neura net!or$s% For exampe, suppose one neura net)!or$ can I$ic$J a a to!ar#s the "oa from any position on a fie#, an# another net)!or$ can #rie the a aroun# the fie# !ithout osin" contro% 'either of these t!onet!or$s aone can pay soccer% o!ever, if !e cou# someho! comine their exper)tise into one, perhaps !e cou# "et a soccer payer out% Cominin" these controersis not a simpe matter of processin" oth their outputs% Oust ecause a root can#rie an# shoot #oes not mean it $no!s where to #rie or when to shoot% :oth

shootin" an# #riin" affect each other as !e% &here you #rie affects ho! easyyour shot is, an# shootin" forces a root to stop #riin"% ;n or#er to optimaycomine the t!o s$is, the hi##en no#es of the t!o net!or$s must share informationso that the ne! comine# expert can ma$e intei"ent #ecisions an# comine the t!os$is effectivey% &e hypothesi/e that '(AT has the capaiity of searchin" for theri"ht interconnec)tions et!een t!o #istinct net!or$s to create an inte"rate#supernet!or$ that ta$es a#vanta"e of the expertise of oth its component net!or$s%

Finay, !e !ou# i$e to estaish a characteri/ation of !hat '(AT is est suite#

for% The experimenta resuts sho! the #ifference et!een (S4 an# '(AT is si"nifi)

canty hi"her on the har#est poe aancin" tas$% This resut impies that evovin" #i)

verse topoo"ies is particuary suite# for proems !here other metho#s are i$ey to

"et stuc$% Such proems may e #eceptive, meanin" oca optima have ar"e asins

of attraction compare# to "oa optima, an# the "oa optima are si"nificanty

#ifferent from the oca optima *.o#er", 18+% :ecause '(AT can a!ays a##more structure, it is not necessariy trappe# even if the current !ei"hts of a net!or$srepresent a oca optimum in fitness space% :y a##in" a##itiona structure, '(AT a##s

ne! #imensions to !ei"ht space, therey openin" up potentia ne! avenues forescape% &e pan to test '(AT on proems !ith varyin" fitness an#scapes to "et a

etter i#ea of the $in#s of proems the metho# tac$es est%



34/37


8 +on"lusion

The main concusion is that '(AT is a po!erfu metho# for artificiay evovin" neura

net!or$s% '(AT #emonstrates that evovin" topoo"y aon" !ith !ei"hts can e ma#e

a maHor a#vanta"e% (xperimenta comparisons verify that such evoution is severa

times more efficient than the neuroevoution metho#s so far% Aation stu#ies sho!

that historica mar$in"s, protection of innovation throu"h speciation, an# incrementa

"ro!th from minima structure a !or$ to"ether to pro#uce a system that is capae of

evovin" soutions of minima compexity% '(AT stren"thens the anao"y et!een

.As an# natura evoution y oth optimi/in" and complexifying soutionssimutaneousy% &e eieve that the capacity to compexify soutions over the course

of evoution offers the possiiity of continua competitive coevoution an# evoution of

cominations of experts in the future%

A"knowle#gments

This research !as supporte# in part y the 'ationa Science Foun#ation un#er "rant

;;S)


35/37


Fumer, :% an# 0ii$$uainen, ?% *12+% Usin" mar$er)ase# "enetic enco#in" of neura net)!or$s to evove finite)state ehaviour% ;n Earea, F% O% an# :our"ine, 4%, e#itors, Toward a"ractice of Autonomous Systems/ "roceedings of the 5irst European (onference on

Artificial 'ife,pa"es 255232, 0;T 4ress, Camri#"e, 0assachusetts%

.o#er", D% (% *18+% !enetic Algorithms in Search; @ptimi$ation and 8achine 'earning%A##ison)&esey, ?ea#in", 0assachusetts%

.o#er", D% (% an# ?ichar#son, O% *187+% .enetic a"orithms !ith sharin" for mutimo#a func)tion

optimi/ation% ;n .refenstette, O% O%, e#itor, "roceedings of the Second 7nternational (onfer-enceon !enetic Algorithms, pa"es 1G815G, 0or"an 6aufmann, San Francisco, Caifornia%

.ome/, F% an# 0ii$$uainen, ?% *17+% ;ncrementa evoution of compex "enera ehavior%Adap-tive )ehavior, 5=-17-G2%

.ome/, F% an# 0ii$$uainen, ?% *1+% Sovin" non)0ar$ovian contro tas$s !ith neuroevou)tion% ;n Dean, T%, e#itor, "roceedings of the Sixteenth 7nternational oint (onference on

Artificial7ntelligence, pa"es 1-531-31, 0or"an 6aufmann, San Francisco, Caifornia%

.ome/, F% an# 0ii$$uainen, ?% *2


36/37


0ontana, D% O% an# Davis, 9% *18+% Trainin" fee#for!ar# neura net!or$s usin" "enetic a"o)rithms% ;n Sri#haran, S%, e#itor, "roceedings of the Eleventh 7nternational oint(onference onArtificial 7ntelligence, pa"es 732737, 0or"an 6aufmann, San Francisco,Caifornia%

0oriarty, D% (% *17+% Symbiotic Evolution of Neural Networs in Se%uential #ecision Tass+4h%D% the)sis, Department of Computer Sciences, The University of Texas at Austin%Technica ?eport UT)A;7)257%

0oriarty, D% (% an# 0ii$$uainen, ?% *13+% (fficient reinforcement earnin" throu"h symioticevoution% 8achine 'earning, 22=11-2%

0oriarty, D% (% an# 0ii$$uainen, ?% *17+% Formin" neura net!or$s throu"h efficient an#

a#ap)tive co)evoution Evolutionary (omputation, 5*G+=-7--%

>pit/, D% &% an# Shavi$, O% &% *17+% Connectionist theory refinement= .eneticay searchin"

the space of net!or$ topoo"ies% ournal of Artificial 7ntelligence &esearch, 3=1772n reinforcement earnin" of contro actions in noisy an# non)0ar$ovian#omains% Technica ?eport U'S&)CS()T?)G1xfor# University 4ress, >xfor#, U6%

Thierens, D% *13+% 'on)re#un#ant "enetic co#in" of neura net!or$s% ;n "roceedings of the7EEE 7nternational (onference on Evolutionary (omputation, pa"es 571575, ;(((4ress, 4iscata!ay,'e! Oersey%

&atson, O% D% et a% *187+% 8olecular )iology of the !ene 5ourth Edition% The :enHaminCummin"s 4uishin" Company, ;nc%, 0eno 4ar$, Caifornia%



37/37


&hitey, D% et a% *1-+% .enetic reinforcement earnin" for neurocontro proems% 8achine'earning, 1-=2528G%

&iean#, A% *11+% (vovin" neura net!or$ controers for unstae systems% ;n "roceedings of

the 7nternational oint (onference on Neural Networs, pa"es 33737-, ;((( 4ress,4iscata!ay,'e! Oersey%

ao, X% *1+% (vovin" artificia neura net!or$s% "roceedings of the 7EEE, 87*+=1G2-1GG7%

ao, X% an# 9iu, % *13+% To!ar#s #esi"nin" artificia neura net!or$s y evoution% Applied8athematics and (omputation, 1*1+=8-

Date post:	06-Mar-2016
Category:	Documents
Upload:	themasterofpiestuff
View:	220 times
Download:	2 times

Evolving Neural Networks Through Augmenting Topologies

Documents