koza - Kachergis · Title: koza.dvi Created Date: 191000908155607

Genetic Programming: Biologically Inspired

Computation that Creatively Solves

Non-Trivial Problems

John R. Koza, Forrest H. Bennett III, David Andre, and Martin A. Keane

Abstract. This paper describes a biologically inspired domain-independent tech-nique, called genetic programming, that automatically creates computer programsto solve problems. Starting with a primordial ooze of thousands of randomly cre-ated computer programs, genetic programming progressively breeds a populationof computer programs over a series of generations using the Darwinian principle ofnatural selection, recombination (crossover), mutation, gene duplication, gene dele-tion, and certain mechanisms of developmental biology. The technique is illustratedby applying it to a non-trivial problem involving the automatic synthesis (design) ofa lowpass �lter circuit. The evolved results are competitive with human-producedsolutions to the problem. In fact, four of the automatically created circuits ex-hibit human-level creativity and inventiveness, as evidenced by the fact that theycorrespond to four inventions that were patented between 1917 and 1936.

1 Introduction

One of the central challenges of computer science is to get a computer tosolve a problem without explicitly programming it. In particular, it would bedesirable to have a problem-independent system whose input is a high-levelstatement of a problem's requirements and whose output is a working com-puter program that solves the given problem. Paraphrasing Arthur Samuel(1959), this challenge concerns

How can computers be made to do what needs to be done, withoutbeing told exactly how to do it?

As Samuel also explained (Samuel 1983),

\The aim [is] ... to get machines to exhibit behavior, which if doneby humans, would be assumed to involve the use of intelligence."

Three questions arise:

� Can computer programs be automatically created?

� Can automatically created programs be competitive with human-producedprograms?

� Can the automatic process exhibit creativity and inventiveness?

16 J. R. Koza, F. H. Bennett III, D. Andre, and M. A. Keane

This paper provides an a�rmative answer to all three questions.Section 2 describes genetic programming. Section 3 presents a problem

involving the automatic synthesis (design) of an analog electrical circuit,namely a lowpass �lter. Section 4 details the circuit-constructing functionsused in applying genetic programming to the problem of analog circuit syn-thesis. Section 5 presents the preparatory steps required for applying geneticprogramming to the lowpass �lter problem. Section 6 shows the results.

2 Background on Genetic Programming

Genetic programming is a biologically inspired, domain-independent methodthat automatically creates a computer program from a high-level statementof a problem's requirements.

John Holland's pioneering book Adaptation in Natural and Arti�cial Sys-

tems (1975) described a domain-independent algorithm, called the geneticalgorithm, based on an evolutionary process involving natural selection, re-combination, and mutation. In the most commonly used form of the geneticalgorithm, each point in the search space of the given problem is encodedinto a �xed-length string of characters reminiscent of a strand of DNA. Thegenetic algorithm then conducts a search in the space of �xed-length charac-ter strings to �nd the best (or at least a very good) solution to the problemby genetically breeding a population of character strings over a number ofgenerations. Numerous practical problems can be solved using the genetic al-gorithm. Recent work in the �eld of genetic algorithms is described in Gold-berg 1989, Michalewicz 1996, Mitchell 1996, Gen and Cheng 1997, and Back1997.

Genetic programming is an extension of the genetic algorithm in whichthe population consists of computer programs. The goal of genetic program-ming is to provide a domain-independent problem-solving method that au-tomatically creates a computer program from a high-level statement of aproblem's requirements. Starting with a primordial ooze of thousands ofrandomly created computer programs, genetic programming progressivelybreeds a population of computer programs over a series of generations us-ing the Darwinian principle of natural selection, recombination (crossover),mutation, gene duplication, gene deletion, and certain mechanisms of de-velopmental biology. Work on genetic programming is described in Koza1992; Koza and Rice 1992; Kinnear 1994; Koza 1994a; Koza 1994b; An-geline and Kinnear 1996; Koza, Goldberg, Fogel, and Riolo 1996; Koza etal 1997; Koza et al 1998; Banzhaf, Poli, Schoenauer, and Fogarty 1998,Banzhaf, Nordin, Keller, and Francone 1998; Spector, Langdon, O'Reilly,and Angeline 1999; Koza, Bennett, Andre, and Keane 1999, and on the WorldWide Web at www.genetic-programming.org. The computer programs arecompositions of functions (e.g., arithmetic operations, conditional operators,problem-speci�c functions) and terminals (e.g., external inputs, constants,

Genetic Programming: Biologically Inspired Computation 17

zero-argument functions). The programs may be thought of as trees whosepoints are labeled with the functions and whose leaves are labeled with theterminals.

Genetic programming breeds computer programs to solve problems byexecuting the following three steps:

1. Randomly create an initial population of individual computer programs.2. Iteratively perform the following substeps (called a generation) on the

population of programs until the termination criterion has been satis�ed:

(a) Assign a �tness value to each individual program in the populationusing the �tness measure.

(b) Create a new population of individual programs by applying the fol-lowing three genetic operations. The genetic operations are applied toone or two individuals in the population selected with a probabilitybased on �tness (with reselection allowed).i. Reproduction: Reproduce an existing individual by copying it intothe new population.

ii. Crossover : Create two new individual programs from two exist-ing parental individuals by genetically recombining subtrees fromeach program using the crossover operation at randomly chosencrossover points in the parental individuals.

iii. Mutation: Create a new individual from an existing parental in-dividual by randomly mutating one randomly chosen subtree ofthe parental individual.

3. Designate the individual computer program that is identi�ed by themethod of result designation (e.g., the best-so-far individual) as the resultof the run of genetic programming. This result may represent a solution(or an approximate solution) to the problem.

Genetic programming starts with an initial population (generation 0) of ran-domly generated computer programs composed of the given primitive func-tions and terminals. The creation of this initial random population is a blindrandom search of the space of computer programs.

The computer programs in generation 0 of a run of genetic programmingwill almost always have exceedingly poor �tness. Nonetheless, some individu-als in the population will turn out to be somewhat more �t than others. Thesedi�erences in performance are then exploited so as to direct the search intopromising areas of the search space. The Darwinian principle of reproductionand survival of the �ttest and the genetic operation of crossover (augmentedby occasional mutation) are used to create a new population of o�springprograms from the current population of computer programs.

The reproduction operation involves probabilistically selecting a computerprogram from the current population of programs based on �tness (i.e., thebetter the �tness, the more likely the individual is to be selected) and allowingit to survive by copying it into the new population.


The crossover operation creates new o�spring computer programs fromtwo parental programs selected probabilistically based on �tness. The parentalprograms in genetic programming are typically of di�erent sizes and shapes.The o�spring programs are composed of subexpressions (subtrees, subpro-grams) from their parents.

For example, consider the following computer program (presented here asa LISP S-expression):

(+ (* 0.234 Z) (- X 0.789)),

which one would ordinarily write as

0:234Z + x� 0:789.

This program takes two inputs (X and Z) and produces a oating pointoutput.

Also, consider a second program:

(* (* Z Y) (+ Y (* 0.314 Z))).

One crossover point is randomly and independently chosen in each parent.Suppose that the crossover points are the * in the �rst parent and the + in thesecond parent. These two crossover fragments are the subexpressions rootedat the crossover points and are underlined in the above two parental computerprograms.

The two o�spring resulting from crossover are

(+ (+ Y (* 0.314 Z)) (- X 0.789))

and

(* (* Z Y) (* 0.234 Z)).

Crossover creates new computer programs using parts of existing parentalprograms. Because entire sub-trees are swapped, the crossover operation pro-duces syntactically and semantically valid programs as o�spring regardless ofthe choice of the two crossover points. The two o�spring here are typical ofthe o�spring produced by the crossover operation in that they are di�erentfrom both of their parents and di�erent from each other in size and shape.Because programs are selected to participate in the crossover operation witha probability based on �tness, crossover allocates future trials to regions ofthe search space whose programs contain parts of promising programs.

The mutation operation creates an o�spring computer program from oneparental program selected based on �tness. One mutation point is randomlyand independently chosen and the subtree occurring at that point is deleted.Then, a new subtree is grown at that point using the same growth procedureas was originally used to create the initial random population.

For example, consider the following parental program (presented as aLISP S-expression) composed of Boolean functions and terminals:


(OR (AND D2 D1 (NOR D0 D1))).

Suppose that the AND is randomly chosen as the mutation point (out of theseven points in the program tree). The three-point subtree rooted at the ANDcorresponds to the underlined portion of the LISP S-expression above. Thesubtree rooted at the chosen mutation point is deleted. In this example, thesubtree consists of the three points (AND D2 D1). A new subtree, such as

(AND (NOT D0) (NOT D1)),

is randomly grown using the available functions and terminals and in-serted in lieu of the subtree (AND D2 D1). The result of the mutation oper-ation is

(OR ((AND (NOT D0) (NOT D1)) (NOR D0 D1))).

The o�spring here is typical of the o�spring produced by the mutationoperation in that they is di�erent than its parent in size and shape.

After the genetic operations are performed on the current population, thepopulation of o�spring (i.e., the new generation) replaces the old population(i.e., the old generation). Each individual in the new population of programs isthen measured for �tness, and the process is repeated over many generations.

The dynamic variability of the computer programs that are created duringthe run are important features of genetic programming. It is often di�cultand unnatural to try to specify or restrict the size and shape of the eventualsolution in advance.

Scalable automated programming requires some hierarchical mechanismto exploit, by reuse and parameterization, the regularities, symmetries, ho-mogeneities, similarities, patterns, and modularities inherent in problem en-vironments. Subroutines provide this mechanism in ordinary computer pro-grams. Automatically de�ned functions (Koza 1994a, 1994b) implement thismechanism within the context of genetic programming. Automatically de�nedfunctions are implemented by establishing a constrained syntactic structurefor the individual programs in the population. Each multi-part program in thepopulation contains one (or more) automatically de�ned functions and one(or more) main result-producing branches. The result-producing branch usu-ally has the ability to call one or more of the automatically de�ned functions.An automatically de�ned function may have the ability to refer hierarchicallyto other already-de�ned automatically de�ned functions.

The initial random generation is created so that every individual programin the population consists of automatically de�ned function(s) and result-producing branch(es) in accordance with the problem's constrained syntacticstructure. Since a constrained syntactic structure is involved, crossover andmutation are performed so as to preserve this syntactic structure in all o�-spring.


Architecture-altering operations enhance genetic programming with au-tomatically de�ned functions by providing a way to automatically determinethe number of such automatically de�ned functions, the number of argumentsthat each automatically de�ned function possesses, and the nature of the hier-archical references, if any, among such automatically de�ned functions (Koza1995). These operations include branch duplication, argument duplication,branch creation, argument creation, branch deletion, and argument deletion.The architecture-altering operations are motivated by the naturally occur-ring mechanism of gene duplication that creates new proteins (and hencenew structures and new behaviors in living things) as described by SusumuOhno in Evolution by Gene Duplication (1970). Details are found in Koza,Bennett, Andre, and Keane 1999.

Genetic programming has been applied to numerous problems in �eldssuch as system identi�cation, control, classi�cation, design, optimization, andautomatic programming.

3 Statement of the Illustrative Problem

Design is a major activity of practicing engineers. The design process entailscreation of a complex structure to satisfy user-de�ned requirements. Sincethe design process typically entails tradeo�s between competing considera-tions, the end product of the process is usually a satisfactory and compliantdesign as opposed to a perfect design. Design is usually viewed as requiringcreativity and human intelligence. Consequently, the �eld of design is a sourceof challenging problems for automated techniques of machine intelligence. Inparticular, design problems are useful for determining whether an automatedtechnique can produce results that are competitive with human-producedresults.

The design (synthesis) of analog electrical circuits is especially challeng-ing. The design process for analog circuits begins with a high-level descrip-tion of the circuit's desired behavior and characteristics and entails creationof both the topology and the sizing of a satisfactory circuit. The topologycomprises the gross number of components in the circuit, the type of eachcomponent (e.g., a capacitor), and a list of all connections between the com-ponents. The sizing involves specifying the values (typically numerical) ofeach of the circuit's components.

Although considerable progress has been made in automating the syn-thesis of certain categories of purely digital circuits, the synthesis of analogcircuits and mixed analog-digital circuits has not proved to be as amenableto automation. There is no previously known general technique for automat-ically creating an analog circuit from a high-level statement of the designgoals of the circuit. Describing \the analog dilemma," O. Aaserud and I.Ring Nielsen (1995) noted


\Analog designers are few and far between. In contrast to digital de-sign, most of the analog circuits are still handcrafted by the expertsor so-called `zahs' of analog design. The design process is charac-terized by a combination of experience and intuition and requires athorough knowledge of the process characteristics and the detailedspeci�cations of the actual product.\Analog circuit design is known to be a knowledge-intensive, multi-phase, iterative task, which usually stretches over a signi�cant periodof time and is performed by designers with a large portfolio of skills.It is therefore considered by many to be a form of art rather than ascience."

This paper focuses on one particular problem of analog circuit synthe-sis, namely the design of a lowpass �lter circuit composed of capacitors andinductors. A simple �lter is a one-input, one-output electronic circuit thatreceives a signal as its input and passes the frequency components of theincoming signal that lie in a speci�ed range (called the passband) while sup-pressing the frequency components that lie in all other frequency ranges (thestopband). In particular, the goal is to design a lowpass �lter that passes allfrequencies below 1,000 Hertz (Hz) and suppresses all frequencies above 2,000Hz.

The approach described in this paper has been applied to many otherproblems of analog circuit synthesis, including the design of a ampli�ers,computational circuits, a temperature-sensing circuit, a voltage reference cir-cuit, a time-optimal robot controller circuit, a di�cult-to-design asymmetricbandpass �lter, a crossover �lter, a double passband �lter, bandstop �lter,frequency discriminator circuits, and a frequency-measuring circuit (as de-scribed in detail in Koza, Bennett, Andre, and Keane 1999).

4 Applying Genetic Programming to the Problem

Genetic programming can be applied to the problem of synthesizing circuitsif a mapping is established between the program trees (rooted, point-labeledtrees | that is, acyclic graphs | with ordered branches) used in geneticprogramming and the labeled cyclic graphs germane to electrical circuits.The principles of developmental biology provide the motivation for mappingtrees into circuits by means of a developmental process that begins with asimple embryo. For circuits, the embryo typically includes �xed wires thatconnect the inputs and outputs of the particular circuit being designed andcertain �xed components (such as source and load resistors). Until these wiresare modi�ed, the circuit does not produce interesting output. An electricalcircuit is developed by progressively applying the functions in a circuit-con-structing program tree to the modi�able wires of the embryo (and, duringthe developmental process, to new components and modi�able wires).


An electrical circuit is created by executing the functions in a circuit-con-structing program tree. The functions are progressively applied in a devel-opmental process to the embryo and its successors until all of the functionsin the program tree are executed. That is, the functions in the circuit-con-structing program tree progressively side-e�ect the embryo and its successorsuntil a fully developed circuit eventually emerges. The functions are appliedin a breadth-�rst order.

The functions in the circuit-constructing program trees are divided into�ve categories: (1) topology-modifying functions that alter the circuit topol-ogy, (2) component-creating functions that insert components into the circuit,(3) development-controlling functions that control the development processby which the embryo and its successors is changed into a fully developedcircuit, (4) arithmetic-performing functions that appear in subtrees as argu-ment(s) to the component-creating functions and specify the numerical valueof the component, and (5) automatically de�ned functions that appear in theautomatically de�ned functions and potentially enable certain substructuresof the circuit to be reused (with parameterization).

Each branch of the program tree is created in accordance with a con-strained syntactic structure. Each branch is composed of topology-modifyingfunctions, component-creating functions, development-controlling functions,and terminals. Component-creating functions typically have one arithmetic-performing subtree, while topology-modifying functions, and development-controlling functions do not. Component-creating functions and topology-modifying functions are internal points of their branches and possess one ormore arguments (construction-continuing subtrees) that continue the devel-opmental process. The syntactic validity of this constrained syntactic struc-ture is preserved using structure-preserving crossover with point typing. Fordetails, see Koza, Bennett, Andre, and Keane 1999.

4.1 The Embryonic Circuit

An electrical circuit is created by executing a circuit-constructing programtree that contains various component-creating, topology-modifying, and development-controlling functions. Each tree in the population creates one circuit. Thespeci�c embryo used depends on the number of inputs and outputs.

Figure 1 shows a one-input, one-output embryonic (initial) circuit inwhich VSOURCE is the input signal and VOUT is the output signal (theprobe point). The circuit is driven by an incoming alternating circuit sourceVSOURCE. There is a �xed load resistor RLOAD and a �xed source resis-tor RSOURCE in the embryo. In addition to the �xed components, thereare two modi�able wires, Z0 and Z1. All development originates from thesemodi�able wires.


Fig. 1. One-input, one-output embryonic (initial) circuit .

4.2 Component-Creating Functions

The component-creating functions insert a component into the developingcircuit and assign component value(s) to the component.

Each component-creating function has a writing head that points to anassociated highlighted component in the developing circuit and modi�es thatcomponent in a speci�ed manner. The construction-continuing subtree ofeach component-creating function points to a successor function or terminalin the circuit-constructing program tree.

The arithmetic-performing subtree of a component-creating function con-sists of a composition of arithmetic functions (addition and subtraction) andrandom constants (in the range {1:000 to +1:000). The arithmetic-performingsubtree speci�es the numerical value of a component by returning a oating-point value that is interpreted on a logarithmic scale as the value for thecomponent in a range of 10 orders of magnitude (using a unit of measurethat is appropriate for the particular type of component).

The two-argument resistor-creating R function causes the highlighted com-ponent to be changed into a resistor. The value of the resistor in kilo Ohmsis speci�ed by its arithmetic-performing subtree.

Figure 2 shows a modi�able wire Z0 connecting nodes 1 and 2 of a partialcircuit containing four capacitors (C2, C3, C4, and C5). Figure 3 shows theresult of applying the R function to the modi�able wire Z0 of �gure 2.

Similarly, the two-argument capacitor-creating C function causes the high-lighted component to be changed into a capacitor whose value in micro-Farads is speci�ed by its arithmetic-performing subtree. In addition, the two-argument inductor-creating L function causes the highlighted component tobe changed into an inductor whose value in micro-Henrys is speci�ed by itsarithmetic-performing subtree.


Fig. 2. Modi�able wire Z0.

Fig. 3. Result of applying the R function.

4.3 Topology-Modifying Functions

Each topology-modifying function in a program tree points to an associatedhighlighted component and modi�es the topology of the developing circuit.

The three-argument SERIES division function creates a series compositionof the modi�able wire or modi�able component with which it is associated,a copy of the modi�able wire or modi�able component with which it is asso-ciated, one new modi�able wire (with a writing head), and two new nodes.Figure 4 shows the result of applying the SERIES function to the resistor R1from �gure 3. After execution of the SERIES function, resistors R1 and R7and modi�able wire Z6 remain modi�able. All three are associated with thetop-most function in one of the three construction-continuing subtrees of theSERIES function.

The reader is referred to Koza, Bennett, Andre, and Keane 1999 for adetailed description of all the circuit-constructing functions mentioned herein.


Fig. 4. Result after applying the SERIES division function.

The four-argument PARALLEL0 parallel division function creates a parallelcomposition consisting of the modi�able wire or modi�able component withwhich it is associated, a copy of the modi�able wire or modi�able componentwith which it is associated, two new modi�able wires (each with a writinghead), and two new nodes. There are potentially two topologically distinctoutcomes of a parallel division. Since we want the outcome of all circuit-constructing functions to be deterministic, there are two members (calledPARALLEL0 and PARALLEL1) in the PARALLEL family of topology-modifyingfunctions. The two functions operate di�erently depending on degree andnumbering of the preexisting components in the developing circuit. The useof the two functions breaks the symmetry between the potentially distinctoutcomes.

The one-argument polarity-reversing FLIP function reverses the polarityof the highlighted component.

The two-argument TWO GROUND (\ground") function enables any part ofa circuit to be connected to ground. The TWO GROUND function creates anew node and a composition of two modi�able wires and one nonmodi�ablewire such that the nonmodi�able wire makes an unconditional connection toground.

The eight two-argument functions in the TWO VIA family of functions(called TWO VIA0, ..., TWO VIA7), each create a new node and a composition oftwo modi�able wires and one nonmodi�able wire such that the nonmodi�ablewire makes a connection, called a via, to a designated one of eight imaginarynumbered layers (0 to 7) of an imaginary silicon wafer on which the circuitresides. the TWO VIA functions provide a way to connect distant parts of acircuit.

The zero-argument SAFE CUT function causes the highlighted componentto be removed from the circuit provided that the degree of the nodes at both


ends of the highlighted component is three (i.e., no dangling components orwires are created).

4.4 Development-Controlling Functions

The one-argument NOOP (\No Operation") function has no e�ect on the mod-i�able wire or modi�able component with which it is associated; however,it has the e�ect of delaying activity on the developmental path on whichit appears in relation to other developmental paths in the overall circuit-constructing program tree.

The zero-argument END function makes the modi�able wire or modi�-able component with which it is associated non-modi�able (thereby endinga particular developmental path).

4.5 Example of Developmental Process

Figure 5 is an illustrative circuit-constructing program tree shown as a rooted,point-labeled tree with ordered branches. The overall program consists oftwo main result-producing branches joined by a connective LIST function(labeled 1 in the �gure). The �rst (left) result-producing branch is rootedat the capacitor-creating C function (labeled 2). The second result-producingbranch is rooted at the polarity-reversing FLIP function (labeled 3). This�gure also contains four occurrences of the inductor-creating L function (at17, 11, 20, and 12). The �gure contains two occurrences of the topology-modifying SERIES function (at 5 and 10). The �gure also contains �ve oc-currences of the development-controlling END function (at 15, 25, 27, 31, and22) and one occurrence of the development-controlling \no operation" NOP

function (at 6). There is a seven-point arithmetic-performing subtree at 4under the capacitor-creating C function at 4. Similarly, there is a three-pointarithmetic-performing subtree at 19 under the inductor-creating L function at11. There are also one-point arithmetic-performing subtrees (i.e., constants)at 26, 30, and 21. Additional details can be found in Koza, Bennett, Andre,and Keane 1999.

5 Preparatory Steps

Before applying genetic programming to a problem of circuit design, sevenmajor preparatory steps are required: (1) identify the embryonic circuit, (2)determine the architecture of the circuit-constructing program trees, (3) iden-tify the primitive functions of the program trees, (4) identify the terminals ofthe program trees, (5) create the �tness measure, (6) choose control param-eters for the run, and (7) determine the termination criterion and method ofresult designation.


– 0.880 END FLIP L END – L -0.657 END

-0.875 -0.113 END -0.277 END -0.640 0.749 -0.123 END

–0.963 FLIP SERIES L L

– SERIES NOP

C FLIP

LIST1

2 3

4 5 6

8

7

9 1 0 1 1 1 2

1 3 1 4 1 5 1 7 1 81 6 1 9 2 0 2 1

2 2

2 3 2 4 2 5 2 6 2 7 2 8 2 9 3 0 3 1

Fig. 5. Illustrative circuit-constructing program tree.

5.1 Embryonic Circuit

The embryonic circuit used on a particular problem depends on the circuit'snumber of inputs and outputs. A one-input, one-output embryo with twomodi�able wires (�gure 1) was used.

5.2 Program Architecture

Since there is one result-producing branch in the program tree for each modi�-able wire in the embryo, the architecture of each circuit-constructing programtree depends on the embryonic circuit. Two result-producing branches wereused for the �lter problems.

The architecture of each circuit-constructing program tree also dependson the use, if any, of automatically de�ned functions. Automatically de�nedfunctions provide a mechanism enabling certain substructures to be reusedand are described in detail in Koza, Bennett, Andre, and Keane 1999. Au-tomatically de�ned functions and architecture-altering operations were notused here.

5.3 Function and Terminal Sets

The function set for each design problem depends on the type of electricalcomponents that are to be used for constructing the circuit.

The function set included two component-creating functions (for induc-tors and capacitors), topology-modifying functions (for series and paralleldivisions and for ipping components), one development-controlling function(\no operation"), functions for creating a via to ground, and functions for con-necting pairs of points. That is, the function set, Fccs, for each construction-continuing subtree was


Fccs =fL, C, SERIES, PARALLEL0, PARALLEL1, FLIP, NOOP, TWO GROUND,

TWO VIA0 , TWO VIA1, TWO VIA2, TWO VIA3, TWO VIA4, TWO VIA5,

TWO VIA6, TWO VIA7g.

The terminal set, Tccs, for each construction-continuing subtree was

Tccs = END; SAFE CUT.

The terminal set, Taps, for each arithmetic-performing subtree consisted of

Taps = <,where < represents oating-point random constants from {1:0 to +1:0. Thefunction set, Faps, for each arithmetic-performing subtree was,

Faps = +; -.

The terminal and function sets were identical for all result-producingbranches for a particular problem.

5.4 Fitness Measure

The evolutionary process is driven by the �tness measure. Each individualcomputer program in the population is executed and then evaluated, usingthe �tness measure. The nature of the �tness measure varies with the prob-lem. The high-level statement of desired circuit behavior is translated intoa well-de�ned measurable quantity that can be used by genetic program-ming to guide the evolutionary process. The evaluation of each individualcircuit-constructing program tree in the population begins with its execu-tion. This execution progressively applies the functions in each program treeto an embryonic circuit, thereby creating a fully developed circuit. A netlistis created that identi�es each component of the developed circuit, the nodesto which each component is connected, and the value of each component. Thenetlist becomes the input to our modi�ed version of the 217; 000-line SPICE(Simulation Program with Integrated Circuit Emphasis) simulation program(Quarles, Newton, Pederson, and Sangiovanni-Vincentelli 1994). SPICE thendetermines the behavior of the circuit. It was necessary to make consider-able modi�cations in SPICE so that it could run as a submodule within thegenetic programming system.

The desired lowpass �lter has a passband below 1; 000 Hz and a stopbandabove 2; 000 Hz. The circuit is driven by an incoming AC voltage source witha 2 volt amplitude. In this problem, a voltage in the passband of exactly 1volt and a voltage in the stopband of exactly 0 volts is regarded as ideal. The(preferably small) variation within the passband is called the passband ripple.Similarly, the incoming signal is never fully reduced to zero in the stopbandof an actual �lter. The (preferably small) variation within the stopband iscalled the stopband ripple. A voltage in the passband of between 970 millivolts


and 1 volt (i.e., a passband ripple of 30 millivolts or less) and a voltage inthe stopband of between 0 volts and 1 millivolts (i.e., a stopband ripple of1 millivolts or less) is regarded as acceptable. Any voltage lower than 970millivolts in the passband and any voltage above 1 millivolts in the stopbandis regarded as unacceptable.

Since the high-level statement of behavior for the desired circuit is ex-pressed in terms of frequencies, the voltage VOUT is measured in the fre-quency domain. SPICE performs an AC small signal analysis and reports thecircuit's behavior over �ve decades (between 1 Hz and 100; 000 Hz) with eachdecade being divided into 20 parts (using a logarithmic scale), so that thereare a total of 101 �tness cases.

Fitness is measured in terms of the sum over these cases of the absoluteweighted deviation between the actual value of the voltage that is producedby the circuit at the probe point VOUT and the target value for voltage.The smaller the value of �tness, the better. A �tness of zero represents an(unattainable) ideal �lter.

Speci�cally, the standardized �tness is

F (t) =100X

i=0

W (d(fi); fi)d(fi);

where fi is the frequency of �tness case i; d(x) is the absolute value of thedi�erence between the target and observed values at frequency x; andW (y; x)is the weighting for di�erence y at frequency x.

The �tness measure is designed to not penalize ideal values, to slightlypenalize every acceptable deviation, and to heavily penalize every unaccept-able deviation. Speci�cally, the procedure for each of the 61 points in the3-decade interval between 1 Hz and 1; 000 Hz for the intended passband is asfollows:

� If the voltage equals the ideal value of 1:0 volt in this interval, the devi-ation is 0:0.

� If the voltage is between 970 millivolts and 1 volt, the absolute value ofthe deviation from 1 volt is weighted by a factor of 1:0.

� If the voltage is less than 970 millivolts, the absolute value of the deviationfrom 1 volt is weighted by a factor of 10:0.

The acceptable and unacceptable deviations for each of the 35 points from2; 000 Hz to 100; 000 Hz in the intended stopband are similarly weighed (by1:0 or 10:0) based on the amount of deviation from the ideal voltage of 0volts and the acceptable deviation of 1 millivolts.

For each of the �ve \don't care" points between 1; 000 and 2; 000 Hz, thedeviation is deemed to be zero.

The number of \hits" for this problem (and all other problems herein) isde�ned as the number of �tness cases for which the voltage is acceptable orideal or that lie in the \don't care" band (for a �lter).


Many of the random initial circuits and many that are created by thecrossover and mutation operations in subsequent generations cannot be sim-ulated by SPICE. These circuits receive a high penalty value of �tness (108)and become the worst-of-generation programs for each generation.

5.5 Control Parameters

The population size, M , was 320; 000. The probability of crossover was ap-proximately 89%; reproduction 10%; and mutation 1%. Our usual controlparameters were used (Koza, Bennett, Andre, and Keane 1999, AppendixD).

5.6 Termination Criterion and Results Designation

The maximum number of generations, G, is set to an arbitrary large number(e.g., 501) and the run was manually monitored and manually terminatedwhen the �tness of the best-of-generation individual appeared to have reacheda plateau. The best-so-far individual is harvested and designated as the resultof the run.

5.7 Implementation on Parallel Computer

The problem was run on a medium-grained parallel Parsytec computer sys-tem consisting of 64 80-MHz PowerPC 601 processors arranged in an 8 by8 toroidal mesh with a host PC Pentium type computer. The distributedgenetic algorithm (Andre and Koza 1996) with unsynchronized generationswas used with a population size of Q = 5; 000 at each of the D = 64 demes(semi-isolated subpopulations) for a total population,M , of 320; 000. On eachgeneration, four boatloads of emigrants, each consisting of B = 2% (the mi-gration rate) of the node's subpopulation (selected on the basis of �tness)were dispatched to each of the four adjacent processing nodes.

6 Results

The creation of the initial random population is a blind random search of thesearch space of the problem. The best circuit from generation 0 has a �tnessof 61:7 and scores 52 hits (out of 101).

Figures 6, 7, 8, and 9 show the behavior of the best circuits from genera-tion 0, 10, 15, and 49, respectively, of one run of genetic programming. Thehorizontal axis represents �ve decades of frequencies from 1 Hz to 100; 000Hz on a logarithmic scale. The vertical axis represents output voltage ona linear scale. Excluding the �xed source and load resistors of the test �x-ture of the embryonic circuit, the best-of-generation circuit from generation0 consists of only a lone 358 nF capacitor that shunts the incoming signal


to ground. A good �lter cannot be created by a single capacitor. However,even a single capacitor di�erentially passes higher frequencies to ground andperforms a certain amount of �ltering. Figure 6 shows that the best circuitfrom generation 0 bears some resemblance to the desired lowpass �lter inthat it passes frequencies up to about 70 Hz at nearly a full volt and it al-most fully suppresses frequencies near 100; 000 Hz. However, its transitionregion is exceedingly leisurely. Nonetheless, in the valley of the blind, theone-eyed man is king. Moreover, as will be seen momentarily, this modestbeginning serves as a building block that will become incorporated in the100%-compliant lowpass �lter that will eventually be evolved.

The evolutionary process produces better and better individuals as therun progresses. For example, the best circuit from generation 10 has inductorsin series with the incoming signal as well as a single capacitor shunted toground. Figure 7 shows that that the frequencies up to about 200 Hz arepassed at nearly full voltage and that frequencies above 10; 000 Hz are almostfully suppressed. Figure 8 shows that the best circuit from generation 15 (withtwo inductors in series with the incoming signal and three capacitors shuntedto ground) comes closer to meeting the requirements of this design problem.

1.0Hz 10Hz 100 Hz 1.0 KHz 10K Hz 100 KHz

Fre que ncyV(RLOAD:1)

1.0V

0.5V

0V

Fig. 6. Frequency domain behavior of the best circuit of generation 0.

6.1 Campbell 1917 Ladder Filter Patent

The best circuit (�gure 10) of generation 49 from this run is 100% compli-ant with the problem's design requirements in the sense that it scored 101hits (out of 101). It has a near-zero �tness of 0:00781 (about �ve orders ofmagnitude better than the best circuit of generation 0). As can be seen, thisevolved circuit consists of seven inductors (L5, L10, L22, L28, L31, L25, and


1.0Hz 10Hz 100 Hz 1.0 KHz 10K Hz 100 KHz

Fre que ncyV(L10:2 )

1.0V

0.5V

0V


1.0H z 10Hz 100Hz 1 .0KHz 10KHz 1 00KHz

F requencyV(L12:2)

1 .0V

0.5V

0 V


L13) arranged horizontally across the top of the �gure \in series" with theincoming signal VSOURCE and the source resistor RSOURCE. It also con-tains seven capacitors (C12, C24, C30, C3, C33, C27, and C15) that are eachshunted to ground. This circuit is a classical ladder �lter with seven rungs(Williams and Taylor 1995).

Figure 9 shows the behavior in the frequency domain of this evolvedlowpass �lter. As can be seen, the 100%-compliant lowpass �lter deliversa voltage of essentially 1 Volt in the entire passband from 1 Hz to 1; 000Hz and suppresses the voltage of essentially 0 Volts in the entire stopbandstarting at 2; 000 Hz. There is a sharp drop-o� from 1 Volt to 0 Volts in thetransitional (\don't care") region between 1; 000 Hz and 2; 000 Hz.


1.0Hz 10Hz 100Hz 1.0KHz 10KHz 100KHz

FrequencyV(RLOAD:1)

1.0V

0.5V

0V

Fig. 9. Frequency domain behavior of 100%-compliant seven-rung ladder circuitfrom generation 49.

Fig. 10. Evolved seven-rung ladder lowpass �lter.

The circuit of �gure 10 has the recognizable features of the circuit forwhich George Campbell of American Telephone and Telegraph received U.S. patent 1; 227; 113 in 1917 (Campbell 1917). Claim 2 of Campbell's patentcovered,

\An electric wave �lter consisting of a connecting line of negligible at-tenuation composed of a plurality of sections, each section including acapacity element and an inductance element, one of said elements ofeach section being in series with the line and the other in shunt acrossthe line, said capacity and inductance elements having precomputedvalues dependent upon the upper limiting frequency and the lowerlimiting frequency of a range of frequencies it is desired to trans-mit without attenuation, the values of said capacity and inductanceelements being so proportioned that the structure transmits withpractically negligible attenuation sinusoidal currents of all frequen-cies lying between said two limiting frequencies, while attenuatingand approximately extinguishing currents of neighboring frequencieslying outside of said limiting frequencies."


An examination of the evolved circuit of �gure 10 shows that it indeed consistsof \a plurality of sections" (speci�cally, seven). In the �gure, \Each sectioninclude[es] a capacity element and an inductance element." Speci�cally, the�rst of the seven sections consists of inductor L5 and capacitor C12; thesecond section consists of inductor L10 and capacitor C24; and so forth.Moreover, \one of said elements of each section [is] in series with the lineand the other in shunt across the line." Inductor L5 of the �rst section isindeed \in series with the line" and capacitor C12 is indeed \in shunt acrossthe line." This is also true for the circuit's remaining six sections. Moreover,�gure 10 herein matches �gure 7 of Campbell's 1917 patent. In addition, thiscircuit's 100% compliant behavior in the frequency domain (�gure 9 herein)con�rms the fact that the values of the inductors and capacitors are suchas to transmit \with practically negligible attenuation sinusoidal currents" ofthe passband frequencies \while attenuating and approximately extinguishingcurrents" of the stopband frequencies.

In short, genetic programming evolved an electrical circuit that infringeson the claims of Campbell's now-expired patent.

In addition to possessing the topology of the Campbell �lter, the evolvedcircuit of Figure 10 also approximately possesses the numerical values de-scribed in Campbell's 1917 patent (Campbell 1917). In fact, this evolvedcircuit is roughly equivalent to what is now known as a cascade of six identi-cal symmetric �-sections (Johnson l950). To see this, we modify the evolvedcircuit of Figure 10 in four ways.

First, we delete the 9:68 �H inductor L5 near the upper left corner ofthe �gure. The value of this inductor is more than �ve orders of magnitudesmaller than the value of the other six inductors (L10, L22, L28, L31, L25, andL13) in series across the top of the �gure. The behavior of the evolved circuitis not noticeably a�ected by this deletion for the frequencies of interest inthis problem.

Second, we replace each of the �ve identical 202 nF capacitors (C24, C30,C3, C33, C27) by a composition of two parallel 101 nF capacitors. Since thecapacitance of a composition of two parallel capacitors equals the sum ofthe two individual capacitances, the behavior of the evolved circuit is notchanged at all by these substitutions.

Third, we note that the two 86:1 nF capacitors (C12 and C15) at the twoends of the ladder are each approximately equal to the (now) ten 101 nFcapacitors. Suppose, for sake of argument, that these 12 approximately equalcapacitors are replaced by 12 equal capacitors with capacitance equal to theiraverage value (98:5 nF). The behavior of the evolved circuit is only slightlychanged by these substitutions.

Fourth, we note also that the six non-trivial inductors (L10, L22, L28,L31, L25, and L13) are approximately equal. Suppose, for sake of argument,that these six approximately equal inductors are replaced by six equal induc-


tors with inductance equal to their average value (200; 000 �H). Again, thebehavior of the evolved circuit is only slightly changed by these substitutions.

The behavior in the frequency domain of the circuit resulting from theabove four changes is almost the same as that of the evolved circuit of Fig-ure 10. In fact, the modi�ed circuit is 100%-compliant (i.e., scores 101 hits).The modi�ed circuit can be viewed as what is now known as a cascade ofsix identical symmetric �-sections. Each �-section consists of an inductor ofinductance L (where L equals 200; 000 �H) and two equal capacitors of ca-pacitance C=2 (where C equals 197 nF). In each �-section, the two 98:5 nFcapacitors constitute the vertical legs of the � and the one 200; 000 �H induc-tor constitutes the horizontal bar across the top of the �. Such �-sections arecharacterized by two key parameters. The �rst parameter is the characteristicresistance (impedance) of the �-section. This characteristic resistance shouldmatch the circuit's �xed load resistance RLOAD (1; 000 ). The second pa-rameter is the nominal cuto� frequency which separates the �lter's passbandfrom its stopband. This second parameter should lie somewhere in the tran-sition region between the end of the passband (1; 000 Hz) and the beginningof the stopband (2; 000 Hz). The characteristic resistance, R, of each of the�-sections is given by the formula

pL=C. This formula yields a characteristic

resistance, R, of 1; 008 . This value is very close to the value of the 1; 000 load resistance of this problem. The nominal cuto� frequency, fc, of eachof the �-sections of a lowpass �lter is given by the formula 1=(�

pLC). This

formula yields a nominal cuto� frequency, fc, of 1; 604 Hz (i.e., roughly inthe middle of the transition region between the passband and stopband ofthe desired lowpass �lter).

The legal criteria for obtaining a U. S. patent are that the proposedinvention be \new" and \useful" and

... the di�erences between the subject matter sought to be patentedand the prior art are such that the subject matter as a whole would[not] have been obvious at the time the invention was made to aperson having ordinary skill in the art to which said subject matterpertains. (35 United States Code 103a).

George Campbell was part of the renowned research team of the Ameri-can Telephone and Telegraph Corporation. He received a patent for his �lterin 1917 because his idea was new in 1917, because it was useful, and becausesatisifed the above statutory test for unobviousness. The fact that geneticprogramming rediscovered an electrical circuit that was unobvious \to a per-son having ordinary skill in the art" establishes that this evolved result sat-is�es Arthur Samuel's criterion (Samuel 1983) for arti�cial intelligence andmachine learning, namely

\The aim [is] ... to get machines to exhibit behavior, which if doneby humans, would be assumed to involve the use of intelligence."


6.2 Zobel 1925 \M-Derived Half Section" Patent

In another run of this same problem, a 100%-compliant circuit was evolved ingeneration 34. This evolved circuit is roughly equivalent to what is now knownas a cascade of three symmetric T-sections and an M -derived half section(Johnson 1950). To see this, we modify this evolved circuit from generation34 in three ways.

First, we insert wires in lieu of two 0:138 �H inductors (whose value isabout six orders of magnitude smaller than the value of the other inductorsin the circuit). The behavior of this slightly modi�ed evolved circuit (�gure11) is not noticeably a�ected by these changes for the frequencies of interestin this problem.

Fig. 11. Slightly modi�ed version of the evolved lowpass �lter circuit consisting ofthree symmetric T-sections and an M -derived half section.

Second, we replace each of the three 198; 000 �H inductors in the �gure(L16, L13, and L10) with a series composition of two 99; 000 �H inductors.Since the inductance of two inductors in series is equal to the sum of theirinductances, this change does not a�ect the behavior of the circuit at all. Thecircuit can now be viewed as having one incoming 85; 400 �H inductor (L5)and six 99; 000 �H inductors in series horizontally at the top of the �gure.

Third, we note also that the values of the (now) seven inductors in serieshorizontally across the top of the �gure are approximately equal. Suppose, forsake of argument, that each of these seven approximately equal inductors arereplaced by an inductor with inductance equal to their average value (97; 000�H). This change does not appreciably a�ect the behavior of the circuit forthe frequencies of interest.

After the above changes, the evolved lowpass �lter can be viewed as con-sisting of a cascade of three identical symmetric T-sections and an \M -derivedhalf section." In particular, each T-section consists of an incoming inductor ofinductance L=2 (where L equals 194; 000 �H), a junction point from which acapacitor of capacitance C (where C equals 194 nF) is shunted o� to ground,and an outgoing inductor of inductance L=2. The two inductors are the hor-izontal arms of the \T." The �nal half section (so named because it has only


one arm of a \T") has one incoming inductor of inductance 2 and a junctionpoint from which a capacitive-inductive shunt (C3 and L11) is connected toground.

The �rst three symmetric T-sections are referred to as \constant K" �ltersections (Johnson l950, page 331). Such �lter sections are characterized bytwo key parameters. The characteristic resistance, R, of each of each of thethree T-sections is given by the formula R =

pL=C. When the inductance,

L, is 194; 000 �H and the capacitance, C, is 194 nF, then the characteristicresistance, R, is 1; 000 according to this formula (i.e., equal to the valueof the actual load resistor). The nominal cuto� frequency, fc, of each of thethree T-sections of a lowpass �lter is given by the formula fc = 1=(�

pLC).

This formula yields a nominal cuto� frequency, fc, of 1; 641 Hz (which is nearthe middle of the transition band for the desired pass �lter).

In other words, both of the key parameters of the three T-sections arevery close to the canonical values of \constant K" sections designed with theaim of satisfying this problem's design requirements.

The �nal section of the evolved circuit closely approximates a section nowcalled an \M -derived half section". This �nal section is said to be \derived"because it is derived from the foregoing three identical \constant K" proto-type sections. In the derivation, m is a real constant between 0 and 1. Letm be 0.6 here. In a canonical \M -derived half section" that is derived fromthe above \constant K" prototype section, the value of the capacitor in thevertical shunt of the half section is given by the formula mC (116:4 nF). Theactual value of C3 in the evolved circuit is 117 nF. The value of the inductorin the vertical shunt of an \M -derived half section" is given by the formulaL(1 � m2)=4m. This formula yields a value of 51; 733. The actual value ofL5 in the evolved circuit is 52; 200 �H. The frequency, f1, where the atten-uation �rst becomes complete, is given by the formula f1 = fc=

p1�m2.

This formula yields a value of f1 of 2; 051 Hz (i.e., near the beginning of thedesired stopband).

Taken as a whole, the topology and component values of the evolvedcircuit are reasonably close to the canonical values for the three identicalsymmetric T-sections and a �nal \M -derived half section" that is designedwith the aim of satisfying this problem's design requirements.

Otto Zobel of American Telephone and Telegraph Company invented theidea of adding an \M -derived half section" to one or more \constant K"sections. As Zobel (1925) explains in U. S. patent 1; 538; 964,

The principal object of my invention is to provide a new and im-proved network for the purpose of transmitting electric currents hav-ing their frequency within a certain range and attenuating currentsof frequency within a di�erent range. . . . Another object of my in-vention is to provide a wave-�lter with recurrent sections not all ofwhich are alike, and having certain advantages over a wave-�lter withall its sections alike.


The advantage of Zobel's approach is a \sharper transition" in the frequencydomain behavior of the �lter. Claim 1 of Zobel's 1925 patent covers,

A wave-�lter having one or more half-sections of a certain kind andone or more other half-sections that are M-types thereof, M beingdi�erent from unity.

Claim 2 covers,

A wave-�lter having its sections and half-sections so related that theycomprise di�erent M-types of a common prototype, M having severalvalues for respectively di�erent sections and half-sections.

Claim 3 goes on to cover,

A wave-�lter having one or more half-sections of a certain kind andone or more half-sections introduced from a di�erent wave-�lter hav-ing the same characteristic and the same critical frequencies and a dif-ferent attenuation characteristic outside the free transmitting range.

Viewed as a whole, the evolved circuit here infringes the claims of Zobel's1925 patent.

6.3 Johnson 1926 \Bridged T" Patent

In another run, a 100% compliant recognizable \bridged T" arrangementwas evolved. The \bridged T" �lter topology was invented and patented byKenneth S. Johnson of Western Electric Company in 1926 (Johnson 1926).As U. S. patent 1; 611; 916 (Johnson 1926) states,

In accordance with the invention, a section of an arti�cial line,such as a wave �lter, comprises in general four impedance paths, threeof which are arranged in the form of a T network with the fourth pathbridged across the transverse arms of the T. The impedances of thisnetwork, which for convenience, will be referred to as a bridged Tnetwork, bear a de�nite relationship to a network of the series shunttype, the characteristics of which are well known.

In the forms of the invention described herein, the arms of thebridged T network consist of substantially pure reactances. Its mostuseful forms are found to be wave �lter networks in which there is asubstantially in�nite attenuation at a frequency within the band to besuppressed and the network may be designed so that this frequencyis very near the cut-o� frequency of the �lter, thus producing a verysharp separation between the transmitted and suppressed bands.

Claim 1 of patent 1; 611; 916 covers,


An electrical network comprising a pair of input terminals and a pairof output terminals, an impedance path connected directly betweenan input terminal and an output terminal, a pair of impedance pathshaving a common terminal and having their other terminals con-nected respectively to the terminals of said �rst path, and a fourthimpedance path having one terminal connected to said common ter-minal and having connections from its other terminal to the remaininginput terminal and output terminal, each of said paths containing asubstantial amount of reactance, the impedances of said network hav-ing such values that said network is the equivalent of a series-shuntnetwork having desired transmission characteristics.

The \bridged T" of �gure 12 involves L14, C3, C15, and L11. In particular,L14 is the \impedance path connected directly between an input terminal andan output terminal" that is referred to later as \the �rst path." The junctionof C3, C15, and L11 is the \common terminal." C3 and C15 are the \pair ofimpedance paths having a common terminal and having their other terminalsconnected respectively to the terminals of said �rst path." L11 is the \fourthimpedance path having one terminal connected to said common terminal andhaving connections from its other terminal to the remaining input terminaland output terminal" (namely, the input and output terminal of the sectionthat are both grounded).

Fig. 12. \Bridged T" circuit from generation 64.

6.4 Cauer 1934{1936 Elliptic Patents

In a run of this same problem using automatically de�ned functions (de-scribed in Koza, Bennett, Andre, and Keane 1999), a 100% compliant circuitemerged in generation 31. After all of the pairs and triplets of series inductorsin the evolved circuit are consolidated (as shown in �gure 13), it can be seen


that the circuit has the equivalent of six inductors horizontally across thetop of the circuit and �ve vertical shunts. Each vertical shunt consists of aninductor and a capacitor.

Fig. 13. Evolved Cauer (elliptic) �lter topology from generation 31.

This circuit has the recognizable elliptic topology that was invented andpatented by Wilhelm Cauer (1934, 1935, 1936). The Cauer �lter was a sig-ni�cant advance (both theoretically and commercially) over the earlier �lterdesigns of Campbell, Zobel, Johnson, Butterworth, and Chebychev. For ex-ample, for one commercially important set of speci�cations for telephones,a �fth-order elliptic �lter matches the behavior of a 17th-order Butterworth�lter or an eighth-order Chebychev �lter. The �fth-order elliptic �lter has oneless component than the eighth-order Chebychev �lter. As Van Valkenburg(1982, page 379) relates in connection with the history of the elliptic �lter:

Cauer �rst used his new theory in solving a �lter problem for theGerman telephone industry. His new design achieved speci�cationswith one less inductor than had ever been done before. The world�rst learned of the Cauer method not through scholarly publicationbut through a patent disclosure, which eventually reached the BellLaboratories. Legend has it that the entire Mathematics Departmentof Bell Laboratories spent the next two weeks at the New York Pub-lic library studying elliptic functions. Cauer had studied mathematicsunder Hilbert at Goettingen, and so elliptic functions and their ap-plications were familiar to him.

Genetic programming did not, of course, study mathematics under Hilbert oranybody else. Instead, the elliptic topology invented and patented by Cauer


emerged from this run of genetic programming as a natural consequenceof the problem's �tness measure and natural selection | not because therun was primed with domain knowledge about elliptic functions or �ltersor electrical circuitry. Genetic programming opportunistically reinvented theelliptic topology because necessity (�tness) is the mother of invention.

7 The Illogical Nature of Creativity and Evolution

Many computer scientists and mathematicians unquestioningly assume thatevery problem-solving technique must be logically sound, deterministic, logi-cally consistent, and parsimonious. Accordingly, most conventional methodsof arti�cial intelligence and machine learning are constructed so as to possessthese characteristics. However, in spite of this strong predisposition by com-puter scientists and mathematicians, the features of logic do not govern twoof the most important types of complex problem solving processes, namelythe invention process performed by creative humans and the evolutionaryprocess occurring in nature.

A new idea that can be logically deduced from facts that are known ina �eld, using transformations that are known in a �eld, is not considered tobe an invention. There must be what the patent law refers to as an \illogicalstep" (i.e., an unjusti�ed step) to distinguish a putative invention from thatwhich is readily deducible from that which is already known. Humans supplythe critical ingredient of \illogic" to the invention process. Interestingly, ev-eryday usage parallels the patent law concerning inventiveness: People whomechanically apply existing facts in well-known ways are summarily dismissedas being uncreative. Logical thinking is unquestionably useful for many pur-poses. It usually plays an important role in setting the stage for an invention.But, at the end of the day, logical thinking is the antithesis of invention andcreativity.

Recalling his invention in 1927 of the negative feedback ampli�er, HaroldS. Black of Bell Laboratories (1977) said,

Then came the morning of Tuesday, August 2, 1927, when the conceptof the negative feedback ampli�er came to me in a ash while I wascrossing the Hudson River on the Lackawanna Ferry, on my way towork. For more than 50 years, I have pondered how and why the ideacame, and I can't say any more today than I could that morning.All I know is that after several years of hard work on the problem,I suddenly realized that if I fed the ampli�er output back to theinput, in reverse phase, and kept the device from oscillating (singing,as we called it then), I would have exactly what I wanted: a meansof canceling out the distortion of the output. I opened my morningnewspaper and on a page of The New York Times I sketched a simplecanonical diagram of a negative feedback ampli�er plus the equationsfor the ampli�cation with feedback.


Of course, inventors are not oblivious to logic and knowledge. They do notthrash around using blind random search. Black did not try to construct thenegative feedback ampli�er from neon bulbs or doorbells. Instead, \severalyears of hard work on the problem" set the stage and brought his thinkinginto the proximity of a solution. Then, at the critical moment, Black madehis \illogical" leap. This unjusti�ed leap constituted the invention.

The design of complex entities by the evolutionary process in nature isanother important type of problem-solving that is not governed by logic. Innature, solutions to design problems are discovered by the probabilistic pro-cess of evolution and natural selection. There is nothing logical about thisprocess. Indeed, inconsistent and contradictory alternatives abound. In fact,such genetic diversity is necessary for the evolutionary process to succeed.Signi�cantly, the solutions evolved by evolution and natural selection almostalways di�er from those created by conventional methods of arti�cial intelli-gence and machine learning in one very important respect. Evolved solutionsare not brittle; they are usually able to grapple with the perpetual noveltyof real environments.

Similarly, genetic programming is not guided by the inference methods offormal logic in its search for a computer program to solve a given problem.When the goal is the automatic creation of computer programs, all of ourexperience has led us to conclude that the non-logical approach used in theinvention process and in natural evolution are far more fruitful than the logic-driven and knowledge-based principles of conventional arti�cial intelligenceand machine learning. In short, \logic considered harmful."

8 Conclusion

We illustrated genetic programming by applying it to a non-trivial problem,namely the synthesis of a design for a lowpass �lter circuit. The results werecompetitive with human-produced solutions to the problem. The results ex-hibited creativity and inventiveness and correspond to four inventions thatwere patented between 1917 and 1936.

References

1. Aaserud, O. and Nielsen, I. Ring. 1995. Trends in current analog design: Apanel debate. Analog Integrated Circuits and Signal Processing. 7(1), 5{9.

2. Andre, David and Koza, John R. 1996. Parallel genetic programming: A scal-able implementation using the transputer architecture. In Angeline, P. J. andKinnear, K. E. Jr. (editors). 1996. Advances in Genetic Programming 2. Cam-bridge: MIT Press.

3. Angeline, Peter J. and Kinnear, Kenneth E. Jr. (editors). 1996. Advances in

Genetic Programming 2. Cambridge, MA: The MIT Press.4. Back, Thomas. (editor). 1997. Genetic Algorithms: Proceedings of the Seventh

International Conference. San Francisco, CA: Morgan Kaufmann.


5. Banzhaf, Wolfgang, Nordin, Peter, Keller, Robert E., and Francone, Frank D.1998. Genetic Programming | An Introduction. San Francisco, CA: MorganKaufmann and Heidelberg: dpunkt.

6. Banzhaf, Wolfgang, Poli, Riccardo, Schoenauer, Marc, and Fogarty, TerenceC. 1998. Genetic Programming: First European Workshop. EuroGP'98. Paris,

France, April 1998 Proceedings. Paris, France. April l998. Lecture Notes inComputer Science. Volume 1391. Berlin, Germany: Springer- Verlag.

7. Black, Harold S. 1977. Inventing the negative feedback ampli�er. IEEE Spec-

trum. December 1977. Pages 55{60.8. Campbell, George A. 1917. Electric Wave Filter. Filed July 15, 1915. U. S.

Patent 1,227,113. Issued May 22, 1917.9. Cauer, Wilhelm. 1934. Arti�cial Network. U. S. Patent 1,958,742. Filed June

8, 1928 in Germany. Filed December 1, 1930 in United States. Issued May 15,1934.

10. Cauer, Wilhelm. 1935. Electric Wave Filter. U. S. Patent 1,989,545. Filed June8, 1928. Filed December 6, 1930 in United States. Issued January 29, 1935.

11. Cauer, Wilhelm. 1936. Unsymmetrical Electric Wave Filter. Filed November10, 1932 in Germany. Filed November 23, 1933 in United States. Issued July21, 1936.

12. Gen, Mitsuo and Cheng, Runwei. 1997. Genetic Algorithms and Engineering

Design. New York: John Wiley and Sons.13. Goldberg, David E. l989a. Genetic Algorithms in Search, Optimization, and

Machine Learning. Reading, MA: Addison- Wesley.14. Holland, John H. 1975. Adaptation in Natural and Arti�cial Systems. Ann Ar-

bor, MI: University of Michigan Press.15. Johnson, Kenneth S. 1926. Electric-Wave Transmission. Filed March 9, 1923.

U. S. Patent 1,611,916. Issued December 28, 1926.16. Johnson, Walter C. 1950. Transmission Lines and Networks. New York: NY:

McGraw-Hill.17. Kinnear, Kenneth E. Jr. (editor). 1994. Advances in Genetic Programming.

Cambridge, MA: The MIT Press.18. Koza, John R. 1992. Genetic Programming: On the Programming of Computers

by Means of Natural Selection. Cambridge, MA: MIT Press.19. Koza, John R. 1994a. Genetic Programming II: Automatic Discovery of

Reusable Programs. Cambridge, MA: MIT Press.20. Koza, John R. 1994b. Genetic Programming II Videotape: The Next Generation.

Cambridge, MA: MIT Press.21. Koza, John R. 1995. Evolving the architecture of a multi-part program in ge-

netic programming using architecture-altering operations. In McDonnell, JohnR., Reynolds, Robert G., and Fogel, David B. (editors). 1995. Evolutionary Pro-gramming IV: Proceedings of the Fourth Annual Conference on Evolutionary

Programming. Cambridge, MA: The MIT Press. Pages 695{717.22. Koza, John R., Banzhaf, Wolfgang, Chellapilla, Kumar, Deb, Kalyanmoy,

Dorigo, Marco, Fogel, David B., Garzon, Max H., Goldberg, David E., Iba,Hitoshi, and Riolo, Rick L. (editors). Genetic Programming 1998: Proceedings

of the Third Annual Conference, July 22-25, 1998, University of Wisconsin,

Madison, Wisconsin. San Francisco, CA: Morgan Kaufmann.23. Koza, John R., Bennett III, Forrest H, Andre, David, and Keane, Martin A.

1999. Genetic Programming III : Darwinian Invention and Problem Solving.San Francisco, CA: Morgan Kaufmann.


24. Koza, John R., Deb, Kalyanmoy, Dorigo, Marco, Fogel, David B., Garzon,Max, Iba, Hitoshi, and Riolo, Rick L. (editors). 1997. Genetic Programming

1997: Proceedings of the Second Annual Conference San Francisco, CA: MorganKaufmann.

25. Koza, John R., Goldberg, David E., Fogel, David B., and Riolo, Rick L. (edi-tors). 1996. Genetic Programming 1996: Proceedings of the First Annual Con-ference. Cambridge, MA: The MIT Press.

26. Koza, John R., and Rice, James P. 1992. Genetic Programming: The Movie.Cambridge, MA: MIT Press.

27. Michalewicz, Zbigniew. 1996. Genetic Algorithms + Data Structures = Evolu-

tion Programs. Springer-Verlag. Third edition.28. Mitchell, Melanie. 1996. An Introduction to Genetic Algorithms. Cambridge,

MA: The MIT Press.29. Ohno, Susumu. Evolution by Gene Duplication. New York: Springer-Verlag

1970.30. Quarles, Thomas, Newton, A. R., Pederson, D. O., and Sangiovanni-Vincentelli,

A. 1994. SPICE 3 Version 3F5 User's Manual. Department of Electrical Engi-neering and Computer Science, University of California, Berkeley, CA. March1994.

31. Samuel, Arthur L. 1959. Some studies in machine learning using the game ofcheckers. IBM Journal of Research and Development. 3(3): 210-229.

32. Samuel, Arthur L. 1983. AI: Where it has been and where it is going. Proceed-ings of the Eighth International Joint Conference on Arti�cial Intelligence. LosAltos, CA: Morgan Kaufmann. Pages 1152{1157.

33. Spector, Lee, Langdon, William B., O'Reilly, Una-May, and Angeline, Peter(editors). 1999. Advances in Genetic Programming 3. Cambridge, MA: TheMIT Press.

34. Van Valkenburg, M. E. 1982. Analog Filter Design. Fort Worth, TX: HarcourtBrace Jovanovich.

35. Williams, Arthur B. and Taylor, Fred J. 1995. Electronic Filter Design Hand-

book. Third Edition. New York, NY: McGraw-Hill.

Date post:	26-Feb-2021
Category:	Documents
Upload:	others
View:	1 times
Download:	0 times

koza - Kachergis · Title: koza.dvi Created Date: 191000908155607

Documents