IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS, VOL. SMC-2, NO. 4, SEPTEMBER 1972

[3] J. W. Butler, M. K. Butler, and A. Stroud, "Automatic classifica-tions of chromosomes-II," in 1964 Proc. Rochester Conf. DataAcquisition and Processing in Biology and Medicine. New York:Pergamon, 1965.

[4] J. W. Butler, "Automatic analysis of bone autoradiographs," inPictorial Pattern Recognition, G. C. Cheng et. al., Eds. Washing-ton, D.C.: Thompson Book Co., pp. 75-85, 1968.

[5] T. D. Caspersson, G. Lomakka, G. Svenson, and R. Saftstrom,"A versatile ultramicrospectrograph for multiple-line and surfacescanning high resolution measurements employing automateddata analysis," Expt. Cell Res., vol. 3, pp. 40-51, 1955.

[6] H. J. Conn, Biological Stains, 7th ed. Baltimore, Md.: Williamsand Wilkens, 1961.

[7] G. A. Daland and T. H. Ham, A Color Atlas of MorphologicHematology. Cambridge, Mass.: Harvard University Press,1967.

[8] M. R. Evans and J. W. Sweeney, "A computer technique forinvestigating and rationalization of scintillation scan reading,"Methods Inform. Med., vol. 6, no. 1, pp. 24-27, 1967.

[9] J. Hilditch and D. Rutovitz, "Chromosome recognition," Ann.N.Y. Acad. Sci., vol. 157, pp. 339-364, 1969.

[10] A. Hughes, A History of Cytology. New York: Aberlard-Schuman, 1959.

[11] M. Ingram, P. E. Norgren, and K. Preston, Jr., "Automaticdifferentiation of white blood cells," in Image Processing inBiological Sciences, D. M. Ramsey, Ed. Berkeley, Calif.:Univ Calif. Press, 1968.

[12] M. Ingram and K. Preston, Jr., "Automatic analysis of blood

cells," Sci. Amer. vol. 223, pp. 72-82, 1970.[13] L. W. Koster, "Color photography of biological stains," J. Biol.

Photogr. Ass., vol. 35, no. 1, Feb., 1967.[14] R. S. Ledley, "High-speed automatic analysis of biomedical

pictures," Science, pp. 216-223, Oct. 1964.[15] M. L. Mendelsohn, B. H. Mayall, J. M. S. Prewitt, R. C. Bostrom,

and W. G. Holcomb, "Digital transformations and computeranalysis of microscopic images," in Advances in Optical andElectron Microscopy, V. E. Coslett, Ed. New York: AcademicPress, 1968, pp. 77-150.

[16] N. J. Nilsson, Learning Machines. New York: McGraw-Hill,1965.

[17] J. M. S. Prewitt and M. L. Mendelsohn, "A general approach toimage analysis by parameter extraction," in Proc. Computersin Radiology, Chicago, 1966.

[18] , "The analysis of cell images," Ann. N. Y. Acad. Sci., vol. 128,pp. 1035-1053, 1966.

[19] P. G. Stein, L. E. Lipkin, and H. M. Shapiro, "Spectra II:general-purpose microscope input for a computer," Science,vol. 166, pp. 328- 333, 1969.

[20] G. L. Wied et al., "Taxonomic intracellular analytic system(TICAS) for cell identification, Acta. Cytol., vol. 12, pp. 180-204,1968.

[21] F. Winsberg et al., "Detection of radiographic abnormalities inmammograms by means of optical scanning and computeranalysis," Radiology, vol. 89, pp. 211-215, 1967.

[22] I. T. Young, "Automated leukocyte recognition," Ph.D. disserta-tion, Massachusetts Inst. Technol., Cambridge, 1969.

Simplification of Sequences of OperatorsROBERT J. POLGE, MEMBER, IEEE, AND LOUIS CALLAS, MEMBER, IEEE

Abstract-The output of a system is often expressed as a sequence ofunary and/or binary operations on the input. Analysis or design con-siderations may require manipulation of the sequence of operators toobtain an equivalent sequence either mathematically simpler or easier todesign. For this purpose, available operator algebra must be consider-ably expanded. Several new operators are introduced: the multipleoperator which is a combination of unary operators, and the zero-con-volution and zero-correlation operators. Three commutation tables aredeveloped, one for unary and transform operators, one for binaryoperators, and one for binary operators operating on two functionsalready transformed by unary or transform operators. The commutationtables can be used to manipulate and simplify a sequence of operators.They should be useful in the analysis and design of complex systems.This becomes quite clear when one tries to repeat the examples of thispaper without using the commutation tables. The simplification ofsequence of operators procedure not only permits a faster simplificationof multiple integral transformations, but it also provides such a clearpicture of the operations to perform that the manipulations becomeobvious. This contrasts with the conventional technique of rearrangingthe integrals and performing changes of variables somewhat blindly.Since the purpose of this paper is to make available a practical tool,most of the derivations of the formulas were left out for conciseness.

Manuscript received June 23, 1971; revised March 10, 1972.The authors are with the Department of Electrical Engineering,

University of Alabama, Huntsville, Ala. 35807.

I. INTRODUCTION

THE OUTPUT of a system is often expressed as asequence of unary and/or binary operations on the

input. Analysis or design considerations may require ma-nipulation of the sequence of operators to obtain anequivalent sequence either mathematically simpler or easierto design.

Available operator algebra allows manipulation of se-quences containing only convolution operators, Fourieror Laplace transforms, and transform operators. Mikusinski[I] has shown that the elements (a,b, ) of the class § ofcontinuous real or complex-valued functions over the inter-val 0 < t < oo in which the operators are addition andconvolution of functions are the elements of a commutativering [2]. Brand [3] proved that the functions of class 9form a vector space [4] over the field of complex numbersK because they satisfy the postulates pertaining to scalarmultiplication. The absence of divisors of zero in the con-volution ring makes the ring a quotient field Q whoseelements are ordered pairs (a,b) with b # 0, and which hasthe equivalence relation (a,b) = (c,d) iff ad = bc. The unitelement of the quotient field is the Dirac function 3(x) [5].

526

POLGE AND CALLAS: SIMPLIFICATION OF SEQUENCE OF OPERATORS

Also it is well known that the Laplace transform [6] of theconvolution of two functions f(x) and g(x) is equal to theproduct of the Laplace transform of f(x) by the Laplacetransform of g(x). In the preceding sentence the wordLaplace can be replaced by the word Fourier [7]. Correla-tion differs from convolution only by a folding and complexconjugate operation [8]. The preceding investigations ledto a very useful, but incomplete, operator algebra.

It is the purpose of this paper to develop commutationtables corresponding to an expanded operator algebra andto show how these tables can be used to replace complicatedsequences of operators by simpler sequences. The tablescontain the most common combinations of operators, butmore extensive tables are available in [9].A general sequence of operators can be represented by

an expression such as ((S1f1B1S2f2)B2S3f3)B3S4f4, wherethe fj are functions of a real variable. The symbol S in-dicates a sequence of unary operators and/or transformoperators, B represents a binary operator, andf stands fora function. Sufficient conditions for existence and simplifica-tion of operators are that: all the functions have a Fouriertransform or a Fourier series or are physically realizable,and the binary operators include time averaging when theyoperate on two functions of finite power.A unary operator may be defined as an assignment of a

member F of a second class of functions to each functionf in the given class [10]. For example, the operator S%O isa unary operator which maps a function f(t) into a functionf(t - to). The unary operators which not only modify afunction but also change its domain will be denoted astransform operators. The Fourier transform is a transformoperator. Several unary and transform operators (UTO)are defined, and a commutation table is constructed toshow how a sequence of two UTO can be replaced by anequivalent sequence of two or three UTO which mayinclude as a factor a function or a coefficient.The ordered sequence of unary operators is defined as a

multiple operator which performs some or all of the follow-ing operations: translation, folding, complex conjugate,Hilbert transformation, and differentiation or integration.The binary operator operates on two functions to form a

third function. For example, the output fo(t) of a linearsystem with impulse response h(t) and input fi(t) can beexpressed as a convolution integral

t

fo(t) = h(u)fi(t - u) du

= h(t) * fi(t) (1)

where the asterisk is the binary operator denoted as con-volution. A commutation table is developed for the binaryoperators convolution, correlation, zero convolution, andzero correlation. The last commutation table involves bothUTO and the binary operator convolution. It gives expres-sions equivalent to Sif(x) * Sjg(x). This table is also usefulfor binary operators related to convolution or correlation.Several examples show that a great deal of time and insightis gained by using the sequence simplification procedure,

TABLE IUNARY OPERATORS AND TRANSFORM OPERATORS (UTO)

Fourier operator 3f(x) = f f(x)e jyxdx

Identity operator Df(x) = f(x)

Cf(x) = r(x) - ji(x)Complex conjugate operator (for f(x) = r(x) + ji(x))

d kf(x), for k > 0

Integro-differential operator i f(x) =

A x k <Of(x)dx k < 0

JO f(x) = 0 for x < 0

Inverse Fourier operator -1F(y) ' fF(y)eVX dy

Hilbert operator f(x) f d =1 * f(y)

Translation operator a f(x) f(x- a)

( a b) fp °pi + (n -p)bf(x)

where Cn np(o p)'

Folding operator ?ff(x) f(-x)

T12Averaging operator eSf(x, T) iimJ f(X; T)dx

-T/2

Multiple operaltor %la,0i,0,Y,k)f(x) = a?LC ]iY,&k f(x)

based on the commutation tables, instead of the conven-tional technique of rearranging integrals and performingchange of variables somewhat blindly.

II. UNARY TRANSFORM OPERATORS

The UTO considered in this paper are defined in Table I.The multiple operator is a combination of unary operatorswhich is defined by

aI*-,vflPyeqf(x) = #(aca,f,y,k)f(x). (2)

The superscripts oX, ,B, and y are logical variables which takevalues 0 or 1. The superscript k is an integer. The transfor-mations are to be performed in the order shown. Themultiple operator X has five components a, a, ,B, , and k.The first component a represents the algebraic value oftranslation. The logic variables cx,/3,y indicate which of thefolding, complex conjugate and Hilbert operators are pres-ent. For example, ax = 1, / = 0, y = 0 means that foldingtakes place but that W and X are not used. The fifthcomponent k is a signed integer. The magnitude of kindicates how many differentiations or integrations shouldbe performed; the sign is positive for differentiation andnegative for integration.

III. COMMUTATION TABLES FOR UTO

Assume that one wants to move the operator U2 to theright to simplify the sequence U1U2U3U4U5. This meansthat the pair of UTO, U2 U3, must be replaced by anequivalent sequence, which is read on Table II at the

527

IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS, SEPTEMBER1972

TABLE IIFOR Two UTO OPERATING ON f(X)

\2P a -~ 31 a 2'02

C C Clu Ci'J c C c

C C (jY)Ij (-jy),Jl C C (-1)|'?

A OM a(-jx)' 2rr2' 3(j) 3(eibX) c

_ -1ve 3'C()C- l -jbx

3-1 cyox),~~~~~-31 C C -jsgn(y)-J jsgn(y)d . c -312

2b C C (eibY)J (e JbY0 C '72b a7-bv

V c (-1)OV& c c -.1 0b

Note: The sequence of operators 0201 f, where 0° defines the firstoperator, is equivalent to the sequence(s) of operators contained inthe table at the intersection of row 01 and column 02. The functionsgn (y) is equal to I for y positive and to - I for y negative.

intersection of row U3 and column U2. The symbol Cdenotes commutativity and f is the identity operator.

For example, the table yields the following equivalentsequences:

11Ff(x) = ~F*f(x)

W.Uf(x) = f;WYf(x) or 1-YF'6f(x) or YfY Wf(x)

s#Jbf(x) = &bibYf(x)bXf(X) = e WF(

le-*"f(x) = -* f(x)

9mY/f(x) = (- )mY,Gmf(x).

The equivalent sequences of Table II were derived bymanipulation of expressions such as U1 U2f(x) written instandard form; translation, folding, change of variables,and inversion of the order of integration were performedas necessary.The equivalent sequences for pairs of UTO involving the

multiple operator are derived using the commutation tableof the simple UTO. For example, the sequence of UTOequivalent to ~F,(axa,fl,y,k) is obtained by writing-FJ-(a,a,fl,y,k)f(x) in terms of UTO and shifting F to theright. one step at a time, until it immediately precedesf(x). Similarly, one shows that

-O(a 1 ,al#l ,yl k l)-(a2,a2,l#29Y2,k2)f(x)= (_1)a2(Y1+kl).#(ai + (-1)01a2, ° + fll +2 2

YI + Y2, k1 + k2)f(x) (3)by shifting unary operators until the transformations areperformed in the proper order: integro-differentiation,Hilbert transformation, complex conjugate transformation,folding, and translation.

TABLE IIIBINARY OPERATORS

Convolution operator P12(r) fl(x)f2(T - x)dx fl(x) * f2(x)

Correlation operator Cp12(T) =f fl(X)f2(T + x)dx fl(x) * f2(x)

- LIMl1

T/2Meanconvolution P12(T) = T T fi(x)f2(T -x)dx

f1(x) f0(xoperator -T/2

- LIMl ~~T/2Meancorrelation 12(T) =LMT

f1(X)f2(T + x)dx fl(x) g f2(x)operator -T/2

Zero-convolution P12(0) =LIM

flf(x)f2(T - x)dx fl(x) *0 f2(x)ope rator JT -

Zero-correlation ep (0) =L f f1(x)f0(r + x)dx f1(x) *0 f2(x)

LIM T/2Zero-meancorrelation P(0) = T 1 + x)dx f(x) o f(x)

operator T0 JT/2

If the functionf(x) contains a parameter T, this parameterbehaves as a constant during the transformations. A usefulequivalence not included in the table is '%"Oxf(x; r)=&xf(x; -T).

IV. BINARY OPERATORSA binary operator combines two functions into a third

function. In this paper the only binary operators investigatedare integral transformations related to convolution or

correlation. They are defined in Table III.

The convolution operator is associative, commutative,and distributive. Since the correlation operator is not com-

mutative, it is often advantageous to replace correlation byconvolution before starting to simplify a sequence of opera-tors. The substitution of convolution for correlation, or

correlation for convolution, is based on the followingequivalences:

f(x) * g(x) = V1Kf(x) * g(x)

f(x) * g(x) = Vwf(x) * g(x).

(4)

(5)

The mean convolution or the mean correlation operatorsmust be used to obtain the convolution or the correlationfunction except when at least one of the two functions isaperiodic. The zero-convolution and the zero-correlationoperators perform a convolution or a correlation followedby the substitution of zero for the parameter. These binaryoperators can be used to represent the integral of a product.Clearly f*0g = (f*g)O, f*0g = (f*g)O, and f *0g =

(f * A).Table IV gives expressions involving four binary operators

(convolution, correlation, zero convolution, and zero cor-

relation) and considers equivalent expressions. The row andcolumn correspond to the left and right operator, respec-tively. The parentheses show in which order the operationsshould be performed. The formula

f*g*h = (f*g)*h =f*(g*h) (6)

COMMUTATION TABLE

528

POLGE AND CALLAS: SIMPLIFICATION OF SEQUENCE OF OPERATORS

TABLE IVCOMMUTATION TABLE FOR Two BINARY OPERATORS

TABLE VSEQUENCES SIMILAR TO SI f * S29

0n

c

it:;r,b

V

16 c

ff Vg(x)dx-CDX

Tf S (x)dx_@0

CD

f Cg(x)dx-0

f*g f*Cg i(f*g) M(f*g) FI0(f*g) f*g 9Jf(x) dx

Cf*g C(f*g) | D(Cf*g) l(Cf*) a0,(CfI*9) g*f

S(f*g) | S'(f*Cg) | n(f*g) | '(f*g) |30(f*g) | (f*Vg | fSfx)dx

(f*g) | t(f*Cg) (g)) ] _(f*g) bu 3(f*g) l$(f* fg) ;g ff(x)dx

Xb(f*g) |a(f*cg) | -u(f*g) | M(f*g) | b+c(f g) a1( f*r) 9f3f(x)dx

-(Vf*g)

Vf*g r *f*g b (vf*g) | (irf*9 5' Orf*g) ?t(f*g) .9 ) fdx)

Note: The table shows that the convolution of functions operated on by unary operators 01 f * 029can be replaced by an equivalent convolution of functions contained in the table at the intersectionof row 01 and column 02

ff 9g(x)dx

Right|B0

Le*f tX l | * | *0 *0Left3

* | f * g * h- (f * g) * h (f *g) *hCf* *h (f * g) *o h (f * g * h) (f *g) *o h (Cf* g*h)f * (g * h) f *(g *h) f* g*h f * (g *o h) (g *o h)lf f (g o h) - o h) fall permutations of f, g, h are

valid

(f *g)* h f*g *h (f *g) * h f*Cg*h (f * g) *o h (f* g * h)o (f *g) *o h - (f* g*h)of *(g *h * g *h |f * (g * h) - * h |f * (g *o h) - (g *o h)lf f *(g *o h) - (g *o h)S 1f

||(f *° g) * h - (f *° g)-h | (f *o g) * h - (f *° g)'h(fo)h (fo) (*g* (*g h (f *og) *oh -(f *og) (9lh),O (f *og) *oh - (f *og) (8n)0f *o (g* h) - (f * g * h)o f*o (g*h) (f h) f *o (g *o h) (g *o h) o f * ( *o h - (g *o h) (S1f)o

(f *o g) * h (f *o g)½lh |(f *° g) * h - (f *o g)D h | (f *o g) *o h (f *o g) (D1h)o| (f *o g) *o h - (f *o g) (S- h) |

f *° (g * h) - (Cf * g * h)o f *o (g * h) (Cf *g*h)e f *o(g *o h) - (g *o h) (Dff *o (g oh-(g*oh (b8lf)o o(*h go)( )

9I.L0 5

529

co

f g(x)dx-OD I

v

CD

'Y -7, g (x)dx-0

IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS, SEPTEMBER 1972

means unrestricted commutation of the functions and orderof operation. Therefore, an expression similar to (4) standsfor a set of equivalent sequences. For example, one reads(f* g) * h = f * * h, which can be transformed into(W'Yg * f) * h = (g * f) * h. As usual, (g(x))0 meansg(x = 0). For example, (f"*"f * WVg * h)o indicates thatthe two convolutions are performed in any order, then xis replaced by zero.

V. COMBINATION OF BINARY OPERATORS AND UTO

Consider the sequence of operators such as S1f(x) *S2g(x), where the binary operation convolution is per-formed last. Table V shows whether an equivalent sequenceof the form S3(f(x) * g(x)) where convolution is performedfirst exists, or whatever equivalent sequences are available.Consider two functions f(x) and g(x). Each one is operatedon by a unary operator yielding S1f(x) and S2g(x). Thenthe two functions are combined using the binary operatorconvolution. The first and second unary operators deter-mine, respectively, the row and column of the table. Theintersection of the row and column lists an equivalentsequence. For example, if SI = _m, and S2 = _9n (_9k iSdefined in Table I), one can see that

_imf(t) * -9ng(t) = 9m+f(f(t) * g(t)).In this case, when the convolution is performed first, thetwo integrodifferential operators combine into one.

Table V is also valid for the mean convolution operator.A similar table for correlation and mean correlation is notpresented for the sake of brevity and because correlationcan be replaced by convolution. Recall also that zero-meanconvolution is identical to mean convolution followed bya substitution of T by zero. Note that the inverse Fourieroperator can be replaced by the direct Fourier operator,and vice versa, using Ff(x) = 27r, lf(x).

VI. GENERAL PROCEDURE FOR SIMPLIFICATION OFSEQUENCES OF OPERATORS

Consider several functions where each function is trans-formed by a sequence of unary operators and where pairsof transformed functions are combined by binary operators.This type of problem can be represented by the followingexpression:

((S,f, B1 S2f2) B2 S3f3) B3 S4f4- (7)

The symbol S indicates a sequence of unary operators.The symbol B represents a binary operator such as con-volution or correlation, and f stands for a function. Ageneral procedure for simplifying sequences of operatorsis given in the following.

1) Consider the groups of two transformed functions anda binary operator one at a time starting with the inner-most binary operator.

2) In the selected group the binary operator is first trans-formed from correlation or meancorrelation to con-volution or meanconvolution.

3) Transform operators are shifted to the left usingTable II.

fi't ' h(t) f"f(t) = fi(t) * h(t)

Fig. 1. Block diagram of linear filter.

4) Unary operators are written in terms of multipleoperators.

5) Multiple operators and unary operators are combinedinto a simplified form.

VII. EXAMPLES OF SIMPLIFICATION OF OPERATOR SEQUENCES

A. Simplification by General Procedure

Assume that one wants to simplify the expression

(a I* t'-'f(X) * Ya,jg(x)) * a - h(x) (8)

which is quite complicated when written in the integralform. The parentheses indicate that 9 a1iK*i9,f(x) *

ga,Wg(x) should be simplified first. One obtains succes-sivelyKY1a 'X7''f(x) * Y02a9g(x)

=- a1l6aIf-1f(X) * $-ajwg(x)

= I(-a1,0,1,1,1)f(x) * W(a2,0,l,0,0)g(x)= 4(-a, + a2, 0, 1, 1, l)(f(X) *g(x))- (9)

where [9, table C-2] was used. The expression (8) becomes

,#(-al + a2, 0, 1, 1, l)(f(x) *g(x)) * M(a3,1,1J,,-l)h(x)= A(-al + a2 + a3, 0, 1,0, 0)((f(x) * g(x)) * lth(x))= 4(-al + a2 + a3, 0, I, O, 0)[h(x) * ( f(x)*g(x))]

(10)= gJ(-ai+a2+a3)t [h(x) * (f(x) * g(x))]which is considerably simpler than (8), and much easier towrite in integral form.

B. Minimum Mean-Square Error Linear Filter

Consider the block diagram of the linear filter shown inFig. 1, the input is fi(t) and the output is fo(t). Let thedesired output of the filter be denoted byfd(t). The mean-square error for the filter with nonoptimum impulse re-sponse h(t) is [11]

1 T/2 -o 2

=2(t)= lim _I h(T)fi(t - T) dT - fd(t) dt

(11)

which can be written in operator notation as

Vts(hl(t )) = et[h(t) *ft(1i2)-. (12)

After expanding the square one obtains

le2(h(t')) = [h(t') *.fi(t')] *O [h(t') *.fi(t')]-2[h(t') * Ji(t')] *o.fd(t') + &tfd2(t') (13)

where *o is the zero-mean correlation operator. The opti-mum realizable filter is determined by its impulse responsehop(t). The impulse response of a nonoptimum filter can be

530

POLGE AND CALLAS: SIMPLIFICATION OF SEQUENCE OF OPERATORS

,]_ Square Law

FDetector

O Tywl (U;)_I_ _ (J! yy(t) = a x2(t)

Fig 2nB S ra e .

Fig. 2. Square law detector.

written as

h(t) = h0p(t) + Ag(t) (14)

where g(t) is a causal function and A is a parameter. Bydefinition &tc2(h0p(t)) is the minimum mean-square error.Therefore, for any ). or g(t),

ffe2(h(t')) -&Gt2(hop(t')) . 0. (15)

This difference consists only of those terms of 6te2h(t')which contain i, and (15) becomes

A[gt)*.fi(t')) *io ('h.p(t ) *fi(t ))+ (hop(t') * fA(t')) 1O (g(t') * ))]+ ;A,2[g(t ) *PO(') *o (g(t ) * Ai(t))]-22[(g(t') *fi(t')) *o f(t')] . 0. (16)

Since zero-mean correiation is commutative, the first twoterms are equal. The term in )2 can be written as 2&G[(g(t') *fi(t')]2; therefore, it is always positive. Finally, to satisfythe inequality (16) for any 2. and g(t), one must have

(g(t') * fi(t')) *O (hop(t') * fi(t'))- (g(t ') * fi(t ')) *o fd(t')) = 0. (17)

In the first term of (17) correlation is replaced by con-volution in order to use the commutative and distributiveproperties of the zero-mean convolution operator:

(lVg(t') * ffi(t')) *0 hop(t') * fL(t')I'g(t') *0 Y"fj(t') *fi(t') * hop(t')

g(t') *O (Oii(t') * h,p(t ). (18)

A similar simplification is performed on the second termof (17), and (17) becomes

g(t') lo (Oii(t') * hop(t') - 4id(t')) = 0 (19)for any i. and g(t).The condition (18) is ensured for t < 0 because g(t) is a

causal function (g(t) = 0, for t < 0). This leads to thewell-known Wiener-Hopf equation

ji(t ) * hop(t') - id(t') = 0,

which is normally written as

hop(t')4ii(T - t') dt' - +id(t') = 0,-0

or t > 0 (20)

for t > 0.

C. Power Spectrum at the Output of a Nonlinear Detector

A square law detector y(t) = ax2(t) has an input x(t) =s(t) + n(t) where s(t) = Ccos (wot + 0) and n(t) is anarrow-band Gaussian noise of amplitude A centered aboutwo. The input power density spectrum l$xx(w) = DS(w)) +1),j(w) is shown in Fig. 2. Using the operator notation

q)xx(wO) = (9-Y o)o + 9'0)) (- 2 3(w) + A reCt2OB (w))

find the output power spectral density (Dyy(wo). Assume thatboth the signal and the noise are statistically independent,stationary, and that both have zero-mean value.The output autocorrelation is obtained first starting with

(T)= y(t)y(t + T)) = y(t)y(t + T) (22)

4yy(z) = a2[s(t) + n(t)]2[s(t + 1) + n(t + T)]2. (23)

Expanding (23) and dropping the terms where n(t), n(t + T),s(t), s(t + T) can be factored out, yields [12]

Xyy(T) = a2[s2(t)s2(t + T) + 4n(t)n(t + T) s(t)s(t + T)

+ s;(t) n2(t + T) + n2(t) S2(t + 1)

(24)+ n2(t) n2(t + T)].

Performing the averaging results in

-i~(z =a2C4 a2C44 8

-cos 20)oz + (4a2 4n(T)455(z)

+ a2C2a ) + (2a24,1.(r) + a214) (25)

where the first term is due to signal alone, the third term isdue to noise alone, and the second term is due to signal andnoise. The output power spectral density is equal to theFourier transform of the output autocorrelation function:

(D,y(W) = FOYyj(T) = T1(w) + T2(wo) + T3(w) (26)where Ti(w) represents the Fourier transform of the ithterm of (25). Using a Fourier transform table one obtains

T1(O) = 7ra 3C() + (9-2 + 2 o) ()aC (w)2~~~~~~~~~~

(27)

2a2 2 (T2(w)) = [4$is(wo)* fln(w)] + 27ma2C2r173(w)

=2a2(f-° + 90) 2 6(w) * (9- °

+ Y(,O)A rect2,fB (()] + 27ra2C 2q12 63((o))

2= 2[ [(S-" +Y0t) 2 2A (6(wo) * rect2,fB (o))]

+ 27ra2C2oa2 3(w)

= (9-2,o + 2.Y0 + 92500)a2C2A rect2,B (w))

(21) + 2ma2C r 2 3(w) ())

531

f(

(28)

IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS, SEPTEMBER 1972

a2T3(w) = a2 [s (Ct) * 'F.j(w)] + 2ira2'a4 3(@)

a2- [(&O + g.Yo)A rect2,B (O) *

ir

+ J(0,o)A rect2B(wB)] + 2ira2n 3(w)

- (e-2wo + 2Y0 + Y2eoo)2a2A2BA4wB(w)+ 2ira 2an4 3(w) (29)

where A47rB(w) means a symmetric triangular waveform ofheight unity and width 47rB. Finally,

Ara2C4 C1Dyy(w,)) = 83-2wo1(a) + a2C2A rect2,rB (co)

+ 2a2A2BA4sB(w))

+ 2r2 (C + a2)n 6(w) + 2a2C2A rect2nB (a)

+ 4a2A2BA4rB(wO)7(ra2C4 22Arc+ 2co ( 8 3(w) + a2C2A rect2xB (c)

+ 2a2A2BA4B(C))O) (30)

(\yy(w) is shown in Fig. 3.

D. Simplification of DesignA sequence of operators can also be manipulated to

correspond to a simpler design. For example, one may wantto simplify the block diagram of Fig. 4 so that only onecorrelator will be needed. Referring to Fig. 4

Y(t) = Jd2-dU[(Xl(t) * X2(t') + n,(t') * n2(t'))* (h1(t') * h2(t')) * h3(t')]. (31)

Assuming that the noise has zero mean and is uncorrelatedwith the signal,

X1(t') * x2(t') + n,(t') * n2(t')- X1(t') * X2(t') + xl(t') * n2(t') + n1(t') * x2(t')

+ n,(t')* n2(t)

-(x,(t') + n1(t')) * (x2(t') + n2(t')). (32)When (32) is substituted into (31), the following is obtained:

)(t) = Jd2-dIi(xl(t ) + n1(t ) * (X2(t )+ n2(t')) * (hI(t') * h2(t')) * h3(t')]

- d2-dU[(X1(t') + n,(t')) * h1(t')] * [(X2(t')+ n2(t')) * h2(t') * h3(t')]

- Jd1[Xl(t') + nl(t')) * h1(t'))] * 9Jd2[(X2(t')

2

2TT.(c- + a2) 6(Ui)

- 2ub

Fig. 3. Power spectral density at output of square law detector.

Fig. 4. Block diagram using two correlators.

Linear Dela Cioss-Xn(t) + nj(t). Filter p Line f(t) CorreLator

h, (t) d4

Linear Linear De Ia,X2(t) + n(t)-*- Filter Filter Line

h2(t) h3() d2

Fig. 5. Equivalent block diagram using one correlator.

x(t)

n(t)

( t)

ly(t) ]

Fig. 6. Cross correlation of linear system input and output.

E. Impulse Response in n Dimensions

The preceding techniques for simplification of operatorscan be extended to more than one dimension. For example,the linear system impulse response can be determined fromFig. 6. Let x(t) and n(t) be n x I column vectors repre-senting input signals and noise, respectively. The linearsystem can be defined by an n x n impulse response matrix[h(t)]. When xQt) and n(t) are independent, and if eitheror both have zero mean, the output [Y(t)] is

[y(t')] = n(t') * ((x(t') + n(t'))T * [h(t')])= t'n(t') * ((x(t)T + n(tf)T) * [h(t')])

= ('rn(t') * x(tV)T+ Fn(t') * n(t')T) * [h(t')]+ n2(t')) * h2(t') * h3(t')]. (33)

This sequence of operators corresponds to the block diagramof Fig. 5 which contains only one correlator as desired.

= [0] + [ nn(t')] * [h(t')]. ('34)If the noise vector is selected in such a manner that theelements of the noise covariance matrix are b(t) functions,

532

IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS, VOL. SMC-2, NO. 4, SEPTEMBER 1972

i.e., [4( t)] = [6(t)], then

[y(t')] = [h(t')] (35)and the impulse response matrix of the linear system hasbeen defined, while the system was operating under thecontrol of x(t). Note that the commutative law does nothold for vectors and matrices.

VIII. CONCLUSIONS

Several new operators were introduced: the multipleoperator which is a combination of unary operators, andthe zero-convolution and zero-correlation operators. Threecommutation tables were developed, one for unary andtransform operators, one for binary operators, and one forbinary operators operating on two functions already trans-formed by unary or transform operators.The commutation tables can be used to manipulate and

simplify a sequence of operators. They should be useful inthe analysis and design of complex systems. This becomesquite clear when one tries to repeat the examples of thispaper without using the commutation tables.The simplification of sequence of operators technique

not only permits a faster simplification of multiple integraltransformations, but it also provides such a clear picture ofthe operations to perform that the manipulations becomeobvious. This contrasts with the conventional technique of

rearranging the integrals and performing changes of vari-ables somewhat blindly.The commutation tables and simplification procedure

are used to teach graduate courses in communication. Thestudents learn faster, and much more material can becovered. Since the purpose of this paper is to make availablea practical tool, most of the derivations of the formulas wereleft out for conciseness.

REFERENCES[1] J. Mikusinski, Operational Calculuiis. New York: Macmillan,

1959, p. 342.[2] F. Ayres, Jr., Theory and Problems of Modern Algebra. New

York: McGraw-Hill, 1965, p. 101.[3] L. Brand, Differential and Difjerence Equiations. New York:

Wiley, 1966, p. 519.[4] D. T. Finkbeiner, Introduction to Matrices and Linear Transforinia-

tions. San Francisco, Calif.: Freeman, 1966, pp. 18 and 24.[5] R. Bracewell, The Fourier Transform and Its Applications. New

York: McGraw-Hill, 1965, pp. 67 and 69.[6] B. P. Lathi, Signals, Systems and Communication. New York:

Wiley, 1965, p. 160.[7] A. Papoulis, The Fourier Integral and Its Applications. New

York: McGraw-Hill, 1962, p. 7.[8] S. J. Mason and H. J. Zimmerman, Electronic Circuits, Signals,

and Systems. New York: Wiley, 1960, pp. 206 and 326.[9] R. J. Polge and L. Callas, "Simplification of sequences of opera-

tors," Univ. Alabama, Huntsville, Rep. UARI 116, 1972.[10] W. Kaplan, Operational Methods for Linear Systems. Reading,

Mass.: Addison-Wesley, 1962, p. 64.[I1] Y. W. Lee, Statistical Theory of Communication. New York:

Wiley, 1963, p. 358.[12] W. W. Harman, Principles of the Statistical Theory of Commlnica-

tion. New York: McGraw-Hill, 1963, p. 198.

Training a One-Dimensional Classifier to

Minimize the Probability of ErrorGUSTAV N. WASSEL, MEMBER, IEEE, AND JACK SKLANSKY, SENIOR MEMBER, IEEE

Abstract-Some of the results of a study of asymptotically optimumnonparametric training procedures for two-category pattern classifiersare reported. The decision surfaces yielded by earlier forms of non-parametric training procedures generally do not minimize the probabilityof error. We derive a modification of the Robbins-Monro method ofstochastic approximation, and show how this modification leads totraining procedures that minimize the probability of error of a one-dimensional two-category pattern classifier. The class of probabilitydensity functions admitted by these training procedures is quite broad.We show that the sequence of decision points generated by any of thesetraining procedures converges with probability one to the minimum-probability-of-error decision point.

Manuscript received July 26, 1971; revised April 24, 1972. Thiswork was supported in part by the NSF under Science Faculty Fellow-ship 60196 and in part by the U.S. Air Force Office of ScientificResearch under Grant AFOSR 69-1813B.G. N. Wassel is with the School of Engineering, University of

California, Irvine, Calif., and the Department of Electrical Engineering,California State Polytechnic College, Pomona, Calif.

J. Sklansky is with the School of Engineering, University of Cali-fornia, Irvine, Calif. 92664.

1. INTRODUCTION

THE PRELIMINARY results of a study of asymp-totically optimum nonparametric training procedures

for two-category pattern classifiers are reported. By an"optimum classifier" we mean a classifier which finds theminimum possible error rate, called the "Bayes' rate." Inthis paper the theory is restricted to one-dimensional featurespace. The theory extends to cases where the costs of a falsealarm and a false rest are unequal.The nonparametric training procedures in the literature

on artificial pattern classifiers fall into the following twomajor types.

I) The first type includes training procedures, each ofwhich terminates or converges to a classifier that separatesthe pattern classes perfectly, provided the classes areseparable. If the classes are not separable, these trainingprocedures yield neither minimal-error solutions nor ap-

533