OBJECT-ORIENTED PROGRAMMING FOR LINEAR ALGEBRA
by
John Alan McDonald
TECHNICAL REPORT No. 175
August 1989
Department of Statistics, GN-22
University of Washington
Seattle, Washington 98195 USA
algebra.
ALAN LVH"J LJU
Dept. of Statistics,
July 10,
2 Abstractions
low high mathematical abstractions,losing the level of performance inproblems provided by traditionaltine libraries.
To the results of this experiment (sofar), this paper compares thetation, and of onstandard numerical linear algebra problems withLinpack[4], a classic high-quality Fortran packagefor solving system of linear equations and relatedproblems,
Abstract
numerical analysts and Lisp jSmalltalk proshare the assumption that languages
"'("...,.t"''''1"1 are more appropriate for traditional,scientific programming than object-
To show how straightforward application of object-oriented design to standard algorithms in numerical linear algebra improves clarity and expressiveness, without sacrificing speed or accuracy, I describe parts of Cac-
a system for numerical linear algebra and constrained optimization, implemented in Common
and CLOS,
Keywords: Numerical Analysis, ScientificComputing, Common Lisp Object System
1 Introduction
Choosing appropriate abstractions is an essentialpart both good programming and good mathe-matics. purpose of this paper is to give an ex-
of number-crunching problemsabstraction mechanisms provided
Undergraduate mathematics courses in linear algebra tend to come in two styles: a relatively concrete version often called "Matrix Algebra" a,nd amore abstract version, designed for math majors,that might be called "Finite Dimensional VectorSpaces"[6].
The basic entities in a matrix algebra course arematrices: n x m arrays of numbers. oper-ation is matrLx multiply: = L,kBik * Ckj.
isting numericalcompiled subroutine libraI'iesteractive svstems
dnnelllsliomu vector space course,coaeents are vector spaces
vector space, is a coJle<:tlcmtwo +
aestracnon is to
in
••
• (make-element Abstract-Space &restoptions)
• (element? Object Abstract-Space)
4 Spaces
A space is represented in Cactus by an instance ofa subclass of Abstract-Space. Each space class,therefore, represents a family of spaces (its instances). An inheritance graph for some importantspace classes is given in figure l.
The basic Abstract-Space protocol provides away to test whether a given object is an element ofthe space and a way to create a new object c~Jpa.l~le
of representing an element of the space:
tation isSI. Second, C2 may extendresentation that's validthird, C2 may C1
its instances to represent of a set difijoin.tfrom SI. If class inheritance is used to reT)reSelrtsubsetting, then only the first possibility is
Cactus provides two distinct, roughly l:'a.'<lde''''~'
class hierarchies, described in the next two sections. The classes described in section 4 n1'",.,.v,I1'"
representations for spaces; classes in section 5provide representations for points. There is no simple correspondence between classes in thetwo hierarchies, because a given space may use many pointclasses to represent its elements and a given pointclass may be used by many spaces.
many
introduction to
concepts, encouraging appronrovrdinz a com-
of abstraction is in appuca-because it is more the same concepts
can to study infinite dimensional spacesapplications to differential equations, signal
processing, etc.
3 Representations
Cactus is designed to closely model the abstractions used a course on finite dimensional (vector) spaces[12]. It provides representations for avariety of spaces, points (elements of some space),and mappings between spaces. The linear algebracore of Cactus deals with vector spaces, vectors,and linear transformations; more general spaces,points, and mappings are used are used to support constrained optimization at a similar level ofabstraction[lO].
By space, I mean essentially a set of elements,possibly with a set of operations defined on thoseelements. A representation for a space must provide a way to tell if a given object is an elementof the space (or, more precisely, if a given objectis a representation of an element of the space). Itshould also provide a way to create new elements ofthe space. In addition, we need to be able to tell if
81 a subset (or subspace, in an appropriateof S2.
makes a space sound suspiciously aa natural first thought is to represent spaces
the space operain-
Figure 1: Inheritance graph of some important Cactus space classes (from top to bottom).
1
represents a particularlyfamily of subspaces of euclideanA block
abuse of notation-ego one generic add function isused by all vector spaces. (This restriction impliesthat if a given object is an element of two
one space a linear subspacefar, this has not been a senous llmlta-
tion.)key idea in many fast algorithms is to
problems into small ones that;;)Ulev", ..... separareiy. In numerical linear algebra,
temporarily restricting attention to
Euclideart-Vector-Space)
euclidean vector spaces includes:
should be define mul1tiplespaces over a given set of objects; the vee
adlJJtJlOn and
section 2, we defined a vector space as a set
• (canonical-bas is-elementEuclideart-Vector-Space Integer)
•
• (make-linear-trartsformationVector-Space Vector-Space).
spaces of the home space.in arbitrary subspaces of the home spaceexpensive and usually avoided.
For the purposes of this paper, we are concernedthe subclasses of
stances represent vectors.protocol includes:
• (add Abstract-Vector Abstract-Vector&:key result)
• (scale Number Abstract-Vector &:keyresult)
Figure 2: Inheritance graph of some important element representation classes (from top to bottom).
are Lvectors and Block-Lvectors. Lvectors isthe "standard" representation in Cactus for Rnfamily of vector spaces; Block-Lvectors representthe block subspaces of Rn.
• (linear-mix NumberNumber Abstract-Vector &:key
Linear-mix computes a general linear ....UUllUU1(l,
tion of two vectors.The Euclidean-Vector protocol includes:
• (inner-product Euclidean-VectorEuclidean-Vector)
• (12-norm Euclidean-Vector)
• (11-norm Euclidean-Vector)
spaces.
• (sup-norm Euclidean-Vector)
The only two instantiable classes shownure 2 are Lvector and Block-Lvector.
Lvector instances are the "standard" representation for vectors in Rn, i.e., the elementsLvectors space. An Lvector has aholding its coordinates in a standardsystem.
A Block-Lvector is a compactresentation for an element of itshome space. A Block-Lvectorholding an array justzero coordinates,and
a read-
5 Points
• (element? Abstract-Point
• (home-space Abstract-Point)
Elements of spaces are represented by instancesof subclasses of Abstract-Point. An inheritancegraph for some point classes is in figure 2. Theprotocel for Abstract-Point's includes:
Q 't'~.-" I '''''''''', ~ .nap P l ns : prun" (app"r:d (a 11-"ub"1a"""" 'linear-tran~fornation)
fl
//' <:--- "Lfli
a: 1
Dynamic Liso Listener 2
Figure 3: Inheritance graph of abstract Mapping classes (from left to right) .
6 Mappings for linear transformation classes where aODrc)J:)I-late.
between spaces are represented by instances of subclasses of Mapping, some of whichare shown in figure 3. Each Mapping. has domainand codomain spaces. (The slightly pedantic"codomain" is used here, rather than "range," because, strictly speaking, the range of a transformation is the image of it's domain, which mightnot cover the codomain space.) The protocol forMapping's includes:
• (transform Mapping Abstract-Point &keyresult)
The most important subclassLinear-Transformation is MatriX, whichsents the transformation by a two-dimensionalray and implements the protocol functions inusual way (matrix multiply). An importantrelated to Matrix is Restricted-Matrix, whosedomain and codomain are required to be block subspaces. A Restricted-Matrix is represented bythe appropriate part of a larger two-dimensionalarray; it is used to focus attention on a smallproblem.
point argument must be an element ofthe mapping's domain; the result, if supplied,must be an element of the codomain.
• (compose Mapping Mapping &key result)
There are a fairly large number ofcates that determine whether ahas some property (eg, unitary?)the choosing an algorithm to use, forin solving a vector equation. For some
swercan beIdentity-Transformation is always U.l.<:t.!5''';U<:1..l.:
In cases a Matrixto
transformations h~1twf~n twospaces are a vector
Linear-Transformation
eqto the codomain of the second. The optional
if must be a m,t.PiHl1·j;arguments domain to
f!:UmEmt's codomain.
earspaces,
(9raph-~ubcla~~e~ 'linear-transfornation :prune '(la-annihilator»
Figure 4: Inheritance graph of Linear Transformation classes (from left to right) .
Another important class of transformations isProduct. A product transformation contains a listof other transformation objects-its factors-andrepresents the composition of the factors. Producttransformations arise from matrix decompositionsdiscussed below.
7: Comparison with matrix algebra
point, the apparent simplicity of matrixal!?;e()ra systems may seem to be a significant advant<},.e;e over Cactus.
vector space course).
Further, the apparent simplicity of the matrixgebra system is misleading. The apparent simplicity is actually due to the failure to make explicitthe abstractions essential to the underlying algorithms. Multiple representations of linear transformations are, in fact, an unavoidable part of any numericallinear algebra system. Linpack representslinear transformations by arrays, parts of arrays,and groups of scalar variables in at least a dozen
ways. Linpack is excellent Fortran, but,because it is Fortran, the representations and therules for their use are not in the code.
#'inverse
to
The LU decomposition is based onit is easy to "iT,v",r-t-" a non-singular trIangUlartrix,
In theof a matrix, T-l, is aninstance of either Inverse-Upper-Triangular orInverse-Lo>1er-Triangular. In either case,has a slot holding the same two-dimensional arrayas T (or a copy). The difference between T-lT is the transform method. T, of course, usesmatrix multiply. T-l uses either back substitutionor forward elimination, depending on whether T isupper or lower triangular. Back substitution andforward elimination are comparable in speed andnumerical stability to matrix multiplication[5].
(Back substitution is based on the following:Suppose U is a non-singular upper triangular matrix. Given b, we can find an x such that U-1b = xor, equivalently, Ux = b, by first solving Un,nxn =bn for X n, then substituting X n into Un-1,n-1Xn-1 +Un-1,nXn = bn - 1 to get X n-1, and so on. Forwardelimination is essentially the same.
Back substitution and forward elimination areexamples where thinking in terms of systems ofequations is useful. However, this is not a seriousviolation of the Cactus paradigm, it's encapsulated within the definition of the two inversetriangular classes.)
The issue that remains is how to A. Theactual factoring is more complicated than simply Land U. vVe actually produce A = (G1G2'"
where U is upper triangular the G, aretriangular. G1 is chosen so that GIl. A = Al haszeros in the first column, below the first element.Then we restrict attention to the (n - 1) x (n - 1)
corner A and recurseG, are
LirJ;eaI'-Pl~odtlct;)inverse (
'Linear-Product:factors
To aP1Dreciate
tors
8 An example
we a Matrix.a Linear-Product
factors toare easy to invert.
Linear-Product we invoke a siI1n.pJ,e IIlet.b.o<l:
advantages of Cactus over matrixconsider a standard algorithm,
LU decomposition.Suppose we have A : nn H> nn, a linear trans
and b E nn, and we want to solvex E nn such that: Ax = b. If-to make
easy-we assume that A is square and invertible, then we need to invert A and apply it tob: x =A-lb.
The thing to notice is that it's a bad idea totry to represent by a matrix, The best floatingpoint matrix approximation to A-I, }I-I, may notbe good enough[5}; for some A's and some b's, theapproximate x = }I-1b maybe be far from the truesolution z.
This isn't a problem for Cactus, where manyother representations are available, but it causestrouble for traditional systems, where the matrixis the only abstraction around. The recommendation of standard numerical linear algebra texts isfor users to change how they think-to give up theelegance of a single vector equation and, instead,think about how to "analyze and solve various systems of simultaneous (scalar] linear equations"[4].
The basic idea underlying many algorithms innumerical linear algebra is to decompose the matrix, A = A1 A 2 ... An, in such a way that the factors properties that are convenient for the
at hand. To solve Ax = b, we find facwhose inverses can be computed and ap
to vectors etticientlvA - I•. , 1 •
is exactlyit to Dr,:>d.l1ce
a1t;erl.la.ti,re representations, to improve space,or accuracy of is a of curre~~n~lt;r~,e==-•....................................................................................... 'LrJ;lJl..l;r;se (factol;'.s.tGL}»))7 >.
sparse rna-
10 Conclusion
9 Performance
spaces.
Runninz times for Cactus Iu-decomposeo
qr-decompose, and the most closely comparaoreLinpack subroutines, are given in figuresComparing Lisp and Fortran on a :::Jymt)011CSmachine might seem unfair to Fortran; hOWE~ver.
believe similar results would hold on COjo.Vl~nt;t()l(l.;:tl
architectures. For all but the smallest pTClol,ems,the overhead due to the use ofgramming is negligible compared to thein a set of simple floating point array nnp"'U.lIlf,,,
(eg. computing dot products) essentially theas the Fortran BLAS [9J. The relative performanceof Cactus to Linpack on a given machine will be determined by how efficiently BLAS operations callbe implemented in Common Lisp versus Fortran OIl
that machine. Cactus includes a protocolr.n.0n Lisp BLAS, to permit specializedimplementations for different Common Lisps.ple array loops execute at close to the samein Symbolics Fortran and Symbolics Commonif the Lisp versions are carefully coded
Limited experience with Commonon standard architectures suggests
appropriate
5:Iu-decompose with SGEFA, on aics 3620 FPA, Symbolics Commonera 7.2 and PCL (AAAI'88 versionFortran 77. times are in seconds.
decomposition, a sta-A PIG1 · · • PnGnU.
For
Hc)us:enolcler vector
a twc-mmensionar
implementation in is almost aabove. The factors
are by instances of the obvious GaussPivot classes. Because it's a little easier, Cac
tus actually produces a factoring of the inverse:= U-1G;;1 Pn ... GI" PI, where Pi-
l = Pi and= I - gi ® ei.
the Linpack routine SGEFAas a vector of integers, P and
R. vector of integersthe ith ele-
P the ith pivot, taking advan-taze of the fact, that, in this particular case, theoith pivot must be between i and something else. R
packs together the Gi'S and U in one array. Theupper triangle of R is U; theith sub-diagonal column of R holds the non-zero elements of the gi vectors, taking advantage of the fact that the first ielements of gi must be zero. To use the results ofSGEFA to solve a system of equations, one passes
right side a vector to SGESL. Toresults for other purpose is not practi-
cal.
The QR decomposition is another standard algorithm in numerical linear algebra, similar in outline LU. Orthogonal Householder transfer-
of Pivot. InLinear-P:rO<1U<:'t transfer
in the same way as thereturned by LU decomposi-
of routines W,,"U.\..JU'V
[6J P.R. Halmos, Finite-dimensional VectorSpaces. Van Nostrand, Princeton, NJ,
[3J Tony D. DeRose. A coordinate-free approach
to geometric programming (a geometricbra and its implementation).port, Dept. of Computer Science, U.ington, 1989. course notes for ,,","',,",1''''-'''
Spring
[4J J.J. Dongarra, C.B. Moler, J.R. Bunch,G.W. Stewart. LINPACK Users'SIAM, Philiadelphia, 1979.
[5J G.H. Golub and C.F. Van Loan. MatrixComputations. The Johns Hopkins UniversityPress, Baltimore, 1983.
[2] Tony D. DeRose. Geometric programrmng;coordinate-free approach. ACM ."'.~~~i\."·U.
Tutorial Course No. 25, 1988.
[7J Richard D. A primer: 11new SCRATCHPAD. In John Fitch,iter, EUROSAM'84 , Internationalsium on Symbolic and Algebraiction, Proceedings, pages 123-147, ....,"'...1..4>.•,
Springer-Verlag. Lecture Notes,,",J-.l'''U\'''''', voiume 174.
References
6: Performance comparison of Cactusqz--decompcae with Linpack SQRSL, on a Symbol
with FPA, Symbolics Common Lisp, GenPCL (AAA1'88 version) and Symbolics
r ortran 77. times are given in seconds.
n;n~a.IJl.e and to reuse in con-symptom that error messages
semantic content: "The vector V notthe domain of the transformation TO" as
opposed to "Attempting to as e t beyond thearray bounds of #<art-q 30711>."
Better evidence of the advantages of this levelof abstraction will be obtained as Cactus is extended to be a toolkit for problems in constrainedoptimization, which is an area intrinsically morecomplicated than linear algebra. Constrained optimization requires careful treatment of the un
the primaryen:rphasis hastranstormations rather than on spaces).
will be similar to work in computer graphicsby DeRose (2,3] and otners.
An.otl1er compelling aspect that deserves morecomoimng abstract aigeera
symbolic
11 . Acknowledgments
An outline of arizona.'j.'t:!r;hJl1c:al Report 131, Dept. of Statistics, U.
vvaJ.tE~r Noll. Finite-Dimensional Spaces (AlGeometry, and Analysis), volume 1.,
Steele. Common Lisp, The Language.jJJ.l::'J. ".c;w. Press, 1984.
Wolfram. Mathematica: A System for Doing Mathematics by Computer.Addison-Wesley, Redwood City, CA, 1988.