Some software tools used in thedevelopment of the prototype YorkAda compilerby Colin Runciman

Among the various software tools developed inthe early stages of the York Ada compiler projectare two families of tools described here. Syntax-driven generators, distinctive in their inclusion ofa mechanism for syntax abstraction and theirmode of adaptability, were used to produce avariety of compiler components; interfacecheckers, automatically generated fromspecifications of the information intended topass between components, were used to helpisolate and repair faulty components.

1 Introduction

The prototype York Adaf compiler (Ref. 1) was developed in aGNIX§ environment (Refs. 2 and 3), where the use of standardsoftware tools is normative. UNIX also makes it easy to intro-duce new tools on a par with standard ones, and this paper isconcerned with some of the new tools that were devised for thecompiler project. Use of these tools became a primary mode ofwork during the development of the compiler. The motivatingprinciples behind two particular families of tools will beexplained, the tools themselves will be described, and someassessment will be made of their usefulness.

It is a typical characteristic of compilers that their develop-ment proceeds from unusually full and definitive requirements,in the form of manuals for the source language and for thetarget machine. This characteristic often leads those develop-ing a compiler to consider the use of a genera tor. A generator isa tool that takes as input a specification, and gives as its outputa program, or program component. The first collection of toolsto be described here is a family of generators.

Another common characteristic of compilers is that they arelarge programs built from functionally diverse components.Both because of the wide range of implementation techniquesneeded and because of the sheer amount of work involved,compilers are often developed by a team of people rather thanby an individual. The composition and interfacing of compo-nents is therefore a vital concern, and in this connection check-ers, another kind of software tool, have an important role to play.

t Ada is a trademark of the CIS Government Ada Joint Program Office.§ UNIX is a trademark of AT&T Bell Laboratories.

140

A checker analyses its input to ensure that it conforms to aspecification; the specification may be built in to the checker orit may be a separate input. The second collection of tools to bedescribed here is a family of checkers.

As will be seen in more detail below, each kind of tool sup-ports the other. On the one hand, input specifications for gener-ators may themselves need to be checked; on the other, somechecking tools can be generated from specifications. Note thateach class of tools is concerned with specifications. Presentedwith any new software tool one naturally asks: 'Are distinctivenotations involved in its use?' and 'Do these notations makeprecise any useful information about programs that wouldotherwise be unexpressed?'. For the tools to be described, theanswer to both these questions is: 'Yes.'

2 Generators

Generators applied in the context of compiler development areoften referred to as compiler compilers. They have been usedto assist in the production of whole or part compilers for anumber of years. There are two important factors to be con-sidered when assessing the usefulness of a compiler compiler:

• Relative simplicity: How much easier is it to compose thenecessary specificiations than to compose (the relevant part of)the compiler unaided, and how much simpler is the argument ofcorrectness?• Relative quality: To what extent are expressive power in thespecifying notation and sophistication in the generating tooltogether adequate for the production of results that compete inquality with what can be programmed unaided?

Among the ideal specifications from the point of view of relativesimplicity must be the definitive manual for the source lan-guage. But prose description, even of the most exceptionalquality and accompanied by many examples, does not provide asufficiently precise and accessible specification from which togenerate software; and the only formal document typicallyincluded in the definitive manual of a programming language isa context-free phrase structure grammar defining the legal syn-tax of programs: no comparably precise technique is used todefine semantic properties. The language reference manual(LRM) for Ada (Ref. 4) follows this typical pattern.

Some groups developing Ada implementations have there-fore begun by developing formal definitions of Ada semantics(Refs. 5 and 6). In principle, a formal semantic definition could

Software Engineering Journal July 1987

be supplied as a specification to a compiler compiler; but, inpractice, the construction of necessary generators is still a mat-ter of research, and some parts, at least, of the compiler mustinstead be hand-written, following as closely as possible theframework given by the formal definition.

Formal semantic definitions invariably begin with the intro-duction of a so-called abstract syntax that is used as a definitiverepresentation for programs in the remainder of the definition.The abstract syntax is not strictly derived from the concretesyntax of the LRM and often writers of semantic definitions donot trouble to state explicitly the formal relationship between thetwo.

2.1 A syntax-abstracting generator

At York, we adopted a complementary approach. From thecompiler writer's point of view, abstract syntax is often closelyrelated to, if not identical with, an internal data structure of thecompiler that is used to represent programs and that will bereferred to in what follows as the AST (abstract syntax tree).Abstract syntax may therefore be seen as a component of thecompiler, and, as such, a possible product of a generating tool.

An attraction of this approach is that it promotes a uniformAST design, in which a small number of representational tech-niques are consistently applied. Further, it is a logical first steptowards the automatic generation of AST processors fromwhich a compiler is built. Once the derivation of abstract fromconcrete syntax is embodied in a generating tool, a generalisa-tion of this tool can be used not only for the production of thenecessary AST definition, but also for the production of a parser(accepting program texts that conform to the concrete syntaxand building a corresponding AST syntactic structure) and of anunparser (performing the reverse operation that is useful both inthe production of compiler reports and for consistent format-ting of source programs). The same tool could be adapted toproduce any compiler component whose chief concern is sys-tematic processing of the AST, whether or not this also involvesconcrete syntax.

2.2 Preparing the concrete syntactic specification

The most natural specification document to supply as inputto a syntax-abstracting generator for the AST is the LRM defi-nition of syntax. There must, however, be some non-trivial trans-formation involved because parse-trees of the LRM grammarare too cumbersome for use as ASTs. Although part of thenecessary transformation can be encoded in the syntaxabstraction procedures of the generator itself, the generator'stask is simplified considerably by the preliminary application ofa series of language-preserving transformations to the LRM'srules of concrete syntax. So we used a two-stage process: trans-formation at the concrete level, followed by abstraction. Thetransformations might have been carried out using an ordinarytext-editor, but that would have been error-prone and tedious.So instead, required transformations were implemented in aspecialised tool for editing syntax rules. The command script tocontrol this editor, operating on the LRM rules, had the followingobjectives:

• to reduce the number of syntactic rules to a minimum byin-line substitution, under the constraints that at least thosenon-terminals named in a given wanted list must be preserved(because they are the sites of planned tunnels to semanticinformation) and no new non-trivial common sub-phrases maybe created• to eliminate left-recursion and left-common-factors — thisis more important for top-down parsing than for the derivationof the AST• to redefine non-terminals in a given lexeme list as termi-nals, discarding rules thereby made redundant.

concrete syntacticspecification

interwoven concrete andabstract syntax

adaptablegenerator

generatedproduct

adaptor

J I

serviceroutines

Fig. 1 The adaptable generator with an adaptor fitted

Q

the nth call, shownby the relevant node

Fig. 2 The pattern of adaptor calls for a given syntactic structure

The necessary command script was produced automatically bya simple generator, given the LRM rules and the two lists as aspecification. In a few places, it was observed that there wouldbe some advantage in merging similar phrases, replacing eachby the same weaker, linguistically inclusive, phrase. A commandto perform this transformation was implemented.* The scriptapplying this command was hand-composed, together with acomplementary set of hand-written discrimination routines.These routines are used in the compiler to determine which ofthe original LRM phrases applies in each AST instance of amerged unit, usually by reference to semantic information.

2.3 Structure of the adaptable generator

Transformed rules for concrete syntax were supplied as inputto the adaptable generator, depicted in Fig. 1. In this diagramthe adaptable generator itself is the L-shaped part. Its inputspecification is a concrete syntactic definition. The generatorperforms syntax abstraction to obtain corresponding abstractsyntax, and passes both concrete and abstract forms to theadaptor, interwoven in such a way that the relationship betweenthe two is clear. It is the adaptor that determines what is gener-

• Since equivalence (and hence inclusion) of languages described bycontext-free rules is undecidable, the command requires a strongercondition of recognisable inclusion based on bounded compositionof a smaller repertoire of transformations.

Software Engineering Journal July 1987 141

ated. Different adaptors can be used, for example? to generateparsers and unparsers. In order to generate the required prod-uct, the adaptor may communicate further with the adaptablegenerator by calling a collection of service routines: availableservices include the supply of global syntactic information andhelp with production of formatted program text.

Where information about the concrete syntax was not rele-vant to the generated component, a standard concrete filter'wasused to filter this information out, and only an abstract adaptorhad to be supplied. The adaptable generator was used with thisfilter attached, for example, to generate the definition of abstractsyntax as a data structure, and, with more sophisticated abstractadaptors, to generate some semantic analysis phases. Fordevelopment of the adaptable generator itself, and for the vali-dation of its specification, checking adaptors were found use-ful. These generated not compiler components, but reportsabout the information supplied across internal interfaces.

2.4 Implementation details

The adaptable generator is implemented as a program withthe adaptor as a missing package. Primary information aboutsyntactic structure is passed to the adaptor by calls of a standardcollection of procedures that the adaptor is required to imple-ment. The pattern of procedure calls follows the normal struc-ture of a syntactic definition. For each rule in turn there is a left-hand side call followed by a series of right-hand side callsrepresenting a traversal of the tree structure corresponding tothe right-hand side of the syntax rule. Traversals involve visitingparent nodes both before and after their children, as illustratedin Fig. 2.

Parameters in a left-hand side call are:

• identity, which names this syntactic unit• origin, which indicates whether this syntactic unit arises inthe LRM explicitly or has been introduced by the syntax-abstracting rules of the adaptable generator — this informationis supplied so that components of the compiler can be gener-ated to issue diagnostics only in terms of entities named in theLRM.

Parameters in a right-hand side call are:

• tree address, which is a suffix that, when added to an expres-sion referring to an instance of the current rule in some abstractsyntax tree, forms an expression referring to the currently visitedbranch• origin, which indicates, for example, whether this branchcorresponds directly to a fragment of program source or is atunnel to another part of the AST or to deeper semantic infor-mation—required tunnels are specified in an additional input tothe generator: they must lead from syntactic units appearing inthe wanted list mentioned earlier• abstract descriptor, which is a template for an AST branch• concrete descriptor, which is a template for a parse sub-tree.

The whole series of calls describing the syntactic rules is pre-ceded by a call of an initiator and followed by a call of aterminator. Both initiator and terminator are parameterless, butlike all procedures of the adaptor they can obtain information bycalling service procedures. To give some idea of relative sizes,the source code of the adaptable generator is around 4000 linesof C and sources of adaptors range from 200 to 1200 lines of C.

3 Checkers

Limitations, both of program designers and of the machinesthat they use, dictate that large programs must be constructed

in pieces. A particular decomposition into pieces and the cor-rect co-operation between them can in principle becharacterised by interface descriptions. Recognising this, manyprogramming languages provide some means of defining inter-faces, usually in a way that can be enforced cheaply at compiletime, assuming that program components are compiled in acontext including relevant interface specifications. Forexample, to describe data values a programmer may specifytypes, and to describe procedures he may specify sequence,types and modes of use for parameters. This information isunnecessary for the interpretation of a correct program but is aform of redundancy deliberately introduced for the purpose ofclear presentation and to guard against errors. Some languagesalso allow explicit control of the use of procedures defined inanother component, by such means as import and export lists.Ada itself provides all these things.

Although useful to a certain extent, such provisions lack thedescriptive power required to prevent large classes of errors. Wewanted to specify internal interfaces of the compiler more fullyand with greater precision than is possible using typical pro-gramming language mechanisms alone, and to do this in sucha way that software tools could readily be applied to checkconformity with the specifications and issue clear reports aboutany violations.

3.1 Specifying AIR

A primary medium of communication between componentsof the compiler is a large data structure called AIR (an acronymof Ada intermediate representation). AIR is defined as a networkof strongly typed nodes. Each node belongs to exactly one ofthe lexical, syntactic, semantic and management domains(the syntactic domain, for example, is exactly the AST describedearlier). The notation used to specify AIR has, first of all, familiarelements of data type declarations: a node is either a group ofnamed items, possibly incorporating discriminants and vari-ants, or else a list of items of the same type. Items may besimple, such as truth value, but many are links to other nodes. Inaddition to this conventional type of information, each link maybe annotated in a number of ways, as follows:

• shared — it is possible that other nodes also have links tothe same destination: links not marked shared must haveunique destinations• optional — the item only constitutes a link if it satisfies apredicate non-null: following a link not marked optional mustalways lead to some AIR node• beyond (pocket) — for some purposes it is unnecessary toprocess the entire AIR, and so connected sub-networks or AIRpockets are defined: when a link is marked as shown the sourcenode is inside the named pocket but the destination node isoutside; if a link is not so marked, membership of the pocketmust be the same for source and destination nodes.

Example 1:The concrete syntax rule of the LRM defines a component

association as follows:

component_association =[choice{"|" choice}" = > expression.

As a straightforward example of the annotations used in thespecification of AIR consider the definition for the correspond-ing AST node:

node(component_association)optional list_of (CHOICE) choicesshared EXPRESSION expression

end node

142 Software Engineering Journal July 1987

(Notation has been changed slightly from that actually used, toavoid the need for further explanation.)

Each node definition is also headed by strictly redundant infor-mation indicating, for example, whether such nodes are evershared and which pockets they may belong to. This redundancyis deliberate: it increases the value of the specification as areference document when developing components and itincreases the opportunity of detecting specification errors.

All elements of the specification described so far apply uni-formly to AIR, no matter by which component it is being pro-cessed. A further set of link annotations enable linkspecifications to incorporate dependencies on processinghistory:

• before(phase) — a link must be constituted if and only if theAIR has not been processed by the named compiler phase• after(phase) — a link must be constituted if and only if theAIR has been processed by the named compiler phase• during(phase) — a link may only be constituted if the AIR iscurrently being processed by the named compiler phase.

Example 2:Type checking in Ada often involves overload resolution:

many definitions may co-exist for the same identifier, and theappropriate interpretation for each identifier instance must bedetermined. This resolution process could be regarded as aphase of the compiler that, crudely summarised, establishes atunnel from every expression node (in the AST) to some typenode (in the semantic domain):

node (expression)after (resolution) shared TYPE type

end node

If such phase dependencies were not introduced into the AIRspecification, the second line in this example could only take thefollowing, much weaker, form:

optional shared TYPE type

A straightforward tool was implemented to check AIR specifi-cations for internal consistency. Since the Ada languagechanged frequently, and sometimes significantly, during thedevelopment of AIR, the consistency checker was an indispen-sable safeguard against the introduction of errors into the AIRspecification.

3.2 Checking AIR flow between components

The whole compiler can be viewed as a collection of compo-nents that communicate by the flow of AIR representing the Adaprogram unit being compiled, as depicted in Fig. 3. The pro-gramming primitives used in the compiler are such that there islittle inherent protection against errors in the construction ormanipulation of AIR that constitute violations of the AIR specifi-cation; hence the need for a checking tool to analyse the AIRbeing processed in any part of the compiler and to report anyways in which it deviates from the specification.

This checking requirement is met by a pair of tools: thedeflator is applied to AIR and stores a linear representation of itsinput in an AIR file at the same time as checking that the AIRbeing deflated conforms to the specification; the inflator isapplied to a previously created AIR file and reconstructs from itthe corresponding AIR. Each of the tools can be restricted tooperate only for specified AIR pockets.

source

1parser

AIR flow

— r "\\\\\

4library

manager \\/ \

/ \/ \

declarationanalyser

1i

usageanalyser

11i

unparser

ii

codegenerator

object

Fig. 3 The compiler viewed as a collection of components com-municating by AIR flow

programming environment tools

call

AIR producing and consuming tools

call

AIR manipulation procedures

call

abstract data type operations

call

call

Fig. 4 The compiler viewed as a collection of procedures

Both the inflator and the deflator were produced automati-cally by the same generator. The generator accepts the AIRdefinition as its primary specification. The mechanism used forinflation and deflation requires that special tags can be placedon nodes during a traversal of the AIR. These tag fields areintroduced by the standard GN1X pre-processor that is used totranslate AIR specifications into type and data declarations inthe implementation language C. So the notations present in theAIR specifications have at least two interpretations: so far as thespecially constructed checking tools are concerned notationshave the meanings explained earlier, but so far as the C com-piler is concerned they are just invocations of macros and, assuch, get replaced by valid sub-phrases of type declarations.

Once the checkers were implemented, and in view of thecontinual revisions made necessary by language changes, twofurther annotations, obsolete and projected, were introduced asappropriate in the AIR specification. The inflator and deflatorwere re-designed to act on these annotations so that in somecases 'stale' AIR could be processed or futuristic componentsapplied (with the issue of suitable warning messages) where thiswould not otherwise be possible.

Software Engineering Journal July 1987 143

33 Procedural interfaces between components

Before concluding the discussion of checking tools, it isworth examining briefly the compiler structure induced by therelationship of procedural invocation. Each of the componentgroups represented as a box in Fig. 4 may be viewed as apackage defining some types and procedures and perhapsmaintaining some local objects. Our aim was once again tospecify interfaces more precisely than usual programming lan-guage mechanisms allow, and to implement checkers toenforce these specifications.

Package specifications in a programming language such asAda characterise acceptable procedure calls by their par-ameters alone without reference to other procedures of thepackage: they characterise acceptable parameters by their typealone without reference to other parameters of the procedure.But the constraints of an interface often include both relation-ships between procedures, such as a pattern of calls, and rela-tionships between parameters in any one procedure call. It isarguable that the number and complexity of these interfaceconstraints should be minimised in a good design, but it isexceptional that they can be vacuous. Further, many packagesrely upon implicit communication between their procedures vialocally maintained objects. The properties of these objectswhich must be preserved for such communication form part ofan interface between procedures and should be specified. Theformulation and proof of such properties as data invariants is acentral activity of formal methods such as VDM (Ref. 7).

Pre-conditions for procedures in the prototype compiler wereformulated as ordinary truth-valued expressions; invariantswere formulated as an additional truth-valued procedure in therelevant package. Using the standard UNIX pre-processor, andintroducing a very small compiler component to issue diagnos-tic messages, it was not difficult to arrange both for the checkingof pre-conditions and invariants, and for suitable reports to beproduced when these checks failed.

4 Discussion and conclusions

The adaptable generator was used to produce a number ofcompiler components ranging in source size from a few tens toa few thousand source lines. In total, about half the prototypesource was generated by tools. The remaining half largelyprovided implementations of abstract data types assumed bythe generated code.

It is difficult to assess the relative quality of generated prod-ucts as a whole. The nature of the system was such that theprimary factor limiting quality was the ingenuity of the adaptorwriters. In the case of some generated products, there is littledoubt that we could have obtained unaided a result of equalquality with less effort. However, even in these cases, use of thetool proved to be an advantage. Modification effort was often farless than would otherwise have been the case. This was justwhat we hoped and expected — it was part of the originalmotivation for using a generator. An unexpected advantage wasthat the discipline of designing an adaptor often gave rise to newsystematic views of the component being developed. Also,some components would have been extremely difficult ortedious to write unaided; yet they required little effort to produceusing the adaptable generator. One example in this category is atable representing various relations between lexical phrases:this was first generated itself and then used in an adaptor togenerate the parser. Another example is a display routine forASTs — not a compiler component but a useful developmenttool.

The AIR-checking inflator and deflator proved to be invalu-able weapons with which to attack problems of isolating, fixingand testing faulty compiler phases. Phases could be strungtogether as elements in a UNIX pipe, with inflation and deflationbuilt in in such a way that the pipe composition operator was

effectively secure with respect to the AIR specification. DeflatedAIR representing a particular test case could be held indefinitely,and inflated for processing by a phase as required. The man-agement domain of the AIR contains information about allpreviously applied phases, including time stamps and so on (asimple tool displayed this information on request), and anyattempt to use an AIR file inappropriately caused an informativemessage to be issued.

The usefulness of checking AIR against even a weak specifi-cation is worth stressing. The next step up towards a full formalspecification of internal interfaces would have been a set ofinterpass AIR predicates expressed in a general programmingnotation: to avoid specifications as complex as their subjectsthis would have required a generator to factor out traversal andcommunication methods, leaving the specifier to formulateproperties only for node kinds involved at the relevant stage ofcompilation.

This paper has only described a sub-set of the tools applied inthe compiler project. For example, generating tools were alsoused for the construction of code generators, driven by tem-plate-based specifications of target machines. Several toolswere devised to assist with version and configuration control,problems which exercised us greatly: in this case our own toolsformed a layer on top of the standard UNIX provisions.

Although they were built particularly for use in the compilerproject, the tools described in this paper have potential appli-cations in any large software project that involves several com-ponents operating on a shared complex data structure.Generating such components from a common partial specifi-cation avoids needless multiplication of effort (for both originalconstruction and subsequent modification), ensures a degreeof consistency and encourages a systematic view of each com-ponent's task. Specifying and checking the interfaces betweencomponents enhances the protection against inconsistency,enables faults to be detected early and offending componentsto be correctly identified.

5 Acknowledgments

The author was a member of a team working on the Adacompiler project. Much of the AIR design, in particular the linkannotation system and its interpretation, is due to Chris John-son, who also, as the author of several adaptors, influenced thedesign of the adaptable generator. The design and implementa-tion of the AIR-driven generators that were used to producedeflator and inflator tools is due to Jim Briggs. The project wasfunded by the UK Science and Engineering Research Council.John McDermid commented helpfully on an earlier version ofthe paper.

6 References

1 WAND, I. C. et a/.: 'Facts and figures about the York Ada compiler',ACM Ada Letters, 1987 (to be published)

2 'Special issue on the UNIX time-sharing system', Bell SystemTechnical Journal, 1978, 57, July/Aug.

3 'Special issue on the UNIX system', AT&T Bell Laboratories Tech-nical Journal, 1984, 63, Oct. (8, Part 2)

4 The programming language Ada — reference manual*. LectureNotes in Computer Science 155 (Springer-Verlag, 1983)

5 BJORNER, D., and OEST, O. N. (Eds.): Towards a formal descrip-tion of Ada'. Lecture Notes in Computer Science 98 (Springer-Verlag, 1980)

6 CIHL, J. et a/.: 'An attribute grammar for the semantic analysis ofAda'. Lecture Notes in Computer Science 139 (Springer-Verlag,1982)

7 JONES, C. B.: 'Systematic software development using VDM'(Prentice-Hall, 1986)

Dr. C. Runciman is with the Department of Computer Science, Univer-sity of York, Heslington, York YO1 5DD, England.

144 Software Engineering Journal July 1987