A-0101 953 DISTRIBUTED SYSTEMS TECHNOLOGY SURYEY(U) /CRNEGIE-MELLON UNIY PITTSBURGH PA SOFTWdARE ENGINEERINGINST E C COOPER MAR 97 CMU/SEI-87-TR-5 ESD-TR-S7-iS6

UNCLASSIFIED FIG 12/ NL

W N2.0

L

MICROCOPY RESOLUTION TEST CHARTNATIONIAL UUREAU OF STANDARDS- 1963-A

-- Carnegie-Mellon University

-* Software Engineering Institute

WnDl0trtd O . Tchnoloby Survey

I-

JUN 3 01987

to smog" ." "k

87 6 29, 036' 7

. ....

Technical ReportCMEM4-Th-s

ESOTR-S-106WMh IN7

Distributed Systems Technology Survey

Eric C. Cooper

Aooousion For

DTUC TAIB1hanUMMOMOd 03

.Tustrifitles

AVailability Codes T IF.E .CT~ Eu

Approved for pubic relm.Diutrbutlon unfifflhed.

Software Engineering InstItuteCarnege-Mellon Univrsfty

Pitsbrgh, Pennsylvanla 15213

7;;;;.... ...

This technical report was prepared for the

SEI Joint Program OfficeESD/XRSHanscom AFS. MA 01731

The Idm arid sid1I In this report shoul not be construed as an officialDoO position. it IS pulished In the Interest of scient and technical

Review and Approval

This report has been reviewed and Is approved for publication.

FOR THE COMMANDER,

Karl H. Shingle!,SEI Joint Proga Ofic

This do=umnW Is abo~ Uwtuo Ow Delsm Tedminlod- Isndmn Cme~wr. DTIC provlds s aes w and Wansfer ofIII a W I e do oml mgm on r 0.0 paertel DoD aee - -Figend poordia on fralsr. Ondol@6w U.S. Governmen

qsn~sfd ed 6 cuWrams o balinea ooplpsee -sm OiTI dmoy: Deleio Tecdaol inhrfmnCenkr.SimanoSoo Aleundl. VA 222044145.

Cooisa of ftis docmamet we abo awW~le 11ugh to Nallond Tedwh nio drm6on Services. For kinornion on ordeing.coased TM deadly: Naedoa TedwdMiad loftlo Servics. U.S. Deprinrt of Commnerce. Springfield, VA 22161.

11 9 1 1 il I ,-I I

jA

Tbof C o nt 9

1)bemdudnj)

2

mh rnetarks 3

PO"7ols 3Rsfsrsnce Model 3

4.2. Trarpor Protocols 44.3. HVghe-Level Protocols 4

8) PormIng Languag buep 7

9) s)Reo Procedure COU 8~Ul~lh~U RInoeProcedure Call 9( 9.2. D1s=is ratsof Remote Procdure Cml 9

0) Solwo Tools for Dlstutsd 9alonet

14 DitWbutedl File System'~1

12.3. kftgangV Workstation Disks and File Servers 1212.4. hitgratig File System Name Spaces 1212.5. File Servers versus Database Servers 13

i3) Fult Tolernce 1313.1.T13

2. ssWTraraclons14141 Coclus14

FamwordThe Technol o r e gge and Assessment PxOjeo omnhed a number of related kwestl-

I eu 9n techneology hI specM poblsm -ra I* review research and developmentmnull ad oo-s do ave wdie prodoiw;

* new technologie vuugil eeu~ eier ws ofresearch anddevelpmernt results, ped-odi srveys of speuoftes aw~ aeadstlcloni of paticularly good exaffqfts of thesppion of speclic techologise wtl

*requIrsee for now tedmology 1Vmgi oostiing atudles of software developmentnooke wil fte DoD. and case stdies of bolh successful and unsuccessful pr*j

Tedinuisgy aseset kwslve udsmiwn lte software development proes detemk~tg"S Feei A! UM te8chnology for solvig; %p MC- i p obles a. OVAIUM" ng MR1 oftrtolo sf w, mal 0 0ng gIbn lotecoologies to neede, and! dleninkn the p olewtW payWf ofMawW edholele As-sssment adivile of ft projed focuse oqa c otehnolog aa for

ITi Fpew Is ens of a seriee of an"ve upses. It Is not k 1nded ID prOvide an exhaustivedoussion of opls puiu is US ma Dfts of dotis ss ems Ndtelg. Rather, II is ides

as an Wroovs seview of US ltdwgy suveyed These sue owe omn~ad I BIl 190ad "my~ "n.

Members ONft poda rswde WO -vi umlal leohuwlogl surveys have been Moucled&I ~le heinooftfs The pjd d nt isWe is iU! to o rne surveys, but focused onpleno oftesssdi In thse suewys. 7Me WOa Ini 1 1nA, ing US sur**v was not Io dsr lbeaUS techfoo In hi en d I*A is ephmis ses OWa ~te 9Mw a ong Impad on or are

sla oi 8011110 InIUS 5U1nheU The eOV Own pus h port is Isprovide an oveeview oft t lesh lie US w e Mel deeoigsi e ni sw**Iro-

1IntroductIonOne of US cwr technology areas I whi! project mMoeo were blemied Is dutsAe eys-tealsdhnoog. This reoW surveys the technical wouenhole hi eIn i dlstrlbuled eye.

Pson ona lils US ngforcebehind USpsoraln odf dislrlu ed sytem. oolaftiosaracoaleolsolle mey of provdin conpilin power is WdkAaies. Local mra netwoes are a ot-etlihe my of shing mane to more eupensive and Ame freqetl use resources we lasr

~eir an W"g d"sks a Ufa as a meons fo users; to shae k"lorwtl.

In arbisd sysiems, however, Owre Is a fundaenal dichotom belooeen USe need for heegra-Von (is achieve shtang and USe need for sulony (to control one's local environmient). Many of

$1110 tehIFa SCAL111nSe181 presSedheecanbe valisted i terms of how they balancethese two

Olbbusd-W-tema hawe a nunbera06weil kown" polen Ial beneft:

" Muideriy: Resumes cam be ade and reconhipired easily." Psefoemuic: Puallelemn can be used lo peefon joft more quickly." AwalobIT Pedurmoncy can be uised lo poide non-sop service.

To acheooe these beneft I pravce , however, requires solutlons, for a nurvte of dificult tech-.ici prolems. Also, ther are traoff volalonshois between, for exampfs, using a distutdsyste f rese peeforwne versus usMig ft Increased avallobhIly. The folowing ec-lions discuss; some of the Important tchnolos and bsues Involved in dsriued systems.Refee a es I o opp op INt surveys wre Icuded I the discussion, but two general references areapproprIate here. Firs, ft book e dite by Lamoeon of a. 1181 is anl excelMt overview of manyusoet of dlostuted systems. Second, Tanerims book on coniute networks 136] Is prob-aly em best Wut! 9 point for mnore kdonet on networks and prolocols.

2. Hardware TechnologyEcnomic factor we a majo reonm for the prolierat of dlstbted systoe. Processors,memory, an magne tc ndoptical diss ar sufnt iexpne oallowan oganizationtodeflo a workstation MInevery office, Minaddltonto supportinga macdhn room with maInframnesarvi ft servesa. Curt workstations typically have from I to 10 MIPS (mI1o1n MIstruotlons persecond) of procesing power I to 10 megabytes of memory, 20 to 200 megabytes; of diskstrape, an codt Ion than 20,000 dollars.

A variet 06 network technologies we avallabis for- Voeront n s conort. A distinc-Von Is commionly made betwee local ae and long-hail networks. For both physical and ad-,udnlstrtv0 reasons, local ame networks (LANs) are typically used within, and managed by, asingl organtilon. Currently, most LANs are construced from coaxial cable or fiber optics, withbuiwiAdtl 1 raniging frm 10 to 100 mfin bit per second and longths on the order of Ikilomeoter. The iNoecon-nectlon topologie of such networks include bus, rIng, tee and star struc-

s.

Long-houl networks, on the other hand, can span confinerga, and are usuaNl managed by comn-panies; or goverrmert agencies; for use by others. The technologies used for long-hsul networksftkitje telephone knm and satelkie Inks. Long-hail networks use one or more of the following

ewfth vitwr~s:

" Oaul sol -" MmsopsAg h-" Paicket WAWA~

C=A* swithn Is used I the telephone system. I thI schem, a route through the network IsetIshed1 beore anyW data Is sent. Slice comnmunication between computers tends to come in

M2

bursts. circuit soitchkn doe not provide good utilization of avalable bandwidths, and the timeretqilr to pro-alocate a circuit may be unacceptably long.

Messge-withedand packet-switchd networks awe als called sicre-and-o'waid networks be-couse data Is andepeindentl roued from one swichin node to another. For reasons of refllbity,the ne Wor ~oplg of a storeand-forward network should be connected redundantly, so thatthere we several paths or routs between any two nodes.

In message, swiching, Individuial messages we routed from one switching node to another. Thissilninatse the long set-p tim associated with circut switcin and provides boner utlization ofbandwidths. The disadvantage of this approach s thW fth variable size of messages nukces ktdficuit to allocate node buffering resources efficiently.

I packet switching, messages we first broken up irfo fixed-size packets, which are then Wnividually routed through the network and resseuled at the dstlinastion. Different packets of

the same message mray be routed along different paths, and hnce may arrive out of order.Higher level protocols we used to hande out-of-order packets. Since all packets are of the same(reltvel smal) size, buffering at Intermediate nodles le simplilied. The ARPANET le an examrpleof a long-hail packet-switched network.

3. Intemetworksfecause of their low cost, workstations tend ID proilerafe I organizations. and the need for LANstends Io grow a well. Large organizations are soon faced with the necessity of connetingseveral LANs through a structure called an Internetworc. In fact, someo of the components of anIntemetwork can be long-hail networks. I the MARA Internet, for exampile, the long-haulARPANET Is connected with hundreds of LANs at univeisites; and research laboratories. Thesame etore-and-fouward approach can be used In an Intemnetwork, viewing the Internet gatewaysas the packet switches of a single larger network.

4. ProtocolsProlocols are used to provide virtual cormmunication services with properties different from (andtypically at a higher level than) those provided by the physical network. This leads naturally to alayered model of prolocols, such as the one that has been standlardized i the ISO ReferenceModel for Open Systems Interconnection [171.

Further discussion about protocols may be found in the survey article by Tanenbaum [37].

4.1. 1S0 Reteirenos ModelThe ISO Reference Model consists of the following seven layer:

1. Physical layer low-level commnunication of bits2. Data k layer franing, checksumning

3

& Netork lWer. tmtwolc addrskn and routin4. Transport e relable communicaion, hos-to-hot addressing5. Session layer: connection management, process-to-process addressing6 Preenation layer data formatting, encryplion, compression7. Application laye. user programs

The ISO Reference Model does not match all protocol architectures perfectly. In the DoD Internetfamily of pmtocols, for example, the IP [26] and TCP [271 protocols provide functionality thatranges from the data link layer to the session layer.

4.2. Transport ProtocolsConmunklcon services can be characterized by a number of attrbutes:

" The need for a connection establishment protocol before communication can occur" The number of communicating entities" Relabliy of data delivery

" Client inleface (messages or stream abstraction)" Fixed or variable length of messages

The bli Is a brief characterzatlon of a number of transport prolocols according to the above

" Daagram protocol: connectionless, unreliable delivery, fixed-size packets. Ex-amplas Include the Xerox PARC PUP protocol, the DoD Internet Protocol (IP). andVmh DoD User Diagram Protocol (UDP).

* Byte stream protocol: conection-based, reliable delivery, stream abstraction. Ex-ample Include the Xerox PARC byte stream protocol (BSP) and the DoD trans-mission control protocol (TCP).

" Message protocol: connectionless, reliable delery, varable-length messages. Ex-amples include the protocol used by the Spice system [29.

9 Requetrp-onse protocol: connectionless, reliabledelivery, variable-length alter-nating request/rponse messages. Examples are described by Birrel and

A ecm topic of research has been the Incorporation of many-to-many communication semanticsInto varous trainsport protcols 1, 8, 9. New protocols in each of the above classes wi lkely beeIon-do with many-o-many semantics.

4.3. Hlgher-Level ProtocolsHigher-level protocols, those implemented at the session layer or higher in the ISO model, arecorrspodnly harder to characterize. Examples from the DoD Internet family Include the Tel-net nwork terminal protocol, the fie transfer protocol (FTP), and the mal delivery protocol(SMTP). Other areas of research include protocols for graphics, window managers, voice, multi-media messages, bootstrap ioadng, remote debugging and monitoring, and remote procedurecal, dicussed more fuly below.

4

5. Heterogeneitycurrnly ther IS no0 $Ingle standard machine architecture, opraing system, programming lan-guage Or Programming ekWironmet, and such standards are not "iet t0 appear In the nearfuture. As a result, organizations fin themselves faced with the problem of Integratng a hetero-geneous collection of such resources. As evidenced by a recent workshop in Eastsound,Washington, that was devoted sole" to the problems of heterogeneity, and by current researchprojects In heterogeneity at Institutions such as CMU and MIT, t Is the key problem In distri-buted sysems today [241.

The most Important pittal to avoid Insa heterogeneous system Is the lowest common denominatoreffect. This occurs when Interfaces are only defined for those operations tha are supported by allcc iponents In the systm. As the rnmer of heergneu components Increases, this set ofommon operations may approach the amplty set.

A nurnber of techiques can be used to avoid the loetcommon dnominator effect. Onetechnique Is a common data representation protocol, In which al communicating cormponentstranslate tei Interactions Into a standard external representation. As described below, this canbe handled automatically In remote procedure call system through the use of a stub generator.The main difculty with this technique Is that the representation protocoll itee suffers from thelowest common denominator effect. The advantage. however, Is that such protocols are flexblesOnc they are capable of representing arbitrary programming language data types We arrays andrecords. DeSchon surveys a miunier of data repreetto standards, [101.

Another technique Is called option negotialtion [341. In which each pair of commnunicating pariesInegotiates which protocol options they wil support. This approach alos each pair to commu-nicte with maximal functionality. The option negotiation approach Is appicable at many levels Ina 1"eterogeneU s -1Q_ distributed systm.

The data repreenIationI protocol and option negotiation tectviques can be successfully corn-bined. For exanmple, the reote proecdur call systm at the DEC Systems Relsearch Centeruses negotiation at bindiq ngom to decide betow two possil data represeNio protlocols.

A third and somewhat ad hoc approach to coping with heterogeneity Is the proxy tehiqe Aproxy Is a specialized agent In a remote environmert whose purpose Is to provid an Interlace toth a eirorwnen tha Is more cornpatlble with other comp~onents of the systm. This approachwas firs used6 In remote job entry (RJE) systm to access batch facilties from timnesharingsystems. It has been used successfully In the Looms system [253 to li eg ae I13M mainframestrensparerify Into a distrbuled UNlx' ovirorynerg.

0WOWN

6. Models of Distributed Programs

Although transparency is desrable at the highest lvel of a distributed "stem, at some lowerlevel the fact that te system is distributed must be made avilable to the pogrmmner. How thisis done is brge determined by the model of distribted programs that the system designeradopts.

One of the most well known approaches, developed at Xerox PARC In the 1970s, Is called theclen/server model. The computing environent is assumed to consist of personal workstionsand a coleclion of shared network services inplemented by server machines. Such servicesmight Include fle strage (discussed more fuly below), printing, and electronic mail. The pro-grams runnin on the users workstation are viewed as clients of these servers. The cliert/servermodel is a skple extension of the application prograv/operating system model familiar In central-ked timesharing systems. It is flexible because new services awe easily added, and I supports aheterogeneous environment wel: "Black boxes" can be used as serers as long as some inter-face can be constructed on the client side. A disadvantage of the cllenlserv model is that Itdoes not support bad balancing or multi-machine parallel applications, although such programstructures can be shoe-ornmed into this model by using a pool f "Mcompute servers."

Some of these deficiencies are remedied In the network operating system (NOS) model. In thismodel, a transparnt Interface to all network resources is presented to tho applications program-mer. not just at the user Interface level. The Locus system at UCLA [25] and the Spice system atCMU [29] are successful examples of systems that follow this model. A major disadvanae of thenetwork operating system model is Its difficulty In accommodating heterogenety (in the form ofblack boxes) because It assumes that a common software interface can be installed on all thenetwork resources.

7. Operating System IssuesThis section briefly descrbes a number of operating system features that are particularly impor-tug for supporting distributed systems.

A message-based operating system consists of an efficient kernel imllemen~ttion of processes,virtual memory, and inter-process communication, together with a set of server processes provid-Ing conventional opsing system services such as device drivers and file systems. The Accentkernel is a prime eftamnpe of a message-based system [29.

Message-based kernels alow inter-loess communication to be extended over the network in asimple and transparent fashion. The key is the notion of intefediary processes that interceptremoty destined messages and pertorn the a pro te forwarding.

There gImsno agreement that a ightweight process mechanism is essential to support com-monly used distributed prgram structures. A number of lightweight processe can share a singleaddress space; this allows the conaruction of servers, for example, th correctly handle concur-rent Incoming requests. The lack of such lightweight processes has been a weak point of UNIXand a number of mesg e operang systems.

A proces migration facilty lows a running process to be moved from one machine to another.Such a facility is a valuable mechanism for implamenting bad balancing policies, whereby jobsae moved off heavily loaded machines and onto lightly loaded ones. Variants of proces migra-tion can be used to increase fault tolerance by checkpoinng process state. Process migration isgreatly simpli fed in message-based operating systems [28.

A simpler form of load balancing can be accomp shed at task creation time by starting the taskon a lightly loaded processor. Further experience Is needed to determine whether the full powerof process migration is necessary.

Workstation technology has advanced to the point where most new hiWh-end workstations aremutiprocessors with approximately 10 processors. Operating system support for muftiproces-sors, and In particular for efficient execution of parallel programs, will be an Increasingly Importantrecuirement.

Finally, UNIX compatibilty is often a practical necessity. The wide variety of software tools avail-able under UNIX would be prohibitively expensive to port to an Incompatible environment.

Many of the features mentioned in this section have been Included In the design and implemen-tation of the MACH-1 operating system at CMU [31, a kernel and programming environment thatwill probably serve as the new foundation for DARA-sponsored research in strategic computing.

8. Programming Language IssuesOne approach to Integrating disrbtled programming prmitves into the programming environ-ment Is to Incorporate them Into the programming language Itself. This approach can be accom-plished in two ways: the mechanisms can be buill into the language, or they can be providedextemaly.

CSP [161 and Ada [121 are examples of languages with buit-in communication primitives. Thisapproach extends the benefits of strong typing to distrbuted programs because the language isthe only Interlace to the communication mechanism. Unfortunately, most languages of this typeignore the problem of heierogeneous environments. As discussed previously, in order to copewith heterogeneity, some common data representation protocol or negotiation scheme must beused among the language kplementations on different machines. Without a language-definedstandard, programs produced by different compilers are unlikely to be able to communicate. Adaprovides only a partial solution to this problem In the form of pragma statements that allow controlover the representation of data types.

In message-baed operating systems, primitives for message communication are typically Inte-grated Into the programming language In the form of a subroutine rary. Again, Uttle support forheterogeneity has been provided. Issues of data representaion and type safety are usually theresponsibility of the programmer.

Remote procedure call (RPC) systems represent a compromise between the built-in and the

7

external appmach. By using a stub generator, the remote procedure call mechanism can beclosey coupled to, yet separate from, the compiler. This approach Is described in more detail inthe next section.

9. Remote Procedure CallReote procedure call Is a conbined protocol-level and language-level mechanism for construct-ing distributed programs. A remote procedure call mechanism allows a programmer to write adistributed program in the same way one writes a single-machine program: using procedure callsIn one's favorlie programming language. Remote procedure call meshes well with both theclient/server and network operating system models.

The language-level Wtegration of remote procedure call into a conventional programning lan-guage is typically accomplished by the use of a stub generator, a specialized compiler that trans-Istes a module Interface into stub procedures for the client and server halves of a remote inter-face. The stub procedures handle the details of representing the data types of the programminglanguage in an external form when they are sent in messages, and the converson to and fromthe Internal form. The stub procedures also interface with the lower level request/responseprotocol used to exchange the call and return messages.

The stub generator approach has a number of advantages:

* The stub generator manipulates source-level programs, so strong typing can be pro-vided.

o The stub generator Is separate from the compiler, so the same stub generator can beused with any compiler for that language.

* The stub generator is a natural place to 'hide" knowledge about the external repre-sentation protocols and/or negotiation schemes used between heterogeneousmachines.

To invoke a remote procedure, the client stub builds a call message containing the name of theprocedure to be invoked and the external representation of Its arguments. The client sends thecal message to the server machine, where it is Interpreted by the server stub. The argumentsare converted to their Internal representation and are passed to the named procedure. When theprocedure returns, Its results are externalized In a return message and sent back to the client.Finally, the client stub converts the results back into Internal form and returns them to the clientprogram.

Nelson gives a comprehensive treatment of remote procedure call in his thesis [231. Birrell andNelson describe the transport protocol and binding mechanisms used In an Implementation ofRPC at Xerox PARC [6].

S

9.1. Advantages of Remote Procedure CallThe single biggest advantage of remote procedure cal is that it makes writing distributed pro-grams almost as easy as writing single-machine programs. The same software developmentnetotdoMoge that work well for centralized systems, such as the use of modularity, abstract

data types, and stepwise refinement, continue to work just as well when extended with remoteprocedure call.

9.2. Disadvantages of Remote Procedure CallAlthough remote procedure call has become extremely popular, it Is not a panacea. In particular,ii s not suitable for the transfer of large amounts of data, or for communication over high-latencymedia. Special bulk data transfer protocols are preferred in such cases.

One common criticism of remote procedure call, namely that the synchronous nature of remoteprocedure call does not allow any parallelism, is really not a problem. In fact, remote procedurecall neither helps nor hinders parallelism. The above criticism is usually accompanied by anargument in favor of non-blocing remote calls, where the application can either poll for the returnvalue or have it delivered asynchronously. Such features are actually a poor man's substitute forlightweight processes, and are only desirable in environments where processes are heavyweightand expensive. If Ightweight processes ae well supported in the programring language andenvironment, they become the natural means of achieving parallelism in conjunction with remoteprocedure cal. f not, poling or asynchronous delivery mechanisms can be simulated withremote procedure cal, but use of such features can result in rather convoluted programs. For themost effective match, systems should support both remote procedure call and lightweight proc-eses.

10. Software Tools for Distributed EnvironmentsMaking software tools function transparently in a distributed environment often requires substan-tial effort. Consider some of the tools that have become standard equipment in centralizedenvironments:

*Compilers* Uinkers

SDebuggers*Profiing tools

e Version control and system conflguration tools

A number of Issues must be addressed when extending these tools to distributed environments.

IrogramvIng language ompil and Interpreters must be Integrated with communicationfaclles such as message primitives or mo procedure call. The software engineering issuesm compicated by mechine dependencies, language dependencies, and compiler dependencies,

any one of which can el~ec the representation of programming language data types in messages.

Debuggemust be extended to allow single-stepping across machine boundaries when following

9

* V.

a chain of remote procedure calls. It should be possible to set breakpoints in remote modulesand to trace the flow of control of a distributed program. An advantage of nessae-bsed operat.Ing systems #or ditributed debugging is the abilty to encapsulate the entire environment of aprocess, since all of its interactions occur via messages.

Profiling tools provide the progrmmer with histograms of where time is spent in a program. Thisalows the programmer to detect bottlenecks and to apply optimizations where they will do themost good. In the distributed case, profiling must work correctly when portions of the programexecute at remote nodes.

Version control and system configuration is a particularly difficult problem in a distributed envon-ment. Schmidt describes a variety of techniques for maintaining consistent releases of largesoftware systems In the Xerox PARC environment [311. Shared file servers, discussed below, areessential to the success of such a scheme.

11. ecurlity

A distributed environment raises a runer of securty Issues. First the broadcast nature of mostlocal area networks makes them particularly vunereWl o eavesdrtopng. Anyone with a per-sonal workstation on an Ethemet can easly monitor all networ trafflc. Secondly, the lack ofcontrol over the software no In an Individual workstation mUe masquerades, replays, andsiia acolve threats possbl.

These problem are solved in single-machine or centraized environments by physical security:locked machine rooms and protected teminal lines. Uroreunately, the decentraized nature ofdis tbuted systems precludes such measures. Iodc rathe than physical schemes must beusedinstead.

The siOles problem I* solve Is tha of eaesrpping. The solution uses encryption: two per-sons wishing to communicate do so by encrypting all their message with a secret key knownonly to them. This effoctve constructs a secum private comnunication channel on top of theunderlying Insecure pujlc channel. The Data Encryption Standard (DES) can be used for secret-key encryption and decryption [211. Hardware Implementations of DES am available and shouldbe Inckued in new workstations.

More elaborate eicryption-based schemes can be used to solve the authentication problem, inorder to prevent masquerades and smlr active threats [11. 22. In such a scheme, a personcan securey Identify himel to anothe person by obtaining fm a mutually %Mod authen-tication service an proof of idnt that is unable to be forged. Brrell has dowrod a compre-henve scheme that provides both privacy and authemicaion for rmote pocedum sh [.

The eryption4-as schme that have been proposed In ft fterame do not afford muchprotection against denlmtlof-servce attacks. it has been observed ta passive threats are difficultto deec buteasytoprev whieoactivethreats are eaytodetect butdifficuitto preve

10

12L Distributed File SystemsDisibid fil systems have more Impact on progranuning environments ftn any othe aspect

of distrbuted syoem. A good discssion of file servers and distributed fie systems may befound In the survey m~cds by 8vobodova [3M.

12.. Mes aid DlnactoalasFils we the primary means of Wtn and sharing long-lved Information in computer systems.Flo systems may Impos structure on the conterts of files (Index or record structures or filet"pe) or may Vest the costs merely aequences of bytes. This report takes the latterqsproach and views a Usb as a sequence od uninterpreted bytes; any structure Imposed on filecontents Is viewe a a ogicall higher level. A common approach Is to deal only In machine-sensible unique Identfiers at the Us system level

A separate concept, often limped togethe with the Mie sysm, Is the directory system, whichprovides a mepping from ussr-sensible ames to Usl identifiers. Directordes may themselves beorplemns as fils, containing nami'efWler pairs. The directory system Implements creation,deltIon, lookup, and enumeratlon of namwedttllr pairs. Additional functions may Include ex-pansion of patterns containing wildoard characters.

Tlhe drectoy systemn is responsible for any structuring of file names. A common approach Is amre siuctured directiory system In which the full name of a Usl is a path name consisting of aseqence of =Mepnsre starting with the Woo drectory of the tree. For exmple, In the UNIXdrectory syse -probably the most common tree-structured system) the path name

VW mp 1W O denotes the Usl found by starting at the root directory (the leftmost 1. consuht-Ing the dirictory uarto find the directory sm which In turn contains the entry p~wertex In theUMI system. only the V/ Is Interpreted by the directory sem; Usl extensions such as Atx arepurely wivnvrion. Other directory stm provide more m&por for, and often more restrictionson, the use of Us extensions. Another feature of directory system tha is ntssin- from UNIX isthe provisin of muftbl versions of files. Versions are typically specilled through additional filename syntax, and Usl operations typically use dfferent default versions If none is specified. Forexample, openin a Usl for reading would default to the mnost recent version. while deleting a filewould deloull to the oldest version.

A final component Is the protection system, often subsumed by the directory system. For ex-ample, the direcity system can allow access control list to be, associated with each directoryentry, and can provide detailt acces contl through an Inheritance mechanism. Noe that ansams control mechanism presupposes some method of securiely identifying people. I a distri-builed environment, this can be accomplished with an authentication service a outlined above.

12.2 Sarng Files In a Distrbuted SystemThe ease with which fie can be shared In a distributed stm is a good measure of the overallmacces of Ow sysem. Several approache we possible. The lowest level technique Is the disksewrm. A disk sewver can be viewed as a ffuliported disk controlle whose I/O bus is the net-work. This eppvwah requires minimal changes to the operating system of the client machine,

'r11

since the Interface Is similar to that of a local disk. The abstraction provided Is simply that ofvrtul disk pages. Although read-only sharing of ties Is simple with this technique, write sharingposes difiules.

The disk server's interface Is too low-level to Implement concurrent write operations properly. Forexample, there is no way to lock a file or to enforce access controls. Instead, the client operatingsystems have to negotiate among themselves using a separate protocol.

An Intermediate level approach Is to provide an abstraction of files with unique IDs. The Interfaceto such a file server can alow individual blocks of flies to be read or written, as well as logicaloperations on the entire file such as locking. Fle servers of this type are usually accessed via adirectory system, which must Itself be a shared service.

The highest level approach Is to use a complete file and directory server, functionally equivalentto the file and directory system on a cliet machine. Interfacing is again simple because fileoperations can be intercepted at a high level and redirected to the remote server.

12.3. Integrating Workstation Disks and File ServesAnother Issue that is raisd when workations are netwokd with file servers is how to useworkstation disks most ofecdivey. One succeslul method, used in the Cedar file system (321,considers a shared flies to be imtiable (read-on), and uses each workstation file system as acache for sorne porton of the globally shared fle syslem. Fine e created on the local filesystem and remain prive uil they we glred back on h shared Go server. From that time on,that version of the file may not be modMed i, w.. -y be shmred by other users (sublect to normalprotection mechanism of course). Guararilseng coilency is relatively simple; the shared fileserv must provide atomic creation of a new version of a file.

A different approach Is taken by the designers of the Camegle-Mellon ITC file system [30).Workstation disks we also used as caches, bu shared files are not assumed to be immutable.As a result, cache valdatlon Is required, iNed elther by the workstation before using a cachedfl, or by the e server when a shared e is modffied.

12.4. Integrating File System Name SpamOnce fle servers we used to permit shng of fles In a nAtwork, Inrtgat of many file namespaces becomes an issue. The itegrated name space should allow a file to be named in thesame way from any machine in the network, In order to foster portable programs and minimizeconfusion when users change workstations.

If the different file serms are at the Intermediate or low level described above, integration can beachieved through a single (logically centralized) directory service. A more common case, how-ever, Is that existing workstations, mainframes, and file servers all have their own file and direct-ory systems that must be intgrated into a single name space. For tree-structured name spaces,two schemes we possible. The first uses a super-root that logically contains the roots of all filesystems in the networ Additional syntax Is used lo refer to the super-root in full pathnames.For example, the file name /JAurftb might be used to refer to usrab on machine A from any

12

ote machi e in the nOwork. Advantages of tis scheme are that i Is simple to implement andueranies consistent Interpretaion of ft names anywhere in the network. A disadvantage isa h is no tramparent sice the iocatlion of a remote file is reflected In Is full path

name. This problem can be circumented through the use of symbolic inks, I directory systemfeature which aim a user to kmpose an arbitrary view on top of the actual tree siruure.

The second scheme ailmow remote mount points in each local directory tree, so that each direct-ory system may have a different view of the distributed system. The problem with this approachis that consistent interpretation of names must be Obtained by convention; i is not erorced byany mechanism. The logic of name Iterpretation on the local machine Is also more complicated.On the other hand, I gives Individual machines more control over their view of the name space.

12.5. File Servers versus Database ServersThere Is growing agreement among designers of distributed ie systems that it Is kportant todistinguish between fie system and database system functionaliy. For example, file serversmust support efficient sequential reading of small files and creation of new versions of flies, butprobably do not need to support large files or synchronized modification of portions of files. Data-base severs, on the other hand, can be used for tcional updates to shared Information andefficient access to large files. Making this distinction aim optimized file server and databasesaver designs, rather than compromised designs stretched to ft both classes of needs.

13. Fault ToleranceAnother area In which distributed systems differ from centralized systems Is failure semantics.Partial fabures, in which some but not aN of the conponents of a system continue to function, aremore common In distributed systems and add to their complexity. Various mechanisms are usedIn order to cope with this complexity. The book by Anderson and Lee presents a thoroughoverview of faul tolerance tchniques [2].

13.1. Trammctions for ReliabilityTransactions are used to simplify the construction of reliable distributed programs, ones which donot lose or corrupt data. Transactions were first used in database systems [14], but have sincebeen adopted In operating systems [331 and programming languages [201. A transaction hashr essential properties, each of which must be guaranteed even In the presence of processor

and communication faires.

Serldzability, the first property, means that the concurrent execution of any number of trans-actions is equIvalent to their serial execution In some order. This property isures that I eachtransaction translorms a consitent database state Into another consistent database state, the

verall consistency of the database is preserved when transactions execute conurrntly.

The secon property is alomicky. which guarantees that a transaction Is an all-or-nothing opera-tion; no pertial effects of a transaction am ever visible to other transactions. When more than oneprocessor is Involved, this requires some form of distributed commit protocol, the most well

13I

known of which Is two-phase commit [14.19]. At any time before committing, a transaction mayabort, leaving the system state as I the transaction had never been executed. The fact thatIntermediate effects are not visible to other transactions means that the domino effect (cascadedaborts) cannot occur. When a transaction Is aborted, one can be sure that no other transaction,eilther still running or already committed, could have relied on updates perormed by the abortedtransaction.

The third property Is permanence, which states that once a transaction commits, its effects be-come permanent. Providing permanence In the presence of failures requires some form of stablestorage [19]. This Involves writing each logical page of data onto more than one disk and modi-fying the read and crash recovery operations to take advanlage of the redundancy. it Is stillpossible that the copies of the disk page can become corrupted In such a way that the readoperation would fail; but by Increasing the degree of repication, the probability of such a cata-strophic failure can be made arbitrarily snal.

Crash recovery mechanism use stable storage In two ways: for chedkpoints and logs. A check-point Is a snapshot of a consistent state ta can be rest od after a crash. A log Is a record ofthe events or operations that affect the state of the system; It is replayed after a crash. Check-points provide faster crash recovery, while kos ae less expensive during normal operation. N acomination of these two schemes is used, the log need only be replayed from the most recentcheckpoint, and the time between checkpoints can be used to balance the cost of the normal andrecovery modes of operation.

13.2. Nested TranactionsNested transactions are a generalization of single-lev atomic transactions, in order to allowthem to mesh properly with the concepts of composiion and abstraction supported by program-ming languages. In this scheme, a transaction consists of a tree of subtransactions. with a singletop-level transaction a the mot. The intermedlate effects of a transaction that has not yet com-mitted am visible only to its descendants in the tree. The effects of a committed subtransactionare visible only to ancestors and siblings In the tree. f a transaction aborts, any uncommittedsubtransactions must be aborted, and the effects of any committed subtransactions must beundone. The nested transation model was chosen for the Argus system at MIT 1201.

13.3. Replication for AvailabilityThe avallabilty of a system is the probability that the system will be up (either at a particular timeor on average). Replication is used to Increase the availability of distributed systems, eitherthrough the use of a primay/msandby architecture or via a modular redundancy scheme. In aprima4ry/sandby scheme, only a single component peorms Its normal functions; al the otheroOrrponen are on standby In case the primary falls. In a modular redundancy approach, all

componenta perform the same function, and some form of voting on the outputs Is used to maskfailues.

A classic prmy/fandby archileture Is Tandem's method of process pairs [41. The processesIn a process pair execute on different physical processors. One process Is designated as theprimary, the other as the standby. Before each request Is processed, the primary sends Ior-

14

manc abou t s Internal state to the standby In the torn of a checpoint. The checktpoint enablesthe standby to complete the request I the primary fails.

The s prooec at Cornell uses a phiaystn, architecture for replicated objects (5M. In eachInteraction iwith a replicated objec I Isis, one replica plays the role of coordinator, and only Itperfonn the operation. The coordinator then uses a two-phase commi protocol to update theothesrorplicas.

Triple modular and Nmnodular redundancy have long been familiar to designers of fault-tolerantoorrpuer sy"ems [21. i triple modular redundancy, ever y computation Is carried out by each of

three processors. The results are then ompared. and I at least two agree that vaiue Is used. InIthe Circus system, repliatin was Itegrte with remote procedure cal in order to support mod-ular redundancy A the program module level 191.

Gliords weighted voting schemne uses quonjms and version nutnbers to provide replicationtraninparency for fles[3. hInthis alorithm, read aridwrite quorums (sets of replicas) are chosenso that any read operation will iiciide t most recently written version. Heutlhy extendedGliord algoritm to handle replicated abstract data type [151. I HerlWs *poach, con-strit on quorumn assiginments wre derived ftrm analysis of the semnantics of the abstract data

14. ConclusionWell designe distributed systems should stric appropriate balancs between the needs forInegration and autonomy, and between the needs for Increased peuomiance and increasedavailablity. mhe itemized points beiow represent the features that project menbers recommendfor Iclusion In the operating system, programming languagle, and support environment of anyhLIIJT distributed system.

* Mesg-ad kernel* Transparent networlc Itter-process omruanlcaton* Remote procedure call faciy* Group communicaion 1trate*d with remote procedure calle Conventionael software toole extended for distributed environments0 IOwihprcesses

Distributed daab system& Distributed file system& tx 00 ii Wity

15

(13 Muduqu Aharnad and Arus J. Bernaleln.Mukas Co nca i In iUNIX 4.2M5.hIPme of 10 5h NOwu~lb COnbreewe on Dieltad Conput Systerw,

Powe 80-67. may, 1965.(2 T. Andeson and P. A.Lee.

Fail Toieam:.Ah~ R da wd Po'cIePreteIa, 1961.

33 Robert V. Barmn Richard F. RaUM.d Ellen Siegel. Avadle Tevanlan, anid MichaelW. Young.MACH-I: A mul~oer- Or sd Opening Syste nd Euwronrmr.hi ArthrWoiA (editor), New Con yut EnhraWlft: PWanWVecOr"arWSyfftOlc.

SIAM, 19m.143 Joel F. Balilett.

A NonStop Kernel.hIAAWO M~o*e1. 81h S~lrPoAim On OPerAVIhw 8)vaten. Pdn*C*A pages 22-29.

Deceridw, 1961.PtLONhd as OPrig Syaterm Review, I15(5).

(5M Kenneth P. Brmn, Thomas A. Joeeph. Thomaes RaeucWe and Anm El Abbad.hInlementlng F&au-Tolerunt Distrb Wed ObjectsI PMOPOe.W of Me 4th SPFPOSfm On Relllyt I Dibbed Soft* &we Datba

SyatehM pages 124-133. Ooer 1984.(63 Andrew D. Skrell and Bruce Jay Nelson.

h100smmng Rmoft Prorae Cab.ACM TraneactM On Con We *Mmwi 2(l):3-U, February, 1964.

173 Andrew D. brell.Secue Commuricoaon LWg Remo*e Pmmoe Cab.ACM TraCtlo On Covputer Sj&Vns 3(1 ):1-14, February, 1965.

[6" David R. Chedo and Wily Zmaenepoel.Dlstrbaed Proes Groups I te V Kernel.ACM Tamweottns on Conwute Sps 3(2):77-10o7, may, 1965.

(93 Eue C. Cooper.RPAPetaed bam ProgveneI PNXWoee~np of t 10th ACM Syriposhim on Operat 8>wteni Pfftc*Is, pages

63-78. Deceuiber, 1965.PrIhdas OPe6raso I SYaeWM Re Vew 19(5).

1101 Annals L DeSchon.A Survey ofO Datso bfeetan 5trdwV.Tedwicel Repor RFC 971, SRI Netwoarlk honnon Certer, January. 1988.

1113 Whiuld DNe and Marli E. Helinn.Privacy and Autenllonft: An Inroduallon lo CrlIrp hyf.AOceeWWW of Me IEEE 67(3):397--427, March, 1979.

(123 1 WRoem~ MwwM fo te Ads PlOpwroft Lwpm"jUkted Staest Deprtrnert of DeO~ee 1963.U.&. Goernmnet Printin Offoe, ANSUIIL-STD-ISISA-19613.

1e

[133 DOW~ K. 60&od.VfWIe Vlig for Replcated OatahIn.01me o'f te 7th S$iripsorn~ on Operating System Princ#*us pages 150-162.

Dee=er 1979.Piluhe as (peraifir System Review, 13(5).

[143 J. N. Gra.Nowe on Datae Operating System.I R. Byer and R. M. Graham and G. Segmueler (edi) Operating Systems An Ad-

wieCowae, pages 393-481. Sprn~ -Veft.g 1978.Volumne W of Leckwae tes I Conriafer Scence.

[153 MaM Ios Y.A Quonrnm-Consenaus Rpication Metthod for Abstract Data Types.ACM Trmnsactbns on Computer Sy&enw 4(l):32--53, February, 1986.

1163 C.A. R. Hoare.Co ilmlivialnc luelPcse.Cwnmua*tW~n of te ACM 21(8):66-67, August. 1978.

1173 Reteve AVW of Open Systm ---eeconnnI9STC97SC16, 1979.Document N227.

[183 B. W. Larv~eo nd M. Paul and H. J. Seageit (editor)Uctaw Anoe In Ccnpfle Science. Volumne 105: Dbtvtjuted SystemsArd~ecture

and khp*,ement al n. An A&Wanced Cours.Spine-Veda, 1981.

1191 Buer W. Luimieo and Howard E. Sturgis.Crash Recvery In a DWIsrbuted Data Storage System.Compuer Science Laboiratory, Xerox PARC.

[201 Barbara Usiov anid Robedt Schellier.Guardians and Actions: Linguistic Suppor for Robst Distributed Programs.ACM TMaNeCOMn on Pforamfi Languages and System 5(3)381 -404, July, 1983.

[211 Dat Encrypton StandardNatina Bureau of Standards. 1977.Federal Inlormation Processing Standards Publication 46.

[2) Roger M. Needhiam and Michael D. Schroeder.Using Encryption for Authentication In Large Networkts of Computers.

Iomta*r of t ACM 21(12)M93-99, December, 1978.2) bruce Jay Nelson.

Remote Proceduro CalLPhD #*ft.15 Computer Science Department, Carnegie-Mellon University, May, 1981.Published asCMU report CMLLCS-81-119 and Xerox PARC report CSL-81 -9.

[243 David Nolki, Norm Hutchinon, Jan Sanlsi, and Michael Schwartz.Report on the ACM SIOPS Workshop on Acoomodatig Heterogeneity.Opeat Systen Rew 20(2).-24, April. 1986.

[253 G. Popek B. Wallier, J. Chow, D. Edwards, C. Kllne, G. Rudisin, and G. Thiel.LOCUS: A Network Traruperenf, ig Reliablity Distributed System.I Proosivallg of tOe 91h Synwskim on Operat System Pdnc*ies pages 1 69--1 77.

Decme, 1981.PuLahed as Operatin Systemns Review, 15(5).

17

(21 Jon Postal.I*Nert Protcol.RFC 791. SRI Network Information Center, September, 1981.

1271 Jon Postel.Tnhrrdslon Control ProtocolTechnical Report RFC 793, SRI Network Inboration Center, September, 1981.

1281 Michael L. Powell and Barton P. Mille.Process Migration I DEMOSAMP.I Proceedings otf Oe 91h ACM Synposlum on Operating Systemns Pfincies, pages

110-119. October, 19633.Published as Operating Swen Review, 17(5).

* (1291 Richard F. Rashi and Georg G. Robertson.Accent: A Commnunication Oriented Network Operating System Kernel.In Proceedings of the 81h Symuposim on Operating System Prnc*s, pages 64--75.

* December, 1981.Published as Operang System Review, 15(5).

130] M. Satyanarayean, John H. Howard, David A. Nichols, Robert N. Sidebotham, AlfredZ Spector, and Michasel J. West.The ITC Distrbuted File System: Principles and Design.In Proceedirg of the 101h ACM Symiposium on Operating Systems Principes, pages

35-50. Decebe, 1985.Published as Operating System Review, 19(5).

1311 Eric Emerson Schmidt.Controftn Large Software Devekpmsent In a Distrbuted EnvfronmentPhD thesis, Computer Science Division, University of California, Berkeley, December,

1962.Published as Xerox PARC report CSL-82-7.

(321 Michael D. Schroeder, David K. Gifford. and Roger M. Needham.A Caching Fio System for a Programmers Workstation.In Proceedings of the 101h ACMW Symposkm on Operating Systems Principles, pages

25-34. Decemer 1985.Published as Operatng System Review. 19(5).

133 Alfred Z. Spector, Dean Daniels, Daniel Duchamp. Jeffrey L. Eppinger, and Randypascil.Distrbuted Transactions for Reliable Systemns.In Pftcedng of the 10th ACM Syrrpoahbm on Operating Systems Princ*iles, pages

127-146. December, 1985.* Published as Operat Systemns Review, 19(5).

1341 Robert F. Sproull and Dan Cohe.High-Level Prtocols.Phvee" of the IEEE 88(1 1):1371 -1386, November, 1978.

(351 Wea Svobodova.File Serwes for Network-Based Distributed System. 1

ACM Compuin SUrveys 16(4):353-396, December, 1984.(36 ndrew S. Tanrtau.

Prergicell 1981.

f371 Andrew S. Tanorbim.Network~mAC~M Con pat Suiwys 13(4).453-489, Decorir, 1961.

19

-- al-q

jgUfl CLASSIFICATION OF THIS PAGE 00'4/ j-I REPORT DOCUMENTATION PAGE

lal REPORT SECURITY CLASSIFICATION lb. RESTRICTIVE MARKINGS

UNLIITED, UNCLASSIFIED NONE2. SECURITY CLASSIFICATION AUTHORITY 3. DISTRIBUTION/A VAILABILITY OF REPORT

N/A -- UNCLASSIFIED, UNLIMITED, DTIC, NTIS2b. OECLASSI FICATION/0OWNGRADING SCHEDULE

4. PERFORMING ORGANIZATION REPORT NUMBER(S) 5. MONITORING ORG NIZATION REPORT NUMBER(S)

CMU/ SEI-87-TR-5 ESD-TR-87-1'

Gal NAME OF PERFORMING ORGANIZATION Eb. OFFICE SYMBOL 7a. NAME OF MONITORING ORGANIZATION

SOFTWARE ENGINEERING INST. (f tapplicable)

SEI SEI JOINT PROGRAM OFFICE

Or- ADDRESS (City. State and ZIP Cod) -7b. ADDRESS (City. State and ZIP Code)

CARENGIE-HELLON UNIVERSITY ESD/XRSIPITTSBURGH, PA 15213 HANSCOM AIR FORCE BASE

HANqSCOM, MA 01731Se. NAME OF FUNDINO/SPONSORING Ib. OFFICE SYMBOL 9. PROCUREMENT INSTRUMENT IDENTIFICATION NUMBER

ORGANIZATION (if applicable)

SEI JPO IESD/XRS1 F19628 85 0003Sc. ADDRESS (City. State and ZIP Code) 10. SOURCE OF FUNDING NOS.

CARNEGIE-MELLON UNIVERSITY PROGRAM PROJECT TASK WORK UNtT

PITTSBURGH, PA 15213 ELEMENT NO. NO. No. NO.

_______________________________ 63752F N/A N/A N/A11. TITIE (include Security- Classiiction)

DISTRIBUTED SYSTEMS ECHNOLGY SURVEY

13a. TYPE OF REPORT 13b. TIME COVERED 114. DATE OF REPORT (Yr.. Mo.. Day) ji5. PAGE COUNT

FINAL IFROM ... TO .. MARCH 87 2016. SUPPLEMENTARY NOTATION

N/A

____. ______________________ 18I. SUBJECT TERMS (Continue on reverse if necessary and identify by block number)

F IE L D I G R O U P _ SU B . G R . - -I19. ABSTRACT (Continue on reverse if necessary and identify by block number)

THIS REPORT IS ONE OF A SERIES OF SURVEY REPORTS. IT IS NOT INTENDED TO PROVIDEAN EXHAUSTIVE DISCUSSION OF TOPICS PERTINENT TO THE AREA OF DISTRIBUTED SYSTEMSTECHNOLOGY. RATHER, IT IS INTENDED AS AN INFORMATIVE REVIEW OF THE TECHNOLOGYSURVEYED. THESE SURVEYS WERE CONDUCTED IN LATE 1985 AND EARLY 1986.

20. DIST RI BUT ION/AVAI LABI LIT Y OF ABSTRACT 21. ABSTRACT SECURITY CLASSIFICATION

UNCLASSIFIED/UNLIMITED C SAME ASRPT. W OTIC USERS UNCLASSIFIED, UNLIMITED, DTIC, NTIS

22a. NAME OF RESPONSIBLE INDIVIDUAL 22b TELEPHONE NUMBER 22c OFFICE SYMBOL

KARL H. SHINGLER4126-30 EJP

DO FORM 1473, 83 APR E DITION OF I JAN 73 IS OBSOLETE.

SECURITY CLASSIFICATION OF THIS 'All

11 ~ ~ ~ ~ -------------- il.Fl 'Il I.lilil U111 l!1114919 l

uMau