July 1983 Report No. STAN-B-83-973Also twtdered C’SL 246

The Distributed V Kernel and ItsPerformance for Diskless Workstations

bY

David R. Cheriton and Wily Zwaenepoel

Department of Computer Science

Stanford UniversityStanford, CA 94305

.The Distributed V Kernel

and its Performance for Diskless Workstations

David R. Cheriton and Willy Zwaenepoel

Computer Systems LaboratoryDepartments of Computer’ Science and Electrical Engineering

Stanford University

Abstract

The distributed V kernel is a message-oriented kernel thatprovides uniform local and network interprocess communication.It is primarily being used in an environment of disklessworkstations connected by a high-speed local network to a set offile servers. We describe a performance evaluation of the kernel,with particular emphasis on the cost of network file access. Ourresults show that over a local network:

1. Diskless workstations can access remote files with minimalperformance penalty.

2. The V message facility can be used to access remote files atcomparable cost to any well-tuned specialized file accessprotocol.

We conclude that it is feasible to build a distributed system withall network communication using the V message facility evenwhen most of the network nodes have no secondary storage.

. .

I . IntroductknThe distributed V kernel is a message-oriented kernel thatprovides uniform local and network inter-process communication.The kernel interface is modeled after the ‘Thoth [3,5] andVerex (4, S] kernels with some modifications to facilitate efficientlocal network operation. It is in active use at Stanford and atother research and commercial establishments. The system isimplemented on a collection of MC68OO&based SUNworkstations [2] interconnected by a 3 Mb Ethernet [9] or 10 Mb

This work was sponsomd in pm by the Defense Advanmd Rcseatch Projects @IICyunder contracts MDA903-8OC4102 and NKU39-83-K0431.Permission to copy without fee all or part ofthis material is granted provided that thecopies are not made or distributed for directcommercial advantage, the ACM copyrightnotice and the title of the publication andits date appear, and notice is given thatcopying is by permission of the Associationfor Computing Machinery. To copy otherwise,or to republish, requires a fee and/orspecific permission.

Ethernet [7]. Network interprocess communication ispredominantly used for remote file access since most SUNworkstations at Stanford are configured without a local disk

This paper reports our experience with the implementation anduse of the V kernel. Of particular interest are the controversialaspects of our approach, namely:

1. The use of diskless workstations with all secondary storageprovided by backend file servers.

2. The use of a general purpose network interprocesscommunication facility (as opposed to special-purpose fileaccess protocols) and, in particular, the use of a Thoth-likeinter-process communication mechanism.

The more conventional approach is to configure workstationswith a small local disk, using network-based file servers forarchival storage. Diskless workstations, however, have a numberof advantages, including:

1. Lower hardware cost per workstation.

2. Simpler maintenance and economies of scale with sharedfile servers.

3. Little or no memory or processing ovcrhcad on theworkstation for file system and disk handling.

4. Fewer probiems with replication, consistency anddistribution of files.

The major disadvantage is the overhead of performing all fileaccess over the network. One might therefore expect that we usea carefully tuned specialized tile-access protocol integrated intothe transport protocol layer. as done in LOCUS [ll). Instead, ourfile access is built on top of a general-purpose interprocesscommunication (IPC) facility that scrvcs as lhc transport layer.While this approach has the advantage of supporting a variety ofdifferent types of network communication, its generality has thepotential of introducing a significant performance penalty overthe “problem-oriented” approach used in LOCUS.

Furthermore. because sequential file access is so common, it isconvcnlional to use streaming protocols to minimize the effect ofnetwork latency on performance. Instead. WC adopted asynchronous “request-response” model of mcs.sagc-passing anddata transfer which, while simple and efficient to implement as

well as relatively easy to use, does not suppon application-leveluse of streaming.

These potential problems prompted a performance evaluation ofour methods, with particular emphasis on the cfliciency of fileaccess. This emphasis on file access diztin:uishes our work fromsimilar studies [IO, 131. The results of our stud! strongly supportthe idea of building a distributed vstem using disklessworkstations connected by a high-speed local network to one ormore file servers. Furthermore, we show that remote file accessusing the \’ kernel IPC facility is on]! slightly more expensivethan a lower bound imposed by the basic mst of networkcommunication. From this we conclude that relatively littleimprovement in performance can be achieved using protocolsfurther specialized to file access.

2. V Kernel Interprocess CommunicationThe basic model provided by the I’ kernel is hat of many smallprocesses communicating by messages. A process is identified bya 32-bit globally unique process idcrzri/ier or pid Communicationbetween processes is protided in the form of short fixed-lengthmessages, each uith an associated reply message, plus a datatransfer operation for maying larger arnounE of data betweenprocesses. In particular, all messages are a fixed 32 bytes inlength.

The common communicaLion scenario is as follows: A cfienrprocess executes a Send to a seTseT process \vh;ch then completesexecution of a Receive to receive the missage and eventuallyexecutes a Reply 10 respond uith a rep!! message back to thechent We refer to thus sequence as a mcssrge exchange. lherecciier ma! execute one or more :\lo\*eTu or .IloveFrom datatransfer operations beliveen the ume the message 1s received andthe Ume the reply message is sent.

The following sections describe the primitives relevant to thispaper. The interested reader is referred to the V kernelmanual [6] for a complete description of the kernel facilities.

2.1. PrimitivesScnd( message, pid )

Send the 32-b\te message specified b! message to theprocess specified by prd Ihe sender blocks until therecciler has rcceiied the message and has sent back a

. 32-byte rep11 using Reply. The rep]\ messageoverwnlcs the onglnal mcssagc areaUsing the kernel mc4tigc format con\entions, aprocess sJ)cciJics in the nlejtige the scgmcnr of itsaddress space t!ar the mesG;e recipient may access

pid =

and uhether the reciJlient may read or write thatsegment- A segment is specified b! the last two wordsof a mehsase. gilin its Stan addre% and its lengthrespecti\el>. Resened fins bl& aL the &sinning of themessage lndlcare uhcther a ScgmcnI is specified and ifso, 11s access permissions.

Recci\ c( message )Hock the Invoking process. if ntycLqn. to receive a32-b!tc mc’+a;c in 11s mesage VccIor. ~Icssages arcqueued In ~rst-cornc-first-senc’d (J-Cl-S) order until

received. .

( pid, count ) = RcceivcWithScgmcnt( message, scgptr, scgsize )Block the inI oking process to receli e a message as withReceive except, if a segment is spccit’icd in the messagewith read access, up to the first segsize bl tcs of thesegment may bc transferred to the array starting atsegptr. with counr specifying the actual number ofbytes received.

Reply( message, pid )Send a 32-byte reply cbntained in the message bufferto the specified process providing it is awaiting a replyfrom the replier. The sending process is readied uponreceiving the reply; the replying process does notblock.

ReplyWithSegmcnt( message, pid. destptr, segptr, segsize )Send a reply message as done by Reply but alsotransmit the short segment specified by segprr andsegsize to desfptr in the destination process’ addressspace.

MoveFrom( srcpid, dest, src, count )Copy counf bytes from the segment starting at src inthe address space of srcpid to the segment starting atdesr in the active process’s space. The srcpid must beawaiLing rcpl) from the active process and must haveprovided read access to the segment of memory in itsaddress space using the message con\entions describedunder Send

MoveTo( destpid, dest, src, count )Copy counr bytes from the segment starting at src inthe active process’s space to the segment starting at desrin the address space of the dcsrpid procebs. The desrpidmust be auairlng reply from the active process andmust ha\e provided wntc access to the segment ofmcmop in i& address space using the messageconventions described under Send

SetPid( logicalid, pid, scope )Associate pid with the specified logicalid in thespecified scope, which is one of local. remote or both.Example loglcalid’s are fileserver, nameserver, etc.

pid = GctPid( logicalid. scope )Return the process idcntificr associated with (ogicalidin the specified scope if any, else 0.

2.2. DiscussionThe V kernel’s interproccss communication is modeled after thatof the Thoth and Verex kernels, which have been used in multi-user systems and real-time applications for several years. Anextensive discussion of this design and its motivations isavailable [5], although mainly in the scope of a single machinesystem. We summarize here the highlights of the discussion.

1. Synchronous request-response message communicationmakes programming easy because of the similarity toprocedure calls.

2. The distinction between small messages and a separate datatransfer facility ties in well with a frcquenlly observed usagepattern: A last amount of interprocess communication istransfer of small amounts of control information (e.g. de&completion) while occasionally there is bulk data transfer(e.g. program loading).

3. Finally, sl nchronous communication and small, fixed-sizemesugcs reduce queuing and buffering problems in thekernel. !n particular. only small, fixed-slrc mcssazc buffers

must be allocated in the kcrnc! and Large amounts of dataare transferred dircctlg bctwccn users’ address spaceswithout extra topics. Morcovcr. by virtue of the synchronyof the communication. the kcrncl’s message buffers can bcstatically allocated. As excmp!i!icd in Thoth. these factorsmake for a small, efficient kemc!. .

The V message primitives appear ill-suited in several ways for anetwork environment. at least on first observation. The shoifixed-length messages appear to make inefficient use of largepacket sizes typically available on local networks. The ’synchronous nature of the message sending would seem tointerfere with the true parallelism possibk between separateworkstations. And the economies of message buffering affordedby thcsc restrictions in a single machine implementation are lessevident in a distributed environment. Finally, the separate datatransfer operations MoveTo and MoveFrom appear only toincrease the number of remote data transfer operations that mustbe implemented in the distributed case.

However, our experience has been that the V message primitivesare easily and efficiently implemented over a local network.Moreover, we have found that the semantics of the primitivesfacilitated an efficient distributed implementation. The onlymajor departure from Thoth was the explicit specification ofsegments in messages and the addition of the primitivesReceive WithSegment and Reply WithSegment. This extensionwas done for efficient page-level file access although it has provenuseful under more general circumstances, e.g. in passing characterstring names to name servers.

3. implementation IssuesA foremost concern-ii the implementation of the kernel has beenefficiency. Before dcscri’>ing sonic of the implementation det&of the individual primitives, we list several aspects of theimplementation that are central to the efficient operation of thekernel.

1. Remote operations are implemented directly in the kernelinstead of through a process-level network server. Whenthe kernel recognizes a request directed to a remote process,it immediately writes an appropriate packet on the network.The alternative approach whereby the kernel relays aremote request to a network server who then proceeds towrite the packet out on the network incurs a heavy penaltyin extra copying and process switching. (WC measured afactor of four increase in the remote message exchangetime.)

2. Interkcmc! packets use the “raw” Ethernet data link level.The overhead of layered protocol implementation has beendescribed many times [lo]. An alternative impkmcntationusing intcrnct (II’) headers showed a 20 percent increase inthe basic mcssagc exchange time, cvcn without computingthe IP hcadcr checksum and with only the simplest routingin the kernel. While we recognize the tradeoff bctwccn

intcrnct functionality and local ‘net performance, we havechosen not to burden the dominant (local net) operationwith any more overhead than is strictly necessary.

3. The synchronous request-response nature of a replyassociated with each message is exploited to build reliablemessage transmission directly on an unreliable datagramservice. i.e. without using an exVa layer (and extra packets)to implement reliable transport The reply message servesas an acknowledgement as we!! as carrying the replymessage data

4. The mapping from process id to process location is aided byencoding a host specification in the process identifier. Thekernel can thus determine quickly whether a process iseither local or remote. and in the latter case on whichmachine it resides.

5. There are no per-packet acknowledgements for large datatransfers (as in MoveTo and in MoveFtom). There is only asingle acknowledgement when the transfer is complete.

6. File page-level transfers require the minima! number ofpackets (i.e. two) because of the ability to append shortsegments to messages using Receive WithSegment andReply WithSegment.

The following sections look at particular aspects of theimplementation in greater detail.

3.1. Process NamingV uses a global (flat) naming space for specifying processes, incontrast to the local port naming used in DEiiOS [l] andAccent (12). Process identifiers are unique within the context of alocal network. On the SUN workstation, it is natural for the Vkernel to use 32-bitvrocess identifiers. The high-order 16 bits ofthe process identifier seme as a logical host identi/ier subfieldwhile the low-order 16 bits are dsed as a locally unique identifier.

In the current 3 Mb Ethernet implementation, the top 8 bits ofthe logical host identifier are the physical network address of theworkstation, making the process identifier to network addressmapping trivial. In the 10 Mb implementation, a table mapslogical hosts to network addresses. When there is no table entryfor the specified logical host. the message is broadcast New“logical host-to-network address” correspondences can bediscovered from messages received. However, each node must atleast know or discover its own logical host identifier during kernelinitialization.

The use of an explicit host field in the process identifier allowsdistributed generation of unique process identifiers betweenmachines and allows an efficient mapping from process id tonetwork address. In particular. it is very cfficicnt to determinewhether a process is local or remote. This “locality” test onprocess idcntificrs serves as the primary invocation mechanismfrom the local kernel software into the network IPC portion. Ingcncral, most V kcrnc! operations differ from their Thothimplcmcntation by a call to a “non-local” handler when one ofthe .proccss idcntificr parameters fails to validate as a local

process. With the exception of GetPid, kernel operations with noproccm identifier parameters are implicitly local to theworkstation.

GetPid uses network broadcast to determine the mapping of alogical process identiIier to real process identifier if the mappingis not known to the local kernel. Any kernel knowing themapping can respond to the broadcast request The addition oflocal and remote scopes was required to discriminate betweenserver processes that serve only a single workstation and thosethat serve the network.

3.2. Remote MessagelmplementationWhen a process identifier is spcciIied to Send with a logical hostidentifier different from that of the local machine. the local pidvalidation test fails and Send calls h’onL.ocufSend which handlestransmission of the message over the network

The NonLudSend routine writes a interkeme! packet on thenetwork addressed to the host machine of this process or elsebroadcasts the packet if the host machine is not known. Whenthe host containing the recipient process receives the packet itcreates an alien process descriptor to represent the remotesending process using a standard kernel process descriptor’ andsaves the message in the message buffer field of the alien processdescriptor. H’hen the receiving process replies to the message, thereply is transmitted back to the sender as well as being saved for aperiod of time in the alien descriptor. If the sender does notreceive a reply within the timeout period T. the original messageis retransmitted by the sender’s kernel. The receiving kernelfilters out retransmissions of received messages by comparing themessage sequence number and source process with those

. represented by the aliens. The kernel responds to a retransmittedmessage by discarding the message and either retransmitting thereply message or else sending back a “reply-pending” packet tothe sending kernel if the reply has not yet been generated. It alsosends back a reply-pending packet if it is forced to discard a newmessage because no (alien) process dcsaiptors are available. Thesending kernel condudes the receiving process does not exist (andthus the Send has failed) if it receives a negativeacknowledgement packet or it retransmits N times withoutreceiving either a reply message or a reply-pending packet

This description supports the claim made above that reliablemessage transmission is built immediately on top of an unreliabledatagram protocol with the minima1 number of network packetsin the normal case

3.3. Remote Data TransferMoveTo and MoveFrom provide a means of transferring a largeamount of data between remote processes with a minima! timeincrease over the time for transferring the Shme amount of data in

.raw network datagrams. MoveTo transmits the data to bc movedin a scqucncc of maximally-sized packets to the destinationworkstation and awaits a single acknowlcdgemcnt packet when a!!the data has been rcccivcd. Given the observed low error rates oflocal networks, full retransmission on error introduces only aslight performance degradation. WC have, however, implcmcntedretransmission from the .!ast correctly received data packet inorder to avoid the pitfall of repeated identical failures arisingwhen back-to-back packets are consistently being dropped by thereceiver. The implementation of MoveFrom is similar except aMoveFrom request is sent out and acknowledged by the requesteddata packets, essentially the reverse of MoveTo.

As in the local case, major economics arise with MoveTo andMoveFrom because, by their definitions, there is no need forqueuing or buffering of the data in the kernel. The V kernelmoves the data directly from the source address space into thenetwork interface, and directly from the network interface to thedestination address spa&.

3.4. Remote Segment AccessThe message and data transfer primitives provide efficientcommunication of small amounts and large amounts of data, lessthan 32 bytes or several tens of network packets. However, page-level file access requests involve an intermediate amount of datathat is not efficiently handled by the Thoth primitives whenimplemented over a local network.

V file access is implemented using an I/O protocol developed forVerex [4]. To read a page or block of a file, a client sends amessage to the file server process specifying the file. blocknumber. byte count and the address of the buffer into which thedata is to be retumcd. The server reads the data off disk, transfersit into the remote buffer using MoveTo, and then replies,confirming the amount of data read. In the cOmmon case of thedata fitting into a single network packet, this requires 4 packettransmissions (assuming no rctransmissions): one for tbc Send, 2for the MoveTo and one for the Rep&. This is double the numberof packets required by a specialized page-level file access protocolas used for instance, in LOCUS [ll] or WFS [14].

To remedy this problem, we made some modifications to theoriginal Thoth kernel interface. First, we added the primitivesReceive WithSegment and Reply WithSegment. Second, werequired explicit specification of segments in messages (asdcscribcd in Section 2.1). Using this explicit spcci!icat.ion, Sendwas then modified to transmit as part of the network packetcontainmg the message, the first part of a segment to which readaccess has been granted. if any. Using the Receive WithSegmentoperation, the recipient process is able to receive the request,message and the first portion of the segment in a single operation.By setting the size of the initial portion of the segment xnt to be

‘USC of aandard kcrad praen &scriptw fa Jicru rcduca the amount ofcpakdizcd code for hdling rcnxxc mcssaga Howvu. alien pracaa do nueraue and can reasonably be thought of as marage buffcn

2Thir is possible wilh a programmed l/O intcrfaa, as used by the SUN 3 hfbFdhcma inwrfacc as well as the 3COM 10 Mb Ethernet interface. A anwcntiorulDMA inerfaa may require a packet asembly bulk for trvlsnissh and napth.

at lcast as large as a file block. a file write operation is Fcduccd toa single two packet exchange. In this fashion, read access to smallsegments (such as single block disk write operations) is handledefficiently. The Repb WirhSegmcnr operation eliminates theextra packets in the case of a file read by combining the replymessage packet with the data to be returned.

Advantages of this approach include:1. The advantages of the Thoth IPC model with the network

performance of a WFS-style file access protocol.

2. Compatibility with efficient local operation. For instancc,segments may be accessed dirtily if in the same addressspace or if the recipient process operates a DMA device.

3. Use of Receive WithSegment and Reply WthSegment isoptional and transparent to processes simply using Send

An expected objection to our solution is the apparent asymmetryand awkwardness of the message primitives. We feel theasymmetry may be appropriate given the prevalent asymmetry ofcommunication and sophistication between client and serverprocesses. Also, it is not unexpected that there be some conflictbetween efficiency and elegance. Given that applicationscommonly aaxss system services through stub routines thatprovide a procedural interface to the message primitives, it is notinappropriate to make some compromises at the message level forefficiency.

We now turn to the discussion of the performance evaluation ofthe kernel. We first define the tern nerwork penal@ as areasonable lower bound on the cust of network communication.Subsequently we discuss the efficiency of the kernel..both interms of message passing and file access.

4. Network PenaltyOur measurements of the V kernel are primarily concerned withtwo comparisons:

LThe cost of remote operations versus the cost of thecorresponding local operations

2 The cost of file access using V kernel remote operationsversus the cost for other means of network file access.

An important factor in both comparisons is the cost imposed bynetwork communication. In the first comparison, the cost of aremote operation should ideally be the cost of the local operationplus the cost of moving data across the network (data that is inshared kernel memory in the local case). For instance. a localmessage Send passes pointers to shared memory buffers anddescriptors in the kernel while a remote message Sendmust movethe same data across the network. In the second comparison, thebasic cost of moving file data across the network is a lower boundon the cost for any network file access method.

To quantify the cost of network communication, WC define ameasure we call the nework penalty. The network penalty isdefined to be the time to transfer n bytes from one workstation toanother in a ndwork datagram on an idle network and assuming

.no errors. The network penalty is a function of the processor, thenetwork. the network inLcrface and the number of bytestransferred. It is the minimal time penalty for interposing thenetwork belwecn two software modules that could otherwisetransmit the data by passing pointers. The network penalty isobtained by measuring the time to transmit n bytes from the mainmemov of one workstation to the main memory of another andvice versa and dividing the total lime for the experiment by 2The experiment is repeated a large number of times for statisticalaccu racy. The transfers are implemenled at the data link layerand at the interrupt level so that no protocol or process switchingoverhead appears in the results. The assumptions of error-freetransmission and low network utilization are goodapproximations of most local network environments.

Network penalty provides a more realistic minimum achievabletime for data transfer than that suggested by the physical networkspeed tiuse it includes the processor and network interfacetimes. For instance, a 10 Mb Ethernet can move 1000 bits fromone workstation to another in 100 microseconds. However, thetime to assemble the packet in the interfaoz at the sending endand the time to transfer the packet out of the interface at thereceiving end are comparable to the time for transmission. Thus,the time for the transfer from the point of view of thecommunicating software modules is at least two or three times aslong as that implied by the physical transfer rate.

Measurements of network penalty were made using theexperimental 3 Mb Ethernet In all measurements. the networkwas essentially idle due to the unsociable times at whichmeasurements were made. Table 4-1 lists our measurements ofthe 3 Mb network penalty for the SUN workstation using the 8and 10 MHz processors with times given in milliseconds. Thenetwork time column gives the time for the data to be transmittedbased on the physical bit rate of the medium, namely 2.94megabits per second.

Network Penalty

Bytes3 Network Time Network Penalty8 MHz 10 MHz

64 .174 0.80 0.65128 348 L20 0.96256 .696 2.00 1.62512 1.392 3.65 3.00

1024 2784 6.95 5.83

Table 4-l: 3 Mb Ethernet SUN Network Penally (times in mscc.)

The network penalty for the 8 MHz processor is roughly given byp(n) = .0064 l n + .3!90 milliseconds whcrc n is the number ofbytes transferred. For the 10 MHz processor. it is .OOS4 l n +251 milliseconds.

be only amsidu packet d7.rs that fit in a single network datagram.

The diffcrcncc between the network time, computed at thenetwork dala rate, and the measured network penalty time isaccounted for primarily by the processor time to generate andtransmit the packet and then receive the packet at the other end.For instance, with a 1024 byte packet and an 8 MIIz processor,the copy time from memory to the Ethernet interface and viceversa is roughly 1.90 milliseconds in each direction. Thus, of thetotal 6.95 milliseconds. 3.80 is copy time, 278 is networktransmission time and .3 is (presumably) network and interfacelatency. If we consider a 10 Mb Ethernet with similar interfaces,the transmission time is less than 1 millisecond while the copytimes remain the same, making the processor time 75 percent ofthe mst in the network penalty. The importance of the processorspeed is also illustrated by the difference in network penalty forthe two processors measured in Table 4-1.

With our interram, the processor is required to copy the packetinto the interface for transmission and out of the interface onreception (with the interface providing considerable on-boardbuffering). From the copy times given above, one might arguethat a DMA interface would significantly improve performance.However, we would predia that a DMA interface would notresult in better kernel performance for two reasons. Firs& thekernel interprets a newly arrived packet as it copies the packetfrom the network interface, allowing it to place much of thepacket data immediately in its final location. With a DMAinterface. this copy would be required after the packet had beenD.MA’ed into main memory. Similarly, on transmission thekernel dynamically constructs a network packet from disparatelocations as it copies tic data into the network interface. MostDIMA interfaces require the packet to be assembled in onecontiguous area of memory, implying the need for a comparablecopy operation. FmaUy, there is not currently available (to ourknowledge) a network D.UA interface for the Ethernet thatmoves data faster than a 10 MHz Motorola 68000 processor asused in the SUN workstation. In general, the main benefit of asmart network interface appears to be in offloading the mainprocessor rather than speeding up operations that make use ofnetwork communication.

5. Kernel PerformanceOur first set of kernel measurements focuses on the speed of localand network intcrproarss communication. The kernel IPCperformance is prcscntcd in terms of the times for messageexchanges and the data transfer operations. WC first describe howthese measurements were made.

5.1. Measurement MethodsMeasurements of individual kernel operations were performed byexecutmg the operation N times (typically 1000 times). recordingthe total time required. subtracting loop ovcrhcad and otherartifact. and then dividing the total time by N. Mc&uremcnt oftotal time relied on the software-maintained V kernel time whichis accurate plus or minus 10 milliseconds

.Mcasurcmcnt of processor utilization was done using a Iow-priority “busywork” process on each workstation that rcpcatcdlyupdates a counter in an infinite loop. All other processorutilization reduces the processor allocation to this process. Thus,the prmr time used per operation on a workstation is theelapsed time minus the processor time allocated to the“busywork” process divided by N. the number of operationsexecuted.

Using 1000 trials per operation and time accurate plus or minus10 milliseconds. our measurements should be accurate to about.02 milliseconds except for the effect of variation in network load.

5.2. Kernel MeasurementsTable 5-l gives the results of our measurements of messageexchanges and data transfer with the kernel running onworkstations using an 8 MHz processor and connected by a 3 MbEthernet Note that GetTime is a trivial kernel operation,included to give the basic minimal overhead of a kerneloperation, The columns labeled Local and Remote give theelapsed times for these operations executed locally and remotely.The Difference column. lists the time difference between the localand remote operations. The Penalty column gives ‘the networkpenalty for the amount of data transmitted as part of the remoteoperation. The Client and Server columns list the processor timeused for the operations on the two machines involved in theremote execution of the operation. Table 5-2 gives the samemeasurements using a 10 MHz processor. The times for bothprocessors are given to indicate the effect the processor speed hason ICXXI and remote operation performance. As expected, thetimes for local operations, being dependent only on the processorspeed, are 2S percent faster on the 25 percent faster processor.However, the almbst 15 percent improvement for remoteoperations indicates the processor is the most significantperformance factor in our configuration and is not renderedinsignificant by the network delay (at least on a lightly loadednetwork).

A significant level of concurrent execution takes place betweenworkstations even though the message-passing is fullysynchronized. For instance, transmitting the packet, blocking thesender and switching to another process on the sendingworkstation proceeds in parallel with the reception of the packetand the readying of the receiving process on the receivingworkstation. Concurrent execution is indicated by the fact thatthe total of the scrvcr and client processor times is grcatcr thanthe elapsed time for a remote mes.sage exchange. (See the Clientand Server columns in the above tables.)

5.3. Interpreting the MeasurementsSome are is required in interpreting the implications of thesemeasurements for distributed applications. Superficially, the fat!that the remote Send-Receive-Reply scqvcnce takes more than 3times as long as for the 1~1 cast suggests that distributedapplications should be designed to minimize inter-machinecommunication. In general, one might consider it impractical to

. Kernel Performance

Kernel Operation Elapsed Time NetworkPenalty

Processor Time

Local Remote Difference Client ServerGetTime 0.07 - 0.07 -Send-Rcccivc-Reply 1.00 3.18 2.18 1.60 1.79 2.30MoveFrom: 1024 bytes 1.26 9.03 ’ 7.77 i.15 3.76 5.69MoleTo: 1024 bytes 1.26 9.05 7.79 8.15 3.59 5.87

Table 5-l: Kernel Performance: 3 Mb Ethernet and 8 MHz Processor (times in milliseconds)

Kernel Operation Elapsed Tie NetworkPenalty

Processor Time

GetTimeSend-Receive-ReplyMoveFrom: 1024 bytesMoveTo: 1024 bytes

LQcal Remote Difference * Client Server0.06 - 0.060.77 2.54 1.77 1.30 1.44 1.790.95 * 8.00 7.05 6.77 3.32 4.780.95 8.00 7.05 6.77 3.17 4.95

Table 5-2: Kernel Performance: 3 Mb Ethernet and 10 MHz Processor (times in milliseconds)

view interprocess communication as transparent across machineswhen the speed ratio is so large. However. an alternativeinterpretation is to recognize that the remote operation adds adelay of less than 2 milliseconds, and that in many cases this timeis insignificant relative to the time necessary to process a requestin the server. Furthermore. the sending or client worksrationprocessor is busy with the remote Send for only 1.44 millisecondsout of the total 2.54 millisecond time (using the 10 MHzprocessor). Thus, one can offload the processor on one machineby. for instance, moving a server process to another machine if itsrequest processing generally requires more than 0.67 millisecondsof processor time, i.e. the difference between the local Send-Receive-Reply time and the local processor time for the remoteoperation.

5.4. Mult i-ProcessTraff icThe discussion so far has focused on a single pair of processescommunicating over the network. In reality, processes on severalworkstations would be using the network concurrently tocommunicate with other processes. Some investigation isrcquircd to determine how much message traffic the network cansupport and also the degradation in response as a result of othernetwork traffic.

A pair of workstations communicating via Send-Receive-Reply atmaximum speed generate a load on the network of about 400.000bits per second. about 13 pcrccnt of a 3 Mb Ethernet and 4percent of a 10 Mb Ethemct Measurements on the 10 MbEthcmct indicate that for the packet size in question nosignificant network delays arc to bc expcctcd for loads upto 25percent. Thus. one would cxpcct minims1 degradation with saytwo separate pairs of workstations communicating on the same

network in this fashion. Unfortunately, our measurements of thisscenario turned up a hardware bug in our 3 Mb Ethernetinterface, a bug which causes many collisions to go undetectedand show up as corrupted packets. Thus, the response time forthe 8 MHz processor workstation in this case is 3.4 milliseconds.The increase in time from 3.18 milliseconds is accounted foralmost entirely from the umeouts and rctransmissions arising(roughly one per 2000 packets) from the hardware bug. Withcorrected network interfaces, we estimate that the network cansupport any reasonable level of message communication withoutsignificant performance degradation.

A more critical resource is processor time. This is especially truefor machines such as servers that tend to be the focus of asignificant amount of message traffic. For instance, just based onserver processor ‘time, a workstation is limited to at most about558 message exchanges per second, independent of the number ofclients. The number is substantially lower for file accessoperations, particularly when a realistic figure for file serverprocessing is included. File access measurements are examined inthe next section.

6. File Access Using the V KernelAlthough it is attractive to consider the kcmcl as simply providingmessage communication. a predominant use of the messagecommunication is to provide file access, especially in ourenvironment of diskless personal workstations. File access takesplace in several different forms, including: random file pageaccess. scqucntial file reading and program loading. In thissection, we assume the file scrvcr is csscntially dcdicatcd to

serving the client process WC are mcasuring and otherwise idle: A diflcrence between the client processor time for remote pagelater section considers multi-client load on the file server. access and for local page access, namely 1.3 milliseconds. A

We firsi dcscribc the performance of random page-level file processor cost of more than 1.3 milliseconds per rcqucsi can be

access. expcctcd from the estimation made earlier using I..OCUS figures.

These measurements indicate the performance when file reading6.1. Page-level File Access and writing USC explicit segmcnl specification in the message andTable 6-1 list the times for reading or writing a 512 byte block Receive WYthSegmenr and Reply WithSegment. However, a filebetween two processes both local and remote using the 10 MHz write can also be perfotmea in a more basic Thoth-like way usingprocessor. The times do not include time to fetch the data from the Send-Receive-MoveFrom-Reply sequence. For a 512 bytedisk but do indicate expected performance when data is buffered write. this costs 8.1 milliseconds: file reading is similar usingin memory. A page read involves the sequence of kernel MuveTo. Thus, the segment mechanism saves 3.5 milliseconds onoperations: Send--Receive-Reply WithSegmenf. A page write is every page read and write operation. justifying this extension toSend-Receive WithSegment-Reply. the message primitives.

Random Page-Level Access

Elapsed Tie NetworkPenalty

Processor Tie

Operationpage readpage write

Local Remote Difference Client Server131 5.56 4.25 3.89 250 3.2%1.31 5.60 4.29 3.89 258 3.32

Table 6-1: Page-Level File Access: 512 byte pages (times in milliseconds)

The columns are to be interpreted according to the explanationgiven for similarly labeled columns of Tables 5-l and 5-2. Notethat the time to read or write a page using these primitives isapproximately 1.5 milliseconds more than the network penalty forthese operations. .There are several considerations that compensate for the u>st ofremote operations being higher than local operations. (Some arespecial (3ases of those described for simple message exchanges.)

. First, the extra 4.2 millisecond time for remote operations isrelatively small compired to the time cost of the file systemoperation itself. In particular disk access time can be estimated 3t20 milliseconds (assuming minimal seeking) and file systemprocessor time at 25 milliseconds.’ This gives a local file readtime of 23.8 milliseconds and a remote time of 28.1 milliseconds,making the cost of the remote operation 18 percent more than thelocal OpeIatiOn,

This comparison assumes that a local file system workstation isthe same speed as a dedicated file server. In reality, a shared fileserver is often faster because of the faster disks and more memoryfor disk caching thai come with economy of scale. If the averagedisk access time for a file scmer is 4.3 milliseconds less than theaverage local disk access time (or better). there is no time penalty(and possibly some advantage) for remote file operations

Second. remote file access oflloads the workstation processor ifthe file system processing overhead per request is greater the

%hir is based on mcasumncnu of IBCUS 1111 thr: give 61 and 43 milliacamb aprcxxsa time ass far 512.byte file read and write opera&r rcspazivcly 011 a PDF11/4S. which is roughly halftie spad of the 10 MHz Moloroia68OllO praeaor used inthtSUNwU-bU&L

6.2. Sequential File AccessSequential file access is the predominant pattern for file activityin most systems. Efficient file systems exploit this behavior toreduce the effect of disk latency by prefctching file pages (read-ahead) and asynchronously storing modified pages (write-behind). File access and file transfer protocols typicallyimplement streaming to reduce the effect of network latency onsequential file access.

Using the V kernel message communication between aworkstation and a file server, the file server can implement read-ahead and write-behind to reduce the effect of disk latency.However, there is no streaming in the network IPC to deal withnetwork latency.

Two factors suggest that streaming can be done without in a localnetwork environment. First, local networks have a low latency asa consequence of their high speed and low delay. Second,

‘although V kernel IPC is logically synchronous, significantconcurrency arises in the network implementation, furtherreducing the effects of nciwork latcncies. The presence ofstreaming adds a significanl cost to the protocol in terms of bulTcrspace. copying time and complexity of code. Moreover. bufferingeffectively puts a disk cache on the workstation, thus raisingproblems of cache consistency between the different workstationsand the file server.

To get a realistic measure of sequential file access pcrfomlance,we modified the test program used for measuring page read timesby the addition of a delay in the server process corresponding tothe disk latency. IIccausc we assume the file scrvcr is doing rcad-ahead operations. tic delay is interposed between the reply to one

rcqucst and the rcccipt of the next rcquc% WC contend that thisprogram closely approximates the performance of a wdrkstalionsequentially reading a file a< rapidly as possible from an otherwiseidle file scrvcr. The results arc shown in Table 6-2.

Sequential Page-Level Access

Disk Latcncv Elanscd Time per Pace Read10 12.02 ’l.5 17.1320 22.22

Table 6-2: Seq. File Access: 512 byte pages (times in msec.)

These results indicate that. for reasonable values of disk latency,the elapsed time for sequential file access is within 10 to 15percent from the minimum achievable (the disk latency). Itfollows that a streaming protocol cannot improve on theperformance measured for V kernel file access by more than 15percent.

Moreover, consider the two cases for the application, namely:Reading faster than the disk latency and reading slower than thedisk latency. Suppose an application is reading file pages over thenetwork using a streaming protocol. If it is reading faster than thedisk can deliver. it will operate much like the V kernel model inattempting to read a page not yet available, possibly requestingthis page from the file server, blocking waiting for the page,haiing the page returned into the local page buffers, copying thepage into its application buffer and then continuing. Thus,performance of a streaming file access implementation can beexpected to be similar to our results. For instance, comparing ourresults with the LOCUS figures for remote sequential file accesswith a disk latency of- IS milliseconds. the average elapsed timeper page is essentially equal to the LOCUS figure of 17.18milliseconds.

If an application is reading pages sequentially slower than thedisk latency time, with say 20 milliseconds between every readrequest, the page should be available locally on each read with astreaming protocol. In this case, the elapsed time for the readshould be L3 mill&co&s compared to 5.6 milliseconds remotely.However, because read operations occur at most every 20milliseconds, the spcedup by replacing non-streamed file accesswith streamed file access is limited to 20 percent or less.

Program Loading

Elapsed Time NetworkPenalty

Processor Tie

Transfer unit Local1 Kb 71.74 Kb 625

16 Kb 60.264 Kb 59.7

Remote Difference518.3 446.5 434.5368.4 305.8 8344.6 284.3 l

335.4 275.1 l

Table 6-3: 64 kilobyte Read (times in milliseconds)

Client Server207.1 297.9176.1 225.2170.0 216.9168.1 2127

Moreover, a streaming protocol would introduce extra processingovcrhcad for copying and buffering rcadahcad pages in thiscircumstance. Assuming the application ,was reading slowlybecause it was compute-bound bclwccn read operations, thestreaming protocol processing overhead would further slow downthe application From this analysis, it is clear that streaming haslimited potential for spccdup over non-streamed file access whenpages are accessed from disk wilh the latencics we have discussed.

In most systems. scqucntial file access is used extensively forprogram loading. llowever. program loading can bc performedmore efficiently with the V kernel using MoveTo. It is thereforenot reliant on the performance figures of this section and isdiscussed below.

6.3. Program LoadingProgram loading differs as a file access activity from page-levelaccess in that the entire file containing the program (or most of it)is to be transferred as quickly as possible into a waiting programexecution space. For instance, a simple command interpreter wehave written to run with the V kernel loads programs in two readoperations: the first read accesses the program headerinformation; the second read copies the program code and datainto the newly created program space. The time for the first readis just the single block remote read time given earlier. The secondread, generally consisting of several tens of disk pages, usesMoveTo to transfer the data. Because MoveTo requires that thedata be stored contiguously in memory, it is often convenient toimplement a large 1 read as multiple MoveTo operations. Forinstance, our current VAX file server breaks large read and writeoperations into MoveTo and MoveFrom operations of at most 4kilobytes at a time. Table 6-3 gives the time for a 64 kilobyteRead. (The elapsed time for file writing is basically the same asfor reading and has been omitted for the sake of brevity. Notealso that network p&alty is not defined for multi-packettransfers.) The transfer unit is the amount of da& transferred perMoveTo operation in satisfying the read request.

The times given for program loading using a 16 or 64 kilobytetransfer unit corr&ponds to a data rate of about 192 kilobytes persecond, which is within 12 percent of the data rate we can achieveon a SUN workstation by simply writing packets to the networkinterface as rapidly as possible. Moreover, if the file serverretained copies of frequently used programs in memory, much as

many current timesharing systems do, such program loadingcould achieve the same performance given in the table,independent of disk speed. Thus. we argue that MoveTo andMoveFrom with large transfer units provide an efficient programloading mechanism that is as fast as can be achieved with thegiven hardware.

7. File Server IssuesFile server performance is a critical issue for disklessworkstations. Unfonunately. we do not yet have experience witha V kernel-based file server. Thus, this section describes what webelieve are the key issues and estimates performance withoutproviding conclusive data In general. we view the processor asthe key resource to consider in file server performance because. asargued earlier, the network bandwidth is plentiful and diskscheduling and buffering issues are identical to those encounteredin conventional multi-user systems.

Ihe number of workstations a file server aan support can beestimated from processor requirements. If we estimate page reador write processing overhead as roughly 3.5 milliseconds for filesystem processing (from LOCUS) plus 3.3 milliseconds for kerneloperation (from Table 6-l]), a page request costs about 7milliseconds of processor time. Program loading appears to costabout 300 milliseconds for an average 64 kilobyte program.Estimating that 90 percent of the file requests are page requests,the average request costs 36 milliseconds Thus, a file serverbased on the SUN workstation prmr aould support about 28file requests a second. From this we estimate that one file servercan serve about 10 workstations satisfactorily, but 30 or moreactive workstations would lead to excessive delays. However, adiskless workstatiorr system can easily be extended to handlemore workstations by adding more file server machines since thenetwork would not am to be a bottleneck for less than 100workstations.

For some programs, it is advantageous in terms of file serverprocessor requirements to execute the program on the file server,rather than to load the program into a workstation andsubsequently field remote page requests from it me programs,executing for a short time and doing a lot of file access whileexecuting are in this class, especially if they require only limitedinteraction with the user.

On this basis, a file server should have a general programexecution facility and the ability to xlectively execute certainprograms. The need for this execution facility is a furtherargument for using a general interprocess communicationmechanism in place of a specialized page-level file accessprotocol. With the V kernel. all inter-program communicationand interaction takes place through the IPC facility, including:file access. argument passing, debugger control and teninationnotification. Thus. cxmtion of a program in the file serverrather than the client’s workstation dots not change the program’sexecution environment nor the client’s interaction with the

.program, i.c. it is transparent cxccpt for performance.

8. Measurementswiththe 10 MbEthernetOur limited access to a 10 Mb Ethernet has precluded basing ourmeasurements on this standard local network. Ilowever, somepreliminary figures using-the 10 Mb Ethernet indicate the effectof using a faster network and slightly faster network interfaces.First, the remote message exchange time is 271 millisecondsusing an 8 MHz processor, roughly the time for the 10 MHzprocessor on the 3 Mb network and 5 milliseconds better thanthe 8 MHz processor on the 3 Mb network. Second, the pageread time is 5.72 milliseconds. Finally, the program loading timeis much improved, achieving 255 milliseconds for a 64 kilobyteload using 16 Kb transfer units. We have not identified to whatdegree the improvement is due to the faster network speed versusthe differences in the network interface.

9. Related Work‘Ihere are a number of previous and concurrent efforts inproviding communication mechanisms for distributed systems.For brevity, we compare our work with only a representativesample that characterizes the search for, and evaluation of,suitable models and implementations

Spector’s remote reference study [13] considered the feasibility ofimplementing remote load and store operations over a localnetwork. Nelson’s work on remote procedure calls [lo]investigates network communication for procedure-based systemsanalogous to what the V kernel provides for message-basedsystems. Rashid and Robertson’s Accent kernel [12] implementsa message system with a number of features such as non-blockingmessage sending that are not provided by the V kernel. Finally,LOCUS [ll] integrates network communication into a UNIX-likesystem in the form of transparent remote file access.

Our work has followed the pattern of Spector’s and Nelson’s workin using a request-response form of communication (in place ofstreaming) and stripping away protocol layers for adequateperformance. However, we share Spector’s concern about theutility of an unprotected globally shared memory in a distributedsystem, which is essentially the functionality provided by hisprimitives. The V kernel provides a strong degree of separationbetween processes and supports protected provision of services ina multi-user. multi-workstation environment by limitinginter-process communication to the kernel IPC primitives.

Our approach differs from Nelson’s primarily in our use of aseparate interconnection mechanism from procedure calls,namely messages, and some of the ensuing differences insemantics. Fundamentally, the V kernel provides a base on whichto build a remote procedure call mechanism by the addition ofsuitable compiler and run-time support Under more dctailcdexamination, many slight differences appear that reflect long-established differences in the construction of procedure-based

versus message-based systems. although it is not clear thesediffcrcnces result in any significant difference in overallperformance.

The V kernel performance is roughly comparable to that of thesoftware implementations developed by Spector and Nelson,allowing for the non-trivial differences in operation semantics andhost processors. We would hypothesize that V kernelperformance could be improved by a factor of 30 usingmicrocode. similar to the improvement observed by Spector andNelson for their primitives. Unfortunately, neither Spector norNelson provides results that afford a comparison with our fileaccess results. In general, their work has concentrated on thespeed of the basic mechanism and has not been extended tomeasure performance in a particular application setting.

In comparison to Accent the V kernel provides a primitive formof message communication, and benefits accordingly in terms ofspeed, small code size and ability to run well on an inexpensivemachines without disk or microcode support For instance,Accent messages require an underlying transport protocol forreliable delivery because there’ is no client-level reply messageassociated with every Send as in the V kernel. We do not at thistime have performance figures for Accent.

LOCUS does not attempt to provide applications with generalnetwork interprocess communication but exploits carefully honedproblem-oriented protocols for efficient remote file access. It isdifficult to compare the two systems from measurements availablegiven the differences in network speeds, processor speeds andmeasurement techniques. However, from the. specificcomparisons with LOCUS presented earlier, we would expectoverall file access performance for the V kernel to be comparableto LOCUS running on the same machines and network.

However, the memo; requirements for the V kernel are abouthalf that of LOCUS compiled for the PDP-11 and probably morelike one fourth when LOCUS is compiled for a 32-bit processorlike the 68OfKl. Thus. for graphics workstations or process controlapplicrations, for instance, the V kernel would be more attractivebecause of its smaller size, real-time orientation and its provisionof general interprocess communication. However, the V kerneldoes not provide all the functionality of the LOCUS kernel whichincludes that of the UNIX kernel and more. When required withV, these additional facilities must be provided by server processesexecuting either on client workstations or network semermachines.

10. ConclusionsWe conclude that it is feasible to build a distributed system usingdiskIess workstations connected by a high-speed local network toone or more file servers using the V kernel IPC. In particular, theperformance study shows that V kernel IPC provides satisfactory

.performance despite its generality. 13ccause the performance is soclose to the lower bound given by the network penalty, there isrelatively little room for improvement on the V IPC for the givenhardware regardless of protocol and implementation used

The efficiency of file access using the V IPC suggests that it cannot only replace page-level tile access protocols but also filetransfer and remote terminal protocols. thereby reducing thenumber of protocols needed. We claim that V kernel IPC isadequate as a transport level for all our local networkcommunication providing each machine runs the V kernel or atleast handles the interkernel protocol. We do, however, see aplace for these specific protocols in internetworking situations.

In addition to quantifying the elapsed time for various operations,our study points out the importance of considering processorrequirements in the design of distributed systems. Moreexperience and measurement of file server load and workstationfile access behavior is required to decide whether file secyerprocessing is a significant problem in using diskless workstations.

The V kernel has been in use with the diskless SUN workstations,providing local and remote interprocess communication, sinceSeptember 1982. It is currently 38 kilobytes including wde, dataand stack The major use of the network interprocesscommunication is for accessing remote files. Our file servers arecurrently 6 VAX/UNIX systems running a kernel simulator andfile server program which provides access to UNIX systemservices over the Ethernet using interkernel packets. A simplecommand interpreter program allows programs to be loaded andrun on the workstations using these UNIX servers. Ourexperience with this software to date supports the conclusions ofthe performance study that we can indeed build our nextgeneration of computing facilities [8] using diskIess workstationsand the V kernel. .

AcknowledgementsWe are indebted to all the members of the V research group atStanford, which at this point includes two faculty -members androughly ten graduate students. In particular, we wish to thankKeith Lantz for his patient comments on a seemingly endlesssequence of drafts and Tim Mann for his many contributions tothe design and the implementation of the kernel. We would alsolike to thank the referees whose comments and suggestionshelped to enhance the clarity of the paper.

References

1. F. Basket& J.H. Howard and J.T. Montague. TaskCommunication in DEMOS. Proceedings of the 6th Symposiumon Operating System Principles, ACM, November, 1977, pp. 23-31. Published as Opemring Systems Review 11(S).

2 A. Bechtolshcim. F. Baskett V. Pratt The SUN WorkstationArchitecture. Tech. Rcp~ 229, Computer Systems Laboratory,Stanford University, March, 1982.

3. D.R. Chcriton. MA. Malcolm, IS. Mclcn and G.R. Sager.“Thoth. a Ponablc Real-time Operating System.” Comm ACM22.2 (February 1979) 105-115.

4. D.R. Chcriton. Distributed I/O using an Object-basedProtocol. Tech. Rept 81-1, Computer Science, University ofBritish Columbia, 1981.

5. D.R. Chcriton. The 730th System: Multi-processSttucturingond Portability. American Elsevier, 1982.

6. D.R. Cheriton, T.P. Mann and W. Zwaenepoel. V-System:Kernel Manual. Computer Systems Laboratory, StanfordUniversity

7. Digital Equipment Corporation, Intel Corporation and XeroxCorporation. The Ethernet: A Local Area Network - Data IinkLayer and Physical Layer Specifications, Version LO.

8. K.A. Ian& D.R. Cheriton and W.I. Nowicki. ThirdGeneration Graphics for Distributed Systems. Tech. ReptSTAN-CS-82-958, Department of Computer Science, StanfordUniversity, February, 1983. To appear in ACM Transucrions onGraphics

9. R.M. Metcalfe and D.R. Boggs. “Ethernet: Distributed PacketSwitching for Local Computer Networks.” Comm ACM 19.7(July 1976). 395-404. Also CSL-75-7. Xerox Palo Alto ResearchCenter, reprinted in CSL-80-2

10. B.J. Nelson. Remote Procedure Call. Ph.D. Th., CarnegieMellon U.. 1981. published as CMU technical report CMU-CS-81-119

11. G. Popek, B. Walker, J. Chow, D. Edwards, C Kline,G.Rudisin. G. Thiel. Ix)CUS: A Network Transparenf High

* Reliability Distributed System. Proceedings of the 8thSymposium on Operating Systems Principles, ACM,December, 1981, pp. 169-177.

.12. R. Rashid and G. Robertson. Accent: A CommunicationOriented Network Operating System Kernel. Proceedings of the8th Symposium on Operating Systems Principles, ACM,December, 1981, pp. 64-75.

13 A. Spector. “Performing Remote Operations Efficiently on aLocal Computer Network” Comm ACM 2.S. 4 (April 1982). 246-260.

14. .D. Swinehart, G. McDaniel and D. Boggs. WFS: A SimpleShared File System for a Distributed Environment. Proceedingsof the 7th Symposium on Operating Systems Principles, ACM,December, 1979, pp. 9-17.