IEEE Transactions on Nuclear Science, Vol. NS-30, No. 5, October 1983
NETWORKED VAX/LSI/CAMAC DATAACQUISITION SYSTEM DEVELOPMENT
J.D. Melvin, M.H. Mendenhall, T.A. TombrelloCalifornia Institute of Technology 301-38
Pasadena, California 91125
D.L. ClarkNuclear Structure Research Laboratory
University of RochesterRochester, New York 14627
Summarya. Recent development of theCaltech data acquisition system installed in1981, which runs on a VAX-11/750, a PeritekQ-bus network, LSI-lls, and CAMAC, isdescribed. In this system, the DEC VMS andRT-11 operating systems are supported on theVAX "host" and LSI-11 "front-end" computers bya VMS device driver and network host program,and a bootable RT-11 device driver. Network"utility" and "control" programs providegeneral purpose support for communicationbetween front-end and host software. Dataacquisition software tools are provided forwriting programs to run nuclear physicsexperiments. A system similar to Caltech's wasinstalled at the University of Rochester in1982. The network has been tested for speedand real-time response. After including allsoftware overhead required by dataacquisition,, it was found that the systemcOutld transfer buffers and acknowledge theirreceipt at a net speed of 127 KB per secondwith a 35% load on the host computer. Thenetwork software is currently being rehostedon Ethernet hardwar e at Caltech in a multiplehost - many front-end compiuter configuration.Compatibility with the current Peritek networksoftware will be maintained.
Intr-oductionUpgrade of the nuclear data acquisition
capability at Caltech was started in 1979.From 1979 to 1981 commercial andresearch-laboratory developed systems invarious parts of the USA were examined,. and in1981 it was decided that the most promisingapproach for Caltech wouild be to developsystem of our own design. This project wasinitiated under the direction of the firstauthor in 1981. It has grown into aninter-divisional (Physics/Engineering) andinter-university (Caltech/University o-fRochester) effort because of the generalinstrumentation problems it has addressed.
The overall. objective of theCaltech/Rochester project is to develop toolswith which any given data acquisition andanalysis task carn be implemnented easily onseveral networked cOMPputers, each optimizedfor a different part of the over-all task.Unlike many cu-rrent networked systems, thefocus of the comptuter network is on speed andreal-time responrse.
In the following sections we presentfirst the history of the Caltech/Rochesterproject. We then describe in functional detailthe network hardware and software, utilitysoftware, and data acquisition software toolsdeveloped. Next we present results of a speedtest on the network, and after that we commenton the current effort to rehost the network onthe new Ethernet hardware. Finally1 in theAppendix, we give technical details of networksoftware developed to date.
Proiect Hi stc:ry
Data Acquisition Hardware
Ar-ound 10 years ago Caltech augmented itshardwired data acquisition systems with aprogrammable system based on a computer.Because older general purpose computerscarried out the basic data acquisition tasksof event handling and histogramming much moreslowly than hardwired systems, this earlysystem was built around a special purposecomputer. Over the years we graduallydiscovered that this computer was practicallyimpossible to reprogram. It became clear thatspecial pLurpose computers, because of theirsmall mair-ket, would never support operatingsystems and compilers comparable to thosebecomirng available on general purposecomputers.
During the past six years general purposeComputers have become faster and lessexpenisive, and operating system software hasdeveloped considerably. The recent superminicomputers, such as the VAX (which we use now),offer excellent program development software,virtual memory (which allows computations onVer y lar-ge data arrays),, and concurrentsuppor-t o-f multiple users. Although these newcomputers have computational speed adequatefor- the analysis part of data acquisition,they still handle events too slowly. Wetherefore decided (as have many laboratories)to divide data acquisition tasks between (1)special ized "front-end" computers which handleevernts qulick:ly, and (2) easy to program "host"computers that can analyze buffers full ofevernts quickly. By this division, we hoped totake maxi muLm advantage of commerci al l yavailable hardware and software.
At Cal tech and at the University ofRochester DEC compuAter hardware and operatingsystemns were selected because of theirreliability and because of their wide use atouir uni versities and by the nulclear physicscommunity as a whole. VAX-11/750 computerswere selected as host computers, and LSI-11/2and lSI-1lt/2: computers as front-endcoumBpters. (Faster- front-end computers areavailable buLt are much more difficult toPr og - air.>
Var iotus links between the front-end andtost romputer-s are possible. We found thatdedicated link.s connecting two computers weremost common. Su.ich links use various standardor custom Parallel or serial interfaces. Insome cases, the Front-end computer Lused was ap-ogranmmable CAMAC branch driver, and thecompu.ttev link was the CAMAC interface.
We decided tco consider the alternativenore ge;ier al appr oach of the computer network.Computer- nletworks connect several computerssoftern over a comtFmorn cabl e, and support avarn ety of i rnter- -co1mpusAter cOmmuni cat i on.
Currently available networks (for exampleDECNET) are slow and/or expensive. Therefore,we selected the low cost, fast Peritek HEX-11network hardware and wrote our own software.
A hardware system was assembled in theW.K. Kellogg Radiation Laboratory at Caltechin early 1981 by the first two authors. Thisinitial hardware system is accuratelydescribed in our previous paperlJ] (exceptthat the VAX disk is now a Systems Industriesdisk on the internal VAX bus off the
Unibus). A hardware system similar toKellogg's was assembled at the University ofRochester by the first and fourth authorsduring the spring and summer of 1982. It isdescribed in a separate paper also beingpresented at this conference [2).
During 1983 the network will be rehostedby the first author on the recently available,faster, and more flexible Ethernet networkhardware. This rehosting is a joint effort ofmembers of the Physics and EngineeringDivisions at Caltech.
Network
General purpose network software hasimproved greatly during the last few years as
programmers have developed methods for
organizing the complex commuinication problemsthat occur between independent compUters. Suchsoftware is oriented towards:
1. Sending messages between networkedcomputers;
2. Allowing computers on a network touse each other's resources;
3. Handling accounting andauthorization for resource use and
file protection across severalsystems;
4. Rerouting network messages when a
link of the network is notserviceable.
Solutions to these problems are often complex,,andi have made network software costly andfrequently much sslower than the hardware.
At Cal terh we decided to wri te oLur own
network software. No commercial software woauAldallow us to achieve ouir goals of speed andreal--time response for the network atreasonable cost. In designing the software, we
tried to maximize use of software inter-facesand data strLUc-tuLres in standard computeroperating systems. As a result a networksoftware system was completed r-ather quicklyby the +irst two authors and was found to havemore than adeqLuate speecd and flexibility fordata acqUiisition. he current networksoftware, which is described in this paper,differs in detail from that outtlined in our
previouis paperr l but Lupholds the same designgoals.
Data Ac_uisistion Software
Once th-ie network software was written,some general network related utilities were
written, and then atterttion was turned to data
acqlAisition software. It was decided to
approach this software also in a way whichwatold take maximal adivantage of software toolsirn standard computer operating systems. OuArapproach is explai ned i.n the followingparagr-aphs.
In the days of hardwired arnaly7-ers, setupand control was carr-i ed1 ouit by pressing
3817
buttons on instrument panels. In many dataacquiSition software systems which we studied,this approach was imitated: One was furnishedwith a large number of "buttons" in the formof typed commands which allow for experimentcornfigLuration and operation,, and often thesecommantds could be chained together forautomatic experiment control. (Of course, thenew data acquisition systems, being programs,had tte great advantage that they could bemodi-fied for those rare experiments which werenot hian-idled by the standard systems.) Suchcommnand driven systems offered great ease ofutse, btut also reqtired a large programmingeffor t, beyond that which we could support.
The developmerit of more flexible, easierto program computers has made another approachto data acquisition programming practical. Theexperi,menter could be provided with a goodcommier Cial operating system for programdevelopment and data acquisition specificsoftware tcjl s for implementing experiments.1thie software tools would be designed to allowthe ful01 rrarge of experiments to be handled,frorm the simplest ones of single parameteracqLi sitiown and analysis, to complex onesinclu,iditg multiple parameters and fed-backexperimien-it control, beyond the capability of acommainds1 ddriven system. The experimenter couldalso tbe provided with program templatesshowing how to apply the software tools toc omamrfi dat a acqu isi t ion probl ems. The essenceof thfis, approach to data acquisition software
Si that USEl of programming tools and commandsttrcttJi.-es i n c:ommercial operating systems istarsi#im ;zecd, anid t:he effort of including suchsfrs.tcltures within a. data acquisition softwaresy stem. whi cli can only be done with sacri f iceof flexibllity arnd/or a large programmingeffort, i-s niriimized.
Data acqUili tlon software based on thisappr oach a-td which VLitilized the network wedevelopedJ was dratted by the first authorst-ar-tin-ig in late 1981. This software has beendeveloped further by other Kellogg personnelat Caltech arnd various University of Rochesterper sorinel dtur-ing 1982 and 1983.
Sr a2b!(s1:9
Ihe rtetwor system developed at Caltechanid thVe Urnivet sity of Rochester wOUld clearlysupl:por t cuMpj,-l ex; interactive graphics as well.as high speed data ar:quivsition. Both kinds ofpi trAeseiI n ) i;nvu ve fast front -end response toeverits, hiuffet ti anrsfers, arnd processing poweranid fJ e;. hL I i ty that is best implemented on a
hostt (1 at ger ) co:mputer. In graphicspi ocessinq, the events are requests fordiispl ay -harnge and graphics device controlinterrutpts. Bujffer-s ar-e data to be graphicallydi spl ayedJ whi ch are senxt from the host to thefront -end crimpuiter . The host provides thecomputational power to extract display datafromf experimerit data.
At the UJriversity of Rochester theft ucnt- encl COmpuLters contain small arraypyncessors (miade by Sky Computer) [3). Thisoffer s a uiseful option for programmableonltrol of graph icr:s displays: Such array
processot s cOUld be used to translate, rotate,
otn other-wise transform displays faster than aVAX h-ost cotld, thtts off--loading from the host
sinple bit-t comnptitationa'ly intensive tasks. It
is platitned that these array processors also:ar y oftt sImtple, cotmpitationally intensivedata aralysis in ntc:lear physics experiments.
3818
At Caltech the capab'ility of the networkto support complex graphics came to the
attention of Engineering Division personnel.As a resUrIt, two VAX COMPLtters and some salary
support were made available for rehosting of
thle network software system on the new
Ethernet hardware. Ethernet hardwar-e allowscomputers separ-ated by as m itch as 500 m tocommunicate. Th-e newCai1tec-hnetwcnrk will span
three buAildings and will sLpport- graphics wctrk
stations fo-r- education of studen-its in
computer-aided engineering concurrently with
use for data acqiti-sition.
Network Hardware
Many kinds of local area network hardware
are now available. Some communiicate data in
parallel (severaal bits per clock pullse), as
does the Peritek network (which we currentlyuse), btt are usual1 y limited to
inter-compiter distances of a few tens of
fee-L. Others convert the data i ntio aserisalstream which can be transmittedd much farther.Ethernet (which we wi1ll use soon) is a networkof the latter designgr , and will probably become
the standar-d of the computer indu.sistry; It is
presenitly acc-eptedi by Xerox, DEC, and Intel as
their fuLt.Lire standarid.ThVee (a.-itetech network installed in 1981
uses Perit.ek hardware. DuLring 1983 the Caltechnetwork will herehosted onEthec-net. hardware
built by Inter-lan,A descr iption crf the Peritek and Ethernet
hardware follows:
Frer-i tel Hardware
Peritek hardware costs about 207. as much
as Ethernet hardware per computer (around $500per- computer) and of fers speeds up to 163 K'.'per second (see the section or network speedibelow). It is restricted to a 40
f
loti nter -Computer separation. One computer serves
as the network master (the host computer inour systems) all commuFnication is
arbi trated by and1 m.slet; pass through t..henetwork master. Eachi front-end comptter isconnected by a separate 40 line r-ibbon cableto the networ-k master, and connectiorns are
dJi-ect withi rto electrical isolation. Front-endco0mputers cart be corirected and disconnected
from their-rcables while the n)etwork isrLtnn i r-ig.
Ether-net Har-dware
Et hernet hardwar-e al lows rtp to a hundred
computters tco communicate on each Ethernetcabl e segment , whi ch may he utp t -c 500 in long.
Several cable segments miay he interconnectedby pui nt-ito-point 1inks Uip to 10bCoO.) m l ong. No
compirter serves as a network master (except as
desired in so-ftware); each F-thernet interfacehandles arbitration for the cable, message
transmission, and retries if messages colli-deauitonomously. Any interface may directcommiunication to any other interface. Allcommunicationr occurs on coax-ial cable withseveral hundred volts of transformer-isolation. Raw speed on Ethernet is 1.25 MD(B1 megabits) per second. Actual speed is
abo ut a third of this because messages are
buffered in thie Ether-net in-terfac-es. (We note
that one coulId modify the Peritek hardwarewith FIFO bLuffers and transmitter-receivers to
outperlorm Ethernet in speed and distance,although not in generality or flexibility.)Ethernet inter'faces may be installed on andremoved fron the Ethernet cable while thenetwor-rk is running.
Frhe Per-tee network hardware operates onthe 0-buLs ofthee LS1-1I front-end computersthat we are using. The particular Ethernethardwar-W-e thata we have selected is built byI rit elan also for the Q-bus. On all VAXccomputters, wehave installed Q-bus extensionsto theIJnribuls by using Able Quniverters; thehosts netwot f inrterfaces reside on these0-tbus etensinsions.
Hoth Fer itek and Ethernet (Interlan)uetwor t. har dware are adequate in speed forIII$LI eat diata acquisition, which generallyrequires transfer rates below 30 KW (60 KB)per secoIId. For- a small laboratory notr-equitirng long rut.ns between host and front-end(no front- end computer in the target area!)9,Per i tek is the most cost effective choice.
T'he E'therrnet project at Caltech hasdifferernt requirements and goals from those ofa typi cal small nuLtc-lear physics laboratory:Data acqi sitiorn front-end computers locateduear a tarndemn accelerator in one building willbe connected or-n a 500 m Ethernet cable segmentto tw-) VAX computters in two other buildings.Tlese compuiters, in addition to dataacqulisition1 , run compuLter - aided engineeringsoftware and ar-e expected to use the networkcocwcur rent ly for non data acquisitionarit i Pt Personnel of both the Physics and
Engigneering Divisions are involved in thisEthernet pr.oject. as both divisions have
rcomptutational needs in data acquisition and
graphics. The project represents a feasibilitystudy ton demonstrate that high speed,real time computtational needs can be satisfiedbhy a nietwork of distributed resources. (Itshotutcl be not e t-ere that there is currently a
par-al lel ramapus-wide Caltech effort to provideesuurtl41 re shar risg anid terminal hookUps U-sing
I f4er Cnetter.:htinology, We anticipate that the
F1iysIc: /9 Engineering network described in
tih5is palp1er will be integrated into the campus
Ietwor- by direct connection, connectiontlt-C)ugh a gateway, or connection through one
or nute oputter which have nodes on eachetwor .>
8oftware
h-tie Caltec.h software is written in layersin ot deri to facilitate transportability andmudcajtt. The f-irst ayer is specific to
rinet wor- k hat-dwar-e A second layer providesgeneral 1ut ilities for carrying ou.t softwaretasks shared between host and front-endcomtjpltersa A thi rd layer supports nuclear data
actqei si't i cun. Thte second and thi rd layers are
trLetwov. h1'1 ardware independent, and communicatewi thl tIrle first layer through standard so-ftwarecnuslrt$ACts of thle VMS and RT--II operatingsystems. We describe these software layersstLar ing with the first layer.
Netwnor Sgtiftware Layer
tlhe 'alter 1h network software implements-rntet coptpuLer ccommurnicatiorn required by highSpo[eu1 nf ataacqIivii tin4. Speci ficily, we havechosen to SLIppOrt:
3819
1. Sending messages from the host tothe front-end computers;
2. Sending buffers between the host andfront-end computers (data buffers goto the host, graphical and extensivecontrol information go to thefront-end).
In implementing item 2, we supported one othervery useful kind of network communication:Emulation of disk transfers for the front-endcomputer on the network. This capability wasdeveloped with the help and experience of thesecond author with computer networks atWashington University in St. Louis. It allowslow cost, disk-less. front-end computers torun complete disk-based operating systems.Also, it allows us to take maximum advantageof commercial software structures: Instead ofintroducing a special message protocol fornetwork communication, we draw on the hardwareindependent disk read-write call structureoffered by the front-end computer operatingsystem (RT-1t in our case).
"Pseudo-disks" for the network diskemulation are set up within virtual memoryarrays of a network "host" program on the VAX.Because of the way the VAX VMS operatingsystem handles virtual memory, the pseudo-disktransfers are cached in VAX memory. Theresulting disk emulation performance oftenexceeds that of a hard disk because the latterrequires seek time to locate data.
Utili.ty Software Layer
In addition to the basic network softwarethat allows commtunication of messages anddata, a layer of network utilities has beenwritten which supports the following dataacquisition relevant software actions:
1. Control of programs in the front-endcomputer by programs on the hostcomputer and vice-versa;
2. Awakening of a program on the hostcomputer each time a special databuffer transfer occurs from thefront-end computer into the host.(When the data acquisition hostprogram is not executting, themul ti-uiser host can service otherprograms. In fact, several dataaCqUisition programs with separatefront-end compuIters can runconcLurrently on a single hostwithOtrt wr-iting special software).
The above actions are carried ouit rapidly,with guaranteed timing, so that an experimentcan be contrl led properly and so that bul fersof acqUired data are analyzed before thehuffers are nr-eciede fnir new data.
In addition tn these actiors, thefollowing convenience fematures are supportedby the network utility software:
~.< Initiation of data acquisition fromanv host terminal in suLch away thatthe acquisition can continuLeaustonomMoslIY, independient ofstubsequent t erminal t.isage;
4. Connecting to and disconnecting fromon-going data acquisition from anyterminal;
5. Common data areas in the host(generally Used for histogramming)which can be concurrently accessedby the data acquisitiorn program andby experiment display and (:cntrol
programs. Common data areas allowone or more people to examine one
on-going experiment, and allow anexper-imenter to write, compile, andrun display and analysis programsthat act on incoming data while anexperiment is running.
Data AcqUisition Software Layer
The above utility software is describedin terms of data acquisition, but could justas well be? used to support any software taskshared between host and front-end computers.Outside of: the utility software layer lies athird software layer specific to dataacquisition. Part of this third software layermu-st be tailored to the specific datacollection hardware of the laboratory thatuses it. For- example, Caltech uses KineticSysteins memory mapped CAMAC crate controllerswhile tthe lUni versi ty of Rochester uses Jorwayparallel /serial CAMAC highway branch driverswith DMA interfaces.
As discussed above, we have not written acomplete, command driven data acquisi tionsystem. Instead, the experimenter uses thesoftware dcevelopment tools of the operatingsystem arid is pr-ovided with data acquisitionspecif if soft ware tools in the form of (1)macro and stibroUtine libraries,, and (2)selecnted programiring langLuages.
Macros and subroLtines are provided toimplement: tthe following basic data acquisitionrelated srtftware actions:
1. Iransfers to and from datacollIection interfaces, e.g. CAMAC(a,arr-os sr'hbrortines);
2. IFilling of buffers with collecteddata (macros, subro)Utines);
i..* Sending of filled buffers to a dataanalysis pr-ogram (subroutines);
4. Coimmuitnication of messages betweenthe collection and analysis programs(sbrhotitines) .
Macro aessembl y langUage and Fortran languagesare provided fur- coding:
1. Response to real-time events(nuclear events in detectors), which
i espo:urise rnormal I y consists ofi eadiu1g data acqLi sition interfaces,e.g. CAtAMAE, an-d stor-ing data in altbId fec (Iacro);
2. Anlalysis of buf fers, includingyating and histogramming (Fortrani);
.. If rieedled, experiment control basedutr fteedback fronm data collected,elapsecd t imie, etc. (Fortr-an).
Ilhe softwar-e is structured so thatdishtu ibUtion of the above tasks betweenfront- end and host computers hardly needcturic erri the experimenter- Both computersstippor-t s iii I ar- Ma:ro assemblers and Fortrancompi IIer s, and message and buffer transfersu-brolutines handtle all network communicationt,r an-sspair er t I y
N1etwork Sp2e!
[her e are two speeds in the describednetwork software system which are importantfor nuc. lear dataa-cq-uisitionr One is the speedat whichi data buffers are transferred from afr ont end compter- to the host. The other istire speedi at which a sigrial from a front-endcompuiter- ( for- readet-s of the Appendix, write
to block 0 of NT7) can pass through allsoftware layers (NTDRIVER, NETHOST, andNETUTIL) to awaken a host program, and cancause that program to signal back to thefront-end computer (write to SATmNETMB whichCaLlses NETHOS-r to sernd an unsolicited messageover the network). We call this the"turn-around message" speed, and designateturn-aro:und message by TAM for short.
A simple benchmark was written: TheLSI-1l program for the henchmark carries oQit apseudo-diski write (buIffer transfer), followedby a VAK program wake-up signal (write toblock 0 of NT7). and then program loopswaiting for a return message from the VAX(waiting for the command word to be filled byan unsolicited network message). When thereturn message arrives, the program startsover, repeating the above operations. Thecorresponding VAX program for the benchmarkcalls the library subroutine which awaits awake-up message from the LSI-ll (mail boxmessage from NETUTIL initiated by NETHOST whena write by the LSI-11 to block 0 of NT7 isreceived) and then sends a message back(writes a command to SATmNETMB which istransmitted by NETHOST over the network as anunsolicited message). 2000 iterations of theseprograms were timed for different sizes ofbuffers transferred. The results are asfo] lows:.
LSI-11/2 FRONT-END
BufEfer size Ti me for(bytes) 2000 cycles
01024
73 sec104 sec
284 sec
LSI-ll/23 FRONT-END
Buffer size Time for
(bytes: 2000) cycles
0 58 sec
1024 84 sec
16384 252 sec
Conclusion
37ms / TAM37ms / TAM +
65 KB/sec buffertranisfer rate(Effective transferrate: 19 KB/sec)37ms / TAM +
During the operation of this benchmark, thetime in user mode on the Kellogg VAX-11/750dropped from around 860% to around 45.. 'i'hus,we estimate that about 357. of the VAX CPU timewas being used for network operation. Based on
this measurement we compuLte that for 30 KW/sec
(60 LB/se). the maximum reasonable rate in anuclear physics experiment, and for 8 KWbufffer-s, 60/127 x .35 .= 17% of the VAX CPUtime would be used to support the networktransfers.
Parerithetically we add that theVAX-11/750 can histogram, flat-out, 75000single par-ameter events per second. Thus anexperiment where 30000 events per second areacquired by a front-end computer, transmittedover the network in 8 KW buffers, andthistogrammed would use 17-. (network) +30000/75000 (histogramming) = 577. of the VAXCPU time. Of course, this is poor use of thepower of a VAX'
At the University of Rochester a morecomplete benchmark was written thiat includedthe utility software layer (SETIJF andCONTROL), and passed buffers and messagesbetween the LSI-11 and the VAX with the dataacquisitiorn library routines provided (seeabove). The result for two thousand 1024 byteplus TAM transfers using an LSI-11/23 was 84seconds, identical to the Caltech result.Ihus, the utility and data-acquisitionsoftware layer-s add insignificant executiontime to the system.
Rehosting on Ethernet
The rehosting o-f the network on Ethernetrequtires rewriting the RT-11 NT device driver,and the VMS NTDRIVER and NETHOST programs (seethe Appendix for a description of NT,NTDRIVER, and NETHOST). By preserving all mailbox and event flag interfaces, all othersoftware layers developed are expected to runwi thOUt ch-ange.
Bec-at-use Of the greater flexibility ofEt herrnet as compared to the Peritek network,some enhanicements of the system are planned.One enharcement is the ability to select oneof several hosts with which network transfersare to OccUr f-rom a given front-end computer.ne application7 of this capability is that twocopies cf the F;T-1l network device driver-couild be loaded into the front-end computeranid dir-ected to two different hosts forsimultareous oper-ation with the two hosts.Another application is that if one host-omputer goes down, an on-going experimentLuOld be c-r,ntinUed with another host. A secondplanned Ethernet enhancement is theelimination of t-he serial lines betweenfront-end and host computers by adding a
"pseu-do-termninal" communication channel to thenetwork 5software. This pseudo-terminalrommtqruicatiori would be added to the RT-11niwl-worv device driver and VAX network host
[programi s in a manner which operatedasynchrunoiisly with respect to pseudo-disk andmiessage transfers. Using this channel it isexpeLLed tthat each computer will be able toemulate terminal communication on the other.
Appendix'Technical Descri2tion
of the SoftwareNetwor-k Software Layer
Iwo kinds of network communication are
sUpported. One k:ind is disk emLLlating readsan-id writes controlled by the front end LSI-11compt.lter. 'rhis communication consists of DMAtransfers of I to 8 lt word buffers. The other
3820
16384
-------------------------------------------------
3821
kind of communication is transmission of a twoword message from the VAX host to an LSI-11 byinterrupt and programmed handshake under VAXcontrol. This kind of communication is called'unsolicited network message", unsolicitedbecause is is not initiated by the LSI-11.
We have written a bootable RT-11 devicedriver NT to control the. Peritek networkhardware from the LSI-11 end. On boot, ROMcode on the networli interface signals the hostto DMA a boot blIck into LII-1l memory. Thisboot block has code to boot the RT-l1operating system. The RT-ll device driverSuApports pseudo-di-sk reads and writes to eightnetworkpseupdoi-disk units, NTO-, NT1, ..., NT7.Besides reads and writes,f the device driversupports two sperial function calls. One callis the standard RT-11 call for variable sizedevices; it returns the device size of the
uinit specified, The other call suppliesaddresses of two words, called "command" and"flags" words, and (optionally) of aninterruipt roLutin-ee, which areuLsed as followswhen anLiunsolicited message comes over thenetwork from a program ruLnning on the VAX: Thenetwork message contains two words, one ofwhich is loaded into the command word in
1.ST-l memory provided that this word iscurrently zero (which signifies that the
L[SI-11 program has picked Lup the previouscommand word). The other message word is or-edinto the1ST-I11 flags word. Typically,, thecommand word is used for control of an LSII-program by a VAX program, and the flags word
is uLsed to signal to theL1S-I1 program whenthe VAX has finished wi th abuf+fer-. After themessage words are loadcid, the suppliedinterrtutpt rou,ite (if any) is cal led.
Oin the VAX, a device driver- N-DRIVER waswritten which fields interrupts from1.SI-11romptiters requiesting network transfers, andhandles networlk DMlA readand writte trans4-ers.All otter network control is carried OUt in aFortran host. program NETHOSI which runs atpriority 15. NETHOST maniptulates the networkhardware to (1) obtain specific networktransfer requests from the LEII-1ls and (2)return tr-anisfer status to the LSI-lIs.
In additionr to network hardware contr-ol,NETHOCST maintains global sections which act asthe pseudo-disks for the SLSI tls. NETHOSTcommUnicates thr-ough a VAX mail, box NETIAOSTMEr)Xwith a program HS ("host-set") which is rLun bya system operator to map and unmappseudo-disks from global sections. Flexiblepseutdo-disk mapping is supported: A globalsectijon may be mapped to one or morepsetido-di sk uinits of onre or mor-e t-51lis sothat, each l SI-11 has uinit specific read orread-write access to each global section. In
addition, programs have been written for theVAX which allow one to directory and copy
T-if1files -from within pseudo-disk formatglobal sections.
Note: A modification was made to RT-11 onthe L[SI-Jts scoi that all console commands drawutility programs, cormpilers, and librariesfromn a device with logical name SL rather thanfrom SY. Ofne psed(oa-disk whi ch contains all ofthe above mentioned software is mapped withread-only access to NT1I of all LSI-1ls, and
NT1 s as.ignedto Lw Tvzs! dul iandNi'l is assigned to SL. -rhLiS. du-plicatepseudo-disk space with RT-11t utility programs,cocmpilers, and l-ibraries need not be providecdfor m-ultiple LESI-Ils.
NEiIOST also provides the followingser-vices:OnO apseuddo-disk write to block zeroof NT7f a flag in VMS event flag clusterNET7FLAGS is set toawak:en the utility programNETUTIL andinfor-m it which LSI-11 did thewrite (see below for action of NETUTIL).NETHOST watches for input from mail boxesSATmNETMB where m is the number of thefront--end LSI-1i1 computer, often called a"satellite". Wher4 a message arrives, two wordsare extracted from it and sent to theindicated Id--SIl as an unsolicited messageover thte network.
Litilii yotywat re Laye
A Fortran program NETUT IL was writtenwhich runs onthte VAX at priority 6. It awaitsa signa'l fromn NETHOST ir NET7FLAGS that tellsit when an LEd-ill writes to block zero ofpseudo-disk. unit NT7. NETLITIL has arraysmiiapped tothle global sections used by NETHOSTfor N-1-7 o-f each LIS-11. After receiving a
NEiT7FLAGS signail , NETUTIL picks up the firstniu.1ll termniniated character string in block Ci ofNT7 and creates a VAX process rLinning theprogram ini the file given by that string i-fS5Chi a pj-rocess has not already been created.Ti. theu sends a 16 character mal.1 box messageto that process, the first 1tfU character givingthe uIser nlatmleOf the owner of the LSI--1l (seeheowIa n-d last foLur characters being "SATinwhereni is theL-i5t11's number. This messageiS typicLally uAsed to awaken a VAX dataanalysis program waiting for new data buffersfrom arn t dI-T1l data collection program.(Usually, return mestss ages from the VAX dataarlalysis progranm to theL[1-l data collectionprogram are senit by wri-ting to mail boxSAlmNFTIMlB. As described in the previoussection,, NETHCOST picks up suLch messages andser-nds ttlem as Unsolicited messages over theiietwori3.
lST-I-1 ownership is established asf l Iows: Each 1 5-T11 has a serial linecu:rinecting it with the VAX. Tht-is serial linemutst be allocated by (and therefore owned by)a VAX user- prior to any LSI-11 operation whichinvokes NETUTIL or else an error message willbe delivered to the LSI-11. The owner of theX:eri al linie is cornsidered the owner of theLSE 11 by NETUTIL.
Another utility program, CONTf'OL, waswritten to control 0151-1i operation from adetached process on the VAX and supportLcLmmrlnication between t his detached processand arn, interactiveuLser (user at a terminal).(CONTROL is operated as fol lows: Theexper- imeriter lorgs on to the VAX and runs a
pr- i vi eged set'Llp pr ogram which starts a
de-t.achecl pr oce=s un-nder his or her accounttrini} rig the pirogram CONIRPOL. CONTROL is toldthe rumber- of an -L-SI-11 and the UIC (useridertificEtiou-i code) of the experimenter whostarted CONTROL. CONTROL allocates the LSI-11serial line and thereby becomes owner of the1ST-iT.Interactive L(Ser initiated messages tobe sent to the LST-11 over the serial line orvia rminsl ixci ted u-.etwork messages ares,bserieret I y ro.tted throuLgh CONTROL: Aninteract iVe user may communicate with CONTROL,y sediifgti essages tho-ugh VAX mail boxS3A T1rrC MI (IB O(m-LSI --Il number). Communicationacmkk low]ledgemer-it. is senit throuigh event *lagsSAT'in, If 1ti1e masen 1-has the saime UIC as CONTN7iOL,I,he t-i5er tmiay c-unr;ect to COINTROL with a connect
3822
message. CONTROL saves the PID (processidentification) of the user. Thereafter, until
the user logs off of the VAX, the user may
issue requests to CONTROL to send messages tothe LSI-11. The PID sent with any requests is
checked by control, and if it is the connecteduiser's PID, the messages are forwarded to the
LSI-1 1.
CONTROL serves several other useful
functions. The VAX program created by NETUTILto communicate with an LSI-11 program is not
attached to a terminal. CONTROL allows this
program to communicate with a user who is
connected to CONTROL as follows: Mail boxes
SATmMESSMB are provided on the VAX for eachLSI-11. A write to the mail box correspondingto the LSI-11 that CONTROL is operating causes
CONTROL to check if there is a user connectedto itself. If so, the message is broadcast to
that user's terminal, appearing on the
terminal screen regardless of the program the
user is currently running.If no user is connected to CONTROL, the
message is saved in a file. When a user
subsequently connects to CONTROL, he or sheimmediately receives all pending messages.
CONTROL logs all such messages to the user as
well as all user requested messages to be sentto an LSI-11 in a file.
Data Acuisit Layer
A program SETUP is provided which acceptsfrom an input file:
1. The name of an LSI-11 front-endprogram;
2. The data buffer size and degree ofmultibuffering in the LSI-11 and VAXmemories;
3. Code for the VAX data analysisprogram which is to communicate withthe front-end program.
The experimenter must previously have created,compiled, and linked with data acquisitionsubroutine and macro libraries the LSI-11program. The SETUP program (1) applies a
preprocessor (discussed below) to the VAXcode, (2) compiles and links this code with a
VAX subroutine library, (3) starts a process
under the account of the experimenter whichruns the program CONTROL, (4) directs CONTROLto to allocate the serial line to the LSI-11thereby taking ownership of the LSI-1 andvia CONTROL, (5) starts the LSI-11 program bysending "RUN <LSI-11 program name>" over theserial line, and (6) sends to the LSI-l1program the name of the VAX data analysisprogram with which it will communicate as wellas (7) the specified buffer size and degree of
multibuffering. The LSI-11 program begins bycalling a setup subroutine on the LSI-11. This
setup subroutine sets up data areas for
buffers, and writes to block 0 of NT7 the name
of the VAX data analysis program with which it
is to communicate. On receiving this write to
block 0 of NT7, NETHOST flags NETUTIL and
NETUTIL starts the specified VAX program. The
LSI-11 program then sends a message which is
received by the VAX program it started, and
this VAX program passes the message on throughSATmMESSMB to CONTROL and thereby to the
experimenter's terminal, informing the
experimenter that all programs are running and
that data acquisition can start. The VAX and
LSI-11 programs will thereafter communicate
with each other by:
1. Pseudo-disk reads and writes fordata transfer;
2. Writes by the LSI-11 program toblock 0 of NT7, which cause NETHOSTto signal NETUTIL to send a mail boxmessage to the VAX program;
3. Writes by the VAX program toSATmNETMB, which get passed byNETHOST to the LSI-11 program viaunsolicited network message.
The LSI -11 and VAX programs are quiteflexible and can easily be tai 1 ored tospecific experiment needs. The onlyrestrictions are that the programs must startby calling library setup subroutines, and thatthey must pass buffers and messages to eachother and to user terminals with the librarysLIbroutines provided.
In order that the VAX data acquisitionp,rogram store analyzed data in a form that maybe simultaneously accessed by other programs,a global section SATmUTIL is provided for eachLSI-11. The VAX code supplied as input toSETUP may include "special declarations"SPECTRUM and WINDOW for arrays. Thesedeclarations are converted by a Fortranpreprocessor in SETUP into integer*4 and bytearray declaratioris, and the arrays so declaredare equivalenced to parts of a memory regionmapped to SATmUTIL and are pointed to bydirrectory entries of Caltech design withinSATmUTIL. Other VAX programs can map toSAI'mUTIL and, by using the directorystrLicture, can locate these same arrays byname.
Ref erences
El] VAX/LSI-11/CAMAC Nuclear Data AcquisitionSystem under Development at the W.K.Kellogg Radiation Laboratory, CaltechpJ.D. Melvin, M.H. Mendenhall, R. McKeown,and T.A. Tombrello, IEEE Trans. on Nucl.Sci. NS-28, 3739 (1981).
[23 VAX-i1Based Acquisition Computer Systemat the Nuclear Structure ResearchLaboratory, D.L. Clark, T.S. Lund, andJ.D. Melvin. Paper presented at theBiennial Conference on Real-Time ComputerApplications in Nuclear and ParticlePhysics, Ber keleys, CA (May 16-19, 1983).
(3] The Performance of an LSI-11/23 withArray Processor as High Speed Front EndProcessor, D.L. Clark. Paper presented atthe same conference as reference 2]3.
Acknowl edgements
Thanks are due to the followingLontributors to the systems described in thispaper: At the W.K. Kellogg RadiationLaboratory, Caltech, K. McLoughlin, D.Potterveld, R. McKeown; in the EngineeringDivision at Caltech, P. Dimotakis; at the
Nuclear Structure Research Laboratory,Uni versi ty of Rochester, T. Lund and P.
Blazer.This project has been supported in part
by NSF grants PHY79-23638 at the CaliforniaInstitute of Technology and PHY79-23307 at the
