IEEE Transactions on Nuclear Science, Vol. NS-34, No. 4, August 1987 NEUTRAL PARTICLE BEAM DISTRIBUTED DATA ACQUISITION SYSTEM * R. T. Daly, M. R. Kraimer, and A. H. Novick Argonne National Laboratory 9700 S. Cass Avenue Argonne, IL 60439 (312) 972-6966 Abstract A distributed data acquisition system has been designed to support experiments at the Argonne Neutral Particle Beam Accelerator. The system uses a host VAXstation II/GPX computer acting as an experimenter's station linked via Ethernet with multiple MicroVAX IIs and rtVAXs dedicated to acquiring data and controlling hardware at remote sites. This paper describes the hardware design of the system, the applications support software on the host and target computers, and the real-time performance. Introduction The Neutral Particle Beam Test Stand (NPBTS) of Argonne National Laboratory (ANL) utilizes the 50 Mev Hinnjector from the former ANL Zero Gradient Synchrotron (ZGS). The injector operates at a maximum repetition rate of 30 Hertz and a pulse width of 60 microsec.11] The NPBTS facilities, located in the ring building previously used by the ZGS, consist of three beam lines designated as Phase 0, A, and B and the test areas or stands at the end of these beam lines. Users from government laboratories and industry will conduct experiments at the NPBTS primarily involving physics measurements, engineering measurements of components and subsystems related to a space based Neutral Particle Beam, and beam diagnostics measurements. Due to the diverse nature of the planned experiments and the differing backgrounds of the experimenters, the data acquisition system must support a wide variety of instrumentation located at the test stands. Although only one experiment may be on-line at one time, multiple experiments may be in a check-out stage; therefore the data acquisition system must support the simultaneous acquisition of data from multiple experiments. Hardware System The data acquisition system, NPBDAS, as shown in Figure 1 provides at each test stand and in the NPB control room a target data acquisition system (TDAS) consisting of a single board MicroVAX II computer with interfaces to a number of standardized instrumentation systems including CAMAC, IEEE 488, RS232, and VME and to the Ethernet used for host computer comnunications. Each TDAS is responsible for acquiring data from and controlling the instrumentation at its test stand, for buffering and preprocessing the data prior to transmission to the host, and for the transmission of the data over Ethernet to the host computer. A 500 meter coax cable provides the only data link between Figure 1. NPBDAS Hardware System. * Work performed under the auspices of U. S. Dept. of Energy. 0018-9499/87/0800-0816$01.00 C 1987 IEEE 816
Transcript
IEEE Transactions on Nuclear Science, Vol. NS-34, No. 4, August 1987
NEUTRAL PARTICLE BEAM DISTRIBUTEDDATA ACQUISITION SYSTEM *
R. T. Daly, M. R. Kraimer, and A. H. NovickArgonne National Laboratory
9700 S. Cass AvenueArgonne, IL 60439
(312) 972-6966
Abstract
A distributed data acquisition system has beendesigned to support experiments at the Argonne NeutralParticle Beam Accelerator. The system uses a hostVAXstation II/GPX computer acting as an experimenter'sstation linked via Ethernet with multiple MicroVAX IIsand rtVAXs dedicated to acquiring data and controllinghardware at remote sites. This paper describes thehardware design of the system, the applicationssupport software on the host and target computers, andthe real-time performance.
Introduction
The Neutral Particle Beam Test Stand (NPBTS) ofArgonne National Laboratory (ANL) utilizes the 50 MevHinnjector from the former ANL Zero GradientSynchrotron (ZGS). The injector operates at a maximumrepetition rate of 30 Hertz and a pulse width of 60microsec.11] The NPBTS facilities, located in thering building previously used by the ZGS, consist ofthree beam lines designated as Phase 0, A, and B andthe test areas or stands at the end of these beamlines. Users from government laboratories andindustry will conduct experiments at the NPBTSprimarily involving physics measurements, engineering
measurements of components and subsystems related to a
space based Neutral Particle Beam, and beamdiagnostics measurements. Due to the diverse nature ofthe planned experiments and the differing backgroundsof the experimenters, the data acquisition system mustsupport a wide variety of instrumentation located atthe test stands. Although only one experiment may beon-line at one time, multiple experiments may be in a
check-out stage; therefore the data acquisition systemmust support the simultaneous acquisition of data frommultiple experiments.
Hardware System
The data acquisition system, NPBDAS, as shown inFigure 1 provides at each test stand and in the NPBcontrol room a target data acquisition system (TDAS)consisting of a single board MicroVAX II computer withinterfaces to a number of standardized instrumentationsystems including CAMAC, IEEE 488, RS232, and VME andto the Ethernet used for host computer comnunications.Each TDAS is responsible for acquiring data from andcontrolling the instrumentation at its test stand, forbuffering and preprocessing the data prior totransmission to the host, and for the transmission ofthe data over Ethernet to the host computer. A 500meter coax cable provides the only data link between
Figure 1. NPBDAS Hardware System.
* Work performed under the auspices of U. S. Dept. of Energy.
0018-9499/87/0800-0816$01.00 C 1987 IEEE
816
817
the TDAS systems and the host. The host computersystem is located in the NPB control room and includesa VAXstation II/GPX color workstation, a variety ofmagnetic storage peripherals, a video imagingsubsystem, an Ethernet interface to the TDAS systems,and a second Ethernet interface to a LAN whichincludes the NPB MicroVAX II Control Computer and theMicroVAX II Remote Experiment Control Computer.
Support Software
The software system developed to support theNPBDAS has the hierarchical structure depicted inFigure 2. The lowest level or Instrument SupportLevel is completely located in each TDAS. It is aVAXELN system which provides all the general purposereal-time functions needed to support the attachedinstrumentation and to communicate over aDecnet/Ethernet link with the host computer runningthe VMS operating system. The System Support Level onthe host provides a buffer management scheme and adefined interface to the Instrument Support Level fromthe User levels which, in general, makes thedistributed nature of the NPBDAS transparent to theUser levels.
The highest level or User Support Level providesa comnnand level interface to utilities which displaydata, manipulate disk data files, and analyze datastored on disk files.[2] In addition, an ExperimentDefinition Utility is used to define for eachexperiment the control functions to be executed by theTDAS, the parameters to be collected by the TDAS, thedata conversions to be applied to the collectedparameters at the host before storage or display, andthe record structure for the data file used to storeselected parameters. The experiment is initiated by aRUN command from the command level. A future paper
will describe the User Support Level in more detail.
Users who wish to write their own dataacquisition programs must use the System Support Levelsoftware to interact 6i4h the TDAS systems and areprovided with the RS/1 3 comnercial software productto analyze and display their experimental data.
Instrument Support Level
The Instrument Support Level software is run oneach TDAS. The TDAS MicroVAX II computers run theVAXELN kernel and use the VAXELN Network Services toprovide a real-time operating system environment forthe VAXELN application programs, and to providesupport for the application level logical link(program to program) communication over Ethernet tothe host ccmputer system.
Two application programs, INIT and VMS_LINK,provide all the run time support at each TDAS. Afunctional block diagram of the application programsis shown in Figure 3. When the TDAS is booted (eitherat power-up or by a host DCL command) the INIT jobcreates a global data area shared by all VAXELN jobs,initializes the instrumentation hardware, anddynamically creates ten jobs, each running the codecontained in the single program image VMS_LINK. Aseach job is created it is passed a unique messageidentifying two logical links, a COIMAND LINK and aDATA LINK, which will be used to comnunicate with anapplication program on the host. Once established theCOMMAND LINK provides a communications path overEthernet for a host application program to passCOMMAND LISTS to the VMS_LINK job on the target. TheCOMMAND LISTS direct data acquisition and controlfunctions at the TDAS. As each COMMAND LIST isreceived, a VAXELN process is created to execute it.
.ny data collected by the TDAS during the execution ofthe COMMAND LIST will be returned to the host over theDATA LINK by a separate COMM process in the VMS LINKjob. Thus, real-time data acquisition can proceedconcurrently with data transmission to the host.
With ten VMS_LINK jobs running on each TDAS, upto ten application programs may be concurrentlydirecting data acquisition at each TDAS viaCOMMAND LISTS. In the NPBTS environment, this featureallows one experiment to be collecting data on-line ata TDAS while multiple experiments are collecting datain a check-out or background mode using instrunentsinterfaced to the same TDAS. Also, it is typical for asingle experiment to establish links to more than onetarget, i.e., a link to TDAS at Phase B beam line anda TDAS in the control room, in order to collect datafrom instruments interfaced to more than one TDAS.
The structure of a COMMAND LIST is shown inFigure 4. The COMMAND LIST consists of a four wordheader and a variable number of integer commands. Theheader words are:
1. UNIT. The identifier of a queue maintained bythe System Support Level software which iscreated by an application program on the hostprior to sending a COMMAND LIST to a TDAS, and towhich any buffers returned from the TDAS as aresult of executing the commands in theCOMMAND_LIST will be queued. The applicationprogram retrieves buffers from the UNIT queueusing the System Support Level software.
HEADER4 WORDS
COMMANDSINTEGER
COMMAND_LIST:
UNIT
TYPE
SIZE
REPEAT
COMMAND 1
COMMAND 2
COMMAND 'N'
ID OF HOST RECEIVE BUFFER QUEUE
EXECUTE NOW, EXECUTE-WAIT, STORE LST
MAXIMUM SIZE OF RETURNED BUFFER
HOW MANY IMES TO REPEAT LIST EXECUTION
_ CMD BYTE 3 BYTE 2 BYrE 1
BYTE DEFINITIONS ARECMD DEPENDENT
COMMAND_LIST STRUCTURE
DO I NUMBER OF ITERATIONS e1000
IWAIT INT. I I CRATE ISTATION
]1
4 i *
UDF I WHICH ONE =1
READ I FUNCTION I CNA
READM I TIMES =10 CNA
ESL I WHICH ONE=2
ENDDO I
DO 1000 TIMES
WAIT FOR INTERRUPT
USER FUNCTION #1
SINGLE CAMAC READ
10 CAMAC READS (FO)
EXECUTE STORED LIST#2
ENDDO
IEND I ENU OF- L161
EXAMPLE COMMANDS IN COMMAND_LIST
2. TYPE. Specifies to the TDAS how to manage thecommands in the COMMAND LIST. For example, theTYPE specifies whether to store the commands inthe global data area or to execute the commands;and, if the commands are to be executed whetherto execute them before or after a reply is sentto the host acknowledging receipt of theCOMMAND LIST, i.e., synchronous or asynchronousexecution. At the host control is returned to theprogram sending the COMMAND LIST only after areply is received.
3.
4L.
SIZE. Specifies the maximum SIZE of the databuffer that the host can receive from the TDAS.REPEAT. Directs the TDAS to execute the commandsin the COMMAND LIST a REPEAT number of times. Avalue of 0 means forever. Each time the end ofthe commands in the COMMAND LIST is reached anydata collected will be transmitted to the host.Thus, this word when used in COMMAND_LISTs whichcollect data and indicates how many buffers ofdata will be returned.
There are presently over twenty-five commandsavailable within the VMS_LINK program. Since eachcommand is implemented as a Pascal procedure, newcommands are easily added. The commands fit into fourcategories:
1. Hardware specific - Examples are commands tocontrol CAMAC modules in single transfer andblock transfer modes and commands to read andwrite strings to an RS232 controlled device.
2. Structural - Examples are the Do loop commandwhich executes a group of commands repeatedly,and the command Execute Stored List whichexecutes command list previously stored in theglobal data region.
3. Control - Examples are the command to Wait ForInterrupt which waits for an interrupt from aspecified CAMAC module, and the command to Waitfor a fixed period of time.
4. Special Purpose - Examples are the command toexecute a User Defined Function to handle aunique device, and the command Time which insertsthe VAXELN system time into the returned databuffer.
An example of the commands contained in a typicalCOMMAND LIST is shown in Figure 4. Together with thecomments to the right of the commands, the list isself-explanatory.
TDAS Performance
The maximum rate at which a COMMAND LIST, whichcontains only a single CAMAC write, can be sent fromthe host to a TDAS and a reply received at the host is70 COMMAND LISTs per second (see Figure 5). Alsolisted are time measurements for reading from a CAMACmodule using three available methods: consecutivesingle CAMAC read commands, a single command whichincludes a number indicating how many reads are to bemade from a single CAMAC address, and optimized readsfrom CAMAC which can be programmed into a User DefinedFunction. With consecutive commands it takes50 microsecs per read. With a single command it takes15 microsecs per read. The maximum rate at which a
CAMAC read can be executed from Pascal is4.7 microsec. This type of CAMAC read could be usedin the procedures invoked by the User Defined Function
Figure 4. COMMAND_LIST Structure.
MKIMnENC I-r
819
command.
The interrupt response time, which is the timefran a CAMAC interrupt until the first instruction inthe interrupt routine is executed, is in 98% of theinterrupts between 23 and 26 microsecs but can extendout to 75 microsecs. The time fran when an interruptoccurs until a VAXELN process waiting for theinterrupt can proceed, i.e., a Wait for Interruptcmnmand is canpleted, is in 98% of interrupts between265 and 275 microsecs but can extend out to2000 microsecs.
MINIMUM SEGMENT 58 USEC 0-> 16 DATA BYTES
The last box in Figure 5 lists the amount of timespent in the VAXELN Network Services procedure SENDfor various sizes of buffers sent to the host. Ifthis procedure were put in-line in the processexecuting the COMMAND LIST, the process would beblocked for the amounts of time shown. In the TDASsystem, a separate COMM process is used for sendingdata to the host. This process runs concurrently withprocess executing the COMMAND LIST, thus reducing theblocking effect of the VAXELN SEND procedure.
Shown in Figure 6 is a observation of theactivity on Ethernet for large messages sent fran theTDAS to the host. Over a circuit or logical link,VAXELN Network Services autanatically breaks up largemessages into multiple segments which are sent overEthernet one at a time. Each segment contains amaximum of 1460 bytes of user data and takes 1.4millisec to transmit over the Ethernet. Anintersegment gap of .3 millisec occurs between eachsegment. In addition, after every eight segments sentfrom a TDAS to the host, a 9 millisec acknowledge
98% 265->275 USECSIGNAL PROCESS FROM INTR. 2% 275->2000 USEC
TIME SPENT IN NETWORK SERVICES BUFFER SIZEMILLISEC BYTES
4.5 2K7.0 4K12.5 8K18.5 16K23.5 64K
Figure 5. TDAS Performance.
cycle occurs before VAXELN starts sending the nextsegment. Therefore, it takes about 22 millisec totransmit 8 segments of user data for an effective bittransmission rate of 4.3 megabits per second. SinceEthernet has a 10 megabit per second bandwidth, themaximun bandwidth utilization which can occur whilesending large messages fran the TDAS to the host is43%. In Data Transfers over Ethernet at the data-linklevel bandwidth utilizations as high as 70% have beenreported.
In actual tests using the NPBDAS data acquisitionsoftware, we have been able to send 64 Kbyte messagesevery 165 millisec (every 5 beam spills) fran a TDASto a host for a rate of 384 Kbytes per second. Thetests did not try to send data at the maximum ratepossible.
System Support Level Software
A library of functions is provided by the SystemSupport Level to interface the User levels to theInstrument Support Level for data acquisition and toimplement a buffer management scheme at the host fordata buffers returned fran the TDASs. There are onlyseven functions supplied in the System Support Levelsoftware. An outline of an application progran (seeFigure 7) using all the functions, will be used toexplain the System Support Level software. Eachnumbered line in the program outline is explainedbelow.
1. Parameter definitions used in System Supportfunction calls.
STATUS=GETW UNIT (UNITID, TIMEOUT, EVENTFLAG, BUFFER, IOSB)
ENDDO
STATUS= FREE_UNIT (UNITID)
STATUS= FREE_LINK
END
Figure 7. Program Outline Using System Support LevelSoftware.
2. The CREATE LINK function creates a buffer pool ofBUFNUM buf'ers each with a size of BUFSIZE andlocks the buffer pool into memory. These buffersare managed by the System Support software andare used to store data buffers received from a
TDAS. In addition two logical links, a DATA LINK(EZFORTO DATA) for receiving data buffers from a
TDAS and a COMMAND LINK (EZPORTO CMD) fortransmitting COMMAND LISTS to and receivingreplies from a TDAS, are established. The linksare established with node RNPB2, which in theNPBDAS is the TDAS associated with Test Stand A.An identifier LINKID is returned to the callerand will be used in subsequent calls to refer to
the two logical links just established.
3. The CREATE UNIT function creates a System SupportLevel maintained queue for data buffers returnedfrom a TDAS. The identifier of the queue is
returned to the caller as UNITID. The UNITIDidentifier will be used later in the header of a
COMMAND LIST. Any data buffers returned from a
TDAS as a result of executing the COMMAND LIST
will be queued to UNITID by the System SupportLevel.
4. A COMMAND LIST (pictorially shown in Figure 6)which reads 512 words from a CAMAC module at
crate 0, station 1, subaddress 0 with a functioncode of 0.
5. The EXECUTE_LIST function transmits the COMMAND
LIST over the EZPORTO CMD link to node RNPB2 as
identified by the LINKID parameter. Since the
COMMAND LIST header TYPE is EXECUTE_NOW, the TDAS
at RNPB2 will reply to the EXECUTE_LIST function
call prior to executing the commands in theCOMMAND LIST. Thus, control will return to thecaller while the COMMAND LIST is being executedat the TDAS, and both systems will be operatingin parallel. If the header TYPE hadbeenEXECUTE WAIT, control would only be returned tothe caller after the TDAS had executed theCOMMAND LIST. The ELNPID parameter is returned tothe caller and identifies the process created atthe TDAS to execute the list. Any data buffersreturned from the TDAS at RNPB2 node as a resultof executing the commands in this COMMAND LISTwill be queued to the UNITID queue identified inthe COMMAND LIST header word, UNIT. The headerREPEAT parameter is equal to 1000 so the listwill be executed 1000 times returning to the host1000 buffers each containing 512 words. Had theheader REPEAT parameter been 0, the commands inthe COMMAND LIST would be repeatedly executeduntil explicitly stopped by another EXECUTE LISTfunction call with a header TYPE of EXECUTE KILLand a single command containing the ELNPIDidentifier.
6. The GETW UNIT function retrieves a data bufferfrom the queue identified by UNITID. If a bufferis queued to UNITID or no buffer is receivedwithin a TIMEOUT period, control will return tothe caller. The IOSB four word block contains thecompletion status and, if a buffer was received,the length of the buffer. A pointer to thebuffer is returned in BUFFER. Another form of thecall, GET_UNIT, is available. In the alternateform, control is immediately returned to thecaller and an event flag, EVENTFLAG, is setwhenever a TIMEOUT occurs or a buffer is queuedto UNITID. The GETW_UNIT function is called 1000times from within a do loop to retrieve all 1000buffers sent over link EZPORTO_DATA from nodeRNPB2.
7. The FREE UNIT function call returns to the SystemSupport Level buffer pool, all bufferspreviously retrieved from the UNITID queue bycalls to GETW UNIT. In this example, 1000buffers will be returned to the buffer pool.
8. The FREE LINK function disconnects all logicallinks in an orderly manner.
Conclusion
System implementation from conception toinstallation at the NPBTS took approximately 10 monthswith a total software effort of 10 man months and a
hardware effort of 2 man months. The use of high levellanguages throughout the software implementation, theuse of an excellent VAXELN debugger to debug softwarerunning at the TDASs, and the use of VMS Decnet andVAXELN Network Services were key factors in bringingthe system into operation quickly. Much of the UserSupport Level software was easily ported from an
existing RSX11-M system since it was written inFortran.
In the immediate future we plan to addsignificantly to the commands available in the COMMANDLIST as new experiments come on-line and to addhistogramnming functionality at the Instrunent SupportLevel. At the User Support Level our efforts will befocused on making the system more user friendly.
821
References
[1] Neutral Particle Beam Test Stand, Users Handbook,1986.
[2] NPBDAS--Neutral Particle Beam Data AcquisitionSystem, Users Handbook, 1987.
[3] RS/1 is a trademark of Bolt Beranek and Newman,Inc.
[4] "Performance Characteristics of Ethernet," ACMSignmetrics, 1985. Timothy A. Gonsalves,Stanford University, Computing System Lab., Dept.of E.E.
