IEEE Transactions on Nuclear Science, Vol. NS-26, No. 4, August 1979 A DISTRIBUTED PROCESSING DATA ACQUISITION SYSTEM AT BROOKHAVEN NATIONAL LABORATORY* Roger C. Lee and Robert A. Scheetz** ABSTRACT DATA ACQUISITION The Tandem Van de Graaff at BNL has been using a computerized data acquisition system for more than eight years. A brief history of its philosophy and performance will be discussed. Also to be presented is the new data collection system which is designed around a high-speed bus with multiple processors. This new system is capable of handling event rates of over 200 kHz for add-one to memory (PHA) type spectra totalling 512k channels. List-mode, special sorting involving gating, or arithmetic operations of the data are supported in addition to one or two parameter PHA types. The system allows eight con- figurations of eight devices to be defined by the user. The devices are typically analog-to-digital converters (ADCs), multi-channel scalers, or routing devices. However any device with digital informa- tion could be interfaced. INTRODUCTION The Brookhaven National Laboratory Tandem Facility supports research in heavy-ion and light- ion physics. Some of the many areas of interest are nuclear structure, elastic and inelastic scattering, fusion evaporation and fission reaction, beam-foil spectroscopy, and molecular structure. The Facility itself consists of two in-line Emperor accelerators which provide a wide variety of energetic ion-beams of precise and easily variable energy. The research is performed on an approximately equal basis between local staff members and outside users. Presently there are twenty in-house scientific staff members and the number of outside personnel is perhaps even larger. In 1967 when the Tandem was being installed, the decision was made to acquire a general-purpose computer (of considerable capability) for data acquisition and data analysis.1 Thus the system started with an SDS (later Xerox, now Honeywell) Sigma 7. Today that same computer is still being used although it has been expanded considerably. A block diagram of the computer system is shown in Figure 1. The idea was to have a system capable of supporting data acquisition and several simultaneous users doing data analysis and reduction. Data ac- quisition was accomplished by designing and building a data collection interface and graphics display system for the Sigma. Data analysis and reduction is done using Xerox's sophisticated time-sharing, real-time, batch operating system. There can be as many as sixteen time-sharing and eight batch users using the system while real-time data collection is taking place. In 1979 there will be two data col- lection interfaces running concurrently. ORIGINAL SYSTEM When the original data collection interface was designed, the philosophy was to provide a versatile, flexible, and expandable system which was easy for the users to setup.2 Therefore it is not just a simple interface to ADCs but instead is capable of being interfaced to any device with digital infor- mation. The user can define from one to eight com- binations (event lines) of eight logical devices. Devices can be combined on several event lines to be utilized in different ways with different resolu- tions, e.g. stored on list-mode tape, sorted into a coincidence spectrum, or used as a single parameter event. Each logical device is coupled to one or more physical devices by a simple interface. Thus an event line can produce an unlimited number of data bits. Typically the data bits are produced by ADCs, scalers, routing devices, or binary pattern generators. A second major function of the data collection interface is to process the data before sending it to the computer. When an event line is triggered, each device connected to that event line is read, the appropriate data are packed together to form a convenient descriptor, e.g. channel number for PHA types, and the data is transmitted to the computer. Each event line has its own input buffer. Thus the software receives separate lists of channel numbers for each spectrum. This presorting by event line and packing the data removes some of the data col- lection burden from the computer. Currently average rates of 30,000 per second can be handled without using appreciable central processor time. The size of the spectra collected on-line is limited to 80,000 channels. NEW SYSTEM The new data collection interface appears ex- ternally identical to the original interface but internally the design is radically different. A diagram of the new system is shown in Figure 2. The new system is designed around a high-speed-sychron- ous bus with specialized processors for handling different functions. There can be twelve processors or devices attached to the bus. The bus can trans- fer data between devices at a rate of 24 megabytes per second. Currently the devices consist of the data collection interface, 512 kilobytes of spectra memory (expandable to 1 megabyte), two interfaces to the Sigma 7, and 16 kilobytes of scratch-pad memory (expandable to 256 kilobytes). Soon to be added are processors for handling list-mode tapes, sorting involving gating or arithmetic calculations, and graphics displays. 0018-9499/79/0800-4405$00.75c1979 IEEE Work performed under the auspices of the U. S. Department of Energy. **Brookhaven National Laboratory, Upton, NY 11973 4405
Transcript
IEEE Transactions on Nuclear Science, Vol. NS-26, No. 4, August 1979
A DISTRIBUTED PROCESSING DATA ACQUISITION SYSTEMAT BROOKHAVEN NATIONAL LABORATORY*
Roger C. Lee and Robert A. Scheetz**
ABSTRACT DATA ACQUISITION
The Tandem Van de Graaff at BNL has been usinga computerized data acquisition system for more thaneight years. A brief history of its philosophy andperformance will be discussed. Also to be presentedis the new data collection system which is designedaround a high-speed bus with multiple processors.This new system is capable of handling event ratesof over 200 kHz for add-one to memory (PHA) typespectra totalling 512k channels. List-mode, specialsorting involving gating, or arithmetic operationsof the data are supported in addition to one or twoparameter PHA types. The system allows eight con-figurations of eight devices to be defined by theuser. The devices are typically analog-to-digitalconverters (ADCs), multi-channel scalers, or routingdevices. However any device with digital informa-tion could be interfaced.
INTRODUCTION
The Brookhaven National Laboratory TandemFacility supports research in heavy-ion and light-ion physics. Some of the many areas of interest arenuclear structure, elastic and inelastic scattering,fusion evaporation and fission reaction, beam-foilspectroscopy, and molecular structure. The Facilityitself consists of two in-line Emperor acceleratorswhich provide a wide variety of energetic ion-beamsof precise and easily variable energy. The researchis performed on an approximately equal basis betweenlocal staff members and outside users. Presentlythere are twenty in-house scientific staff membersand the number of outside personnel is perhaps evenlarger.
In 1967 when the Tandem was being installed,the decision was made to acquire a general-purposecomputer (of considerable capability) for dataacquisition and data analysis.1 Thus the systemstarted with an SDS (later Xerox, now Honeywell)Sigma 7. Today that same computer is still beingused although it has been expanded considerably. Ablock diagram of the computer system is shown inFigure 1. The idea was to have a system capable ofsupporting data acquisition and several simultaneoususers doing data analysis and reduction. Data ac-quisition was accomplished by designing and buildinga data collection interface and graphics displaysystem for the Sigma. Data analysis and reductionis done using Xerox's sophisticated time-sharing,real-time, batch operating system. There can be as
many as sixteen time-sharing and eight batch usersusing the system while real-time data collection istaking place. In 1979 there will be two data col-lection interfaces running concurrently.
ORIGINAL SYSTEM
When the original data collection interface wasdesigned, the philosophy was to provide a versatile,flexible, and expandable system which was easy forthe users to setup.2 Therefore it is not just asimple interface to ADCs but instead is capable ofbeing interfaced to any device with digital infor-mation. The user can define from one to eight com-binations (event lines) of eight logical devices.Devices can be combined on several event lines to beutilized in different ways with different resolu-tions, e.g. stored on list-mode tape, sorted into acoincidence spectrum, or used as a single parameterevent. Each logical device is coupled to one ormore physical devices by a simple interface. Thusan event line can produce an unlimited number ofdata bits. Typically the data bits are produced byADCs, scalers, routing devices, or binary patterngenerators.
A second major function of the data collectioninterface is to process the data before sending itto the computer. When an event line is triggered,each device connected to that event line is read,the appropriate data are packed together to form aconvenient descriptor, e.g. channel number for PHAtypes, and the data is transmitted to the computer.Each event line has its own input buffer. Thus thesoftware receives separate lists of channel numbersfor each spectrum. This presorting by event lineand packing the data removes some of the data col-lection burden from the computer. Currently averagerates of 30,000 per second can be handled withoutusing appreciable central processor time. The sizeof the spectra collected on-line is limited to80,000 channels.
NEW SYSTEM
The new data collection interface appears ex-ternally identical to the original interface butinternally the design is radically different. Adiagram of the new system is shown in Figure 2. Thenew system is designed around a high-speed-sychron-ous bus with specialized processors for handlingdifferent functions. There can be twelve processorsor devices attached to the bus. The bus can trans-fer data between devices at a rate of 24 megabytesper second. Currently the devices consist of thedata collection interface, 512 kilobytes of spectramemory (expandable to 1 megabyte), two interfaces tothe Sigma 7, and 16 kilobytes of scratch-pad memory
(expandable to 256 kilobytes). Soon to be added are
processors for handling list-mode tapes, sortinginvolving gating or arithmetic calculations, andgraphics displays.
0018-9499/79/0800-4405$00.75c1979 IEEE
Work performed under the auspices of theU. S. Department of Energy.
**Brookhaven National Laboratory, Upton, NY 11973
4405
The data collection interface retains the samefunctions as the original interface - sort the databy event lines and pack the data into descriptors.In this design, devices are scanned and the datapacked at a rate of one microsecond each. In addi-tion the new data collection interface has the addedcapability to decide which processor or system onthe bus will receive the data and to automaticallycount the number of events transferred.
The memory systems on the bus utilize semi-conductor memory with 400 ns cycle time. Eachmemory system contains the logic necessary to add-one to either a 16 or 32 bit word. The largermemory system is capable of addressing 512k, 16 bitwords and contains only sorted spectra. For a PHAtype event, the data collection interface suppliesthe memory address to which the memory does an add-one in less than one microsecond. In theory eventrates as high as 1 MHz could be handled but inpractice rates will not exceed 200 kHz since a realevent typically involves more than one device andevent line.
The two interfaces to the Sigma 7 provide linksfor transferring data and control information be-tween the data acquisition system and the computer.The high-speed link (3 megabytes/sec) is used toread the contents of the spectra memory every fiveminutes. The spectra are then stored on a fixed-head disc as permanent backup. Thus if there is asystem crash, power failure, or other catastrophe,only the data collected in the last five minuteswill be lost. The other link is used for controland for data which can not currently be handled bythe processors on the data acquisition bus. Eventu-ally, all data will be handled by the bus and onlycontrol and backup functions will remain in theSigma computer.
The processors to be added in the future willcomplete the task of removing all data collectionfrom the main computer. The tape processor willconsist of a tape drive and the logic necessary toformat and buffer the data for list-mode events.The display processor will take spectra directlyfrom the bus memory and send it to the graphicsdisplay unit, replacing the fixed-head display disc.Finally the sort processor will be capable of handl-ing special cases where gating on one or more
descriptors is necessary or arithmetic calculationsmust be done before the final spectrum is sorted anddisplayed.
DISPLAY SYSTEM
The display system in use at the TandemFacility was also designed to minimize central proc-essor time and yet provide very powerful capabili-ties. Refreshing the display is handled by a fixed-head display disc which sends the stored data to a
display decoder. The decoder provides for selectingfrom one to eight independent display groups. Eachdisplay group has independent local hardware con-
trols for intensity, counts full scale, and lightpen activation. Additional controls are providedfor rotating, tumbling, and changing overall gain or
Z amplitude. Special functions for grid X, grid Y,light pen tags, or overflows can change the charac-teristics of any dot on the display.
The display is cabable of displaying 49,000points at a refresh rate of 15 times per second.These points may be diplayed anywhere in an apparentthree-dimensional cube of 12,288 addressable loca-tions on each axis. Thus it is possible to have aflicker free, simultaneous display of two 128 by 128channel spectra and several singles spectra.
DATA ANALYSIS
The general philosophy of data collection atthe Tandem Facility has been to provide as muchanalysis on-line as possible. Also the system mustbe convenient for the user to setup and acquire hisdata. To this end, we have developed the softwareand hardware tools necessry to do the many differenttypes of experiments encountered and to allow forfast change from one experiment to another (typicalruns last only a few days). Techniques have beendeveloped for multi-channel scaling, multi-spectrumscaling, particle identification, time-of-flightmeasurements, angular distributions, polarimeterstudies, position-sensitive detectors, list-modetape recording, and gating on one or two ADCs whilerouting a third and/or fourth ADC. Many experimentscan be run on-line with total accumulation of allevents. Even those which require list-mode record-ing still retain considerable on-line analysis inorder to monitor and control the experiment.
One of the advantages of our system is in itsuse of a commerical operating system. Support ofsophisticated processors by a large computer systemmakes program development by scientific staff effi-cient. Excellent examples are the editor andFORTRAN compiler which, working together, allow inmost cases a program to be brought to operationalstatus in a few hours. As mentioned before therecan be sixteen users using the programs provided.These include languages such as FORTRAN, BASIC, APL,or assembly, programs for editing, sorting, merging,or copying files, programs for word processing,cross assembly for microcomputers, plotting and manyothers. In addition there are a whole range ofprograms available for data analysis and nucleardata calculations.
SUMMARY
From conception we have tried to provide asystem which could assimilate large quantities ofdata both rapidly and efficiently and also to pro-vide for concurrent display and analysis of data.To this end, we have distributed various functionsto specialized systems wherever possible. The datacollection interfaces and display system are primeexamples. The system has been able to grow to meetthe demands of ever increasingly complex experi-ments. In light of the positive response from a
large user base, the objectives have indeed beenmet.
4406
FIGURE 1 - COMPUTER SYSTEM AT THE TANDEM FACILITY
XEROX SIGMA 7
Word-oriented 32 bit memoryHardware memory mappingByte(8-bit), halfword(16-bit), word(32-bit), anddoubleword(664-bit) addressing and instructionsDirect addressing of entire memoryIndirect addressing with or without post indexingReal-time priority interrupt systemFour blocks of 16 general-purpose registersHardware floating point instructions
Two separate input/output processorsMultiplexor I/O processor with 32 channelsSelector I/O processor for data rates approaching4 megabytes per second
PERIPHERALS
- Three fixed-head, high-speed discs5.4 Mbytes each, 3 Mbyte/sec transfer rates,30 ms maximun access time
- Fixed-head display disc672 kbytes, 3 Mbytes/sec transfer rate,30 ms maximun access time
- Three moving-head disk packs49.8 Mbytes each, 246 kbytes/sec transfer,10 to 55 ms seek time
Packed/unpacked/BCD data- Line printer, 800-1000 LPM- Two Versatec printer/plotters, 500 LPM printing,plots 4,000 channel spectra per minute
- Card reader, 400 CPM
4407
II
A. -ISORT OTHER
PROCESSOR DEVICES
FIGURE 2 - DATA ACQUISITION SYSTEM
BUS SPECIFICATIONS
The high-speed-synchronous bus is used totransfer data at high rates between various devicesused for data collection. The bus is clocked at a 6MHz rate resulting in a 167 ns cycle time. The busis housed in a standard 19 inch chassis and is offixed physical length to avoid clock skew problems.Each device interface is a 15.4 inch by 11.75 inchboard with two 80 pin connectors. There can betwelve device interfaces in addition to the busmaster. Data transfers are done on 32 bidirectionaldata lines requiring only one bus cycle to initiateand complete the transfer. Up to 512k, 32 bit wordsor 1M, 16 bit words can be addressed in any device.There are four lines each for command code, masterdevice address, and slave device address.
The bus master consists of the logic necessaryto arbitrate priority and provide clock signals inaddition to a microprocessor. The microprocessor isused to handle fault conditions and for off-linediagnostics. The master remembers the last twelveactive bus cycles which may be used as a debuggingaid.
DEVICE SPECIFICATIONS
- Bus master6 MHz clock generator16 level priority levelsM6800 microprocessor for fault handling and
diagnostics- Data collection interface
16 level event bufferdevice selection by event lineautomatic counting of events by event lineadds offset to descriptor for each event
line to produce memory address- Memory systems
400 ns read/write cycle time, 800 ns add-one timeRead/write/add-one to 16 or 32 bitsSpectra memory expandable to 1 megabyteScratch pad memory expandable to 256 kilobytes
- SIOP interface to Sigma 7High-speed block word transfers (3 Mbytes/sec)
- MIOP interface to Sigma 7Medium-speed individual 16 bit transfers
REFERENCES
1P. C. Rogers, "The Data Acquisition and AnalysisComplex for the BNL Low Energy AcceleratorFacility", IEEE Transactions for Nuclear Science,February, 1969.
2G. E. Schwender and P. C. Rogers, "The DataCollection Interface Used by the BNL DataAcquisition and Analysis Complex", IEEETransactions for Nuclear Science, February, 1969.
