areaDetector: A new module for EPICS area detector

supportMark Rivers

GeoSoilEnviroCARS, Advanced Photon Source

University of Chicago

• Motivation

• Quick overview of architecture and implementation status

• Some details of implementation from bottom up

• Demo of simulated and real detectors

Outline

• ccdApp synApps module– MAR-CCD– Roper via WinView– ccdApp limitations

• No images exported to EPICS• Relies on vendor GUI• SNL code means no real device support, EPICS is in charge of

everything

• PilatusROI for Pilatus 100K– Adds features like image export to EPICS, region-of-interest

calculations– Unique to that detector, users want same features on MAR-CCD,

etc.– Hard to add additional features – other types of ROIs, for example

Current status of area detector support in EPICS

• Brian Tieman’s (APS) CCD Image Server – Supports dozens of cameras on Windows platforms

– Limited to Windows

– Using PCAS means no EPICS database

– Not easily extended

• Others (Flea camera driver, etc.)

• Really need a uniform approach – platform-independent

– consistent interface

– can support any detector

– easily extensible

• I’ve spent the last few months working on developing such a new model

Current status of area detector support in EPICS

New Approach - Goals

• High performance– Handle detectors ranging from >500 frames/second to <1 frame/second

• Basic parameters for all detectors– E.g. exposure time, start acquisition, etc.

– Allows generic clients to be used for many applications

• Easy to implement new detector– Single device-driver file to write. EPICS independent. Inherits from

ADDriverBase C++ class.

• Easy to implement detector-specific features– Driver understands additional parameters beyond those in the basic set

• EPICS-independent at lower layers.

• Middle-level plug-ins to add capability like regions-of-interest calculation, file saving, etc.

– Below the EPICS layer for highest performance

ADBase.template

Standard asyn device support(device-independent)

NDPluginROI

Vendor API

Driver

Channel Access Clients (medm, IDL, SPEC, etc.)

EPICS Area Detector Architecture

xxxDriver.template

Hardware

C++ Base classes (NDArray, asynPortDriver, asynNDArrayDriver, ADDriver, NDPluginDriver)

High speedGUI

Layer 5 Standard EPICS records

Layer 4EPICS device support

Layer 3 Plug-ins

Layer 1 Hardware API

Layer 2 Device drivers

Layer 6EPICS CA clients

NDPluginStdArrays

NDPluginFile

NDPluginXXX.template

Channel access

Record/device support

asynInt32, Float64, Octet

asynGenericPointer (NDArray)

asynXXXArray

C library calls

NDPluginBase.template

New Approach - Implementation

• Layers 1-3 are independent of EPICS except for libCom and asyn

• libCom– OS independent libraries for threads, queues, mutex,

events, etc.

• asyn– OS-independent libraries for thread-safe interfaces, plug-

in servers.

• Layer 4 is all written, as are records at layer 5. Only simple databases with virtually no logic are required at layer 5.

Status: Layer 2 (device drivers)• ADDriver

– Base C++ class from which detector drivers derive. Takes care of all details of asyn interfaces

• drvSimDetector– Simulation detector, produces calculated images at a very high rate.

Implements nearly all basic parameters.• drvProsilica

– Prosilica GigE and Firewire cameras. – High resolution, high speed. 1360x1024 at 30 frames/second = 40MB/second.

• drvPilatus– Pilatus 100K pixel-array detector. 487x195 at over 200 frames/second.

• drvADSC– ADSC CCD detector control, done by Lewis Muir

• To be done:– Drivers for Roper, MAR-CCD, MAR-345.– Perkin-Elmer amorphous silicon detector (Brian Tieman has working version)– Drivers for other cameras that Brian Tieman’s ImageServer supports?– Drivers for Flea Firewire cameras (Stephen Mudie?)

Layer 2 Device Driver Screen

Status: Layer 3 (plug-ins)• Common features from NDPluginDriver (C++ base class)

– Receive NDArray data over callbacks – Plug-ins can execute in their own threads (non-blocking) or in

callback thread (blocking)• If non-blocking then NDArray data is queued

– Can drop images if queue is full• If executing in callback thread, no queuing, but slows device driver

– Allows • Enabling/disabling• Throttling rate• Changing data source for NDArray callbacks

• NDPlugInStdArrays– Receives arrays (images) from device drivers, converts to standard

asynXXXArray interface (asynInt8Array, etc). • Those interfaces are supported by EPICS waveform records, which can

be displayed by Channel Access clients. – Converts data types. (8-bit camera to DOUBLE waveform, etc.)

Status: Layer 3 (plug-ins)• NDPluginFile

– Receives arrays from device drivers, saves to disk file in netCDF file format. (Portable, self-describing binary format)

– File saving can also be implemented by device driver via vendor API using the same database

• NDPluginROI– Receives arrays from device drivers. Selects region-of-interest. Computes

statistics. – Can be used as NDArray data source for other plugins. It is thus an asyn server

as well as client.

• To be done:– Add other common file formats to drvADFile

• TIFF, HDF– Other ROI drivers: more sophisticated regions, other statistics– Other plugins: e.g. Stephan Mudie’s cursor display.– High-speed GUI:

• Possible replacement for Brian Tieman’s program. Independent of EPICS, gets images directly from device driver callbacks.

• Brian already has a prototype working, written in Java

Layer 3 (plug-ins) Screens

Status: Layer 4 (asyn device support)• Added 3 new array interfaces.

– Previously there were only asynInt32Array, asynFloat64Array – Added asynInt8Array, asynInt16Array, asynFloat32Array for

efficiency– No need to convert datatypes in NDPluginStdArrays if waveform

datatype matches driver datatype. Minimize bandwidth and memory requirements

• Added new interface for (void *) pointers, asynGenericPointer.

• Added new convenience interface, asynStandardInterfaces. – Takes care of all bookkeeping of registering interfaces and interrupts

for drivers using the standard asyn interfaces (i.e. those defined in the asyn module).

• These are now part of asyn R4-10• To be done: ??

Status: Layer 5 (EPICS databases)• Common features:

– Virtually no logic (no links, calculations). Everything is done in drivers.– Almost all readbacks are done with SCAN=I/O Intr. Drivers must callback whenever anything

changes.• ADBase.template

– Database for parameters that all drivers understand• NDPluginBase.template

– Database for record common to all plugins• NDStdArrays.template.

– Database for NDPluginStdArrays plug-in• NDFile.template.

– Database for NDPluginFile plug-in• NDPluginROI.template, NDPluginROIN.template

– Databases for NDPluginROI plugin• simDetector.template.

– Database for features specific to simulation driver• prosilica.template.

– Database for features specific to Prosilica camera driver• Pilatus.template

– Database for features specific to Pilatus detector• To be done: New databases for features specific to new drivers

Status: Layer 6 (CA clients)

• medm screens for all databases

• IDL image display client (epics_ad_display)– Displays images with several IDL tools

• To be done: – Data collection clients (e.g. SPEC)

– Other image display clients

– Improved medm displays

– Improved IDL client using IDL object graphics (e.g. what Steve Mudie has done). Resizeable, etc.

Details: NDArray.h

• Defines NDArray and NDArrayPool classes. • NDArray

– N-Dimensional array. • Everything is done in N-dimensions (up to 10), rather than 2. This is

needed even for 2-D detectors to support color. – This is what plug-ins callbacks receive from device drivers. First 3

fields are used for freelist management. All fields are read only, they can only be modified by NDArray and NDArrayBuff functions.

• NDArrayPool– Allocates NDArray objects from a freelist– Drivers allocate an NDArray object and pass it to callbacks– Clients (e.g. plug-ins) increase the reference count with reserve()

when the buffer is in use, decrease it with release() when they are done.

– When reference count=0 buffer is returned to free-list.– Eliminates need to copy data when sending it to callbacks.

Details: NDArray.h

/* Enumeration of array data types */typedef enum{ NDInt8, NDUInt8, NDInt16, NDUInt16, NDInt32, NDUInt32, NDFloat32, NDFloat64} NDDataType_t;

typedef struct NDDimension { int size; int offset; int binning; int reverse;} NDDimension_t;

Details: NDArray

class NDArray {public: /* The first 3 fields are used for the freelist */ ELLNODE node; int referenceCount; /* The NDArrayPool object that created this array */ void *owner;

int uniqueId; double timeStamp; int ndims; NDDimension_t dims[ND_ARRAY_MAX_DIMS]; NDDataType_t dataType; int dataSize; void *pData;

Details: ADStdDriverParams.h –ADImageMode, ADTriggerMode

Other enums. These can be implemented by device driver, or it can “replace” this with the list of functions it actually supports. The EPICS enum strings are overwritten in xxxDriver.template

typedef enum{ ADImageSingle, ADImageMultiple, ADImageContinuous} ADImageMode_t;

typedef enum{ ADTriggerInternal, ADTriggerExternal} ADTriggerMode_t;

Details: ADStdDriverParams.h –ADStdDriverParam_t/** This is an enumeration of parameters that affect the behaviour of the detector. These are the values that asyn will place in pasynUser->reason when the standard asyn interface methods are called.*/typedef enum{ /* Name asyn interface access Description */ ADManufacturer_RBV, /* (asynOctet, r/o) Detector manufacturer name */ ADModel_RBV, /* (asynOctet, r/o) Detector model name */

ADGain, /* (asynFloat64, r/w) Gain. */ /* Parameters that control the detector binning */ ADBinX, /* (asynInt32, r/w) Binning in the X direction */ ADBinY, /* (asynInt32. r/w) Binning in the Y direction */ /* Parameters the control the region of the detector to be read out. * ADMinX, ADMinY, ADSizeX, and ADSizeY are in unbinned pixel units */ ADMinX, /* (asynInt32, r/w) First pixel in the X direction. */ ADMinY, /* (asynInt32, r/w) First pixel in the Y direction. */ ADSizeX, /* (asynInt32, r/w) Size of the region to read in the X direction. */ ADSizeY, /* (asynInt32, r/w) Size of the region to read in the Y direction. */…

Details: Base class asynPortDriver• asynPortDriver class

– Implements all details of asyn interfaces and callbacks– Greatly simplifies drivers and plugins which derive from it. – Drivers and plugins typically only need to implement writeInt32, writeOctet, etc.– With a small amount of work this can be made completely general, and then should be moved to asyn (R4-11?)

class asynPortDriver {public: asynPortDriver(const char *portName, int maxAddr, int paramTableSize, int interfaceMask, int

interruptMask); virtual asynStatus getAddress(asynUser *pasynUser, const char *functionName, int *address); virtual asynStatus findParam(asynParamString_t *paramTable, int numParams, const char

*paramName, int *param); virtual asynStatus readInt32(asynUser *pasynUser, epicsInt32 *value); virtual asynStatus writeInt32(asynUser *pasynUser, epicsInt32 value); virtual asynStatus getBounds(asynUser *pasynUser, epicsInt32 *low, epicsInt32 *high); virtual asynStatus readFloat64(asynUser *pasynUser, epicsFloat64 *value); virtual asynStatus writeFloat64(asynUser *pasynUser, epicsFloat64 value); virtual asynStatus readOctet(asynUser *pasynUser, char *value, size_t maxChars, size_t *nActual, int *eomReason); virtual asynStatus writeOctet(asynUser *pasynUser, const char *value, size_t maxChars, size_t *nActual); virtual asynStatus readInt8Array(asynUser *pasynUser, epicsInt8 *value, size_t nElements, size_t *nIn); virtual asynStatus writeInt8Array(asynUser *pasynUser, epicsInt8 *value, size_t nElements); virtual asynStatus doCallbacksInt8Array(epicsInt8 *value, size_t nElements, int reason, int addr); …

Details: Base class asynPortDriver - Example from prosilica.cppasynStatus prosilica::writeFloat64(asynUser *pasynUser, epicsFloat64 value){ int function = pasynUser->reason; int addr=0; int status = asynSuccess; tPvUint32 intVal; tPvFloat32 fltVal;

status |= setDoubleParam(addr, function, value); switch (function) { case ADAcquireTime: /* Prosilica uses integer microseconds */ intVal = (tPvUint32) (value * 1e6); status |= PvAttrUint32Set(this->PvHandle, "ExposureValue", intVal); break; case ADAcquirePeriod: /* Prosilica uses a frame rate in Hz */ if (value == 0.) value = .01; fltVal = (tPvFloat32) (1. / value); status |= PvAttrFloat32Set(this->PvHandle, "FrameRate", fltVal); break; case ADGain: /* Prosilica uses an integer value */ intVal = (tPvUint32) (value); status |= PvAttrUint32Set(this->PvHandle, "GainValue", intVal); break; default: break; }

Details: Simulation driver

Generates calculated images using the following algorithm• On reset: pixel[i, j] = (i * gainX + j* gainY)• On each subsequent “exposure” the intensity in each pixel is

increased by gain * exposureTime * 1000.• The images are generated by a thread that waits for an

acquisition start event.• After each acquisition it sleeps for the maximum of the

exposureTime or the exposurePeriod minus the compute time.

• Calls back any registered clients on standard asyn interfaces when any parameters change, and any registered clients on the asynHandle interface when there is a new image (NDArray).

Details: Prosilica driver• GigE and Firewire CCD cameras.• Uses Windows DLL from Prosilica that implements their PvAPI library for

controlling the camera• Their driver has a callback thread that calls a user-written function for each new

image. Uses a frame queue.– The device driver thus does not need to create a new epicsThread.

• Implemented a number of device-specific parameters to read GigE statistics. – Could also implement more to control external triggering, auto-exposure, etc.– Done with polling, not I/O Intr because they change constantly and are for information

only.

• Has a vendor-supplied function for saving images as TIFF files. – This is implemented in the driver. – Can also use the NDFile plug-in to save images in other formats.

• Driver and EPICS database override the ADTriggerMode and ADFileFormat enums to implement the features that this camera actually supports.

Details: NDPluginBase class• Receives NDArray callbacks from device drivers.• Handles connection to an array port driver, enable/disable, minimum callback time (throttling)• Derived class only need to implement processCallbacks

– That will be called either in driver callback thread or plugin thread depending on state of callbacksBlock flag

• Can change NDPlugBaseArrayPort at run-time, switching from one device to another– NDPluginRoi plug-in provides sub-images that other plugins can connect to

• Implements following parameters:typedef enum { NDPluginBaseArrayPort /* (asynOctet, r/w) NDArray interface port */ = ADFirstDriverParam, NDPluginBaseArrayAddr, /* (asynInt32, r/w) The address on the port */ NDPluginBaseArrayCounter, /* (asynInt32, r/w) Number of arrays processed */ NDPluginBaseDroppedArrays, /* (asynInt32, r/w) Number of dropped arrays */ NDPluginBaseEnableCallbacks, /* (asynInt32, r/w) Enable callbacks from driver (1=Yes,

0=No) */ NDPluginBaseBlockingCallbacks, /* (asynInt32, r/w) Callbacks block (1=Yes, 0=No) */ NDPluginBaseMinCallbackTime, /* (asynFloat64, r/w) Minimum time between file writes */ NDPluginBaseUniqueId_RBV, /* (asynInt32, r/o) Unique ID number of array */ NDPluginBaseTimeStamp_RBV, /* (asynFloat64, r/o) Time stamp of array */ NDPluginBaseDataType_RBV, /* (asynInt32, r/o) Data type of array */ NDPluginBaseNDimensions_RBV, /* (asynInt32, r/o) Number of dimensions in array */ NDPluginBaseDimensions, /* (asynInt32Array, r/o) Array dimensions */} NDPluginBaseParam_t;

Details: NDStdArrays plug-in• Implements standard asynXXXArray interfaces (asynInt8, asynInt16, etc.)• Designed primarily to convert images from NDArray stucture to asynXXXArray interface for

EPICS waveform records.• Can change port at run-time, switching from one device to another

– NDPluginROI provides sub-images that NDStdArrays can connect to

• Only 270 lines of codeclass NDPluginStdArrays : public NDPluginBase {public: NDPluginStdArrays(const char *portName, int queueSize, int blockingCallbacks, const char *NDArrayPort, int NDArrayAddr, size_t maxMemory);

/* These methods override the virtual methods in the base class */ void processCallbacks(NDArray *pArray); virtual asynStatus readInt8Array(asynUser *pasynUser, epicsInt8 *value, size_t nElements, size_t *nIn); virtual asynStatus readInt16Array(asynUser *pasynUser, epicsInt16 *value, size_t nElements, size_t *nIn); virtual asynStatus readInt32Array(asynUser *pasynUser, epicsInt32 *value, size_t nElements, size_t *nIn); virtual asynStatus readFloat32Array(asynUser *pasynUser, epicsFloat32 *value, size_t nElements, size_t *nIn); virtual asynStatus readFloat64Array(asynUser *pasynUser, epicsFloat64 *value, size_t nElements, size_t *nIn); asynStatus drvUserCreate(asynUser *pasynUser, const char *drvInfo, const char **pptypeName, size_t *psize);

Details: NDPluginFile• Saves data to disk files• Currently supports netCDF file format. Portable, self-describing binary format.• Can change NDArrayPort at run-time• Implements following parameters:typedef enum { NDPluginFileModeSingle, NDPluginFileModeCapture, NDPluginFileModeStream} NDPluginFileMode_t;

typedef enum{ NDPluginFileWriteMode /* (asynInt32, r/w) File saving mode */ = NDPluginBaseLastParam, NDPluginFileNumCapture, /* (asynInt32, r/w) # of arrays to capture */ NDPluginFileNumCaptured_RBV, /* (asynInt32, r/o) # of arrays captured */ NDPluginFileCapture, /* (asynInt32, r/w) Start/stop capturing */ NDPluginFileLastParam} NDPluginFileParam_t;

Details: NDPluginROI• Selects subset of image data• Computes statistics• Computes histogram• Implements following parameters:typedef enum { NDPluginROIName /* (asynOctet, r/w) Name of this ROI */ = NDPluginROIFirstROINParam, NDPluginROIUse, /* (asynInt32, r/w) Use this ROI? */ NDPluginROIComputeStatistics, /* (asynInt32, r/w) Compute statistics for ROI? */ NDPluginROIComputeHistogram, /* (asynInt32, r/w) Compute histogram for ROI? */ . . . /* ROI definition */ NDPluginROIDim0Min, /* (asynInt32, r/w) Starting element of ROI in each

dimension */ . . ./* ROI statistics */ NDPluginROIBgdWidth, /* (asynFloat64, r/w) Width of background region when

computing net */ NDPluginROIMinValue_RBV, /* (asynFloat64, r/o) Minimum counts in any element */ NDPluginROIMaxValue_RBV, /* (asynFloat64, r/o) Maximum counts in any element */ NDPluginROIMeanValue_RBV, /* (asynFloat64, r/o) Mean counts of all elements */ NDPluginROITotal_RBV, /* (asynFloat64, r/o) Sum of all elements */ NDPluginROINet_RBV, /* (asynFloat64, r/o) Sum of all elements minus background */ . . .

Details: ADBase.templaterecord(ao, "$(P)$(R)AcquireTime"){ field(PINI, "1") field(DTYP, "asynFloat64") field(OUT, "@asyn($(PORT),$(ADDR),$(TIMEOUT))ACQ_TIME") field(PREC, "3")}record(ai, "$(P)$(R)AcquireTime_RBV"){ field(PINI, "1") field(DTYP, "asynFloat64") field(INP, "@asyn($(PORT),$(ADDR),$(TIMEOUT))ACQ_TIME") field(PREC, "3") field(SCAN, "I/O Intr")}

All input records are SCAN=I/O Intr• Provides definition of enums for DataType, FileFormat, etc.

– These can be overwritten by the device-dependent database that gets loaded later. Same record is redefined, as long as record type is the same, any fields can be redefined. prosilica.template redefines enums for FileFormat and TriggerMode

Details: IDL epics_ad_display

• Simple IDL GUI, 400 lines of code

• Talks to NDStdArrays database.

• Uses ImageSizeX_RBV, ImageSizeY_RBV to know how much data to request from the waveform record and to dynamically resize the window

• 3 display modes:– tv (Fastest)

– image_display (Slower, row/column profiles, zoom, mouse values)

– iImage (IDL iTools, very powerful, needs work for real-time)

• Seems like an architecture that works well• Lots more work to do, especially on drivers (MAR, Roper, etc.)• Can get code from CARS

– http://cars.uchicago.edu/software/epics/areaDetector

• Can also get code on SourceForge– http://epics.svn.sourceforge.net/viewvc/epics/applications/trunk/areaDetector/

• I hope others will write drivers, plug-ins and CA applications• Base classes, asynPortDriver, asynNDArrayDriver, asynPluginDriver

actually are generic, nothing “areaDetector” specific about them.• They can be used to implement any N-dimension detector, e.g. the

XIA xMAP (16 detectors x 2048 channels x 512 points in a scan line)• Thanks for your attention!!!

Wrap-up