UFO – Ultra fast X-ray imaging 05K10VKE / 05K10CKB

Final Report

’UFO – Ultra fast X-ray imaging of scientific processes withon-line assessment and data-driven process control’

Institution: Karlsruhe Institute of Technology (KIT)

Principal investigator: Prof. Dr. Marc Weber

SummaryRecent progress in X-ray optics, detector technology, and the tremendous increase ofprocessing speed of commodity computational architectures gives rise to a paradigmshift in synchrotron X-ray imaging. The UFO project aims to enable a novel class ofexperiments combining intelligent detector systems, vast computational power, and so-phisticated algorithms. The on-line assessment of sample dynamics will make activeimage-based control possible, give rise to unprecedented image quality, and will pro-vide new insights into so far inaccessible scientific phenomena.

A demonstrator for high-speed tomography has been developed and extensively used.The system includes critical components like computation infrastructure, reconstructionalgorithms and detector system and proved that time-resolved tomography is feasible.Based on these results the final design of the UFO experimental station has beenrevised and several upgrades have been included to enable further imaging techniques.

A flexible and fully automated detector system for a set of up to three complementarycameras has been designed, constructed and commissioned. A new platform for smartscientific cameras, the UFO-DAQ framework, has been realized. It is a unique rapid-prototyping environment to turn scientific image sensors into intelligent smart cam-era systems. Central features are the modular sensor interface, an open embeddedprocessing framework and high-speed PCI Express links to the readout server. TheUFO-DAQ framework seamlessly integrates in the UFO parallel computing framework.

The UFO project demonstrated that high-end graphics processor units (GPUs) are anideal platform for a new generation of online monitoring systems for synchrotron appli-cations with high data rates. A powerful computing infrastructure based on GPUs andreal-time storage has been developed. Optimized reconstruction algorithms reach athroughput of 1 GB/s with a single GPU server. Generalized reconstruction algorithmsinclude also laminography with tilted rotation axis.

Highly optimized reconstruction and image processing algorithms are key for real-timemonitoring and efficient data analysis. In order to manage these algorithms the UFO

Das diesem Bericht zugrundeliegende Vorhaben wurde mit Mitteln des Bundesministeriums fur Bildungund Forschung gefordert. Die Verantwortung fur den Inhalt der Veroffentlichung liegt beim Autor.

1 / 24

UFO – Ultra fast X-ray imaging 05K10VKE / 05K10CKB

parallel computing framework has been developed. It supports the implementation ofefficient algorithms as well as the development of data processing workflows basedon these. It automatically selects the best code depending on the available comput-ing resources. With its clear modular structure the framework is ideally suited as anexchange platform for optimized algorithms for parallel computing architectures. Thecode published under open source license is well-recognized by the synchrotron com-munity.

The UFO project has been performed in close collaboration with three Russian part-ners. Various collaborating meetings have been organized and a number of scientistsvisited the partners partner institutions. The focus of the Russian contribution has beenthe smart camera platform and algorithm development. The results of the UFO projecthave been reported at several national and international workshops and conferences.The UFO project contributes with developments like the UFO-DAQ framework or itsGPU computing environment to other hard- and software projects in the synchrotroncommunity (e.g. Tango Control System, High Data Rate Processing and Analysis Initia-tive, Nexus data format, Helmholtz Detector Technology and Systems Initiative DTS).

In summary, within the UFO project it was possible to developed key componentsfor future data intense applications. Most important are the X-ray detector system,a smart camera platform, GPU-based computing infrastructure and the parallel com-puting framework including various optimized algorithms. The potential and feasibilityof high-speed X-ray tomography has been demonstrated by prototypes of experimentalstations at the ANKA beamlines TOPO-TOMO and IMAGE.

2 / 24

UFO – Ultra fast X-ray imaging 05K10VKE / 05K10CKB

Report

1 Introduction

X-ray imaging permits spatially resolved visualization of the 2D and 3D structure inmaterials and organisms which is crucial for the understanding of their properties. Ad-ditional resolution in the time domain gives insight in the temporal structure evolutionand thus access to dynamics of processes that are crucial to understand functionalityof devices and organisms and to optimize technological processes. The UFO projectaims to develop and integrate tools and instrumentation for intelligent X-ray imaging ofprocesses with high spatio-temporal resolution and high sample throughput.

The intelligent and interactive imaging system will combine process control, samplemanipulation, and photon detection. A key requirement of the UFO project is a newdata handling strategy: Instead of storing the data in local camera memory, transfer-ring it to mass storage and then processing and analyzing it off-line, the data will be re-constructed on-line by a heterogeneous DAQ system with programmable components(FPGA) and high-performance graphics processor units (GPU).

1.1 State of the art

In recent years, synchrotron radiation imaging has seen a tremendous increase of X-ray flux. The high X-ray flux available at modern synchrotron sources has reduced theexposure times down to the microsecond regime, in particular if broad band radiation isemployed. Accordingly the repetition rates increased. High-speed cameras based onCMOS detectors can cope with this frame rates and therefore allow the investigationof fast processes. In [1] frame rates of 5000 images per second were achieved, usinga filtered white beam from ESRF’s ID19 wiggler source with flux densities in excessof 1015 ph/s/mm2. Using a larger effective pixel size, frame rates of 40 000 images persecond were reported [2].

Tomographic imaging on the timescale of several minutes has been reported in [3],where also a pipelined data acquisition system combining a fast detector system,high-speed data networks and massively parallel computers was employed to acquireand reconstruct a full tomogram in tens of minutes. A specialized visualization com-puter was used for quick data evaluation purposes. The authors of [4] used the quasimonochromatic radiation from an undulator insertion device at SPring8 to acceleratetomographic data acquisition to six seconds, while [5] reported acquisition times of lessthan ten seconds using the white beam of ID15’s wiggler. In [6] grating interferometrywas performed with broad band radiation, using a high-speed CMOS based detec-tor system to exploit the high flux available and to perform high-speed phase gratingtomography. Total acquisition times of 500 ms were reached.

Computed tomography requires reconstruction algorithms which can be classified intoalgebraic and analytic ones [7]. A variety of algebraic reconstruction algorithms exists:

3 / 24

UFO – Ultra fast X-ray imaging 05K10VKE / 05K10CKB

ART (Algebraic Reconstruction Techniques), SIRT (Simultaneous Iterative Reconstruc-tive Technique), SART (Simultaneous Algebraic Reconstruction Technique) [8, 9, 10,11]. Algebraic algorithms are computationally expensive and are currently rarely ap-plied. Analytic reconstruction algorithms exploit the Radon transformation and FourierSlice theorem. Due to their simplicity and robustness, algorithms like the Filtered-backprojection (FBP) are widely used for reconstruction [12]. Depending on the re-quired precision of the measurement the reconstruction is still computational demand-ing.

Graphic processors are increasingly used in scientific computing. They offer a highlyparallel architecture and are optimized for data parallel computations. Over the lastyears standard computing environments have been developed to increase the perfor-mance and ease of development for “non-graphic” applications. The breakthrough wasthe release of the Nvidia’s Compute Unified Device Architecture (CUDA) tools [13].Applications written in CUDA out-performed earlier Open Graphics Library (OpenGL)implementations [14]. Driven by the success of the graphics adapter, the Open Com-pute Language (OpenCL), a vendor-independent programming interface for paralleldevices, was standardized by the Khronos group [15]. But a flexible and open frame-work with scientific algorithms for online processing of streamed data was not availablebefore starting the UFO project.

1.2 Results from third parties

During the course of the UFO project an intensified use of parallel computing archi-tectures could be recognized. Several groups proposed concepts to simplify parallelprogramming, especially on GPUs. Some examples are given below.

Sponge is a GPU implementation on top of the StreamIt Domain Specific Language(DSL) [16]. Dubach et al. presented a Java-based approach called Lime which pro-duces host and kernel code from single Java sources for one OpenCL device [17].GStream is a C++ library to define filter pipelines for execution on single GPUs or mul-tiple GPUs distributed via the Message Passing Interface (MPI) but lacks single-nodemulti-GPU execution [18].

Several frameworks targeted to the GPU tomography appeared as well. ESRF re-leased the second version of the PyHST framework providing several reconstructionalgorithms for parallel beam tomography [19]. PyHST2 is implemented in Python anduses CUDA to offload computations to GPUs. The Vision Lab of the University ofAntwerp has recently open-sourced the ASTRA tomography framework. ASTRA fo-cuses on custom geometries and provides MATLAB interface for convenience of users[20]. To offload computations to GPUs, ASTRA is relying on the CUDA toolkit aswell [21]. The Reconstruction Toolkit (RTK) is based on the open-source and cross-platform Insight Toolkit (ITK) and provides basic filters for reconstruction [22]. E.g.filtering, scatter correction, forward and back-projection, tools for respiratory motioncorrection, etc. For some filters, parallel CUDA implementations are provided. TheCONRAD framework uses OpenCL to support high-speed cone-beam imaging in radi-

4 / 24

UFO – Ultra fast X-ray imaging 05K10VKE / 05K10CKB

ology [23].

Marone and Stampanoni presented a novel approach (gridrec algorithm) for fast recon-struction optimized for conventional CPU technology and adopted in particular to thebeam geometry of the third-generation synchrotron facilities [24]. Fast CT reconstruc-tion of huge datasets using GPU cluster were reported in [25].

Despite of the great success in the field of fast reconstruction, one has to notice thatmost of the implementations are limited to parallel or cone-beam computed tomogra-phy. In the literature optimized implementations for the generalized laminography arestill not available.

High-speed imaging is currently also being deployed at other synchrotrons. The effortis motivated by a large variety of applications that benefit or even require high temporalresolution. For example, geoscientific studies with temporal resolution of 200 ms havebeen reported in [26]. Tomographic investigations of nylon rod-packing structures with300 fps and 5.5 µm pixel size are reported in [27]. Even short exposure times in therange of several µs were used for material science research in [28, 29]. Still the UFOproject aims to go beyond this examples and provide fast time-resolved X-ray imagingwith online data processing and data-driven feedback.

2 Results

Within the UFO project a demonstrator for time-resolved X-ray imaging has been de-veloped. Key elements of the demonstrator are a fast detector, including a set of high-speed and high resolution cameras as well as a novel high-throughput smart cameraplatform. A parallel computing environment for streamed data has been designed pre-sented to the community as a platform for exchange of highly optimized algorithms.The tomography reconstruction algorithms include tomography as well as laminogra-phy and belong to the most efficient implementations in the world. The results obtainedmeet the initially specified goals and are presented in detail below according to the fourwork packages.

The demonstrator has already been used extensively for numerous applications. Thehigh temporal resolution enables in-vivo studies. So called 4D-cine-tomography isrevealing the morphological dynamics inside small organisms. An example recentlypublished in PNAS is shown in Figure 1. The intelligent combination of high-speedand high-resolution images as well as advanced image processing prove that real-timeanalysis is beneficial for life science applications.

2.1 WP1: Overall system development and integration

The focus of this work package was the specification and the overall system designof the UFO experimental stations to be built at ANKA. Subsequently a similar stationwill be developed at the synchrotron SIBIR-2 at the Kurtchatov Institute in Moscow.The system specifications of the experimental stations have been defined based on

5 / 24

UFO – Ultra fast X-ray imaging 05K10VKE / 05K10CKB

Figure 1: In-vivo X-ray cine-tomography revealing the morphological dynamics of a movingweevil screw joint. a-c: Time-lapsed sequence of tomographic slices, corresponding to 0,400 and 800 ms; d: Tomographic slice of post mortem scan with increased exposure time; e:Manual labeling of coxa (green) and trochanter (yellow); f: 3D model of the screw joint based onmanual labeling; g: 3D motion field computed from 0 ms and 130 ms; h: In-vivo morphologicaldynamics of the screw joint based on the 3D model (f) and automated motion estimation (g);i: Kinematics analysis: global displacement of the whole screw-and-nut system with respectto the main body (green plot); sudden translation of the trochanter inside the coxa (blue plot);linear rotational movement of the trochanter (yellow plot) [30].

the experimental demands of critical scientific applications like ultrafast radiographyof reaction kinetics of catalytic processes or high-speed tomography of insect motion.The feasibility of key requirements has been proved by first-hand experience with thedemonstrator high-speed tomography setup at the TOPO-TOMO beamline at ANKA.

The specification includes fast radiography, fast tomography and laminography withspatial resolution of better than 1 µm using a high-speed rotary stage operating at upto 3000 rpm for fast tomography, and a tiltable rotary stage operating at up to 60 rpmfor laminography, res pectively. With an X-ray energy bandwidth dE/E from 10−1 to 10−4

imaging experiments using absorption, propagation based phase, grating based phaseand spectroscopic contrast are planned. Furthermore diffraction imaging is foreseenwith a resolution of better than 0.5 µrad and a diffraction scan axis of −90° to +5°towards incoming beam, thus broadening the scope of experiments possible at the

6 / 24

UFO – Ultra fast X-ray imaging 05K10VKE / 05K10CKB

UFO experimental station.

The system includes an automated sample exchanger for samples with lateral dimen-sions up to 30 mm and 10 s maximum cycle time for tomography experiments, whichallows for remotely selectable samples and a high sample throughput to guarantee effi-cient use of the available beam time. During the evaluation of the demonstrator setup itbecame evident that manual operation when changing between configurations is timeconsuming and error-prone. Thus, a fully motorized setup is foreseen to ensure themost efficient use of beam time at the UFO experimental station.

The final station design with the supplier, as well as construction, installation and com-missioning of the station at the imaging beamline at ANKA, is part of UFO2 project.

2.2 WP2: X-ray imaging set-up

The focus of this work package was the selection of mechanical components for thededicated experimental station and design and construction of its detector system.The mechanical component selection has been critical for the above described sta-tion specification. The detector system has been worked out in detail, manufactured,installed, and tested. The final detector system is shown in Figure 2.

The detector system has two optical configurations: one for high-resolution and onefor high-speed. The system consists of three visible light cameras, an array of up totwelve scintillators and an array of spectral and neutral density filters. The design is fullymotorized, allowing one to combine any optical configuration, camera, scintillator andfilter. It also features protective lead glass to reduce the detrimental effect of X-raysscattered towards the objective lenses and a cooling mechanism for the scintillatorsand the first visible light folding mirror. Finally it includes intense UV LEDs to be ableto partially recover the full transmission of the objectives after long use. The detectorsystem has been successfully tested and is fully functional. All design goals werereached. The damage threshold for LuAG scintillators has been measured to be at0.7 W/mm2 absorbed power for scintillators of 25 mm diameter, which is consistent withvalues calculated by finite element analysis (FEA).

2.3 WP3: Data processing, evaluation and visualization

Development Platform. A collaborative development platform was set up to supportusers and developers alike. It is based on the ”Trac” project management system andhosts Bazaar and Git repositories, a Jenkins-based continuous integration system, abug tracker, wiki-based documentation, and mailing lists. Stable versions of the soft-ware are published on GitHub. Binary packages for multiple distributions are availablethrough the openSUSE build services. A project website presenting the project andsummarizing current achievements is installed within the KIT infrastructure (see Ta-ble 1 for details).

7 / 24

UFO – Ultra fast X-ray imaging 05K10VKE / 05K10CKB

Figure 2: Detector system with space for three cameras (A), two opticals setups (B) and upto twelve scintillators (C). Front and side plates are removed to provide view to the internalstructure (left). Camera revolver (top right); front view of the optics (center right); scintillatorholder (bottom right).

8 / 24

UFO – Ultra fast X-ray imaging 05K10VKE / 05K10CKB

Table 1: Summary of reference to the UFO project in the web.

Homepage http://ufo.kit.edu

GitHub™ https://github.com/ufo-kit

openSUSE™ https://build.opensuse.org/project/show/home:ufo-kit

GPU Computing Platform. A GPU computing platform has been commissioned atthe TOPO-TOMO beam line at ANKA, see Figure 3. The core of the platform is aSuperMicro 7046GT-TRF server. To enhance cooling and scalability, the GPUs andstorage devices are assembled in stand-alone boxes and connected using externalPCIe and SAS interfaces. The setup is equipped with 96 GB RAM, 6 NVIDIA GTX580graphic adapters, 2 Intel Xeon 5650 CPUs and 20 TB of local storage. For big datasets, the second-level cache of the storage system is organized in a RAID 0 fashionwith four SSD drives and provides a read capability of up to 1.3 GB/s with latenciesof around 0.15 ms. After preliminary evaluation, the data is moved to the Large ScaleData Facility (LSDF) [31].

Figure 3: GPU Computing Platform is a single server solution. It consists of 3 boxes: the server,the storage connected using external SAS interface, and GPU extension box connected withexternal PCIe interface.

2.3.1 Software Framework for Parallel Computation

Camera Support. A new robust Linux driver for PCO™ cameras [32] was designed.To support the development of the UFO smart camera platform, a Linux framework(Alps) for rapid prototyping and debugging of PCIe-based electronics was developed[33]. The architecture of the framework is presented on Figure 4. The framework pro-vides pluggable support for different DMA architectures. Support for the DMA protocolused in the prototype of UFO Camera is provided and guarantees uninterrupted opera-tion at the maximum camera speed of about 1.2 GB/s. Stable drivers for the prototype

9 / 24

UFO – Ultra fast X-ray imaging 05K10VKE / 05K10CKB

are provided as well. The Unified Camera Access Library (LibUCA) is developed tostandardize access to the cameras in the UFO project and the variety of other cam-eras used for imaging at ANKA.

Figure 4: The architecture of the Alps framework for rapid prototyping of Linux drivers for PCIe-based electronics: The tiny kernel driver (green) manages memory operations and interlocking.The user-space library (blue) provides direct access to the device memory and access to regis-ters defined in an XML configuration. DMA support and device-specific operations are providedby plugins. All functions are available by a command line interface and can be easily includedin scripting languages.

Graphical User Interfaces. A GUI interface for camera management was released.It provides a simple interface to manage standard and custom camera parameters aswell as showing video previews, see Figure 5.

UFO Parallel Computing Framework. The UFO parallel computing framework is atoolkit to build arbitrary data processing workflows by combining basic building blocksin a graph structure [34]. The sample graph structure for tomography workflow is pre-sented on Figure 6. It is based on OpenCL to offload massively parallelizable problemsto accelerators such as GPUs. By incorporating knowledge about hardware details atrun-time, the system scales with the number and types of GPUs. It is written in Cand provides bindings to glue languages such as Python. The framework containsa rich set of functions typically used for X-ray imaging. This functions include filtersfor common I/O tasks (reading and writing files, accessing cameras through LibUCA),pre-processing (Non-local means and Gaussian denoising, flat-field correction, phase

10 / 24

UFO – Ultra fast X-ray imaging 05K10VKE / 05K10CKB

Figure 5: Camera GUI

reconstruction), tomographic and laminographic reconstruction, Fourier transforms, im-age arithmetics and supports arbitrary, user-defined OpenCL kernels. The functionalitywill be continuously enhanced. But already now, the framework has proven to be validconcept to simplify development as well as usage of optimized algorithms for parallelarchitectures.

Figure 6: Sample graph structure of UFO framework for tomography workflow

11 / 24

UFO – Ultra fast X-ray imaging 05K10VKE / 05K10CKB

2.3.2 Algorithms for Parallel Data Processing

Tomography. The filtered back projection algorithm has been optimized for highlyparallel GPU architectures [35]. We provided multiple implementations tuned for thelatest GPU generations from the two main manufacturers AMD and NVIDIA [36]. Usingthe reference GPU computing platform installed at beam-line, we achieve an outboundreconstruction bandwidth of about 1 GB/s, see Figure 7.

Figure 7: The throughput of tomographic workflow on the target computing platform using theprojection data stored in memory and in the fast SSD-based cache (left), the scalability (right),and the performance of different GPU cards using default and optimized algorithms (bottom)

Laminography. A generalized version of a filtered backprojection algorithm (FBP)used for laminographic reconstruction was ported for execution on graphic cards andintegrated into the current UFO framework as a series of stand-alone nodes in a com-putational graph. The outbound reconstruction bandwidth of 64 MB/s is achieved onthe reference computing platform which measures a 15 times speed-up compared toCPU implementation running on the same platform.

12 / 24

UFO – Ultra fast X-ray imaging 05K10VKE / 05K10CKB

2.4 WP4: Development of on-line process control

2.4.1 Smart camera platform

A new smart camera platform provides embedded camera-side processing and fastdata transmission. The readout architecture can be divided into three main parts: adaughter card containing a CMOS image sensor, the main readout card with a Xil-inx Virtex-6 FPGA and DDR3 memory, and a PC used for camera control and dataacquisition. The block diagram of the architecture and the data flow is presented inFigure 8.

Figure 8: Smart camera platform

Several image sensors are available with resolutions of 2.2 megapixels or 4 megapixels,respectively. The maximum frame rate at 2.2 megapixels is 330 frames per second.Higher frame rates can be achieved in sub-sampling mode. Monochromatic and colorsensors are available.

The main readout board consists of the Xilinx Virtex-6 FPGA and DDR3 memory forimage processing stage and data storage. While Xilinx FPGA provides the physicallayer (PHY) for DDR3 devices, a custom module is developed to overcome the limita-tions present in the available DDR3 IP core and to saturate the link at maximum speed.A standard PCIe cable connection with four or eight lanes is used to transfer the datafrom the camera directly to the main computer memory. Passive copper cables andactive optical links are available. A custom PCIe-DMA (Direct Memory Access) mod-ule is developed to enable the continuous data streaming. The DMA operates as aBus Master, which indicates the ability to initiate PCIe transactions, i.e. write and readtransactions. Unlike commercial cameras, the new module enables the frames to bestreamed directly to the main computer memory. Additionally, using PC memory re-moves the necessity for an on-camera storage buffer, thus allowing recording timesthat are by an order of magnitude higher than what modern commercial cameras can

13 / 24

UFO – Ultra fast X-ray imaging 05K10VKE / 05K10CKB

provide. Furthermore, dead acquisition time required to transfer data to the PC is re-moved. Addressable 32-bit user bank registers are implemented in the dedicated BaseAddress Register (BAR) block. Bank registers are used to read or write status and con-figuration of the DMA engine, CMOS sensor and FPGA logic. Further bank registersare reserved for additional user applications.

2.4.2 Fast reject algorithm

As first example of embedded data processing within the smart camera platform animage-based self-trigger, called fast reject has been implemented. The image-basedself-trigger records fast spontaneous events by evaluating the information content ofthe images. The incoming frames are compared with the reference one, and the re-gions with events are identified and marked for acquisition. The automatic reductionand acquiring of the modified parts of the images reduces the effective amount of dataand at the same time increases the frame rate. Detection of multiple events in differentregions in the same image is possible. Benefits of the algorithm are simplification andacceleration of the data analysis, optimization of the effective bandwidth, and a signif-icant increase of the time-resolution. Several threshold parameters exist to customizethe algorithm.

Several beam test setups were devised at ANKA’s IMAGE beamline to test the suitabil-ity of the algorithm for radiography and tomography applications. A rotating aluminumplate with a small drilled hole (300 µm) was used to ensure the reproducibility of theexperiment. The frame rate in relation to different algorithm parameters is shown inFigure 9.

Figure 9: Performance of the fast reject implementation as function of the interlacing sequence.

For a given object size, the figure shows both the theoretical frame rate (continuouscurves) for three different settings and the experimental points measured. From the plotis well visible that the experimental measurement are close to the theoretical behavior

14 / 24

UFO – Ultra fast X-ray imaging 05K10VKE / 05K10CKB

and the achievable frame rate is increased by factor of five compared to the maximumframe rate of the CMOS sensor at full resolution.

2.4.3 Camera case and cooling

The image sensor cooling concept and its realization, shown in Figure 10 was de-veloped in cooperation with the A.V. Shubnikov Institute of Crystallography (IC RAS),Moscow. The complete image sensor daughter card is housed a vacuum environ-ment to avoid condensation. Using a Peltier cell element, the temperature could bedecreased down to −5 ◦C using 30 % of the maximal cooling power.

Figure 10: Housing of the image sensor

The camera system is currently fully operative at IMAGE beamline with different sensorresolutions.

3 Scientific benefit

UFO is an ambitious project with several technically innovative elements and a cor-respondingly high scientific potential. The competence of the collaboration partnersmatches the technical requirements of the project. The strength of the project teamis the combination of institutes with a strong history in synchrotron radiation researchand instrumentation combined with institutes specialized in fast data acquisition, triggerand slow-control systems, computer science and automatic control. Only by the addi-tional funding granted by the UFO project it was possible to build this interdisciplinarygroup and involve groups that originally had no or only limited exposure to synchrotron

15 / 24

UFO – Ultra fast X-ray imaging 05K10VKE / 05K10CKB

radiation projects. The results of the project and the feedback from the communityclearly show the great benefit of collaborations with strong instrumentation groups out-side of the synchrotron community. Access to the latest technologies in various fieldsfosters the development of improved instrumentation. The UFO project successfullydemonstrated how to meet this challenge.

The project has studied various aspects of ultrafast tomography. Tomography is cur-rently one of the most challenging methods with high data rates and high computationalneeds. Key elements for ultrafast tomography have been analyzed and resulted in ademonstrator that has been used extensively. The main expenses within the UFOproject therefore split to a large extent between personnel cost and investments forcommercial and custom high-speed cameras and the GPU computing infrastructure.

The project has demonstrated that online reconstruction in tomography is possible,affordable and opens the door to new fields such as dynamic life science studies and4D-cine-tomography, which is a unique tool for in-vivo analysis of small species. TheUFO project laid the foundation for a new imaging beamline at ANKA. The constructionand continuous optimization of the methods will be continued in the UFO2 project.First user communities have started the first projects to adopt the new technologiesto their needs. An example is the ASTOR project for ”Arthropod Structure revealedby Ultrafast Tomography and Online Reconstruction” coordinated by a biological groupat the University of Darmstadt. The new IMAGE beam lines will, after completion,be handed over to user operation. The developed technologies in the fields of dataacquisition, parallel processing frameworks and optimized algorithms for highest datarates are essential for further developments in the synchrotron community and beyond.

References

[1] F. Garcia-Moreno et al. “Fast processes in liquid metal foams investigated by high-speed synchrotron x-ray microradioscopy”. In: Applied Physics Letters 92.13 (2008),pp. 134104–134104–3. ISSN: 0003-6951. DOI: 10.1063/1.2905748.

[2] A Rack et al. “The micro-imaging station of the TopoTomo beamline at the ANKA syn-chrotron light source”. In: Nuclear Instruments and Methods in Physics Research Sec-tion B: Beam Interactions with Materials and Atoms 267.11 (2009), pp. 1978–1988.

[3] Y. Wang et al. “A high-throughput x-ray microtomography system at the Advanced Pho-ton Source”. In: Review of Scientific Instruments 72.4 (2001), pp. 2062–2068.

[4] K. Uesugi, T. Sera, and N. Yagi. “Fast tomography using quasi-monochromatic undulatorradiation”. In: Journal of synchrotron radiation 13.5 (2006), pp. 403–407.

[5] M. Di Michiel et al. “Fast microtomography using high energy synchrotron radiation”. In:Review of Scientific Instruments 76.4 (2005), p. 043702.

[6] A. Momose et al. “High-speed X-ray phase imaging and X-ray phase tomography withTalbot interferometer and white synchrotron radiation”. In: Optics express 17.15 (2009),pp. 12540–12545.

[7] A. C. Kak and M. Slaney. “Principles of computerized tomographic imaging”. In: IEEE,New York (1988).

16 / 24

UFO – Ultra fast X-ray imaging 05K10VKE / 05K10CKB

[8] P. Danielsson. “Iterative techniques for projection and backprojection”. Tech. rep. De-partment of Electrical Engineering, Linkoping University, 1997.

[9] K. Mueller. “Fast And Accurate Three-Dimensional Reconstruction From Cone-BeamProjection Data Using Algebraic Methods”. PhD thesis. The Ohio State University, 1998.

[10] K. Mueller, R. Yagel, and J. J. Wheller. “Fast implementations of algebraic methodsfor three-dimensional reconstruction from cone-beam data”. In: Medical Imaging, IEEETransactions on 18.6 (1999), pp. 538–548.

[11] J. S. M. P. Hentschel A. Lange. “Direct iterative reconstruction of computed tomographytrajectories (DIRECT)”. In: Proc. SPIE 5766.25 (2005).

[12] L. Helfen et al. “Synchrotron-radiation computed laminography for high-resolution three-dimensional imaging of flat devices”. In: physica status solidi (a) 204.8 (2007), pp. 2760–2765.

[13] nvidia. “Cuda zone”. 2014. URL: http://www.nvidia.com/cuda.

[14] J. D. Owens et al. “GPU computing”. In: Proceedings of the IEEE 96.5 (2008), pp. 879–899.

[15] Khronos. “OpenCL: The open standard for parallel programming of heterogeneous sys-tems”. 2014. URL: http://www.khronos.org/opencl.

[16] A. H. Hormati et al. “Sponge: Portable Stream Programming on Graphics Engines”. In:SIGARCH Comput. Archit. News 39 (1 Mar. 2011), pp. 381–392. ISSN: 0163-5964. DOI:http://doi.acm.org/10.1145/1961295.1950409.

[17] C. Dubach et al. “Compiling a High-level Language for GPUs: (via Language Supportfor Architectures and Compilers)”. In: SIGPLAN Not. 47.6 (June 2012), pp. 1–12. ISSN:0362-1340. DOI: 10.1145/2345156.2254066.

[18] Y. Zhang and F. Mueller. “GStream: A General-Purpose Data Streaming Framework onGPU Clusters”. In: International Conference on Parallel Processing 0 (2011), pp. 245–254. ISSN: 0190-3918. DOI: 10.1109/ICPP.2011.22.

[19] A. Mirone et al. “The PyHST2 hybrid distributed code for high speed tomographic recon-struction with iterative reconstruction and a priori knowledge capabilities”. In: NuclearInstruments and Methods in Physics Research Section B: Beam Interactions with Ma-terials and Atoms 324 (2014), pp. 41–48. DOI: 10.1016/j.nimb.2013.09.030.

[20] W. J. Palenstij, K. J. Batenburg, and J. Sijbers. “The ASTRA Tomography Toolbox”. In:Proc. of 13th Int. Conf. on Computational and Mathematical Methods in Science andEngineering. Vol. 3. 2013, pp. 1139–1145.

[21] W. J. Palenstij, K. J. Batenburg, and J. Sijbers. “Performance improvements for iterativeelectron tomography reconstruction using graphics processing units (GPUs)”. In: Jour-nal of Structural Biology 176.2 (2011), pp. 250–253. DOI: 10.1016/j.jsb.2011.07.017.

[22] S. Rit et al. “The Reconstruction Toolkit (RTK), an open-source cone-beam CT recon-struction toolkit based on the Insight Toolkit (ITK)”. In: Journal of Physics: ConferenceSeries 489 (2014), pp. 1–4. DOI: doi:10.1088/1742-6596/489/1/012079.

[23] A. Maier et al. “CONRAD—A software framework for cone-beam imaging in radiology”.In: Medical Physics 40.2 (2013), pp. 111914–1–111914–8. DOI: 10.1118/1.4824926.

17 / 24

UFO – Ultra fast X-ray imaging 05K10VKE / 05K10CKB

[24] F. Marone and M. Stampanoni. “Regridding reconstruction algorithm for real-time tomo-graphic imaging”. In: Journal of Synchrotron Radiation 19 (2012), pp. 1029–1037. DOI:10.1107/S0909049512032864.

[25] X. Liu, S. Boons, and A. Sasov. “Strategies for GPU-based conebeam CT reconstructionfor very large data volumes”. In: Proc. of 12th Int. Meeting Fully 3D Image Reconstruc-tion in Radiology and Nuclear Medecine. 2013, pp. 529–532.

[26] X. Xiao, F. Fusseis, and F. De Carlo. “X-ray fast tomography and its applications indynamical phenomena studies in geosciences at Advanced Photon Source”. In: Proc.SPIE. Vol. 8506. 2012, 85060K.

[27] Z. Xiao-Dan et al. “Fast synchrotron X-ray tomography study of the packing structuresof rods with different aspect ratios”. In: Chinese Physics B 23.4 (2014), p. 044501.

[28] J. Yeager et al. “High-speed synchrotron X-ray phase contrast imaging for analysis oflow-Z composite microstructure”. In: Composites Part A: Applied Science and Manufac-turing 43.6 (2012), pp. 885–892.

[29] M Hudspeth et al. “High speed synchrotron x-ray phase contrast imaging of dynamicmaterial response to split Hopkinson bar loading”. In: Review of Scientific Instruments84.2 (2013), p. 025102.

[30] T. dos Santos Rolo et al. “In vivo X-ray cine-tomography for tracking morphological dy-namics”. In: Proceedings of the National Academy of Sciences 111.11 (2014), pp. 3921–3926.

[31] R. Stotzka. “Data Life Cycle Lab. Key Technologies. Status 2013. Big Data in Science”.Tech. rep. Institute for Data Processing and Electronics (IPE), 2013.

[32] PCO. “CCD and CMOS camera systems”. 2014. URL: http://www.pco.de.

[33] S. Chilingaryan. “ALPS – Advanced Linux PCI Services for Rapid Prototyping of PCI-based DAQ Electronics”. In: 18th IEEE Real-time Conference. 2012.

[34] M. Vogelgesang et al. “UFO: A Scalable GPU-based Image Processing Framework forOn-line Monitoring”. In: Proceedings of The 14th IEEE Conference on High PerformanceComputing and Communication & The 9th IEEE International Conference on Embed-ded Software and Systems (HPCC-ICESS). HPCC ’12. Liverpool, UK: IEEE ComputerSociety, June 2012, pp. 824–829. ISBN: 978-1-4673-2164-8.

[35] S. Chilingaryan et al. “A GPU-based architecture for real-time data assessment at syn-chrotron experiments”. In: Nuclear Science, IEEE Transactions on 58.4 (2011), pp. 1447–1455. DOI: 10.1109/TNS.2011.2141686.

[36] S. Chilingaryan et al. “A GPU-based architecture for real-time data assessment at syn-chrotron experiments”. In: Proc. of SC11 Supercomputing Conference. 2011, pp. 51–52.

18 / 24

UFO – Ultra fast X-ray imaging 05K10VKE / 05K10CKB

Publications

2013

Journal papers

M. Caselle et al. “Ultrafast Streaming Camera Platform for Scientific Applications”. In: NuclearScience, IEEE Transactions on 60.5 (2013), pp. 3669–3677. ISSN: 0018-9499. DOI: 10.1109/TNS.2013.2252528

W. Mexner et al. “A new flexible integraton of NeXus data sets to ANKA by FUSE file system”.In: Proceedings of the 14th International Conference on Accelerator & Large ExperimentalPhysics Control Systems. ICALEPCS ’13. 2013

A. Myagotin et al. “Efficient Volume Reconstruction for Parallel-Beam Computed Laminographyby Filtered Backprojection on Multi-Core Clusters”. In: Image Processing, IEEE Transactionson 22.12 (Dec. 2013), pp. 5348–5361. ISSN: 1057-7149. DOI: 10.1109/TIP.2013.2285600

T. dos Santos Rolo et al. “In vivo X-ray cine-tomography for tracking morphological dynamics”.In: Proceedings of the National Academy of Sciences 111.11 (2014), pp. 3921–3926. DOI:10.1073/pnas.1308650111

R. Stotzka. “Data Life Cycle Lab. Key Technologies. Status 2013. Big Data in Science”. Tech.rep. Institute for Data Processing and Electronics (IPE), 2013

P. Vrsansky et al. “Cockroaches probably cleaned up after dinosaurs”. In: PLoS ONE 8.12 (Dec.2013), pp. 5348–5361. DOI: 10.1371/journal.pone.0080560

T. van de Kamp et al. “Insect imaging at the ANKA synchrotron radiation facility”. In: Entomolo-gie heute 11.25 (2013), pp. 147–160

D. Haas et al. “Implementation of an Overall Data Management at the Tomography Station atANKA”. In: Proceedings of the 14th International Conference on Accelerator & Large Experi-mental Physics Control Systems. ICALEPCS ’13. 2013

M. Vogelgesang et al. “When Hardware and Software Work in Concert”. In: Proceedings of the14th International Conference on Accelerator & Large Experimental Physics Control Systems.ICALEPCS ’13. 2013

H. Yang et al. “Algebraic Reconstruction of Ultrafast Tomography Images at the Large ScaleData Facility”. In: Proceedings of ICALEPS 2013. ICALEPCS ’13. 2013

X. Yang et al. “Data Intensive Computing of X-Ray Computed Tomography Reconstruction atthe LSDF”. In: Parallel, Distributed and Network-Based Processing (PDP), 2013 21st EuromicroInternational Conference on. Feb. 2013, pp. 86–93. DOI: 10.1109/PDP.2013.21

Talks

S. Chilingaryan. “Ultrafast X-Ray Imaging of Scientific Processes”. Seminar at TPU, Tomsk.May 2013

S. Chilingaryan. “High-performance computing hardware for high data rates”. HDRI Workshop.Mar. 2013

19 / 24

UFO – Ultra fast X-ray imaging 05K10VKE / 05K10CKB

S. Chilingaryan. “Ultrafast X-Ray Imaging of Scientific Processes”. Seminar at TPU, Tomsk.May 2013

A. Kopmann. “Photon/neutron community experience and plans with GPUs”. GPUs in HighEnergy Physics Workshop, Hamburg. Apr. 2013

H. Pasic. “Managing Large Scale Data at ANKA”. HDRI Workshop, Hamburg. Mar. 2013

T. van de Kamp et al. “Synchrotron-Rontgenmikrotomographie fossiler Insekten”. 26. West-deutscher Entomologentag, Dusseldorf. Nov. 2013

T. van de Kamp et al. “n vivo X-ray 4D cine-tomography for tracking morphological dynamics ininsects”. Entomology 2013. Nov. 2013

T. van de Kamp and T. Baumbach. “Vom Praparat zum Kunststoffmodell: 3D-Druck fur Mor-phologen”. 6. Graduiertenforum der DZG-Fachgruppe Morphologie, Ulm. Oct. 2013

T. van de Kamp et al. “Synchrotron X-ray microtomography for examining animal morphology”.Bio-Geo-Kolloquium der Friedrich-Schiller-Universitat Jena. Apr. 2013

T. van de Kamp et al. “Zur Funktionsmorphologie der Russelkafergattung Trigonopterus.” En-tomologentagung der Deutschen Gesellschaft fur allgemeine und angewandte Entomologie,Gottingen. Mar. 2013

T. van de Kamp et al. “Rontgenbildgebung in den Lebenswissenschaften: Herausforderungenan die Datenanalyse”. LSDMA Community Forum, DESY, Hamburg. Mar. 2013

T. van de Kamp et al. “Imaging in life sciences.” Konigstein Meeting, Baehrenthal, France. Jan.2013

M. Vogelgesang. “An Extensible Parallel Computing Environment for Ultrafast X-Ray Imaging”.HDRI Workshop, Hamburg. Mar. 2013

M. Vogelgesang. “Experiment Control for High-Speed Tomography”. 27th Tango CollaborationMeeting, Barcelona. May 2013

X. Yang. “Algebraic Reconstruction of X-ray Tomography on Parallel Computing Architecture ofLSDF”. HDRI Workshop, Hamburg. Mar. 2013

Posters

S. Chilingaryan et al. “A GPU-based Architecture for Real-Time Data Assessment at Syn-chrotron Experiments”. 532. Wilhelm and Else Heraeus-Seminar, Development of High-ResolutionPixel Detectors and their Use in Science and Society, Bad Honnef. May 2013

T. Farago et al. “GPU Accelerated High-speed Imaging Using X-rays”. GPU Technology Con-ference, San Jose, CA USA. Mar. 2013

M. Vogelgesang et al. “When Hardware and Software Work in Concert”. In: Proceedings of the14th International Conference on Accelerator & Large Experimental Physics Control Systems.Vol. 60. 5. 2013, pp. 3669–3677. DOI: 10.1109/TNS.2013.2252528

20 / 24

UFO – Ultra fast X-ray imaging 05K10VKE / 05K10CKB

2012

Journal papers

D. Haas et al. “Status of the ultra fast tomography experiments control at ANKA”. In: Proceed-ings of PCaPAC2012, Kolkata, India. 2012

H. Anzt et al. “A unified energy footprint for simulation software”. In: Computer Science-Researchand Development (2012), pp. 1–8

M. Caselle et al. “Ultra-fast streaming camera platform for scientific applications”. In: Real TimeConference (RT), 2012 18th IEEE-NPSS. IEEE. 2012, pp. 1–8

P.-A. Douissard et al. “A versatile indirect detector design for hard X-ray microimaging”. In:Journal of Instrumentation 7.09 (2012), P09016

A. Myagotin et al. “Fast volume reconstruction for parallel-beam computed laminography byfiltered backprojection”. In: International Journal of Materials Research 09 (2012), pp. 170–173

A. Riedel et al. “Sayrevilleinae Legalov, a newly recognised subfamily of fossil weevils (Coleoptera,Curculionoidea, Attelabidae) and the use of synchrotron microtomography to examine inclu-sions in amber”. In: Zoological Journal of the Linnean Society 165.4 (2012), pp. 773–794

R. Shkarin and A. Shkarin. “Organizing tests of Custom Image Filters for UFO”. In: Proceedingsof 18th International Conference of Students and Young Scientists “Modern Technique andTechnologies”. 2012

M. Vogelgesang et al. “UFO: A Scalable GPU-based Image Processing Framework for On-lineMonitoring”. In: Proceedings of The 14th IEEE Conference on High Performance Computingand Communication & The 9th IEEE International Conference on Embedded Software and Sys-tems (HPCC-ICESS). HPCC ’12. Liverpool, UK: IEEE Computer Society, June 2012, pp. 824–829. ISBN: 978-1-4673-2164-8

Talks

M. Balzer et al. “KIT Kamera System fur UFO”. Presented at SEI Tagung Dresden. Mar. 2012

S. Chilingaryan. “ALPS – Advanced Linux PCI Services for Rapid Prototyping of PCI-basedDAQ Electronics”. In: 18th IEEE Real-time Conference. 2012

S. Chilingaryan. “A high-throughput platform for real-time X-ray imaging”. In: 18th IEEE Real-time conference. 2012

S. Chilingaryan. “Practical Experience with GPUs for High-Throughput Computing”. Presentedat a PNI-HDRI workshop in Hamburg, Germany. Feb. 2012

S. Chilingaryan. “Visualization of Tango Historic Archives”. Presented at the Tango collaborationmeeting at MAX-lab, Lund, Sweden. Apr. 2012

T. dos Santos Rolo et al. “High-speed X-ray imaging and image-based control at ANKA”. In:The 11th International Conference on Synchrotron Radiation Instrumentation. 2012

T. dos Santos Rolo et al. “Real-time volumetric imaging using high-speed X-ray microtomogra-phy”. In: Focus on Microscopy Conference. 2012

A. Kopmann. “Application of GPUs for online monitoring in Tomography”. Presented at “Tomog-

21 / 24

UFO – Ultra fast X-ray imaging 05K10VKE / 05K10CKB

raphy, data processing and image reconstruction for medicine and engineering.” First topicalworkshop within the framework of the Helmholtz portfolio project ”Detector technologies andsystems platform”. Sept. 2012

H. Pasic. “Extending NeXus for HDRI¿” Presented at PNI-HDRI and PaNdata Workshop. Feb.2012

T. van de Kamp et al. “Synchrotron-Rontgen-Tomographie und interaktive 3D-Modelle in der In-sektenmorphologie”. Presented at 25. Westdeutscher Entomologentag, Dusseldorf. Nov. 2012

T. van de Kamp et al. “Synchrotron-Rontgen-Tomographie und der Einsatz interaktiver 3D-Modelle fur funktionsmorphologische Untersuchungen”. Presented at 5. Graduiertenforum derDZG-Fachgruppe Morphologie, Tubingen. Oct. 2012

T. van de Kamp et al. “Synchrotron X-ray microtomography and digital 3D imaging for examininginsect morphology”. Presented at XXIV International Congress of Entomology, Daegu, SouthKorea. Aug. 2012

T. van de Kamp et al. “Visualization of biological specimens acquired by X-ray microtomogra-phy”. Presented at Instrumentation and methods development for synchrotron-based biomedi-cal research, DESY, Hamburg. May 2012

T. van de Kamp et al. “Insect imaging at ANKA: methods, recent developments and perspec-tive”. Presented at XXIV International Congress of Entomology, Daegu, South Korea. Aug. 2012

T. van de Kamp et al. “Visualization of multidimensional data acquired by synchrotron X-raymicrotomography”. In: Focus on Microscopy Conference. 2012

Posters

S. Chilingaryan. “ALPS – Advanced Linux PCI Services for Rapid Prototyping of PCI-basedDAQ Electronics”. In: Poster at 18th IEEE Real-time Conference. 2012

U. Stevanovic et al. “High-speed camera with embedded FPGA processing”. In: Proceedingsof Conference on Desing and Architecture for Signal and Image Processing (DASIP). 2012

2011

Journal papers

L. Alaribe et al. “Bridgman: Growth of SrI2”. In: Proc. of SC11 Supercomputing Conference.2011, p. 1341

V. Altapova et al. “X-ray phase-contrast radiography using a filtered white beam with a grat-ing interferometer”. In: Nuclear Instruments and Methods in Physics Research Section A: Ac-celerators, Spectrometers, Detectors and Associated Equipment 648 (2011), pp. 42–45. DOI:10.1016/j.nima.2010.12.218

A. Cecilia et al. “Characterisation of LSO:Tb scintillator films for high resolution X-ray imagingapplications”. In: Nuclear Instruments and Methods in Physics Research Section A: Accelera-tors, Spectrometers, Detectors and Associated Equipment 633 (2011), pp. 292–293

S. Chilingaryan et al. “A GPU-based architecture for real-time data assessment at synchrotron

22 / 24

UFO – Ultra fast X-ray imaging 05K10VKE / 05K10CKB

experiments”. In: Nuclear Science, IEEE Transactions on 58.4 (2011), pp. 1447–1455. DOI:10.1109/TNS.2011.2141686

A. N. Danilewsky et al. “Real-time X-ray diffraction imaging for semiconductor wafer metrologyand high temperature in situ experiments”. In: Physica status solidi (a) 208.11 (2011), pp. 2499–2504. DOI: 10.1002/pssa.201184264

A. N. Danilewsky et al. “Dislocation dynamics and slip band formation in silicon: In-situ studyby X-ray diffraction imaging”. In: Journal of Crystal Growth 318.1 (2011), pp. 1157–1163. DOI:10.1016/j.jcrysgro.2010.10.199

T. van de Kamp et al. “A biological screw in a beetle’s leg”. In: Science 333.6038 (2011), p. 52.DOI: 10.1126/science.1204245

Talks

S. Chilingaryan and A. Myagotin. “Optimized computed tomography and laminography algo-rithms”. ANKA Workshop on IT Research and Development, Karlsruhe. July 2011

S. Chilingaryan. “High Speed Tomography at KIT”. Meeting on tomographic reconstruction soft-ware, ESRF, Grenoble. Mar. 2011

A. Kopmann, S. Chilingaryan, and M. Vogelgesang. “Evaluation of fast Tomography ComputingPlatforms”. Meeting of the High Data Rate Processing and Analysis Initiative (HDRI), Hamburg.May 2011

T. van de Kamp. “Dreidimensionale Visualisierung kleiner Kafer mit dem Synchrotron des Karl-sruher Instituts fur Technologie (ANKA)”. Naturwissenschaftlicher Verein Karlsruhe. June 2011

T. van de Kamp. “Automated and Semi-automated 3D Volume Segmentation”. ISS IT Work-shop, Karlsruhe. May 2011

T. van de Kamp. “A biological screw in a beetle’s leg”. 3rd ANKA/KNMF Joint Users Meeting.Oct. 2011

Posters

S. Chilingaryan et al. “A GPU-based architecture for real-time data assessment at synchrotronexperiments”. In: Proc. of SC11 Supercomputing Conference. 2011, pp. 51–52

M. Vogelgesang, S. Chilingaryan, and A. Kopmann. “Flexible X-Ray Image Processing onGPUs”. Nvidia booth at Supercomputing 2011, Seattle, WA. Nov. 2011

M. Vogelgesang, S. Chilingaryan, and A. Kopmann. “A GPU-accelerated Framework for Real-Time Tomographic Reconstruction”. Programming and Tuning Massively Parallel Systems Sum-mer School (PUMPS), Summer School, Barcelona. July 2011

23 / 24

UFO – Ultra fast X-ray imaging 05K10VKE / 05K10CKB

2010

Journal papers

A. Cecilia et al. “LPE grown LSO:Tb scintillator films for high resolution X-ray imaging applica-tions at synchrotron light sources”. In: Nuclear Instruments and Methods in Physics ResearchSection A: Accelerators, Spectrometers, Detectors and Associated Equipment 633 (2010),pp. 292–293. DOI: 10.1016/j.nima.2010.10.150

Talks

S. Chilingaryan et al. “A GPU-based Architecture for Real-Time Data Assessment at Syn-chrotron Experiments”. In: GPU Technology Conference, San Jose USA. 2010

T. B. dos Santos Rolo T. Zienicke et al. “High speed micro-tomography at ANKA: implementationand first applications”. In: XTOP Conference, Warwick, UK. 2010

Posters

S. Chilingaryan et al. “A GPU-based Architecture for Real-Time Data Assessment at Syn-chrotron Experiments”. PNI-In House research NANO & MICRO Sciences and TechnologiesWorkshop Karlsruhe. Dec. 2010

T. dos Santos Rolo et al. “High-speed micro-tomography at ANKA: applications and visualisa-tion”. ANKA User Meeting, Karlsruhe. Oct. 2010

T. van de Kamp et al. “Insect Imaging at ANKA - Visualization and Applications”. ANKA UserMeeting, Karlsruhe. Oct. 2010

T. van de Kamp et al. “Insect tomography at ANKA: applications and visualization”. XTOP Con-ference, Warwick, UK. Sept. 2010

24 / 24