“Live” Tomographic Reconstructions

Alun Ashton

Mark Basham

IncentiveHigher resolution cameras are forcing larger collection

times.• I12 at DLS will use a 4000x2500 pixel detector (PCO

4000) capable of approximately 2 frames per second (Technically 4-5 frames). Assuming the best case scenario this will require around 30-40 minutes to collect the data(~4000 images i.e. the width of the detector).

• Even with good cluster performance, it could still take around 30 minutes to make the 80GB reconstructed volume (4000x4000x2500)

Having to wait an hour from start to finish before being able to see what has been imaged is frustrating, and could waste beam time due to misalignments etc.

Trying something different

• Recent work has focused on fast and accurate reconstructions.– Make good use of complete data sets to correct for

various anomalies.– This has to happen after the collection has been

completed (on an 80Gb file…)• This work focuses on providing a reconstruction during

the data collection.– Less quality, and smaller reconstructions (e.g.

1000x600) to allow visualisation.– Gives the user the ability to see the volume after only

a few minutes.

Traditional Tomography

Ingoing Path

Outgoing Path

Direction of the beam

Traditional TomographyEach new acquisition

collects the next segments data

Traditional Tomography

Traditional Tomography

Traditional Tomography

Traditional TomographyAll the data has been collected

and the final full size and quality reconstruction

can be produced

Experimental changes

• To make the maximum use of this methodology there needs to be some experimental changes.

• The main change is in the way that the data is collected

• This requires certain hardware requirements mainly a sample stage capable of continuous rotation

• The acquisition is then preformed as follows

Continuous rotation

Ingoing Path

Outgoing Path

Continuous rotationEach new

acquisition skips 4 segments and then collects the

5th

Continuous rotation

Continuous rotationInitial lowest

quality reconstruction

can now be calculated

Continuous rotationBecause its

transmition you can flip the image and then its going the right way…. So becomes red

Continuous rotation

Continuous rotation

Continuous rotationRefined

reconstruction can now be

calculated and replaces the

previous one.

Continuous rotation

This is where skipping 4 becomes important.

Continuous rotation

Continuous rotationRefined

reconstruction can now be

calculated and replaces the

previous one.

Continuous rotation

Continuous rotation

Continuous rotation

Continuous rotationRefined

reconstruction can now be

calculated and replaces the

previous one.

Continuous rotation

Continuous rotation

Continuous rotationAll the data has been collected

and the final full size and quality reconstruction

can be produced

Progressive Collection

• Advantages– A full reconstruction can be preformed after only a

fifth of the acquisition time, albeit at reduced resolution.

– As there is a gap between acquisitions, the full sector can be integrated, then the gap can be used for camera readout.

• Disadvantages– Requires custom software to reconstruct, or convert

to classical data.– If the gap is too large, acquisition time can be

increased.

Integration Time

Sector 1 Sector 2

S1 S2

Traditional

New

Acquisition

Detector tomemory

Acquisition

Detector tomemory

Time

User

Save Images to central Storage

Request Aquisition

Request and Retrieve Images

Request Data from storage

Request full Reconstruction

Display ‘live’ data

Process data and produce live volumes

Process all the data and create the full

volume reconstructions

Display the full data

Architecture

Beamline Control PCCamera Control and Live

Reconstruction PC

Central StorageCluster Computing

Resources

Digital camera

User

Save Images to central Storage

Request Aquisition

Request and Retrieve Images

Request Data from storage

Request full Reconstruction

Display ‘live’ data

Process data and produce live volumes

Process all the data and create the full

volume reconstructions

Display the full data

Architecture

Start-up

User

Save Images to central Storage

Request Aquisition

Request and Retrieve Images

Request Data from storage

Request full Reconstruction

Display ‘live’ data

Process data and produce live volumes

Process all the data and create the full

volume reconstructions

Display the full data

Architecture

Start-up

User

Save Images to central Storage

Request Aquisition

Request and Retrieve Images

Request Data from storage

Request full Reconstruction

Display ‘live’ data

Process data and produce live volumes

Process all the data and create the full

volume reconstructions

Display the full data

Architecture

Collect Images

User

Save Images to central Storage

Request Aquisition

Request and Retrieve Images

Request Data from storage

Request full Reconstruction

Display ‘live’ data

Process data and produce live volumes

Process all the data and create the full

volume reconstructions

Display the full data

Architecture

End of collection

User

Save Images to central Storage

Request Aquisition

Request and Retrieve Images

Request Data from storage

Request full Reconstruction

Display ‘live’ data

Process data and produce live volumes

Process all the data and create the full

volume reconstructions

Display the full data

Architecture

Produce full reconstruction

User

Save Images to central Storage

Request Aquisition

Request and Retrieve Images

Request Data from storage

Request full Reconstruction

Display ‘live’ data

Process data and produce live volumes

Process all the data and create the full

volume reconstructions

Display the full data

Differences to normal setups

Computing Hardware for the live reconstruction

• Standard PC server– Passes the data on to the central storage– Scales and applies the flat field correction to the

images as they come in.– Runs the Host program for the Tesla

• Tesla Graphics Processor Unit– Takes the scaled and corrected images– Filters the images.– Provides the Back Projection.

Why the TESLA

• Tomography is inherently very parallelisable– Tesla requires around 100,000 concurrent threads to

make it effective– This then allow for in general a single GPU to run 40

times faster on these problems than a single CPU or 10 times faster then a QuadCore.

• Space and power are saved in this case as a 1U Tesla Unit can effectively replace 20 dual processor quad core machines, for tomographic reconstruction.

• This also allows our beamline machine to pack the punch required to make the ‘live’ reconstructions possible.

Conclusions

• This methodology for collecting tomographic data should give the users much more insight into there samples and more time to make decisions about collections.

• The Tesla GPU is a good way of increasing the speed of Tomographic reconstruction, and has been proven in various different labs around the world.

• We can modify the way in which the experiment is preformed to make the most use or influence the choice of hardware, such as the modifications made to allow for camera readout and continuous rotation stage.

Acknowledgements

• Manchester University– Valeriy Titarenko, Albrecht Kyrieleis, Phil Withers,

Mark Ibson.• Diamond Light Source

– Michael Drakopoulos, Thomas Connolley• Architecture

– Piercarlo Grandi, Nick Rees, Bill Pullford

• Mark Basham, mark.basham@diamond.ac.uk