Extending the Neutron Scientist’s Toolkit
Michael McKerns
Materials Science and Applied PhysicsCenter for Advanced Computing Research
California Institute of Technology
(Distributed Data Analysis for Neutron Scattering Experiments):
Part I: Overview of DANSE
Part II: The Scientist’s Toolkit
What is DANSE?• a 12M$ five-year NSF IMR-MIP software construction project
• a collaborative effort between software professionals, neutron scattering scientists, and facilities
• a software engineering effort– open-source development environment
– framework for the interoperability of modular components
– integration of legacy codes and community-standard software
– connectivity to facility databases and software repositories
• a scientific endeavor– to develop software modules for different subfields of neutron scattering
– to enhance neutron scattering research and facilitate new science
– to build tools for education, collaboration, and plausibility assessment
• an integration framework for building data analysis, visualization, modeling, and instrument simulation tools for all areas of neutron scattering
Serving a Growing Community• With the availability of SNS, OPAL, and J-PARC fast approaching, the
neutron community has the potential to undergo a large growth spurt.
• Software is a vital part of scattering research, and unless the software is both robust and easy to use, that growth may be limited.
• Mature packages do exist (McStas, ISAW, DAVE, …), and commercial packages are also used (Matlab, IDL, Abaqus, IGOR Pro, …) in the analysis process. However, groups often use cryptic legacy code for at least one step.
• To grow as a community, we need:– a way to cultivate and maintain the valuable portions of these legacy codes– to make legacy and community-standard codes interoperable
• define common data structures and interfaces• stop duplication of effort
– to allow scientists to concentrate on science by lowering barriers to software engineering– to allow programmatic integration of analysis software with facility databases & libraries
There is much to do…• Software is needed to support the massive quantity of data that will be
produced at modern neutron facilities. However, existing software may be incapable of utilizing the full richness of the data that will be produced.
• Although the barrier to developing new software must be reduced, it is also critical that more complex software technologies (i.e. high-performance and grid-based computing) are enabled.
• Time is short – we must use the best existing tools to provide a robust solution… yet be flexible enough to allow for the easy substitution of better future solutions.
• The fundamental commodity for neutron scattering software is found within the cores of time-tested community-standard software. Rather than rewrite or duplicate this software, we can use DANSE to provide an integration path for both new and legacy codes into a common environment.
Building a Scientific Toolkit• Through DANSE, scientists will have access to many tools
– basic data structures, optimization algorithms, numerical libraries
– basic data reduction library [obtain I(Q), S(Q), S(E), S(Q,E)]
– graphical/plotting environments• IDL, Matlab, Matplotlib, Gnuplot, Grace, ParaView, ACIS (AutoCAD), …
– instrument simulation• McStas, VITESS, sample simulation framework, …
– materials simulation• ABINIT, VASP, GAMESS, NWChen, NAMD, CHARMM, …
– crystallography• cctbx, FOX, ObjCryst++, …
– molecular viewers and format translators• OpenBabel, Molden, PyMol, ViewMol, DRAWxtl, VMD, AtomEye, …
– and MORE!• ISAW, texture analysis (MAUD), SLD calculator, scattering intensity, …
• These tools will be available in DANSE as component libraries
Component Implementation• build core engine (Python, Fortran, C++, Java, Matlab, IDL, …)
– legacy or custom code and third-party libraries
– provide life-cycle management and exception handling strategy
• construct Python bindings– select entry points to expose to Python
– modularize entry points to monolithic compiled libraries
• cast as a component– extend and leverage framework services
– describe user-configurable parameters
– provide meta-data that specify the IO port characteristics
• test code– satisfy functional requirements with concurrent test development
– utilize interactive runtime testing within Python interpreter
– demonstrate integration with other components
ComponentComponent
CORECORE
Component-based Applications• granularity allows reusability of object-oriented components
• rebinning application
• modularity provides flexibility and extensibility
NeXusReaderNeXusReader SelectorSelector
BckgrndBckgrnd
SelectorSelector
SelectorSelector
EnergyEnergy NeXusWriterNeXusWritertimestimes
instrument infoinstrument info
raw countsraw counts
filenamefilename
time intervaltime interval
energy binsenergy bins filenamefilename
One Code, Many Interfaces…• DANSE applications are designed to be UI agnostic. A clean separation
between UI and application code allows for:– maximum code reusability
– the use of automated tools for rapid UI construction• automated form and wizard builder
• widget and view configuration
• same look-and-feel generated for a web and GUI application
• DANSE applications can be launched:– from command line executables; or built dynamically in the command shell
– from within a dashboard UI; or built dynamically in a Visual Programming Environment
Broad Scope of Applications• data reduction and experiment simulation
– diffraction, engineering diffraction, and inelastic scattering data reduction
– SANS/USANS and neutron reflectometry data reduction
– instrument and microstructure simulation
• modeling– full profile modeling in real and reciprocal space (GSAS, FullProf, PDFFIT)
– finite element modeling (ABAQUS); self-consistent modeling
– constrained fitting by use of data from other experimental techniques
– 1D/2D model fitting; model independent peak fitting
– direct modeling of physical systems; ab-initio modeling
– scattering kernel; multiple scattering
– neutron weight correction; separation of nuclear and spin scattering
– micromagnetic simulations (OOMMF); disordered spin dynamics
– chemical spectroscopy dynamics (CLIMAX)
Facilitates New & Better Science• better data analysis
– FEM calculations of strains in microstructures
– Monte-Carlo inversions of S(Q,E) to obtain parameters of structure and dynamics models
– model refinements with multiple data sets
• integration of theory– micromechanics using correlations of local strains
– phase diagrams from thermodynamic functions
– ab-initio calculations of spin interactions
– soft matter structure using atomic force fields guided by diffraction
• experiment planning and execution– single crystals on chopper spectrometers
– feedback control and real-time assessment
– plausibility testing and contingency planning
– assessment of science/data trends from previous data
DANSE cycle of life• Every subgroup is interested in the same analysis steps, we should be
able to reuse much of the analysis…
Raw data kernel Real space
reduction inversion
simulation fitting
Part I: Overview of DANSE
Part II: The Scientist’s Toolkit
Building a Scientific Toolkit…• Through DANSE, scientists will have access to many tools
– basic data structures, optimization algorithms, numerical libraries
– basic data reduction library [obtain I(Q), S(Q), S(E), S(Q,E)]
– graphical/plotting environments• IDL, Matlab, Matplotlib, Gnuplot, Grace, ParaView, ACIS (AutoCAD), …
– instrument simulation• McStas, VITESS, sample simulation framework, …
– materials simulation• ABINIT, VASP, GAMESS, NWChen, NAMD, CHARMM, …
– crystallography• cctbx, FOX, ObjCryst++, …
– molecular viewers and format translators• OpenBabel, Molden, PyMol, ViewMol, DRAWxtl, VMD, AtomEye, …
– and MORE!• ISAW, texture analysis (MAUD), SLD calculator, scattering intensity, …
• These tools will be available in DANSE as component libraries
Integration Strategy for Data Reduction
• In October 2006, DANSE & SNS teams began to work
together to define a strategy for data reduction and analysis
• SNS to provide basic data reduction for all SNS instruments• DANSE will couple reduction to analysis and simulation
- post-reduction corrections
- analysis tools used in reduction
• DANSE & SNS must work together to- ensure each other’s requirements are met
- standardize data structures & define a clear interface
- leverage each other’s effort
Several Tools Requested• In November 2006, DANSE representatives met with several SNS instrument
scientists to collect input on desired analysis tools.
• GUI Builder– extract user-settable fields and defaults from applications & components
– configure view & widgets with simple XML description
• Peak Fitter– obtain peak information from noisy data
– allow selection of fitting methods
• Detector Viewer– examine raw data
– look for bad detectors, create masks
• Histogram Viewer– provide physical representation at each reduction step
– aid visualization of slicing and complex operations
Collaboration Through Modular Design
• API creates standardized
interface, and facilitates
modular design
• Transforms from either
SNS or DANSE library
• Standardized data objects
(histograms) passed
between each transform Histogram Viewer
SNS Reduction Transform
SNS Reduction Transform
DANSE Reduction TransformAPI
DANSE Rebinner
API
API
histogram
histogram
histogram
histogram
• Histogram is a fundamental data structure for data reduction
• Reduction can be represented by mapping an I(pixel,tof) histogram to a S(Q,E) histogram over the instrument information [inelastic]
Extensibility by Design Abstraction
Raw Data
Histogram HistogramMapping
Making Reduction Instrument-Agnostic
• Can we separate instrument-specific parts of data reduction,
so that transformations are reusable across instruments?
• First, we identify & isolate instrument-specific pieces- instrument definition migrated to instrument data object
- extracting histograms from instrument run files dissociated to measurement
• Then, we reconnect with abstract interfaces
• Reduction & simulation become instrument-independent- supporting a new instrument now may be very simple…
- add an instrument description & new class to extract data from run files
Standardized Reduction Data Objects
Define Reduction & Standardize Interface
Extend Reduction to Inelastic Instruments
A Closer Look at Instrument• Instrument is a reusable instrument definition
– is built from available instrument elements and their relative positions
– element descriptions are maintained separately from element position
– instrument description may be obtained from NeXus file
• Instrument description is a hierarchy of elements– FermiChopper, T0Chopper, Moderator, Monitor, DetectorArray, DetectorTube, Guide,
– SampleEnvironment: Cryomagnet, Cryogoniometer, …
• Instrument Viewer– provides a 3-D representation of
instrument or instrument elements
• Instrument Builder– drag-and-drop interface for instrument construction (proposed)
Building an Instrument
moderator monitor
Monte-Carlo Instrument Simulation
• Simulation framework connects
sources, samples, detectors, and other
instrument elements- by providing a buffer to handle neutron flow
between components
- by providing an engine to drive the neutrons
• Connecting through the neutron
component API allows exchange of
newly built components with those
provided by McStas, Vitess, …
moderator monitor
McStas moderator
python bindingsneutron component API
Sample Simulation• composite sample assembly required
– a sample is usually inside some kind of container
– a sample simulation needs to take into account a collection of scatterers including the sample and other objects
• composite scatterers also required– available sample simulations usually focus on one kind of scattering mechanism
– a full simulation should take into account all possible scattering mechanisms with similar scattering strength
• clean separation of physics properties and geometry– reuse geometer as coordinate environment
– shapes & sample description
– random number generator
– scattering path calculator
sample container
samplesample
Building a Composite Scatterer
Coherent inelastic phonon scattering kernel
Sample Assembly
Collection of scatterers
Aluminum Can
Collection of scattering kernels
Incoherent inelastic phonon scattering kernel
Fcc Ni Sample
Shape
Collection of scattering kernels
Shapepossible kernels: - phonons, magnons - elastic, inelastic - coherent, incoherent - …
kernels providescattering probability
physics is containedin scattering kernels
Define Instrument & Sample Simulation
DANSE Project Information• milestones for the DANSE software
– project start 2006
– beta release 2008
– release 1.0 2009
– transition to community/SNS 2011
• documentation, tutorials, and further information– the DANSE wiki at http://wiki.cacr.caltech.edu/danse
– the DANSE trac page at http://danse.us/trac/tickets/wiki
– the ARCS software page at http://www.cacr.caltech.edu/projects/ARCS/Software.html
– the Pyre homepage at http://www.cacr.caltech.edu/projects/pyre
• contacts– Brent Fultz [email protected]; Michael Aivazis, Ian Anderson
– Simon Billinge, Ersan Üstündag, Paul Butler, Paul Kienzle, Tom Swain
– Michael McKerns [email protected]
End Presentation
