ISIS: towards a 21st century facility computing environment

(slowly)

Robert McGreevy

Science and Technology Facilities Council

ISIS, Rutherford Appleton Laboratory, Harwell Science and Innovation Campus, Didcot, OX11 0QX, UK

3rd November 2008

Source

Instruments

Sample environment

Data treatment

Scientific results

Source

Instruments

Sample environment

Data treatment

Scientific results

30 years ago …

ISIS Target Station 1

ISIS Target Station 2

StructuresNIMROD Intermediate

range order in liquidsWISH High-resolution

magnetic structureSANS2D Large

molecule structure in multi-component

systems

ReflectometryINTER Air/ liquid/ solid interface interactionsOFFSPEC Structures of membrane, protein and liquid interfacesPOLREF Interface measurements in magnetic sensor devices

DynamicsLET High-resolution measurement of material energy scales

TS2 Instruments Phase 1

StructuresNIMROD Intermediate range order in liquidsLMX Large molecule single crystal diffractionZOOM Focusing small angle scatteringIMAT Imaging / diffraction

Structure/DynamicsLARMOR SESANS, High resolution diffraction, MIEZE, MISANS

TS2 Instruments Phase2

Understanding

Data analysis

Data visualisation

Data treatment

Data diagnostics

Data acquisition

Measurement

The data chain

Data curation Data (scientific)

Data (raw)

Data (publication)

Data (metadata) Software

Hardware

Understanding

Data analysis

Data visualisation

Data treatment

Data diagnostics

Data acquisition

Measurement

Data curation Data (scientific)

Data (raw)

Data (publication)

Data (metadata)Fac

ility

User

Software

Hardware

The data chain

IdeasKnowledge

andUnderstanding

Exploitation(Economic Impact)

GRIDHPC

ES

S

XFE

LILL

ES

RF

IRU

VX

HPC

DA

TA

Time to Market

Proposal system

Fromsingle instrument, single experiment

tomulti-instrument + support facilities + computing

Damian Flannery, Tom Griffin, Hannah Guest

Experiment (controls, electronics, data formats)

Freddie Akeroyd (NeXus)

User Interface, DAE-III

Data catalogue, Data curation, Data Policy

Damian Flannery (ICAT)

Single sign-on

Data reduction/analysis – MANTID

Nick Draper

Data analysis – software ‘instrument scientists’

EPSR, Single crystal inelastic, RMC …

Data analysis – modelling/simulation

Simulation – source and instruments

Tom Griffin (GRID-MP)

SCARF (UK GRID) - CASTEP, Tobyfit

McStas

StructuresNIMROD Intermediate range order in liquidsLMX Large molecule single crystal diffractionZOOM Focusing small angle scatteringIMAT Imaging / diffraction

Structure/DynamicsLARMOR SESANS, High resolution diffraction, MIEZE, MISANS

TS2 Instruments Phase2

Software infrastructure

MANTID Bio-SANS

Single crystalImaging/enginering

Gateway Centres

Imaging Solutions Centre, Hartree Centre

Collaboration

European Data Infrastructure for Neutron and Photon Sources

STFC, ESRF, ILL, DLS, PSI, DESY, ELETTRA, SOLEIL, ALBA, BESSY

What will Mantid do?

Mantid– Manipulation and Analysis Toolkit for ISIS Data

Aims– To provide a framework for Data Analysis that is not

instrument or technique/dependent.– To support multiple target platforms (Windows, Linux).– The framework must be easily extensible by Instruments

Scientists/Users.– The framework must be freely redistributable to visiting

scientists.– The framework should provide low-level functionalities

for Scripting, Visualization, Data transformation, Implementing Algorithms, Virtual Instrument Geometry.

Top Requirements

Easily extensible.– Support all current and future analysis.– Support current and future file formats.– Provide a simple but powerful objects and

services to support user created algorithm code.No user license costs.Supportable.Portable.

– Operating System (Windows, Linux).– Computing Power (Laptop, Server).

Architectural Design - Overview

Mantid Framework

Command line & Scripting interface

QTIPlot Graphing tool

RAW data files

NEXUS data files

Future Data analysis GUI

Instrument log files

APIStandard

Algorithms

User Defined Algorithms

Matlab Interface

DAE direct access

Development Process

Iterative development – Why?– Adaptability - the ability to rapidly respond to

changes in strategy, priorities, and plans – Value - continuous delivery of more useful

functionality – Visibility - stakeholder collaboration and

validation throughout the development life-cycle

– Risk - the reduction in overall project risk as a result of #1-3 above

Software led by Scientists

This is project is intended to support the needs of all of the beamlines.

Scientific Steering Committee– To ensure input from each group of instruments

• To ensure what we build fits what you need.• To ensure good communication between the

Mantid project and the scientists.• To allow each group to have equal input into the

project.– Prioritise the list of tasks to be done over the next

iterations.– Review the progress of each iteration.

Milestones– Architecture design completed October 2007

• Completed on time

– Prototype framework released: March 2008 • Released to a limited audience in May 2008• Purposefully delayed to allow for additional

functionality

– Functional framework + 2-3 complete applications released March 2009

Timeline

What Mantid can do now

Data Input/Export– Read Raw files and associated log files– Read Instrument definition files (Mantid XML format)– Nexus files (limited support – data only)

Internal Data Structures– 1D and 2D workspaces– Managed 2D workspaces for large volumes of data

Instrument Definition– Sufficient definition to allow conversion of units– Loading instrument definition from XML files or RAW files

What Mantid can do now

Algorithms– Simple binary mathematical operations

• +, -, *, /– Integration– Rebin– ConvertUnits

• TOF, , energy, d-Spacing, q, Q2, Energy transfer (meV and 1/cm)

– Grouping of detectors– Marking detectors as dead

User Interfaces

Early Prototypes– QTIPlot-Mantid

• A graphical plotting tool which interfaces directly to Mantid for data interaction and analysis.

– Python• Command line access and scripting

Proof of concept– Matlab Interface

QTIPLOT-MantidLoading a Raw File

QTIPLOT-MantidContour and 3D Plots

QTIPLOT-MantidLine graphs of individual detectors

Matlab Interface