ICAT Integration at ISIS

Tom Griffin, ISIS FacilityICAT Developer WorkshopThe Cosener’s House, AbingdonAugust 2009tom.griffin@stfc.ac.uk

ISIS• Neutrons and Muons• Operating since 1984• 1600 user base; ~200 staff• Volume of data ~6TB• ~5M datafiles (27 instruments,

500/hour)• >6.7GB metadata, >33M rows

Current Pre-Experiment

Data.ISIS

Data.ISIS• Main point of access for ISIS data• Maintains index of Run number -> Folder

– GEM12345– \\isis\inst$\NDXGEM\Instrument\data\cycle_01_1\GEM12345.raw– .txt .log .sav0

• Ranges• Zip files

Data.ISIS Authentication

Data Portal

ICAT API

Data.ISIS

User performs search via application e.g. Data PortalSearch is executed in ICATPermitted results are returned to application

1

1

2

3

3

2

Results are displayed to the user4

4

5

User performs request to download datafile, multiple datafiles or dataset

5

6

ICAT creates http GET link and passes to back to user (routed through application)

sessionIdemail (optional)fileId(s) or datasetIdaction (i.e. download, zip,

compressed)

6

User clicks http link7

7

Data.ISIS call ICAT API to check permissionssessionId & datafileId(s) or

datasetId

8

8

Return Exception on failure or DownloadObject on success- userId- array [filename, cycle, run

number]

9

9

User gets their data!10

10

Current Pre-Experiment

‘To Be’ Pre-Experiment

Problems• RB Numbers• Business System

• Not released to users• Replace DataPortal with TopCat

Related Projects• Grid MP – distributed PC network• Analysis and simulation• New portal required anyway ->

1) Single point of access for Grid MP2) Use ISIS data in Grid jobs3) Store results and process in ICATWhere to store data?

Freddie Akeroyd (Data.ISIS)Chris Moreton-Smith (ISIS DFS/FRS)Hannah Guest (Facilities Business System & ERAS)Matt Clarke (SECI modifications)

Damian FlanneryStuart Ansell (writeRaw)

Laurent Lerusse (NXIngest)

Database Services, Roger Downing, FBU IT Service Desk (servers and hosting)

Core ICAT – Glen, Shoaib, Michael.....

Acknowledgments(who did what at ISIS)

ISIS Computing Group

ISIS Data Analysis Group

Questions...