Innovative services and Innovative services and tools for NorduGrid (NGIn)tools for NorduGrid (NGIn)

Nordunet 2008, 11.04.08Nordunet 2008, 11.04.08

Jon Kerr Nilsene-mail: j.k.nilsen[at]fys.uio.noUniversity of Oslo, Department of PhysicsPhD student, NGIn

04/10/23 www.nordugrid.org 2

The NorduGrid The NorduGrid CollaborationCollaboration

Nordic Grid Neighbourhood

A research A research collaboration collaboration that:that:

Develops and Develops and supports ARC supports ARC middlewaremiddlewareProvides Provides middleware to middleware to research groups research groups and national and national Grids Grids Coordinates Coordinates the various the various inputs to the inputs to the KnowARC codeKnowARC code

A research A research collaboration collaboration that:that:

Develops and Develops and supports ARC supports ARC middlewaremiddlewareProvides Provides middleware to middleware to research groups research groups and national and national Grids Grids Coordinates Coordinates the various the various inputs to the inputs to the KnowARC codeKnowARC code

10/04/23 www.nordugrid.org 3

ARC middleware deployed on 50+ sites in 15+ countries all around the world

http://www.nordugrid.org/monitor

Large Hadron ColliderLarge Hadron Collider

10/04/23 www.nordugrid.org 4

40 million events/s per each of 4 experiments

Reduced to few hundred ”good” events/s

~15 petabytes of data per year

10/04/23 www.nordugrid.org 6

Nordic Data Grid Nordic Data Grid Facility Facility

NDGF pilot project started 2003Goal: Create a Nordic Grid infrastructureBuilds on history and competence of Nordic Grid collaborationNorduGrid/ARC middleware chosen as basisFunded (2 M.EUR/year) by National Research Councils of the Nordic countriesCoordinates and hosts major Grid projects (e.g. the Nordic LHC Tier-1)Operates Nordic storage facility for major projectsDevelops grid middleware and services

NOS-NDK SF N S

Nordic Data Grid Facility

10/04/23 www.nordugrid.org 8

Univ. of Oslo NorwayUniv. of Lund SwedenUniv. of Copenhagen DenmarkUniv. of Uppsala SwedenNIIF HungaryGeneva Univ. Hospitals SwitzerlandUniv. of Kosice SlovakiaUniv. of Lübeck Germanyscience+computing ag GermanySUN Microsystems Hungary

NorduGrid members and research teams in medicine, bioinformatics, physics, engineering, automotive industry apps, IT

EU KnowARC project:EU KnowARC project: 10 partners from 7 10 partners from 7 countriescountries

10/04/23 www.nordugrid.org 9

Goals of KnowARC Goals of KnowARC The mission of KnowARC is to create a novel, powerful Next

Generation Grid middleware– extend and re-design ARC– ARC in major Linux distributions– New platforms: Windows, Solaris,

Mac OS-X to promote Grid standardization and

interoperability– interoperability with other Grid

solutions (EGEE and OGF compliant grids)

– possibilities to include (or to be included in) other infrastructures

to prove usage in Health care, Industry and Science– Bioinformatics: Autoimmune

Diseases– Medical Informatics: Lung Diseases– Automotive industry

Next Generation ARC Next Generation ARC

10/04/23 www.nordugrid.org 10

Developed using SOA and existing and comming standards

Web Services, SOAP, GLUE2.0 ...

Non-intrusive (no need to change excisting computing resources)Client-Server communication mainly through Web Services

DENMARK:

University of Copenhagen (NBI, DIKU)

Southern Danish University, Dep. of Computer Science

Aalborg University, Dep. of Computer Science

SWEDEN:

Lund University, Department of Physics

Linköping University, National Supercomputer Center

Umeå University, Department of Computing Science and HPC2N

Uppsala University, Department of Radiation Sciences

Royal Institute of Technology, NADA/PDC, KTH

Stockholm University and KTH, Department of Physics

NORWAY:

University of Oslo, Physics institute

University of Bergen, Department of Physics

University of Tromsø, Department of Computer Science

NTNU, Trondheim, Department of Computer and Information Science

NTNU, Trondheim, IT division (ITEA)

RUSSIA:

St. Petersburg State University, Dep. of Physics

Petrodvorets Telecommunication Center

Dubna?

ESTONIA:

National Institute of Chemical Physics and Biophysics (NICPB)

Estonian Educational and Research Network (EENet)

Tartu University, Institute of Technology

FINLAND:

University of Helsinki, Helsinki Institute of Physics

LITHUANIA:

Vilnius University, Institute of Materials Science and Applied Research

LATVIA

ICELAND

Nordic Grid Neighbourhood

... 9 countries, 24 institutions

Project: … , NO27, Nabo 05-06, Nordic Grid Neighbourhood Project no.: 172724/V11 We have read and approved your final academic and financial project report for the funding period 2005-2006. We are pleased that most of the planned goals of the network have been reached through, i.a. workshops, conferences and tutorials. We are impressed by the number of partners and participants in the various activities.

10/04/23 www.nordugrid.org 12

http://www.nordugrid.org/ngin http://www.nordunet3.org/

•Innovative services and tools for NorduGrid. PI: Farid Ould-Saada, University of Oslo.

4 year grantOslo, Lund, Uppsala, Copenhagen, Helsinki, ReykjavikMainly:

Data Management: storage, indexing and movement Education and expert training in Grid computing

5 PhD grants starting January 2007Grid school

call for collaboration with Baltic and Russia … ??

NGIn for Research and Education

Nordunet3-funded Internet Research Program

NGIn managementNGIn management

10/04/23 www.nordugrid.org 13

Project leaderProf. Farid Ould-Saada, Univ. of Oslo, Norway

Technical coordinatorProf. Alex Read, Univ. of Oslo, Norway

Steering GroupProf. Farid Ould-Saada, Univ. of Oslo, NorwayProf. Paula Eerola, Univ. of Lund, SwedenProf. Tord Ekelöf, Univ. of UppsalaDr. Miika Tuisku, Helsinki Institute of Physics, FinlandDr. Anders Wäänänen, Niels Bohr Institute, Denmark

Participating institutionsPhysics Department, Univ. of Oslo, NorwayPhysics Department, Univ. of Lund, SwedenPhysics Department, Univ. of Uppsala, SwedenDepartment of Radiation Sciences, Univ. of Uppsala, SwedenNiels Bohr Institute, Univ. of Copenhagen and Univ. of Aalborg, DenmarkHelsinki Institute of Physics (HIP), Univ. of Helsinki, FinlandCSC, Finnish IT Centre for Science, Helsinki, FinlandUniv. of Iceland, Reykjavik, Iceland

NGIn - training new Grid NGIn - training new Grid expertsexperts

10/04/23 www.nordugrid.org 14

From the original application: ”The goal of this project is twofold: To extend the excisting middleware and to train new Grid experts...”NGIn obtained 30% of funding applied forThe primary goal of the NGIn project is now to train new Grid expertsTraining programme includes:

Grid PhD studentsA Nordic Grid school (first held in Copenhagen in September 2007)

The NGIn activities will go in line with the other NorduGrid related projectsBudget 2008:

PhD students: 160 k€

Researcer training: 15 k€

Other expenses: 3 k€

Total: 178 k€

NGIn PhD studentsNGIn PhD students

10/04/23 www.nordugrid.org 15

Four PhD students hired, fifth student soon to be hiredIn chronological order:

PhD 1, Jon Kerr Nilsen, Univ. of Oslo, NorwayPhD 2, Mika Silander, Helsinki Institute of Physics, FinlandPhD 3, Salman Toor, Univ. of Uppsala, SwedenPhD 4, Phillippe Gros, Univ. of Lund, SwedenPhD 5, to be announced soon, Univ. of Oslo, Norway

The students are funded 50% by Nordunet3, 50% by local sourcesAll the PhD projects involves data management

Jon Kerr Nilsen, OsloJon Kerr Nilsen, Oslo

10/04/23 www.nordugrid.org 16

Thesis title: Data Management in Grid Middleware

Address limitations and implement solutions in data management and data handling in ARC middlewareUse the solutions in ATLAS data analysisWork in collaboration with both NGIn and the Scientific Computing Group at Univ. of Oslo

Current statusGot familiar with ARC and data management (and Grid acronyms)Survey on data management in conjunction with Grid technology presented at NorduGrid2007Set up a test cluster with dCache and ARC Storage Element (in collaboration with Salman Toor)

Near futureStorage in conjunction with virtualization

Virtualization provides possibilities to run Linux applications in WindowsChallenge to get data out of virtual machine and store them securely and efficiently

Client side of the next generation ARC storage solution

Will work with KnowARC in implementing client-side tools for accessing data in the next generation ARC

Look at data management of local versus external dCache instances in current ARC

Is there any gain in prefering dCache instances in the same network as the computing resource?

Jon Kerr Nilsen, OsloJon Kerr Nilsen, Oslo

10/04/23 www.nordugrid.org 17

27/03/08 www.nordugrid.org 18

Problem: delegation is too coarse-grained in current middleware, e.g. Users delegate all rights (Impersonation) Limited proxy certificates (exclude shell access) Delegating the least set of rights is desirable

Work so far on Analysing ARC components for types of operations

and objects for rights modeling Analysing SAML for extending it to support fine-

grained rights expressions Evaluation of existing solutions with applicable

features, e.g. SecPAL

Thesis title: Fine-grained rights delegation in Grids

Mika Silander, HelsinkiMika Silander, Helsinki

27/03/08 www.nordugrid.org 19

Potential solution candidates SAML + XACML

Pros: other Grid mw committed to these standards

Cons: complexity in policy definition and evaluation, fine-grained rights not modeled

Security Policy Assertion Language (SecPAL) Pros: broad set of security functionality

including rights restrictions, policies expressed with easily understandable syntax

Cons: restrictive license and implementation languages

Mika Silander, HelsinkiMika Silander, Helsinki

27/03/08 www.nordugrid.org 20

Near future plansNear future plans

Near future plans Explore possibility of combining pros of the two

former for ARC: Easily understandable syntax similar to

SecPAL, but, Implementation relying on C/C++ and

adhering to pure SAML & XACML Ensure this is compatible with the security

architecture of ARC v1.x being developed Evaluate SAML + XACML solution of gLite

Applicable, parts reusable?

Salman Toor, UppsalaSalman Toor, Uppsala

10/04/23 www.nordugrid.org 21

Thesis title: Data indexing and data movement in grid environments

Topic: Management of scientific data in Grid systemsMaking a Grid Storage Resource Manager

A layer on top of various Storage Managers (dCache, DPM, Castor, GPFS...)A complete cost based estimation system to find the best SM amongst the SMs registered

Salman Toor, UppsalaSalman Toor, Uppsala

10/04/23 www.nordugrid.org 22

ProgressDeployed dCache test site in Uppsala (with a pool-node in Oslo)Started study of SRM specifications in conjunction with different Storage ManagersDeployed the full Globus toolkit and implemented test services and clients

Next step write a service that

gather the information (owner, permissions etc) of data stored in dCachereplicate the data in a different Storage Manager

Philippe Gros, LundPhilippe Gros, Lund

Physics oriented project become an expert user of AliEn

more Grid oriented project work on the AliEn-ARC integration

Current status: AliEn expertise

still a basic user, which is already a first step(submitting jobs on the Grid)

AliEn-ARC integration project formulated recently studying the current state, existing simple interface

Thesis title: Analysis and management of high energy physics data with NorduGrid

NGIn Grid SchoolNGIn Grid School

10/04/23 www.nordugrid.org 24

The NGIn training programme includes:Grid PhD studentsA Nordic Grid school

The NorduGrid 2007 Grid school was held at Niels Bohr Institute in Copenhagen 26-27 September 2007Organized by CSC, Finland

More than 35 present for the introductory lectures on first dayTutorials and demos had 15 participantsTutorials in topics from High Energy Physics to bio-informatics and medical imagingFrom beginner to specialized user in two days!Several participants spotted on the Grid later on!

NGIn School programNGIn School program

Day 1 (General Grid Introduction) Intro to Grid Intro to ARC First steps with ARC (tutorials)

Day 2 (Specialized tutorials) HEP distributed analysis ARC in bioinformatics, ARC in medical imaging ARC-gLite interoperability Grid Job Manager Dynamic Runtime Environments

Day 3 (ARC Development) Sys-admin and developer training

Next Grid SchoolNext Grid School

Next Grid School planned to be held during NorduGrid2008 in Budapest, Fall 2008

Similar program as in the first school

See you there!

SummarySummary

10/04/23 www.nordugrid.org 27

NorduGrid is a collaboration developing and deploying the ARC middlewareNDGF provides infrastructure and deploys and contributes to the current ARCKnowARC develops the next generation ARC NGIn educates and trains new Grid experts, both through PhD projects and Grid schoolsThe PhD projects contributes to both current ARC and the next generation ARCNext Grid school in Budapest, Fall 2008

Positions available in Positions available in NorduGrid!NorduGrid!

10/04/23 www.nordugrid.org 28