Int'l HEP Data Grid Work Shop in Daegu 1 Status and Prospective in Japan KEK High Energy Accelerator...

Int'l HEP Data Grid Work Shop in Daegu

1

Status and Prospective in Japan

KEK High Energy Accelerator Research Organization

Atsushi Manabe

Computing Research Center (CRC)

KEK


2

Outline

• Grid projects (budget) in Japan academic region.

• HEP Data Grid activities in Japan.– KEK

• Needs in KEK• Strategy of KEK CRC• Pacifi-Grid• Gfarm project• Grid/HPSS test bed on SupserSINET.

– ICEPP: ATLAS Japan Regional Center


3

Grid Projects(budget) in Japan (academic area only)

• MEXT(Ministry of Education, sports …. and technology) projects– SuperSINET; Gigabit backbone for academic researchers in Japan

– MEXT Informatics Project

• A05: Application Grid projects for science ~ US$ 6M /5 y Astronomy (Virtual Observatory), io-Infomatics/Genome (BioGrid) ……

– Networking supercomputers: ITBL ~US$ 100M /5y– National Research Grid Initiative ~US$ 300M/5y

focused in Nano-tech application– (LHC/ATLAS(ICEPP,KEK)

• METI (Ministry of Economy, Trade and Industry) project– AIST GTRC(Grid Technology Research Center) ~ US$ 140M


4

Gigabit network

• Super SINET– 10 Gbps MEXT DWDM nationwide network– About 40 universities and labs – US$ 60M/yr– 5 application areas: HEP, Nanotech,

Bioinformatics, Astronomy, Grid Computing

• Tsukuba-WAN– 330Gbps DWDM ring in the Tsukuba area– Tens of national labs– 6 supercomputers in TOP100– Testbed for Grid infrastructure and applications

http://www.sinet.ad.jp/english/super_sinet.html


5

Nagoya Hub

Internet

Osaka U

Kyoto U

Nagoya U

NIFS

NIG

KEK

Tohoku U

IMSU-Tokyo

U Tokyo

NII

IP 10 GbpsWDM path

IP router

OXC

Tokyo HubOsaka Hub

ICRKyoto-U

NII Chiba

US， EU

Waseda

TITEC

Tsukuba

Hokkaido

IMS

Kyushu　 UKorea

NAO

ISAS

Network Topology of SuperSINET


6

SuperSINET Backbone

Titech

KEK

ICEPP

7Int'l HEP Data Grid Work Shop in Daegu

Network InfrastructureNII- NY

TRANSPAC

Taiwan-USJapan (NII) -NY : 2.4G x 2 Jan. 2003Japan – US: 622M x 2 (TRANSPAC)Korea – US: 45 MKorea – Japan: 2.4G Jan. 2003Korea – Europe: 2 M (TEIN)China(IHEP) – Japan(KEK): 128 kbps (HEP)China – US: 10 MTaiwan – Japan: 155 MTaiwan – US: 622 M (Dec. 2002)

TEIN


8

MEXT Grid Projects• MEXT “Informatics” Project : A05 “Grid” Area

– Lead by Shinji Shimozo(Osaka U.) & Satoshi Matsuoka(Titech)– 11 groups of universities and labs– Research and development on application, middleware, and infrastruct

ure– Support the Virtual Observatory(astronomy) and the BIO

-Grid(bio-Infomatics/Genome) projects

• Tokyo Institute of Technology “Titech Campus Grid” (US$ 2M)

– Lead by Satoshi Matsuoka(Titech)– Total of 800 CPU PC clusters at 13 locations in a 30km range connect

ed by Super TITANET(1-4Gbps) backbone– Pilot project for virtual parallel computer infrastructure for high end co

mputational e-Science


9

ITBL• ITBL(IT-based Laboratory)

– To Connect Super-Computers in Government Labs: NAL, RIKEN, NIED, NIMS, JST, JAERI

– Project period: 2001-2005 (3-stage project) with total of US$ 105M funding

– Software development project.– Applications: mechanical simulation, computational biology, material

science, environment, earthquake engineering– Step 1: Supercomputer centers of government lab are

networked via SuperSINET– Step 2: “Virtual Research Environment”: Grid-enabling

laboratory applications– Step 3: Sharing information among researchers from widely

distributed disciplines and institutions


10

NAREGI (National Research Grid Initiative)

• NII (The National Institute of Informatics) under MEXT project.

• Making Center of Grid Research & Development– Development of Grid Lower and Middleware

(Grid scheduler,VM, RPC, MPI, GUI tools…)

– Networking, Security and User management(Routing, robust TCP/IP protocol, Traffic management)

– Applications for Nano-science and technology.Joint with Institute for Molecular Science.

– http://www.grid.nii.ac.jp/


11

Grid Technology Research Center• Part of AIST under METI (Ministry of Economy, Trade and Industry) • GTRC (Grid Technology Research Center)

– Established in Jan. 2002– Director: Satoshi Sekiguchi

• US$140M for building, equipment, R&D and operations

• AIST super server; PC cluster2004-2Q 11TFLOPS/ AMD Operon x 2116CPUs

• To be central institute for Grid research in Japan– Not just R&D, but serve as central CA, ApGrid’s principal site– Collaboration with institutions and other Grid projects of both domestic and

international scale– Organizing GRID Consortium Japan

– http://unit.aist.go.jp/grid/

http://unit.aist.go.jp/grid/


12

Grid related Activity at KEK


13

• KEK sponsors projects which use accelerators and also projects in related fields– KEKB accelerator

• BELLE experiment

– Proton synchrotron • nuclear and elementary particle physics • medical

– Synchrotron Radiation Facility • bio, material science and so on

– Also many theorists in related fields

• Support International HEP collaboration– ATLAS Japan

Major projects in KEK


14

KEK Future projects• J-PARC

– will be built in Tokai div. of Japan Atomic Energy Research Institute (JAERI) where ~50km north from KEK start the operation in JFY2005

– 50 GeV high-intensity proton synchrotron – nuclear physics and elementary particle physics

• Super B factory– x30 Luminosity(require x10 CPU power). Under

discussion.

• Linear Collider – discussion is still under way


15

Computing Facilities at KEK

• Computing Research Center offers group oriented computing services – Exclusive computing resource for main groups

• Networking, storage and computing resource

• User group owns their own computer systems – PC farms


16

Main computing facilities

• KEKCC computer system – for experiments at PS, linear collider R&D, theory, AT

LAS and so on• IBM SP3, 120TB HPSS storage

• KEKB computer system for Belle– Sun WSs and PC farms. 500TB storage.

• KEKSC: Super-computer system for mainly lattice QCD G.– Hitachi SR-8000 100 nodes; almost 1TFLOPTS


17

Our needs in Grid Era

• ATLAS Regional Center (Grid is premise)– Being set up at ICEPP (Int’l Center for Particle Physics, U.

Tokyo)

– Still KEK have large portion of ATLAS Japan members and they will make data analysis using computing resource at KEK. We will support them.

• And– BELLE; international collaboration and require more

and more CPU and storage.– J-PARC; joint project of other Inst. regionally

separated.


18

BELLE

• BELLE is the major project at KEK• They have many collaborators world wide • They always needs additional CPU power because

acc. luminosity improving and improving.• They seek a solution of their problems related com

puting issue using GRID. • We interviewed the BELLE computing coordinato

r and got the wish list shown in the following slides


19

Wish lists from BELLE• What BELLE wants:

– A way to integrate CPU powers and way to distribute data among collaborators’ Univ.s and Inst.s for their data production.

– Automatic way to submit analysis jobs in the distributed environment.

• once analysis codes have placed somewhere, process for all of experimental data and Monte Carlo should be done automatically and simply users gain the result

– Automatic Monte Carlo production – Automatic version control for libraries, constants and others on

all of the computer system in BELLE – Automatic update on reconstructed data or Monte Carlo if

libraries or constants are once updated


20

Wish lists from BELLE 2

• It is not easy to achieve their requests technically – simplest solution will be hire more Post-Docs

or find more grad-students, however

• We have decomposed their requests – should be done in application level (BELLE)or

– can be solved by GRID


21

What they need related GRID• An efficient way to distribute jobs and way to

manage/share/distribute data for the jobs.– effective data catalogue

• physical to logical mapping • Seamless and unique file tree• query

– file names, contents and others

• works on the Internet but secure

– efficient and easy to distribute data to remote sites– easy to use

• web interface and GUI also


22

continued

• mass storage management system– not only tapes but also large amount of disks – robust and fast transfer – easy integration with other component

• We already have them partially on LAN, but we need to expand them to the Internet. We hope GRID will solve them.


23

KEK CRC strategy towards GRID


24

Done or On going.• Install GT2(Globus toolkit ver.2) and applications (LSF)

to two CRC Systems (KEKCC & KEKB) ) and will open to some limited users. (remote Job submission)

• Build temporal CA with single Virtual Organization(VO).

• Build a inter-institute wide Grid test bed on SuperSINET and make a performance test. (talk later)

• Trial use of SRB with Belle data in a test system.• Start to touch LCG tools by ATLAS Japan people.• watch technical trends carefully.


25

SRB

• SRB is available today and looks useful even in HEP ( http://www.npaci.edu/DICE/SRB/ )– developed at San Diego Super Computer Center – seamless solution to HPSS

• good for the KEKCC (central computer) users

– Data catalogue and replication– – SLAC is also testing

http://www.npaci.edu/DICE/SRB/


26

Until early 2004• Deploy GT3, evaluate LCG tools and other something new tec

hnology.• Build own VO at each sites and integrate them.• File sharing by SRB (or ?) and seamless job submission betwee

n KEKCC and KEKB• Pacifi-Grid test environment;

Pan-Pacific collaboration (see later)– KNU (Korea), Univ. of Melbourne (Australia), SLAC and KEK will t

est SRB

• Start to develop higher level software (experiment side) on the top of Globus and SRB.– Message passing with GSI authentication ?


27

~2004• KEKB and KEKCC (5yeas rental )system

replacement (2005-4Q) .– GRID will become concrete until this stage or

die– Final specification have to be decided

• Decision of middle ware (Grid or else?)• CA • Security • human resource

– trained engineer on site


28

Other topics discussed internally • CA

– nation wide, institution wide or else? KEK certify all Japanese HEP user?– Rules for certification of users.

• PKI– secure enough? Safety private key management (IC card?)– management costs? – seamless integration with local authentication

• Network– How we can overcome latency?– better connectivity to the Internet, routing and throughput

• middleware– Globus, Avaki or something else – SRB, LCG or other data replication solution

• Storage– SAN– StorageTank (IBM)

• security vs usability


29

Pacifi-Grid:Proposal on

Open Pacific Grid collaboration

Takashi Sasaki (KEK)


30

charter• Give research institutions and IT vendors opportunity

of testing Grid technologies under inter-state environment.– Emphasis on Data Grid– Open to everybody who wants to join– Grid is a newly emerging technology and more experiences

are necessary before deploying at the large scale production systems

• Test on high latency network• robustness tests• interoperability tests among sites and vendors • security


31

Collaboration (under discussion)

• Korea• U of Melbourne, Australia• SLAC, USA• JAPAN

– ICEPP (Univ. of Tokyo)

– KEK

• Some venders– Under discussing with Fujitsu, IBM Japan and so on


32

Gfarm R&D

Development R&DGrid middleware of Cluster File System

with job execution.


33

R&D of Gfarm (Grid data farm)• Gfarm

– Development by CS scientist of AIST, Titech and KEK since 2000

• AIST :Advanced Inst. for Science and Technology

– Architecture• PC farm with large local disk/node as a GRID fabric

• Large data file is divided into fragments and stored in the disks

• Data I/O by parallel file system

• Affinity scheduling of process and storage for data residence


34

Data Access Locality

computingnode

storagenode

high speedswitch

jobmanager

jobA

jobB

user job

jobmanager

jobA

jobB

user job

computingnode

storageelement

Exploit the data access locality as much as possible


35

Gfarm• Grid Data Farm : Beta 3 release is now available.• Parallel I/O: exploit the data access locality, store and access files by “fr

agments”• Parallel Job: program runs on the

nodes where the file fragmentsreside: “owner computes”

• Programs and file fragments,job history, data checksum,file replication are managed withMetaDB

• Fragments are replicated forbackup and load balancing

• User sees the fragmented files asa single Gfarm URL

• Provide system call hooks foropen(), close(), read(), write() etc

http://datafarm.apgrid.org/

• Authentication:Globus GSI and/orShared Private Key


36

FADS/Goofy in ATLAS• Used in sub-detector software developments and physics vali

dations• Detector modules also run in ATLAS mainstream framewor

k (Athena)

• Testbed for ROOT I/O• Bandwidth Challenge in

SC2002• Generated 106 fully simulated

higss 4 events in 2 dayswith ~ 400 CPU


37

Gfarm achievement• “File affinity scheduling” provides parallel processi

ng capability for both I/O-bound and CPU-intensive jobs.

• FADS/Goofy was successfully implemented on Gfarm architecture. In the process, ROOT I/O module works with Gfarm parallel file system using the system call hook, without a changes to the ROOT package.

• ROOT I/O files has been successfully replicated at 2.286 Gbps using the SC2002 Bandwidth Challenge testbed with 12 nodes ~ 190Mbps/node.

• Please refer to CHEP03 reports by Y.Morita and O.Tatebe


38

Status of ATLAS Japan RC (ICEPP U. of Tokyo)

• R&D is ongoing – Test-buy of PC farm and storage

• 1U P3, P4 servers ( >150nodes and more) , LTO (10TB) with CASTOR and >20TB disk servers.

– Nordu GRID was tested first • Data transfer tests between U. Tokyo and KEK was

done

– ATLAS data challenges – LCG 0 and 1


39


40


41

LCG-0 Deployment Status

Site Scheduled StatusTier 1

0 CERN 15/2/03 Done

1 CNAF 28/2/03 Done

2 RAL 28/2/03 Done

3 FNAL 30/3/03 Done

4 Taipei 15/4/03 Done

5 FZK 30/4/03 Done

6 IN2P3 7/5/03 In prep.

7 BNL 15/5/03 Done

8 Russia (Moscow) 21/5/03 In prep.

9 Tokyo 21/5/03 Done

Tier 2

10 Legnaro (INFN) After CNAF Done

These sites deployed the LCG-0 pilot system and will be the first sites to deploy LCG-1


42

LCG Resource Commitments – 1Q04 CPU

(kSI2K)Disk TB

Support FTE

Tape TB

CERN 700 160 10.0 1000

Czech Republic 60 5 2.5 5

France 420 81 10.2 540

Germany 207 40 9.0 62

Holland 124 3 4.0 12

Italy 507 60 16.0 100

Japan 220 45 5.0 100

Poland 86 9 5.0 28

Russia 120 30 10.0 40

Taiwan 220 30 4.0 120

Spain 150 30 4.0 100

Sweden 179 40 2.0 40

Switzerland 26 5 2.0 40

UK 1656 226 17.3 295

USA 801 176 15.5 1741

Total 5600 1169 120.0 4223


43


44

Status of ATLAS Japan RC 2

• Budget requests have been submitted to the funding agency – still no decision has been made

• and no decision will be made within this FY– All of national universities and some national laboratories

including KEK will be non-nationalized (not privatized, but will be a kind of independent governmental agencies)


45

Summary of KEK Grid activity

• KEK had a step forward for the Grid computing • People at ATLAS Japan RC are working on LCG • Realistic environment is necessary to test

middleware and applications – We have started the discussion among Pan-Pacific sites

to collaborate each other

– We want to exchange knowledge and experiences


46

End of my presentation

Thank you

Date post:	18-Dec-2015
Category:	Documents
View:	216 times
Download:	2 times

Int'l HEP Data Grid Work Shop in Daegu 1 Status and Prospective in Japan KEK High Energy Accelerator...

Documents