Application of the EDG Application of the EDG TestbedTestbed
Bockjoo Kim*, Soo-Bong KimSeoul National University (SNU)
Kihyeon Cho, Youngdo Oh, Dongchul SonCenter for High Energy Physics (CHEP)
Kyungpook National University
Jysoo LeeKISTI, Supercomputing Center
International HEP DataGrid Workshop
9 Nov 2002
OutlineOutline
LHC Computing Challenge
European DataGrid
European DataGrid (EDG) testbed in Korea
The EDG testbed and tests
The GDMP (Datamanagement) and tests
Possible Immediate Application of the EDG Testbed
Summary
LHC HEP computing, key LHC HEP computing, key parametersparameters
All LHC experiments at CERN: 10 Peta Bytes/yr data storage; disk: 2 P Byte
Multi-experiment Tier 1: 3 Peta Byte/yr; disk: 0.5 P Byte
Tier 0 & 1 at CERN: 2 M SI 95 (PC today ~ 20SI95)
Multi-experiment Tier 1: 0.9 M SI 95
Networking Tier 0 --> Tier 1: 622 Mbps (4 Gbps) (black fibre: 1 Tbps today)
Online Data Rate vs SizeOnline Data Rate vs Size
Level 1 Rate (Hz)
105
104
103
102
High No. ChannelsHigh Bandwidth(500 Gbit/s)
High Data Archive(PetaByte)
LHCB
KLOE
HERA-B
CDF II
CDF
H1ZEUS
UA1
LEP
NA49
ALICE
Event Size (bytes)
104 105 106
ATLASCMS
107
106
It doesn’t…Factor
O(1000)Online datareductionvia trigger selection
“How can this data reach theend user?”
European DataGrid (EDG)European DataGrid (EDG)
WP1: workloadWP2: data managementWP3: grid monitoring servicesWP4: fabric managementWP5: mass storage managementWP6: Integration and testbedWP7: network servicesWP8: HEP applicationWP9: Earth ObservationWP10:BiologyWP11:DisseminationWP12:Management
OS & Net services
LHCVO common application layer
Other apps
ALICE ATLAS CMS LHCbSpecific app layer Other apps
GLOBUS 2.0
EDG middleware
High level GRID middleware
Basic Services
(Globus)
Concept of EDG
To cope with the data challenges in LHC eraParticipants
EC+(Korea+U.S.+Japan+China…)Operational at EC
The EDG Fabric and TestbedThe EDG Fabric and Testbed
CECE//WNWN (PC Cluster)(PC Cluster)
RPMs repositoryRPMs repository
Profile repositoryProfile repository
LCFGServer
SESE(GDMP)(GDMP)
UIUI
RBRB
LCFG : EDG software + Profile + CE, WN, SE, RB, UI installation&confUI : User Interface : Job submission interface for authorized usersCE : Farms of commodity PCs
Globus Resource Access Management (GRAM) service+GSI
RB : Resource Broker : Core of job schedulingSE : Any storage system: accessible via ftp, nfs, etc & GSIWN : Worker CPU hidden underneath CE ~ clusters
ProfilesProfiles
An EDGAn EDG TestbedTestbed
Listens updateListens update
All the machines are linux red hatAll the machines are linux red hat
EDG Important IngredientsEDG Important Ingredients
X.509 Authentication + Users certs on LDAP Grid Information Index Service (GIIS): not fully used. RB does the job
Virtual Organization(VO) : CMS, Atlas, etcVirtual Users and Groups for VOLDAP services for VO, authentication list with edg schema
Replica Catalog (RC) uses also LDAP server with edg schema
Replica Management uses GDMP (grid data mirroring pkg.)
VO ManagementVO Management
Data ManagementData Management
Resource Locating Resource Locating
Security Security
EDG Testbed in KoreaEDG Testbed in Korea
The EDG testbeds are installed at KNU and at SNUThe globus simple CA is managed at KNU and at SNU to sign certsIn addition to the default VO’s in EDG, a cdf VO is constructedAuthentication and VO LDAP server is running at SNU A replica catalog (RC) LDAP server is running at SNUSee the Korean EDG testbed description web site athttp://neutrino.snu.ac.kr/~bockjoo/EDG_testbed/ and EDG
web
CEVOuser
SEVOuser
WNVOuse
r
디스크
NFS
GSIFTP
MAP on diskWith maximum
security
grid-security
NFS
GSIFTP
NFS
GSIFTP
NFS
GSIFTP
빅 팻 디스크
Big FatDisk
GDMP server
(with new VO)
GDMP client(with new
VO)
GDMP client(with new
VO)
SNUSKKU
KNU
UIReal user
RB In operation
In operation
In preparation
.
.
.
LDAPServer
SNU
Authorization and grid-Authorization and grid-mapfilemapfile
o=testbed,dc=edg, dc=org
CN=Franz Elmer
ou=People
CN=John Smith
mkgridmap grid-mapfile
VOVODirectoryDirectory
““AuthorizationAuthorizationDirectory”Directory”
CN=Mario Rossi
o=xyz,dc=edg, dc=org
CN=Franz ElmerCN=John Smith
Authentication Certificate
Authentication Certificate
Authentication Certificate
ou=People ou=tb1 ou=Admin
local users ban list
X.509 certs by CACA published on LDAP
Check Auth. dir and VO
LDAP Services for Korean HEP DGLDAP Services for Korean HEP DG
There are also the CMS and the CDF RC
More Info: http://neutrino.snu.ac.kr/~bockjoo/EDG_testbed/
Auth LDAPAuth LDAP
LDAP for RB LDAP for RB job job
CDF VO LDAPCDF VO LDAP
CMS VO LDAPCMS VO LDAP
How to Use the EDG TestbedHow to Use the EDG Testbed
Certificate should be obtained
Account on a UI must exist
UI CommandsBasic Globus Commands
grid-proxy-init To Access Resource Broker
globus-url-copy
……………….
EDG Commands(python)dg-job-canceldg-job-get-outputdg-job-get-logging-infodg-job-id-infodg-job-list-matchdg-job-statusdg-job-submit
More Info: http://marianne.in2p3.fr/datagrid/documentation
An Illustration of Job SubmissionAn Illustration of Job Submission
UIJDL
Logging &Logging &Book-keepingBook-keeping
ResourceResourceBrokerBroker
Output “sandbox”
Input “sandbox”
Job SubmissionJob SubmissionServiceService
StorageStorageElementElement
ComputeComputeElementElement
Brokerinfo
Output “sandbox”
Input “sandbox”
Information Information ServiceService
Job Status
LFN->PFN
Data ManagementData ManagementServicesServices
Author.&Authen. Job
Subm
it
Job Q
uery
Job Status
dg-job-canceldg-job-get-outputdg-job-get-logging-infodg-job-id-infodg-job-list-matchdg-job-statusdg-job-submit
Job Submission - UIJob Submission - UI
Job submission Job submission commandcommand
Job IdJob Id
Job Submission - RBJob Submission - RB
Initial job Initial job submission submission
phasephase
Job Job executionexecution
phasephase
Job Submission - JSSJob Submission - JSSJSS Server JSS Server
LogLogJSS parsedJSS parsed
Job Submission - CondorGJob Submission - CondorG
CondorG just located the CondorG just located the resource (CE)resource (CE)
Job Submission – the CE=QUEUEJob Submission – the CE=QUEUE
RB wuftpdRB wuftpd
WN pbs momWN pbs mom
Job Submission – RB ftp+WNJob Submission – RB ftp+WN
GDMP:DatamanagementGDMP:Datamanagement
More Info: http://project-gdmp.web.cern.ch/project-gdmp/
How to use GDMP and RC?How to use GDMP and RC?1. Setup environment, e.g., EDG_VO=cms
2. Subscribe to other site’s SEgdmp_host_subscribe –S $SE1 –P $P1 –r $SE2 –p $P2
3. Create file(s) on SE by hand or by JDL (The file should belong tothe group $EDG_VO)
4. On UI, register the file locallygdmp_register_local_file -R -p /flatfiles/SE00/cms/dummy –S $SE –P $P –V $EDG_VO
5. On UI, publish the file and register it on the RCgdmp_publish_catalogue -V cms -C -L log_ID
6. On UI, replicate file if necessary (gsiftp+ldapadd)gdmp_replicate_get –r $SE –p $P –V $EDG_VO
More Info:
http://neutrino.snu.ac.kr/~bockjoo/EDG_testbed/contents/gdmp_test.html
Data Management using GDMPData Management using GDMPSE SE SubscriptionSubscription
File Creation on File Creation on SESE
Check File on SECheck File on SE
Register and Register and PublishPublish
Check catalog on Check catalog on se2se2
LDAP :RC for se1LDAP :RC for se1
Replicate from se1 to Replicate from se1 to se2se2
LDAP: RC for se1 and LDAP: RC for se1 and se2se2
Check catalog on Check catalog on se1se1
Example Application of EDGExample Application of EDG
Currently running Experiment
Existing Resources
Matter of adding ques and hacking : examples exist
Some works are in progress to include the existing node as a WN for different VO’s within a testbed site
CEVOuser
SEVOuser
WNVOuse
r
디스크
NFS
GSIFTP
MAP on diskWith maximum
security
grid-security
NFS
GSIFTP
NFS
GSIFTP
NFS
GSIFTP
빅 팻 디스크
Big FatDisk
GDMP server
(with new VO)
GDMP client(with new
VO)
GDMP client(with new
VO)
EDG testbed at SNU
UIReal user
RB
Que
Que
CDFSoftwa
re
QueK2K
Software
At the moment, the EDG testbed is operated at EC
EDG testbeds are constructed and working at KNU and SNU in KoreaFor Data Management, an LDAP for RC is constructed and the GDMP is used within Korea. This is working.
The testbed is also operational between KNU and SNUA possible HEP application for currently running experiments is illustrated and we are moving toward that direction
The EDG testbed will be constructed for other institutions in Korea New VO will be tested extensively for a few months
Hopefully, we would like to learn other testbeds and test crossgrid functionalities
SummarySummary
Network to the world
Europe CERNTEIN
APII-TransPac
현해
US FNAL
10Mbps•45Mbps by the end of 2002
2Gbps ( soon )
한 - 일 : 8Mbps한 - 미 : 45Mbps
File transfer test on the network
parallel ftp used.
KNU – SNU : ~50Mbps (155Mbps) KNU – SKKU : ~50Mbps (155Mbps) SKKU – SNU : ~50Mbps (155Mbps)
CHEP – KEK : ~7Mbps (8Mbps) CHEP – Fermilab : ~17 Mbps (45 Mbps) CHEP – CERN : ~8Mbps (10Mbps)
FERMILAB – CERN : ~2Gbps ( DataTAG project of iVDGL )
File transfer test on the network (KNU-SKKU,SNU)
Time for moving file
0
10
20
30
40
1 2 3 4 5 6 7number of ftp
sec
1계열
observed bandwidth
0
10
20
30
40
50
60
1 2 3 4 5 6 7
number of ftp
Mbps
1계열
file size for one ftp= 30MB
File transfer test on the network
US-CERNDataTAG linktest
KNU-USftp test
PlansPlans Route to Europe
1st TEIN : 45Mbps from next year
2nd : via USA Korea FERMILAB CERN % We will use Fermilab’s storage as buffer.
3rd : via Japan and USA Korea KEK Fermilab CERN 2Gbps >1Gbps 2 Gbps
Participation to iVDGL anables 2nd and 3rd route.
Contribution : 2003 CMS summer production We will provide significant CPU power and storage.
Future bandwidth requirement on the Future bandwidth requirement on the networknetwork
Korea – Japan – USA Connection
for experiments in Japan ~ 400Mbps by 2006 ~ 650Mbps after 2006 CDF experiment ~40Mbps by 2004 ~230Mbps after 2004 CMS experiment ~800Mbps ( 1PB/year )
Including other traftic and Japan’s, at least 2Gbps required. Should evaluate to Lambda (10Gbps) connection to USA, Japan, Europe
Future bandwidth requirement on the network
622 Mbps +10 Gbps
