Date post: | 02-Jan-2016 |
Category: |
Documents |
Upload: | galena-hodges |
View: | 25 times |
Download: | 0 times |
1CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab
CMS Commissioning and First Data
Stan DurkinThe Ohio State Universityfor the CMS CollaborationBockjoo Kim
U of Florida
Bockjoo KimU of Florida
CMS Software Installation
2CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab
CMS Commissioning and First Data
Stan DurkinThe Ohio State Universityfor the CMS Collaboration
CMSSW Installation Sites on OSG
T3
T3
T3
T3
T3
T3
T3
T3
T3
T2
T2
T2
T2
T2
T2
T2
T2
T2
9 T39 T2
(Caltech)
(FIU)
3CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab
CMS Commissioning and First Data
Stan DurkinThe Ohio State Universityfor the CMS Collaboration
CMSSW on US T3 Sites(as of 10/21/08)Sitename CE (OSG) CPU slots CMSSW
T3_US_Buffalo?? u2-grid.ccr.buffalo.edu/fork 2112 2_1_10, …
T3_US_Colorado heposg01.colorado.edu/? 16 2_1_10, …
T3_US_Cornell nys1.cac.cornell.edu/fork 120 2_1_9, …
T3_US_FIT uscms1.fltech-grid3.fit.edu/fork 160 2_1_10, …
T3_US_FIU ?/?
T3_US_FNALLPC ?/?
T3_US_Kansas ?/?
T3_US_Minnesota ?/?
T3_US_Olemiss?? umiss001.hep.olemiss.edu/fork 42 2_1_10, …
T3_US_Princeton ?/?
T3_US_Rutgers ?/?
T3_US_TTU antaeus.hpcc.ttu.edu/fork 252 2_1_10, …
T3_US_UCLA saxon.hosted.ats.ucla.edu/fork 44 2_1_10, …
T3_US_UCR top.ucr.edu/condor 40 2_1_10, …
T3_US_UIowa ?/?
T3_US_UMD ?/?
T3_US_Vanderbilt vampire.accre.vanderbilt.edu/fork
1408 2_1_10, …
4CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab
CMS Commissioning and First Data
Stan DurkinThe Ohio State Universityfor the CMS Collaboration
Centralized CMS Installation
CMS requires CMSSW to be installed centrally for T2
For T3, one can install it locally or centrally
non-CMS OSG sites, it can be done centrally
All installations are uniform ($OSG_APP/cmssoft/cms/$SCRAM_ARCH/cms/cmssw/CMSSW_X_Y_Z)
Central Installation on OSG done by me
LCG/EGEE counter part (serveral people) is doing this on LCG/EGEE
5CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab
CMS Commissioning and First Data
Stan DurkinThe Ohio State Universityfor the CMS Collaboration
CMS Software Life Cycle
1. Software Development
2. Release Build
3. APT Packaging
4. Tagging Release in XML for Publication
5. Release Announcement
6. Deployment on Grids/Local
7. Release Deprecation Pre-Announcement
8. Tagging for Release Deprecation Release
9. Release Deprecation Announcement
10. Remove Deprecated Release
6CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab
CMS Commissioning and First Data
Stan DurkinThe Ohio State Universityfor the CMS Collaboration
Features of Software Deployment ToolFeatures of Software Deployment Tool
Condor-G Job Submission with Customized Installation/Verfication Scripts
Central Run and Bookkeeping DB
Possible Install via Grid Proxy Based Portal: Different DN Can Install Different Site Simul.
Cronized Installation In Parallel per Release
Production CMS Soft Release on OSG T2/T3
Twiki Pages : https://twiki.cern.ch/twiki/bin/view/CMS/CMSSoftDeployOSG
Portal : https://dev01.ihepa.ufl.edu:8443/csdogrid/csdogrid/
7CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab
CMS Commissioning and First Data
Stan DurkinThe Ohio State Universityfor the CMS Collaboration
Consideration For InstallationConsideration For Installation
Different Linux Flavors
64-bit at 32-bit mode
Network Insulation (non-CMS sites)
Missing Tools on WN : apt-get, rpm, rpmbuild (non-CMS sites)
Different Shared File Systems (Lustre, AFS, etc)
Dedicated Slot : Needs to be done before any other CMS job starts
Disk Space Issue : ~50GB required typically Many files ==> Installation time gets longer
Keeping many releases ==> Deprecation becomes terrible and should be done on time
8CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab
CMS Commissioning and First Data
Stan DurkinThe Ohio State Universityfor the CMS Collaboration
APT and File SystemsAPT and File Systems
APT (RPM tool) uses lots of locks
At least, four different FS accros OSG sites : NFS, AFS, GPFS, lustre
GPFS, lustre, other not well-known FS requires special treatment for locks using local FS ( not many exotic FS, though)
Many files ==> Installation time gets longer
Keeping many releases ==> Deprecation becomes very important ==> (64-bit OS/8GB required in the worst case)
9CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab
CMS Commissioning and First Data
Stan DurkinThe Ohio State Universityfor the CMS Collaboration
DeprecationDeprecation
Mostly same as installation
Most time consuming part is find dependency and dependency bookkeeping
NFS stale file handle and GPFS : ‘rm -rf’
Recycle dependency calculation from one site
Deprecation is also automated and cronized
10CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab
CMS Commissioning and First Data
Stan DurkinThe Ohio State Universityfor the CMS Collaboration
CMSCMS Software Packaging Tool Software Packaging Tool
CMS employs APT packaging for software distribution
CMS provides packaging tools for initial setup and update
RPMs are installed in a non-root area
CMS provides release publication : this allows deployment people for immediate deployment
11CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab
CMS Commissioning and First Data
Stan DurkinThe Ohio State Universityfor the CMS Collaboration
AutomatedAutomated CMS CMS software Deploymentsoftware Deployment
Well-established OSG CMS sites needs software deployment promptly
CRON is used for automation
List of well-established OSG CMS sites
Database(DB) for bookkeeping
Scripts checks and executes: New release that needs to be deployed Deployment status from DB Deployment job submission/resubmission Installation job self-monitoring Email notification Repeatition of all if necessary
12CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab
CMS Commissioning and First Data
Stan DurkinThe Ohio State Universityfor the CMS Collaboration
Implementation of the Deployment ToolImplementation of the Deployment Tool
Local Scripts
A Remote OSG Site
Application Area
Info. Publication
CMS Grid Users
OSG Software
GridCat
GridCat Client
CMS Pkg Tool
RPMs
CMS Pkg Tool RPMs CMS APT Repository
Tool Design
OSG CMS
X509 Web Portal
MySQL DB
Deploy Script
Condor-G Job
Execution Script
CRON Site Catalog
Site Availability
Site List
DB
Update
DB Check DB Check
Execution Script
Job Status Email
13CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab
CMS Commissioning and First Data
Stan DurkinThe Ohio State Universityfor the CMS Collaboration
Grid Portal Screen ShotGrid Portal Screen Shot
14CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab
CMS Commissioning and First Data
Stan DurkinThe Ohio State Universityfor the CMS Collaboration
SAM and SW Installation MonitoringSAM and SW Installation Monitoring
Related with SW InstallationRelated with SW Installation
SAM monitors SW InstallationsSAM monitors SW Installations
15CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab
CMS Commissioning and First Data
Stan DurkinThe Ohio State Universityfor the CMS Collaboration
List of Problems and SolutionsList of Problems and Solutions
RPM version mismatch -> Rebuild RPM DB
rpm-wrapper error 88 -> insufficient disk rpm-wrapper error 92 -> permission problem“Could not get lock” -> FileSystem, use local disk
memory alloc (4byte)-> Remove releases
memory alloc (8byte) -> Use 64-bit apt-get
16CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab
CMS Commissioning and First Data
Stan DurkinThe Ohio State Universityfor the CMS Collaboration
Statistics of CMS SW DeploymentStatistics of CMS SW Deployment
Installed
Releases
2006
0_4_1 TO 1_2_0
2007
1_2_0 TO 1_7_5
2008
1_6_10 TO 2_1_10
27 50 41
118
Sites OSG T2, T3, and Opportune
33 sites
Installations
(ReleasesxSites)
2006 2007 2008
126 294 297
717
Removals
(ReleasesxSites)
2006 2007 2008
15 325 684
1024
Years 2006 - 2008
17CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab
CMS Commissioning and First Data
Stan DurkinThe Ohio State Universityfor the CMS Collaboration
Timing and FilesTiming and Files Site Tier Time to Install* FS No of Files Size (GB)
Caltech 2 54 min XFS 1.7 x 106 57
Florida 2 868 min NFS/lustre 1.3 x 106 48
MIT 2 66 min NFS 1.5 x 106 57
Nebraska 2 57 min NFS 1.5 x 106 54
Purdue 2 52 min NFS 1.6 x 106 96
SPRACE 2 100 min NFS 1.7 x 106 53
UCSD 2 98 min NFS 1.6 x 106 60
UERJ 2 51 min NFS 1.4 x 106 52
Wisconsin 2 71 min AFS 1.9 x 106 61
TTU 3 80 min lustre 1.3 x 106 45
UCR 3 56 min NFS? 1.5 x 106 54
Mississippi 3 186 min NFS? 1.6 x 106 55
UCLA 3 48 min NFS? 1.6 x 106 55
Vanderbilt 3 78 min GPFS 1.2 x 106 47
Buffalo 3 45 min NFS? 1.1 x 106 43
FIT 3 763 min NFS? 1.1 x 106 41
Cornell 3 - - - -
FIU 3 - - - -
18CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab
CMS Commissioning and First Data
Stan DurkinThe Ohio State Universityfor the CMS Collaboration
SummarySummary
More than 717(1024) installations/removal for CMS software have been deployed on OSG, 2006-2008
Automated installation works quite efficiently with almost no problem theses days
Recently, most installation problem comes from rpm db limitation and can be fixed via 64-bit apt-get
T3 sites are not required to deploy CMSSW centrally. At the moment, there are 9 sites under the regular/central deploy list.
If other T3 sites wish to be included, please let me know