Post on 25-Dec-2019
transcript
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 1
Joint EGEE/OSG VO Management at HPDC '08
GlideinWMS
The CMS pilot infrastructure
by Igor Sfiligoi (Fermilab)
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 2
Outlook
● Grid computing overview● The pilot paradigm● Introducing Condor glideins● glideinWMS description● glideinWMS in real life● Conclusions
BNL Jun 16th 2008 glideinWMS explained I. Sfiligoi 3
Barebones Grid is complex
● Resources grouped in independent pools– Each with its own set of rules (often not published)
● Resources in different pools configured differently– Users expected to adapt (after they discover how they work)
Site C
Site F
Site E
Site A
Site D
Site B
Gatekeeper
GKGK
GK
GK
GK
BNL Jun 16th 2008 glideinWMS explained I. Sfiligoi 4
Scientific user needs
● Wants to spend most of his time thinking about the scientific problems
● Has large computing needs, but should be simple– Computing is just a tool
– Time spent on computing problems is subtracting time available for scientific thinking!
Computing resourcespool
Submit job
Get result
BNL Jun 16th 2008 glideinWMS explained I. Sfiligoi 5
● Maybe by creating an overlay over the Grid sites?– To hide the differences between sites
● Making the Grid look as a single, uniform pool?
Can we get to the middle ground?
Site C
Site F
Site E
Site BSite A
Site D
Computing resourcespool
Submit job
Get result
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 6
The pilot paradigm
● A possible solution– Use pilot jobs to create
the overlaySite C
Site F
Site E
Site BSite A
Site D
Submit job
Get result
Pilot PilotPilot
Pilot
Pilot Pilot
PilotPilotPilot
Computing resourcespool
Pilot factory
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 7
Site C
Site F
Site E
Site BSite A
Site D
Submit job
Get result
Pilot Pilot
PilotPilot
Pilot Pilot
PilotPilotPilot
Computing resourcespool
Pilot factory
The pilot paradigm (continued)
● Never send user jobs directly– Send pilot jobs to create the overlay, instead
● When a pilot lands on a Grid worker node– Validates Grid resource
– Prepares the environment
– Pulls a user job
● Hides Grid heterogeneity– Users see a fairly uniform computing pool
User jobs never sent directly
to Grid sites
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 8
Condor glideins
● Condor is based on a distributed architecture● Condor glideins are Grid jobs that start
regular Condor daemons– i.e. they are pilots
implemented with Condor
Site C
Site F
Site E
Site BSite A
Site D
Submit job
Get result
Schedd Collector
Negotiator
Startd
Startd Startd
Startd
Startd
http://www.cs.wisc.edu/condor/
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 9
Submitting glideins
● Condor provides only a basic command line glidein submission tool– Good for trying out glideins
– But not meant to be used as a glidein factory
● A few groups developed glidein factories– CDF has the CDF-specific GlideCAF
– USCMS@FNAL is developing the glideinWMS Site C
Site F
Site E
Site BSite A
Site D
Submit job
Get result
Schedd Collector
Negotiator
Startd
Startd Startd
Startd
Startd
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 10
Introducing the glideinWMShttp://www.uscms.org/SoftwareComputing/Grid/WMS/glideinWMS/
Site C
Site F
Site E
Site BSite A
Site D
Submit job
Get result
Schedd Collector
Negotiator
Glidein
GF
GF
Collector
VO Frontend
Glidein
Glidein Glidein
Glidein
Glidein
Startd
Legenda
Regular Condor
glideinWMS
● An autonomous glidein submission system
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 11
glideinWMS architecture
● glideinWMS composed of six logical pieces:– A Condor central manager
(collector + negotiator)
– One or more Condor submit machines
– A glideinWMS collector
– One or more VO frontends
– One or more glidein factories
– The glideinsSite D
Sub
mit jo
b
Get resu
lt
Schedd Collector
NegotiatorGlideinFactory
WMS Collector
VO Frontend
GlideinStartd
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 12
glideinWMS architecture (2)
● glideinWMS composed of six logical pieces:– A Condor central manager
(collector + negotiator)
– One or more Condor submit machines
– A glideinWMS collector
– One or more VO frontends
– One or more glidein factories
– The glideins (startd)Glidein
VO Frontend
GlideinFactory
WMS Collector
Sub
mit jo
b
Get resu
lt
Schedd Collector
Negotiator
Startd
User job
Transfer job
Transfer result
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 13
glideinWMS architecture (3)
● glideinWMS composed of six logical pieces:– A Condor central manager
(collector + negotiator)
– One or more Condor submit machines
– A glideinWMS collector
– One or more VO frontends
– One or more glidein factories
– The glideins
Sub
mit jo
b
Get resu
lt
Schedd Collector
Negotiator
Site D
GlideinFactory
WMS Collector
VO Frontend
GlideinStartd
Glidein
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 14
Glidein
VO Frontend
GlideinFactory
WMS Collector
Condor handles user jobs
● A glidein Condor pool is still a Condor pool– Just a very dynamic one
● All Condor features available– ClassAds
– Job requirements
– Fair share
● Users really don't know about the glideinWMS
Sub
mit jo
b
Get resu
lt
Schedd Collector
Negotiator
Startd
User job
Transfer job
Transfer result
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 15
Glidein
VO Frontend
GlideinFactory
WMS Collector
Condor handles user jobs (2)
● Internal VO policy management completely in VO hands– Sites only need to handle
inter-VO policies
● Condor provides– Accounting groups
(with quotas)
– Fair share between users withing a group
Sub
mit jo
b
Get resu
lt
Schedd Collector
Negotiator
Startd
User job
Transfer job
Transfer result
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 16
Sub
mit jo
b
Get resu
lt
Schedd Collector
Negotiator
Glidein submission
● glideinWMS processes are responsible only forstartd startup– A glidein just configures
and starts it
– Once started, startd has full control
● Glideins highly customizable– Glidein factory allows for plugins
Site D
GlideinFactory
WMS Collector
VO Frontend
GlideinStartd
Glidein
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 17
Sub
mit jo
b
Get resu
lt
Schedd Collector
Negotiator
Glidein submission (2)
● Condor-G used for submission to Grid sites– Current implementation
● Glidein submission is a collaborative work– VO frontend decides how
many glideins to submit
– Glidein factory actually does the submission
– WMS collector is used for message passing
GlideinFactory
WMS Collector
VO Frontend
StartdGlidein Gatekeeper
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 18
Startd
Sub
mit jo
b
Get resu
lt
Schedd Collector
Negotiator
Glidein submission (3)
● Based on the principle of constant pressure– As long as there are enough
waiting jobs in the queue, a fixed number of glideins are kept at each suitable Grid site
● Works nicely for systems with lots of waiting jobs– Will waste resources on
seldom used systems
GlideinFactory
WMS Collector
VO Frontend
Glidein Gatekeeper
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 19
Security considerations
● GlideinWMS requires security over the wire– WAN network connections cannot be blindly trusted!
● All network traffic features integrity checks– Prevents man-in-the-middle attacks
● GSI authentication (X509 certificates/proxies) used for all interactions with Condor daemons over the network– Based on access lists (like gridmap file, but with regex)
– Attributes not used right now● But will probably need it
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 20
Security considerations (2)
● Startd not running as a privileged user– Cannot change UID by itself when starting user job– Malicious user job could hijack the startd
if running under the same UID● Condor interfaced to gLExec
– gLExec allows to change UID given user proxy
– Startd protected from the user job● gLExec interfaces with local
security infrastructure– Enforces site policies– When pilots use it
Schedd
Startd
User job
Job + proxy
gLExec
Job + proxy
PDP
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 21
Proxy handling
● A single identity used to submit all the glideins– Should have a pilot role
● Condor ships user proxy to worker node– User can use it access other
resources
● Proxy lifetime management and renewal not handled by glideinWMS– Users expected to do it
Site D
User p
roxy
Schedd Collector
NegotiatorGlideinFactory
WMS Collector
VO Frontend
GlideinStartd
User job
gLExec
Pilot proxy
User proxy
User proxy
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 22
User job monitoring
● Good monitoring a must for most users● Condor provides a plethora of monitoring tools
– Most useful are condor_q and condor_status
– Third parties provide additional Condor monitoring tools
● glideinWMS provides tools for pseudo-interactive monitoring– ls, cat, top on the worker nodes
● The glidein factory also maintains a basic Web based graphical view– plus machine readable XML and rrd data
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 23
glideinWMS monitoring
● Good monitoring a must for most administrators, too● Condor-G provides some tools
– Mostly condor_q
● The glidein factory maintains a rich Web based graphical view– plus machine readable XML and rrd data
● Glideins return comprehensive logs– Useful for low level debugging
– But require some expertise to browse though
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 24
Status of glideinWMS
● Version 1.2.1 released May 30th
● Should be usable out of the box for most users– CMS is using it since v1.1
● Still in active development phase– More monitoring
– More automated error checking
– More automated error recovery
– Better integration with other systems
● Condor also an evolving product
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 25
Glidein deployments in HEP● CMS using glideins for
production jobs at FNAL– Recently across all seven T1s
● CMS used them for analysis jobs in CCRC08
– Across 40 T2s
● CDF and MINOS using them for user analysis
CMS FNAL
CMS CCRC
CDF
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 26
glideinWMS in numbers
● Deployed systems– CMS@FNAL stable 3k glideins for the past 6 months
– CMS@CCRC up to 4k glideins over 40 sites globally
– CDF average 2k glideins with 100s of users for past 2 years(by using the GlideCAF)
● glideinWMS Tested on a dedicated test pool, scaled without major problems to – 10k glideins at any time
– 100k user jobs queued
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 27
CMS @ FNAL experience● Using ProdAgent to submit jobs to local schedd● Gliding into a single site
– over LAN– Using 3 CEs
● Saturating the FNAL T1– ~3200 slots
● Quick job turnaround– >10k jobs per day on average– >150k jpd during CSA07
● Few failures– Mostly storage related 0 5-29
0 5-310 6 -0 2
0 6 -0 40 6 -0 6
0 6 -0 80 6 -10
0 6 -120 6 -14
0 6 -160 6 -18
0
20 0 0
40 0 0
6 0 0 0
8 0 0 0
10 0 0 0
120 0 0
140 0 0
16 0 0 0
18 0 0 0
20 0 0 0
Jo b s p er d ay
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 28
CMS @ FNAL setup
● 3 nodes used (+Grid worker nodes)● No GCB (LAN)● No gLExec (only production team, single logical user)
ScheddCollector Negotiator Glidein
Factory
WMS Collector VO FrontendProdAgent
Site DGlideinStartd
ScheddGScheddGScheddGScheddG
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 29
CMS CCRC08 experience
● Using CRAB to submit to the local schedd(s)
● Submitting to 40 T2s– All over the world
– OSG, EGEE and Nordugrid(a first for CMS)
● Ran 300k jobs over 4 weeks– Mix of CPU intensive and IO
intensive jobs
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 30
CMS CCRC08 experience (2)
● Latencies have bitten us– Condor uses blocking connections for security handshake
● Condor working on fix
– For CCRC solved by using multiple condor daemons● Hierarchy of collectors● Multiple schedds
● Still very successful– CMS pleased with the experience
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 31
CMS CCRC08 setup
● 5 nodes used (+Grid worker nodes)● No gLExec (Only one CCRC user)
Schedd
CollectorNegotiator
GlideinFactory
WMS Collector
VO FrontendCRAB
Site DGlideinStartd
ScheddGScheddGScheddGScheddGScheddScheddScheddScheddScheddScheddScheddSchedd
CollectorCollectorCollectorCollectorCollectorCollectorCollectorCollectorCollector
GCB
GCB
ScheddGScheddGScheddGScheddGScheddG
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 32
CMS glidein plans
● Production over all T1s using glideinWMS should start soon (from FNAL)– Prototype in place
– Need to sort out operational issues
● But should be pretty straightforward● Still only one logical user
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 33
CMS glidein plans (2)
● UCSD offered to host an analysis service– Serving physicists
– Using the CRABServer
– Using gLExec
– Expected to be setup over the summer
● Proxy handling may become an issue● Using a single pilot proxy may not be the best strategy
– Problems when competing with PUSH-based WMSes(like gLite WMS and plain Condor-G)
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 34
Conclusions
● Bare-bones Grid difficult to use– Glideins can hide the Grid complexity and make it look
as a uniform computing pool
● CMS has used glideinWMS for the past 6 months– Great success at FNAL
– Good results in tests over T1s and T2s
● Glideins allow for flexible VO policy managing– CDF had excellent experience over past few years
– CMS will go after this soon
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 36
glideinWMS contact info
GlideinWMS home page:http://www.uscms.org/SoftwareComputing/Grid/WMS/glideinWMS/
Condor home page:http://www.cs.wisc.edu/condor/
email: sfiligoi@fnal.gov
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 37
Working over the firewalls
● Condor uses two-way communication– But incoming connection often blocked by Grid sites
● Can use Condor GCB (Generic Connection Broker) to make all communications one-way– By opening a long lived TCP connection– Outgoing connectivity always needed
Startd
User jobSchedd
Collector
Negotiator
2
4
6
3
GCB
1
5 – Overthe establishedTCP connection
http://www.cs.wisc.edu/condor/gcb/
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 38
Condor Internals
condor_schedd
Submit Machine
Submit
condor_collector
Central Manager
condor_negotiator
Worker Node
condor_startd
Advertize
Advertize
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 39
Condor internals
condor_schedd
Submit Machine
condor_collector
Central Manager
condor_negotiator
Worker Node
condor_startd
Match
Match
HPDC Boston Jun 24th 2008 glideinWMS I. Sfiligoi 40
Condor internals
condor_schedd
Submit Machine
condor_collector
Central Manager
condor_negotiator
Worker Node
condor_startd
Spawn
condor_shadow Spawn, using gLExec
condor_starter
Delegate proxy
Claim