Date post: | 30-Dec-2015 |
Category: |
Documents |
Upload: | iris-lucas |
View: | 213 times |
Download: | 0 times |
Copyright © 2006 University of Alberta. All rights reserved
WestGrid Seminar Series
Integrating GridstoreInto The Job Submission
ProcessWith GSUB
Edmund Sumbar
University of Alberta
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 2
GSUB
Gridstore…• close proximity to vizserver
• does not impose quotas
• has a large capacity
• performs backups
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 3
GSUB
time gcp gridstore:largefile .
glacier lattice *matrix nexus cortex
10MB 4 5 8 6 5
100MB 14 7 70 40 170
1GB 120 55 ~600 440 ~4000
Data collected January 11, 2006.
* Using scp, gcp not installed yet.
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 4
GSUB
Job submission protocol…• copy files from gridstore to execution site (gcp)
• submit job (qsub)
• wait for job to end (qstat)
• copy files from execution site to gridstore (gcp)
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 5
GSUB
desktop
user
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 6
GSUB
{execution nodes
desktop
user
head node
exec site
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 7
GSUB
{execution nodes
desktop
user
head node
exec site
gridstore
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 8
GSUB
{execution nodes
desktop
user
head node
exec site
head node
home site
gridstore
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 9
GSUB
ssh {execution nodes
desktop
user
head node
exec site
head node
home site
gridstore
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 10
GSUB
ssh {execution nodes
desktop
user
head node
exec site
head node
home site
grid-proxy-init
gridstore
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 11
GSUB
ssh {execution nodes
desktop
user
head node
exec site
head node
home site
grid-proxy-init
ssh
gridstore
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 12
GSUB
ssh {execution nodes
desktop
user
head node
exec site
gridstore
head node
home site
grid-proxy-init
ssh
gcp
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 13
GSUB
ssh {execution nodes
desktop
user
head node
exec site
gridstore
head node
home site
grid-proxy-init
ssh qsub
gcp
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 14
GSUB
ssh {execution nodes
desktop
user
head node
exec site
gridstore
head node
home site
grid-proxy-init
ssh qsub
gcp gcp
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 15
GSUB
ssh {execution nodes
desktop
user
head node
exec site
gridstore
head node
home site
grid-proxy-init
ssh qsub
gcp gc
p
1
2
3
4
5
6
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 16
GSUB
Becomes tedious/inefficient/error-prone
Automate process with scripts…
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 17
GSUB
Consider gsub…
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 18
GSUB
Consider gsub…• No changes are required to your existing Torque (PBS)
job scripts
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 19
GSUB
Consider gsub…• No changes are required to your existing Torque (PBS)
job scripts
• All qsub command-line options are supported
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 20
GSUB
{execution nodes
desktop
user
head node
exec site
gridstore
head node
home site
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 21
GSUB
ssh {execution nodes
desktop
user
head node
exec site
head node
home site
gridstore
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 22
GSUB
ssh {execution nodes
desktop
user
head node
exec site
head node
home site
grid-proxy-init
gridstore
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 23
GSUB
ssh {execution nodes
desktop
user
head node
exec site
head node
home site
grid-proxy-init
ssh
gridstore
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 24
GSUB
{execution nodes
head node
exec site
ssh
desktop
user
gridstore
head node
home site
grid-proxy-init
ssh gsub
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 25
GSUB
ssh {execution nodes
desktop
user
head node
exec site
gridstore
head node
home site
grid-proxy-init
ssh gsub
gcp
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 26
GSUB
ssh {execution nodes
desktop
user
head node
exec site
gridstore
head node
home site
grid-proxy-init
ssh gsub
gcp
stage in
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 27
GSUB
ssh {execution nodes
desktop
user
head node
exec site
gridstore
head node
home site
grid-proxy-init
ssh qsub
gcp gcp
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 28
GSUB
ssh {execution nodes
desktop
user
head node
exec site
gridstore
head node
home site
grid-proxy-init
ssh qsub
gcp gcp
stage out
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 29
GSUB
ssh {execution nodes
desktop
user
head node
exec site
gridstore
head node
home site
grid-proxy-init
ssh gsub
gcp
gcp
1
2
3
4
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 30
GSUB
Demonstration…
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 32
GSUB
ssh {execution nodes
desktop
user
head node
exec site
gridstore
head node
home site
grid-proxy-init
ssh gsub
gcp
gcp
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 33
GSUB
ssh {execution nodes
desktop
user
head node
exec site
gridstore
head node
home site
grid-proxy-init
ssh gsub
gcp
gcp
blackhole(gridstore head node)
ssh• prepare input
• analyze output
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 34
GSUB
Submit jobs directly from blackhole…
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 35
GSUB
desktop
user {execution nodes
head node
gridstore
head node
home site
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 36
GSUB
desktop
user {execution nodes
head node
gridstore
head node
home site
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 37
GSUB
desktop
user {execution nodes
head nodeblackhole
gridstore
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 38
GSUB
desktop
user {execution nodes
head nodeblackhole
ssh
gridstore
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 39
GSUB
desktop
user {execution nodes
head nodeblackhole
ssh
myproxy-get-delegation
gridstore
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 40
GSUB
head nodeblackholedesktop
user {execution nodes
ssh
myproxy-get-delegation
gridstore
gsub
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 41
GSUB
desktop
user {execution nodes
head nodeblackhole
ssh
myproxy-get-delegation
gsub gsub
gridstore
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 42
GSUB
desktop
user {execution nodes
head nodeblackhole
ssh
myproxy-get-delegation
gsub gsub
gridstore
gcp
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 43
GSUB
desktop
user {execution nodes
head nodeblackhole
ssh
myproxy-get-delegation
gsub gsub
gridstore
gcp gc
p
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 44
GSUB
desktop
user {execution nodes
head nodeblackhole
ssh
myproxy-get-delegation
gsub gsub
gridstore
gcp
gcp
1
2
3
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 45
GSUB
Demonstration…
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 47
GSUB
Tools for monitoring and managing jobs• gstat
• gdel
• glog
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 48
GSUB
Demonstration…
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 50
GSUB
View WestGrid like a scaled-up execution
site…
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 51
GSUB
desktop
user {execution nodes
head node
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 52
GSUB
desktop
user
{
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 53
GSUB
desktop
user
{
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 54
GSUB
desktop
user
{{
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 55
GSUB
desktop
user
{{{
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 56
GSUB
{desktop
user
{{{
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 57
GSUB
{desktop
user
{{{
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 58
GSUB
{desktop
user
{{{
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 59
GSUB
{desktop
user
{{{
blackhole
super head node
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 60
GSUB
{desktop
user
{{{
blackhole
super head node
myproxy
credential server
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 61
GSUB
{desktop
user
{{{
blackhole
super head node
myproxy
credential server
gridstore
file system
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 62
GSUB
{desktop
user
{{{
blackhole
super head node
myproxy
credential server
gridstore
file system
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 63
GSUB
{desktop
user
{{{
blackhole
super head node
myproxy
credential server
gridstore
file system
WestGrid
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 64
GSUB
DISCLAIMER
The opinions expressed herein are those of
the author and do not necessarily represent
the views of WestGrid management.
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 65
GSUB
Gsub implements programmable workflow
Specify actions to be executed at various
times (phases) during the life of a job• before stage in
• after stage in
• job start
• job end
• before stage out
• after stage out
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 66
GSUB
Workflow programmed as a case statement
case $phase in beforestagein ) exit ;; afterstagein ) exit ;; jobstarted ) exit ;; jobended ) exit ;; beforestageout ) exit ;; afterstageout ) if some condition is true; then resubmit fi exit ;;esac
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 67
GSUB
Predefined workflows• email
• prevent
• tidy
• autoresub
• exclusive
• aggressive
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 68
GSUB
No script modifications required
Specified at submission time• gsub jobscript=workflow
Cascade workflows• gsub jobscript=workflow1+workflow2
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 69
GSUB
Demonstration…
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 72
GSUB
Gsub is unsuitable for…• very short jobs
• developing, testing, debugging programs
• staging very large or very many files
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 73
GSUB
Gsub is not officially accepted by WestGrid
Still needs testing in the real world…
Copyright © 2006 University of Alberta. All rights reserved create deleteJanuary 25, 2006 74
GSUB
Gsub is not officially accepted by WestGrid
Still needs testing in the real world…
…any volunteers?