Date post: | 28-Dec-2015 |
Category: |
Documents |
Upload: | aleesha-carter |
View: | 214 times |
Download: | 0 times |
Scaling NVO Services to the Teragrid
Roy Williams
Conrad Steenberg
Craig Miller
Matthew Graham
Joe Jacob
Julian Bunn
Desired Characteristics of NVO Services• Service oriented architecture
• Services should be easily and quickly deployable and usable on workstations or supercomputers
• Services deployed, managed, and upgraded by their developers• Service developers/deployers are trusted users
• Service developer acts as a broker between computing customer and computer center
• Service users authenticated with “graduated security”• Easy to start, but great power is possible
• Asynchrony for compute intensive jobs• Jobs submitted to batch queue• Unique sessionID may be used to monitor job & return results
• From “clicking” to “scripting”• Services may be accessed by clicking on a web page or with scripted client codes• Authentication for web clicking comes from a certificate store or fat browser• Scripted access requires a certificate (strong or weak) straight from the client
• Services as workflow components• A service user may be another service (a computer, not a human!)
A “Graduated Security” Model
Web form - anonymous access, small jobsSome science....
Get NVO weak certificate - access logged, but identity not verified
More science....
Full TeraGrid account - browser accessBig-iron computing....
Scripted accessPower user
Portal-Based
Traditional Grid Security
client
Show us your Certificate!I will do exactly what you want.
Graduated Security
clientMay I have your Request and your Certificate?
This is a US driver’s licence. In the US it proves identity strongly. It is like a strong certificate.
This is a loyalty card where I buy food.(You can put a false address on the application.)It is like a weak certificate.
This is a $50 gift card at a bookstore.It does not prove my identity in any way.It is like an anonymous certificate.
CertificatesThe Virtual Observatory as a Virtual Organization
"nesssi_strong_cert_max_time" : 216000,"nesssi_weak_cert_max_time" : 3600,"nesssi_anon_max_time" : 900,"nesssi_anon_user" : ”nvo","nesssi_weak_user" : "nvo",
Graduated security• No certificate gets 15 CPU-minutes from community account
•Just switch on Javascript
• Weak certificate gets 1 CPU-hour from community account
• In exchange for registering name/email
• Strong (gridmapped) certificate gets infinity from own account
• Get this one from TeraGrid HQ
service implementationweb formspython APIgraduated security
Certificatesmultiple browserscertificate chainsroot certificatesproxy certificatesproxy certificate chains2nd level proxy chainssecure https redirectionteragrid security policecaltech security policeNCSA security policechown directory ownershipNFS root-squashingpubcookie
Three Interfaces
• Commandline with Python & Java• Cert or proxy in wacko place like .globus or /tmp/u509
• Fat Browser• https: and browser managed PKCS12 certs
• Thin Browser• Web Proxy works dynamically with cert authority
Commandline Portal
client nesssi
node
node
node
node
Teragridcluster
certificatepolicies
queue
select useraccount
sandboxstorage
XML-RPC
proxy
open http
buildproxy
Certificate Authority
get certificate
Fat Browser Portal
Browser nesssi
node
node
node
node
Teragridcluster
certificatepolicies
queue
select useraccount
sandboxstorage
JSON-RPC
certificate
open http
Certificate Authority
load certificate
Web Portal
client
certificateservices
nesssiweb portal
nesssi
node
node
node
node
web form SOAP http queue
fetchproxy
select useraccount
sandboxstorage
open http
certificatepolicies
Exercise: Running a Nesssi Service
see http://us-vo.org/nesssi
SessionID and Sandbox
• Identify which job we are talking about• 32 character hex string eg cb28d0753a7fec9a485981f741d425ec
• Used to monitor a running jobsessionID = remoteserver.cutout.init()msg = remoteserver.cutout.monitor(sessionID)
• Used to form URL where results appear, eg• http://dtf-test1.sdsc.teragrid.org:8080
/clarens/shell/cb/cb28d0753a7fec9a485981f741d425ec/cutouts/index.htm
• If you lose the sessionID, you lose your job
DPOSS Mosaic Service
nesssiServer=nesssi.client('https://envoy.cacr.caltech.edu:8443/clarens/',debug=0)
mosaic_loc = "-ra 49.1 -dec 60.1 -rawidth 0.5 -decwidth 0.5 -filt f -bgcorr 0"
session = nesssiServer.dpossMosaic.mosaic(mosaic_loc)print "Your session ID is %s." % session
msg = dbsvr.dpossMosaic.monitor(session)print msg
Repeat the monitoring
nesssiServer.dpossMosaic.mosaic (“-ra 49.1 -dec 60.1 -rawidth 0.5 -decwidth 0.5 -filt f -bgcorr 0”)
Cutout Service
nesssiServer=nesssi.client('https://envoy.cacr.caltech.edu:8443/clarens/',debug=0)sessionID = nesssiServer.cutout.init()print "Session id is ", sessionID
# Upload locations filenesssiServer.upload_file(“inputfile.xml”, ”inputfile.xml”)
# Arguments for service, surveys to use and cutout sizeargs = "-surveys PQ:gr,PQ:gi,PQ:z1,PQ:z2,SDSS:r,SDSS:i,SDSS:z,2MASS:k,2MASS:h "args += "-size 64"
# Run servicenesssiServer.cutout.run(sessionID, args)
Cutout Monitoring
cutouts from Palomar-Quest, SDSS, 2MASSof sources from Veron quasar catalog
Synoptic Coaddition service
Palomar-Quest Survey
Coverage map
Max=18
Making a Service
• Developer builds script• Keyword-value pairs on command line [+uploaded files]• Sandbox location given on cmdline -- all files staged there• Should make index.htm in sandbox for progress• Make Nesssi connector for init(), upload(), run(), monitor()
• Nesssi admin installs your service• Interview first• Symlink to code• Code is cached, restart server after edit• Developer gets right to restart server (running jobs not affected)
• Service instantiations farmed out to cluster with PBS
Server side code
Application example:dposs.py -dir sandbox \
-ra 123 -dec 22.7 \ -rawidth 0.4 -decwidth 0.4 \ -filt j -bgcorr 1It should:
(1) Use keyword-value arguments and uploaded files(2) Read/Write results in the given sandbox directory(3) Write a progress file in sandbox/index.htm(4) Estimate limits for anon/weak/strong certs
Service code will be symlinked from server code directoryRequires sudo server restart to see the service
Client-side Javascript
<input type="button" onclick="connect_nesssi('dposs')" name="Connect" value="Connect to Nesssi">
Service nameExpect to run remote services called:
dposs.init(), dposs.run(), dposs.monitor()
Client-side Javascript
<form name="Parameters"><input name="ra" value="202.4682”Etc…
function getparams() {var params = "-ra " + document.Parameters.ra.value + " " + "-rawidth " + document.Parameters.rawidth.value + " " + "-dec " + document.Parameters.dec.value + " " + "-decwidth " + document.Parameters.decwidth.value + " " + "-filt " + filt + " " + "-bgcorr " + bgCorr; return params;}
The form for the user
Developer converts the form to a string
Nesssi Assets
• Graduated security• Anonymous, Registered, Known
• Multiple interfaces• Fat browser, Web proxy, Scripted
• Multiple implementations• cacr.caltech.edu and sdsc.teragrid.org
• Some useful services• Hyperatlas mosaic, Cutouts, Synoptic coaddition
• Teragrid acceptance of security model• Server runs a job as somebody else• Anonymous access to TeraGrid!!