Date post: | 28-Mar-2018 |
Category: |
Documents |
Upload: | dinhkhuong |
View: | 214 times |
Download: | 0 times |
3
The Application-Infrastructure Gap
Dynamicand/or
DistributedApplications
A
1
B
1
99
Shared Distributed Infrastructure
4
Bridging the Gap:Service-Oriented Infrastructure
Service-oriented applicationsWrap applications asservices
Compose applicationsinto workflows
Service-orientedinfrastructure
Provision physicalresources to support application workloads
ApplnService
ApplnService
Users
Workflows
Composition
Invocation
Provisioning
5
Globus is Service-Oriented Infrastructure Technology
Software for service-oriented infrastructureService enable new & existing resources
E.g., GRAM on computer, GridFTP on storage system, custom application service
Uniform abstractions & mechanisms
Tools to build applications that exploit service-oriented infrastructure
Registries, security, data management, …
Open source & open standardsEach empowers the other
Enabler of a rich tool & service ecosystem
6
Globus asService-Oriented Infrastructure
IBM
IBM
Uniform interfaces,security mechanisms,Web service transport,
monitoring
Computers StorageSpecializedresource
UserApplication
UserApplication
UserApplication
IBM
IBM
GRAM GridFTPHost EnvUser Svc
DAIS
Database
ToolTool
Workflow
Credent.
Host EnvUser Svc
Registry
7
A Typical eScience Use of Globus:Network for Earthquake Eng. Simulation
Links instruments, data, computers, people
8
An eBusiness Use of Globus:SAP Demonstration @ GlobusWorld3 Globus-enabled applns:
CRM: Internet Pricing Configurator (IPC)
CRM: Workforce Management (WFM)
SCM: Advanced Planner & Optimizer (APO)
Applications modified to:Adjust to varying demand & resources
Use Globus to discover & provision resources
IPCDispatcher
IPCServerRequest:
Price QueryDelegation of
Request
Response: PricelistDepending on:- Time- Discount- Number of Items- …
Web Browsers / Batch Processes(typically several thousand requests)
IPCServer
1
2
2
3
SAP AG R/3 Internet Pricing & Configurator (IPC)
9
A
1
B
1
9
A
1 2
B
1 2
1
7
Strategy A:Reduce TCO
18 Servers 11 Servers
A
1 2
B
1 2
1
14
Strategy B:Improve
Performance9 Servers Each 16 Servers Each
9
Cheaper and/or Faster ...
11
Globus Toolkit
Core Web servicesInfrastructure for building new services
SecurityApply uniform policy across distinct systems
Execution managementProvision, deploy, & manage services
Data managementDiscover, transfer, & access large data
MonitoringDiscover & monitor dynamic services
Data MgmtSecurity CommonRuntime
Execution Mgmt
Info Services
Non-WS Components
Pre-WSAuthenticationAuthorization
GridFTP C CommonLibraries
Globus Toolkit version 2 (GT2)
Grid ResourceAlloc. Mgmt(GRAM)
Monitoring& Discovery
(MDS)
Web Services
Components
Data MgmtSecurity CommonRuntime
Execution Mgmt
Info Services
Web Services
Components
Non-WS Components
Pre-WSAuthenticationAuthorization
GridFTP C CommonLibraries
WSAuthenticationAuthorization
ReliableFile
Transfer
Data Access& Integration
Grid ResourceAlloc. Mgmt(WS GRAM)
MDS3 Java WS Core
CommunityAuthorization
ReplicaLocation
eXtensibleIO (XIO)
Globus Toolkit version 3 (GT3)
Grid ResourceAlloc. Mgmt(GRAM)
Monitoring& Discovery
(MDS)
Data MgmtSecurity CommonRuntime
Execution Mgmt
Info Services
Web Services
Components
Non-WS Components
Pre-WSAuthenticationAuthorization
GridFTP Pre-WSGRAM
Pre-WSMonitoring& Discovery
C CommonLibraries
WSAuthenticationAuthorization
ReliableFile
Transfer
Data Access& Integration
Grid ResourceAlloc. Mgmt(WS GRAM)
MDS-Registry
Java WS Core
CommunityAuthorization
ReplicaLocation
eXtensibleIO (XIO)
CredentialMgmt
WorkspaceManagement
Delegation
Globus Toolkit version 4 (GT4)
DataReplication
MDS-Trigger
Python WS Core
C WS Core
WebMDSCommunitySchedulingFramework
Data MgmtSecurity CommonRuntime
Execution Mgmt
Info Services
Web Services
Components
Non-WS Components
Pre-WSAuthenticationAuthorization
GridFTP Pre-WSGRAM
Pre-WSMonitoring& Discovery
C CommonLibraries
WSAuthenticationAuthorization
ReliableFile
Transfer
Data Access& Integration
Grid ResourceAlloc. Mgmt(WS GRAM)
MDS-Registry
Java WS Core
CommunityAuthorization
ReplicaLocation
eXtensibleIO (XIO)
CredentialMgmt
CommunitySchedulingFramework
Delegation
Globus Toolkit version 4 (GT4)
DataReplication
MDS-Trigger
Python WS Core
C WS Core
WebMDS
WorkspaceManagement
16
Java Services in Apache AxisPlus GT Libraries and Handlers
YourJava
Service
YourPythonService
YourJava
Service RFT
GR
AM
Del
egat
ion
Inde
xTr
igge
rA
rchi
ver
pyGlobusWS Core
YourC
Service
C WS Core
RLS
Pre
-WS
MD
S
CAS
Pre
-WS
GR
AM
Sim
pleC
A
MyP
roxy
OG
SA
-DA
IG
TCP
Grid
FTP
C Services using GT Libraries and HandlersSERVER
CLIENT
InteroperableWS-I-compliant
SOAP messaging
YourJavaClient
YourC
Client
YourPythonClient
YourJavaClient
YourC
Client
YourPythonClient
YourJavaClient
YourC
Client
YourPythonClient
YourJavaClient
YourC
Client
YourPythonClient
X.509 credentials =common authentication
Python hosting, GT Libraries
GT4 Components
17
Our Goals for GT4
Usability, reliability, scalability, …Web service components have quality equal or superior to pre-WS components
Documentation at acceptable quality level
Consistency with latest standards (WS-*, WSRF, WS-N, etc.) and Apache platform
WS-I Basic Profile compliant
WS-I Basic Security Profile compliant
New components, platforms, languagesAnd links to larger Globus ecosystem
18
GT4 Web Services CoreSupports both Globus services (GRAM, RFT, Delegation, etc.) & user-developed services
Redesign to enhance scalability, modularity, performance, usability
Leverages existing WS standardsWS-I Basic Profile: WSDL, SOAP, etc.
WS-Security, WS-Addressing
Adds support for emerging WS standardsWS-Resource Framework, WS-Notification
Java, Python, & C hosting environmentsJava is standard Apache
19
CustomWeb
ServicesWS-Addressing, WSRF,
WS-Notification
CustomWSRF Web
Services
GT4WSRF Web
Services
WSDL, SOAP, WS-Security
User Applications
Reg
istry
Adm
inis
tratio
n
GT4
Con
tain
er
GT4 Web Services Core
20
Grid Security
Control access to shared servicesAddress autonomous management, e.g., different policy in different work-groups
Support multi-user collaborationsFederate through mutually trusted services
Local policy authorities rule
Allow users to set up dynamic trust domainsPersonal collection of resources working together based on trust of user
21
GT4 Security
VO
RightsUsers
Rights’
ComputeCenter
Access
Services (runningon user’s behalf)
Rights
Local policyon VO identityor attributeauthority
CAS or VOMSissuing SAMLor X.509 ACs
SSL/WS-Securitywith ProxyCertificates
Authz Callout
KCA
MyProxy
22
GT4 Security
Extensible authorization framework based on Web services standards
SAML-based authorization calloutAs specified in GGF OGSA-Authz WG
Integrated policy decision engineXACML policy language, per-operation policies, pluggable
Credential management serviceMyProxy (One time password support)
Community Authorization Service
Standalone delegation serviceImplements IETF RFC
24
GT4 Data Management
Stage large data to/from nodes
Replicate data for performance & reliability
Locate data of interest
Provide access to diverse data sourcesFile systems, parallel file systems, hierarchical storage (GridFTP)
Databases (OGSA DAI)
25
GT4 Data Functions
Find your data: Replica Location ServiceManaging ~40M files in production settings
Move/access your data: GridFTP, RFTHigh-performance striped data movement
27 Gbit/s memory-to-memory on a 30 Gbit/s link (90% utilization) with 32 IBM TeraGrid nodes.
17.5 Gbit/s disk-to-disk limited by the storage system
Reliable movement of 120,000 files (so far)
Couple data & execution managementGRAM uses GridFTP and RFT for staging
26
GridFTP in GT4
100% Globus codeNo licensing issues
Stable, extensible
IPv6 Support
XIO for different transports
Striping multi-Gb/sec wide area transport
PluggableFront-end: e.g., future WS control channel
Back-end: e.g., HPSS, cluster file systems
Transfer: e.g., UDP, NetBLT transport
p g
02000400060008000
100001200014000160001800020000
0 10 20 30 40 50 60 70
Degree of Striping
Ban
dwid
th (M
bps)
# Stream = 1 # Stream = 2 # Stream = 4# Stream = 8 # Stream = 16 # Stream = 32
Disk-to-disk onTeraGrid
27Reliable File Transfer:Third Party Transfer
RFT Service
RFT Client
SOAP Messages
Notifications(Optional)
DataChannel
Protocol Interpreter
MasterDSI
DataChannel
SlaveDSI
IPCReceiver
IPC Link
MasterDSI
Protocol Interpreter
Data Channel
IPCReceiver
SlaveDSI
Data Channel
IPC Link
GridFTP Server GridFTP Server
Fire-and-forget transfer
Web services interface
Many files & directories
Integrated failure recovery
Has transferred 900K files
28
Replica Location Service
Identify location of files via logical to physical name map
Distributed indexing of names, fault tolerant update protocols
GT4 version scalable & stable
Managing ~40 million files across ~10 sites
IndexIndex
50 M17575 M
10 M2421 M
1 M2<110K
Bloom filter (bits)
Bloom filter
(secs)
Update send (secs)
Local DB
29
Execution Management (GRAM)
Common WS interface to schedulersUnix, Condor, LSF, PBS, SGE, …
More generally: interface for process execution management
Lay down execution environment
Stage data
Monitor & manage lifecycle
Kill it, clean up
A basis for application-driven provisioning
30
GT4 GRAM
2nd-generation WS implementation optimized for performance, flexibility, stability, scalability
Streamlined critical pathUse only what you need
Flexible credential managementCredential cache & delegation service
GridFTP & RFT used for data operationsData staging & streaming output
Eliminates redundant GASS code
31
GRAMservices
GT4 Java Container
GRAMservices
Delegation
RFT FileTransfer
Transferrequest
GridFTPRemote storage element(s)
Localscheduler
Userjob
Compute element
GridFTPsudo
GRAMadapter
FTPcontrol
Local jobcontrol
Delegate
FTP data
Clie
nt
Job
functions
Delegate
Service host(s) and compute element(s)
GT4 GRAM Structure:WSRF/WSN Poster Child
33
Monitoring and Discovery“Every service should be monitorable and discoverable using common mechanisms”
WSRF/WSN provides those mechanisms
A common aggregator framework for collecting information from services, thus:
MDS-Index: Xpath queries, with caching
MDS-Trigger: perform action on condition
(MDS-Archiver: Xpath on historical data)
Deep integration with Globus containers & services: every GT4 service is discoverable
GRAM, RFT, GridFTP, CAS, …
34
GT4 Container
GT4 Monitoring & Discovery
GRAM User
MDS-Index
GT4 Cont.
RFT
MDS-Index
GT4 Container
MDS-Index
GridFTP
adapter
Registration &WSRF/WSN Access
Custom protocolsfor non-WSRF entities
Clients(e.g., WebMDS)
Automatedregistrationin container
WS-ServiceGroup
36
Working with GT4
Download and use the software, and provide feedback
Join [email protected] mail list
Review, critique, add to documentationGlobus Doc Project: http://gdp.globus.org
Tell us about your GT4-related tool, service, or application
Email [email protected]
38
2005 and Beyond
We have a solid Web services base
We now want to build, on that base, a open source service-oriented infrastructure
Virtualization
New services for provisioning, data management, security, VO management
End-user tools for application development
Etc., etc.
39
Short-Term Priorities: Security
Improve GSI error reporting & diagnostics
Secure password, one-time password, Kerberos support for initial log on
Trust roots, use of GridLogon
Identity/attribute assertions in GT auth. callouts (e.g., Shib, PERMIS, VOMS, SAML)
Extend CAS admin & policy support
Security logging with management control for audit purposes
40
Short-Term Priorities: Data Management
Space & bandwidth management in GridFTP
Concurrency in globus-url-copy
Priorities in RFT
Data replication service
Enhance policy support in data services
Physical file name creation service
Scalable & distributed metadata manager
41
Short-Term Priorities: Execution Management
Implement GGF JSDL once finalized
Advance reservation support
Policy-driven restart of “persistent” jobs
Improved information collection for jobs
Improved management of job collections
Credential refresh
Virtual workspace service
Integration of virtual machines (Xen, VMware) and associated services
42
Short-Term Priorities: Information Services
Many more information sources, including gateways to other systems
Automated configuration of monitoring
Specialized monitoring displays
Performance optimization of registry
Archiver service
Helper tools to streamline integration of new information sources
43
Short-Term Priorities: Core
Streamlined container configuration
Remote management interface
Dynamic service deployment
Service isolation: multiple service instances
WS-Notification, subscription performance
Full functionality in C WS Core
Optimized WS-ServiceGroup support
WS-SecureConversation support
45
How Globus Works
Globus is a distributed open source community with many contributors & users
CVS, documentation, bugzilla, email lists
Modular structure allows many to contribute
Globus Alliance Board provides governance when needed
Meritocracy: individuals who demonstrate ongoing contributions & commitment
Primarily: what to include, when to release
Globus Alliance is an informal partnership of organizations led by Board members
46
Evolution of the Globus Alliance
Argonne/U.Chicago (Childers, Foster): 1995
USC/ISI (Kesselman): 1995
Edinburgh (Atkinson, Parsons): 2003
Swedish PDC (Johnsson, Mulmo): 2003
NCSA (Welch): 2004
Univa (Czajkowski, Tuecke): 2004
Other contributors will surely be added
47
From eScience to eBusiness
Since ~2001, growing interest in Globus for commercial use
Enterprises, IT vendors, ISVs asking Globus leaders to address commercial needs
But hard to do in a research laboratory
In response, we have created two new organizations
Globus Consortium
Univa
48
Globus Consortium(www.globusconsortium.com)
Nonprofit organization funded by companies to advance Globus Toolkit for enterprise use
Initial sponsor members: HP, IBM, Intel, Sun
Initial contributors: Nortel, Univa
First two projects already identifiedMember-driven software quality improvements
Contributions to job submission standards
Other projects to be defined, e.g.Develop new features key to enterprise use
Education & outreach
49
Provider of commercial support, services, & products around open source Globus
Commercial distribution of GT4 & beyond
Integration with enterprise systems
Committed to open source & open standards
Founded by Tuecke, Foster, KesselmanTuecke left Argonne to be CEO
Foster, Kesselman remain at Argonne, ISI
Experienced management team Rich Miller, Vas Vasiliadis, Paul Davé, Bob Mandel
26776 U.S.
2753 China
1318 Japan
1017 India
750 U.K.
495 Italy
488 Germany
391 Brazil
328 S. Korea
306 Taiwan
268 France
241 Canada
211 Viet Nam
211 Spain
202 Russia
187 Pakistan
159 Australia
142 Singapore
131 Greece
119 Colombia
111 Poland
109 Netherlands
107 Thailand
94 Switzerland
77 Chile
74 Sweden
68 Belgium
66 Venezuela
66 Romania
64 Indonesia
62 Mexico
61 Turkey
60 Malaysia
58 Portugal
57 Austria
54 Ireland
44 Hong Kong
40 Hungary
38 Egypt
38 Argentina
34 Uruguay
31 Ukraine
29 Slovakia
25 Israel
23 Yugoslavia
23 Iran
22 Bulgaria
22 Uzbekistan
22 Czech Rep.
22 N. Korea
21 Lithuania
21 Croatia
20 Finland
20 New Zealand
17 Nigeria
17 South Africa
16 Jordan
16 Slovenia
16 Afghanistan
15 Denmark
15 Philippines
14 Vanuatu
14 Luxembourg
14 Tunisia
12 Virgin Is. (U.K.)
12 Peru
12 Yemen
11 Norway
11 Algeria
11 Armenia
10 Iceland
9 Zambia
9 Virgin Is. (U.S.)
9 Uganda
9 Bosnia & Herz.
8 Kenya
7 Zimbabwe
7 Saudi Arabia
7 Ecuador
7 Macedonia
6 Bolivia
6 Comoros
6 Zaire
6 Lebanon
5 Puerto Rico
5 Namibia
5 Togo
5 Tajikistan
5 Paraguay
5 Albania
5 Sudan
4 Estonia
4 Camaroon
4 Ghana
4 Tuvalu
4 Costa Rica
4 Cuba
4 UAE
4 Tonga
4 W. Samoa
4 Tanzania
3 Syria
3 Bahamas
3 Ethiopia
3 Mongolia
3 Sri Lanka
3 Wallis & Futuna Is.
3 Belarus
3 Bangladesh
2 Falkland Islands
2 Kuwait
2 Sierra Leone
2 Trinidad & Tobago
2 Guyana
2 American Samoa
2 Andorra
2 Georgia
2 Cook Islands
2 Turkmenistan
2 Gabon
2 The Gambia
2 Kazakhstan
2 Macau
2 Malta
2 Jamaica
2 Latvia
2 Turks & Caicos
1 Bhutan
1 Ascension Island
1 Cyprus
1 Mozambique
1 Tokelau
1 Greenland
1 Nepal
1 Swaziland
1 Iraq
1 Serbia
1 Barbados
1 Cambodia
1 Qatar
1 Saint Vincent
1 Laos
1 San Marino
1 Libya
1 Benin
1 Angola
1 Chad
1 Gibraltar
1 Haiti
1 Guatemala
1 Malawi
1 Equatorial Guinea
1 Palau
1 Bermuda
1 Botswana
1 Suriname 38669 d
ow
nlo
ads
in 2
004 f
rom
glo
bus.
org
26776 U.S.
2753 China
1318 Japan
1017 India
750 U.K.
495 Italy
488 Germany
391 Brazil
328 S. Korea
306 Taiwan
268 France
241 Canada
211 Viet Nam
211 Spain
202 Russia
187 Pakistan
159 Australia
142 Singapore
131 Greece
119 Colombia
111 Poland
109 Netherlands
107 Thailand
94 Switzerland
77 Chile
74 Sweden
68 Belgium
66 Venezuela
66 Romania
64 Indonesia
62 Mexico
61 Turkey
60 Malaysia
58 Portugal
57 Austria
54 Ireland
44 Hong Kong
40 Hungary
38 Egypt
38 Argentina
34 Uruguay
31 Ukraine
29 Slovakia
25 Israel
23 Yugoslavia
23 Iran
22 Bulgaria
22 Uzbekistan
22 Czech Rep.
22 N. Korea
21 Lithuania
21 Croatia
20 Finland
20 New Zealand
17 Nigeria
17 South Africa
16 Jordan
16 Slovenia
16 Afghanistan
15 Denmark
15 Philippines
14 Vanuatu
14 Luxembourg
14 Tunisia
12 Virgin Is. (U.K.)
12 Peru
12 Yemen
11 Norway
11 Algeria
11 Armenia
10 Iceland
9 Zambia
9 Virgin Is. (U.S.)
9 Uganda
9 Bosnia & Herz.
8 Kenya
7 Zimbabwe
7 Saudi Arabia
7 Ecuador
7 Macedonia
6 Bolivia
6 Comoros
6 Zaire
6 Lebanon
5 Puerto Rico
5 Namibia
5 Togo
5 Tajikistan
5 Paraguay
5 Albania
5 Sudan
4 Estonia
4 Camaroon
4 Ghana
4 Tuvalu
4 Costa Rica
4 Cuba
4 UAE
4 Tonga
4 W. Samoa
4 Tanzania
3 Syria
3 Bahamas
3 Ethiopia
3 Mongolia
3 Sri Lanka
3 Wallis & Futuna Is.
3 Belarus
3 Bangladesh
2 Falkland Islands
2 Kuwait
2 Sierra Leone
2 Trinidad & Tobago
2 Guyana
2 American Samoa
2 Andorra
2 Georgia
2 Cook Islands
2 Turkmenistan
2 Gabon
2 The Gambia
2 Kazakhstan
2 Macau
2 Malta
2 Jamaica
2 Latvia
2 Turks & Caicos
1 Bhutan
1 Ascension Island
1 Cyprus
1 Mozambique
1 Tokelau
1 Greenland
1 Nepal
1 Swaziland
1 Iraq
1 Serbia
1 Barbados
1 Cambodia
1 Qatar
1 Saint Vincent
1 Laos
1 San Marino
1 Libya
1 Benin
1 Angola
1 Chad
1 Gibraltar
1 Haiti
1 Guatemala
1 Malawi
1 Equatorial Guinea
1 Palau
1 Bermuda
1 Botswana
1 Suriname 38669 d
ow
nlo
ads
in 2
004 f
rom
glo
bus.
org
26776 U.S. 2753 China 1318 Japan 1017 India
750 U.K. 495 Italy 488 Germany 391 Brazil 328 S. Korea306 Taiwan
Top 10
53
The Globus Ecosystem
Globus components address core issues relating to resource access, monitoring, discovery, security, data movement, etc.
GT4 being the latest version
A larger Globus ecosystem of open source and proprietary components provide complementary components
A growing list of components
These components can be combined to produce solutions to Grid problems
We’re building a list of such solutions
54
Many Tools Build on, or Can Contribute to, GT4-Based Grids
Condor-G, DAGman
MPICH-G2
GRMS
Nimrod-G
Ninf-G
Open Grid Computing Env.
Commodity Grid Toolkit
GriPhyN Virtual Data System
Virtual Data Toolkit
GridXpert Synergy
Platform Globus Toolkit
VOMS
PERMIS
GT4IDE
Sun Grid Engine
PBS scheduler
LSF scheduler
GridBus
TeraGrid CTSS
NEES
IBM Grid Toolbox
…
55DocumentingThe Grid
Ecosystem
The Grid Ecosystem: Software Components for Grid SystemsAnd Applications
www.grids-center.org
56
Example Solutions
Portal-based User Reg. System (PURSE)
VO Management Registration Service
Service Monitoring Service
TeraGrid TGCP Tool
Lightweight Data Replicator
GriPhyN Virtual Data System
57
The Globus Developer Community
We want to facilitate community contributions to the Globus software
Increasingly open governance & mechanisms
Documentation of larger ecosystem
What else can we do?
Tell us how you think we should evolveE.g., adopt Apache processes?
Contribute more of Globus to Apache?We’ve already made major contributions to Apache Axis, WS-Addressing, and WS-Security
Also WSRF/WSN Apache (Apollo, Hermes)
58
Globus and its User Community
How can “we” best support “you”?We try to provide the best software we can
We use bugzilla & other community tools
We work to grow the set of contributors
How can “you” best support “us”?Become a contributor: of software, bug fixes, answers to questions, documentation
Provide us with success stories that can justify continued Globus development
Promote Globus within your communities
Contact me at [email protected]
60Grid2003 Open Science Grid30 sites (2100-2800 CPUs) & growing400-1300 concurrent jobs8 substantial applications + CS experimentsRunning since October 2003
Korea
http://www.ivdgl.org/grid2003
61
Open Science Grid ComponentsComputers & storage at 30+ sites
3000+ CPUs
Uniform service environment at each siteGlobus Toolkit provides basic authentication, execution management, data movement
Pacman installation system enables installation of numerous other VDT and application services
Global & virtual organization servicesCertification & registration authorities, VO membership services, monitoring services
Client-side tools for data access & analysis Virtual data, execution planning, DAG management, execution management, monitoring
IGOC: iVDGL Grid Operations Center
62
Example OSG Applications
CMS proton-proton collision simulation
ATLAS proton-proton collision simulation
LIGO gravitational wave search
SDSS galaxy cluster detection
ATLAS interactive analysis
BTeV proton-antiproton collision simulation
SnB biomolecular analysis
GADU/Gnare genone analysis
Various computer science experimentswww.ivdgl.org/grid2003/applications