Post on 27-Mar-2015
transcript
The Moab Grid SuiteCSS´ 06 – Bonn – July 28, 2006
Computation in Life Science Research
High-throughput virtual screening (serial)
Molecular dynamics or Monte Carlo simulations (parallel)
Free energy calculations (parallel)
Cheminformatics (serial)
Structure prediction and design (serial/parallel)
Quantum chemistry (serial/parallel)
©2006 anterio consult & research Moab Grid Suite
Multiple sites, continents
Multiple applications, hardware
Different policies, operating systems, security
Problems a Pharma/Biotech May Face
Moab Grid Suite
©2006 anterio consult & research Moab Grid Suite
Moab Grid Suite is a policy-based grid management solution that integrates scheduling, managing, monitoring and reporting of workloads across
independent clusters.
Moab Grid Suite
Boehringer Ingelheim
Biogen Idec
Fred Hutchinson Cancer Research Center
TeraGrid, WestGrid, CMA, CDAC
©2006 anterio consult & research Moab Grid Suite
Exemples of Customers
Technical Overview
Message PassingMessage Passing
SerialSerialParallelParallel ApplicationApplication
Resource ManagerResource Manager
Operating SystemOperating System
Hardware (Cluster or SMP)Hardware (Cluster or SMP)
AdminAdmin UsersUsers
Se
cu
rityS
ec
urity
ApplicationApplication
PortalPortal
CLICLI
GUIGUI
Cluster Workload Manager: Scheduler, Policy Manager, Integration PlatformCluster Workload Manager: Scheduler, Policy Manager, Integration Platform
Grid Workload Manager: Scheduler, Policy Manager, Integration PlatformGrid Workload Manager: Scheduler, Policy Manager, Integration Platform
©2006 anterio consult & research Moab Grid Suite
Cluster Stack / Framework:
Architecture
Technical Overview
©2006 anterio consult & research Moab Grid Suite
What Moab does
Moab Grid Suite
©2006 anterio consult & research Moab Grid Suite
Optimizes Resource Utilization with Intelligent Scheduling and Advanced Reservations
Unifies Cluster Management across Varied Resources and Services
Dynamically Adjusts Workload to Enforce Policies and Service Level Agreements
Automates Diagnosis and Failure Response
What Moab does not do
Moab Grid Suite
©2006 anterio consult & research Moab Grid Suite
Does not does do resource management (usually) Does not install the system (usually) Not a storage manager Not a license manager Does not do message passing
ClusterB
ClusterB
Grid Types
Administrator
ClusterA
Local Area Grid (LAG)
ClusterB
ClusterC
Shared User SpaceShared Data Space
Moab
ClusterA
Wide Area Grid (WAG)
ClusterB
ClusterC
Multiple User SpacesMultiple Data Spaces
Moab (Master)
Moab Moab Moab
ClusterA
ClusterB
ClusterC
Moab (Grid Head Node)
Moab Moab Moab
Centralized Management
ClusterA
ClusterC
Moab
Centralized & Local Management
All Grid Rules
Moab (Grid Head Node)Shared Grid Rules
Local Grid Rules
Moab
Local Grid Rules
Moab
Local Grid Rules
ClusterA
ClusterC
Moab
Local Management“Peer to Peer”
Local Grid Rules
Moab
Local Grid Rules
Moab
Local Grid Rules
Management Scenarios
©2006 anterio consult & research Moab Grid Suite
Grid Relationship Combination
Administrator
ClusterA
ClusterB
ClusterC
Shared User SpaceShared Data Space
Moab
Multiple User SpacesMultiple Data Spaces
ClusterD
ClusterE
Moab (Grid Head Node)Shared Grid Rules
Local Area Grid Rules
Moab
Local Grid Rules
Moab
Local Grid Rules
1. Join local area grids into wide area grids --- Moab is able to facilitate virtually any grid relationship.
1
©2006 anterio consult & research Moab Grid Suite
Grid Relationship Combination
Administrator
ClusterA
ClusterB
ClusterC
Shared User SpaceShared Data Space
Moab
Multiple User SpacesMultiple Data Spaces
ClusterD
ClusterE
Moab (Grid Head Node)Shared Grid Rules
Local Area Grid Rules
Moab
Local Grid Rules
Moab
Local Grid Rules
ClusterF
ClusterG
ClusterH
Moab
Local Grid Rules
Moab
Local Grid Rules
Moab
Local Grid Rules
2. Join wide area grids to other wide area grids (whether they be managed centrally, locally - “peer to peer” or mixed)
12
©2006 anterio consult & research Moab Grid Suite
Grid Relationship Combination
Administrator
ClusterA
ClusterB
ClusterC
Shared User SpaceShared Data Space
Moab
Multiple User SpacesMultiple Data Spaces
ClusterD
ClusterE
Moab (Grid Head Node)Shared Grid Rules
Local Area Grid Rules
Moab
Local Grid Rules
Moab
Local Grid Rules
ClusterF
ClusterG
ClusterH
Moab
Local Grid Rules
Moab
Local Grid Rules
MoabLocal Grid Rules
Hosting Site
Moab
Local Grid Rules
3. Resource sharing can be in one direction for use with hosting centers, or to bill out resources to other sites
12
3
©2006 anterio consult & research Moab Grid Suite
Grid Relationship Combination
Administrator
ClusterA
ClusterB
ClusterC
Shared User SpaceShared Data Space
Moab
Multiple User SpacesMultiple Data Spaces
ClusterD
ClusterE
Moab (Grid Head Node)Shared Grid Rules
Local Area Grid Rules
Moab
Local Grid Rules
Moab
Local Grid Rules
ClusterF
ClusterG
ClusterH
Moab
Local Grid Rules
Moab
Local Grid Rules
MoabLocal Grid Rules
Hosting Site
Moab
Local Grid Rules
4. Have multiple levels of grid relationships (e.g. conglomerates within conglomerates within conglomerates)
12
3
©2006 anterio consult & research Moab Grid Suite
4
Administrator
Sovereignty: Local vs. Centralized Management Policies
©2006 anterio consult & research Moab Grid Suite
Local Admin
Each Admin can manage their own cluster
Submit to either:• Local cluster• Specified cluster(s) in the grid • Generically to the grid
Local Cluster AResources
Grid AllocatedResources
Portion Allocated
to Grid
Local Admin can apply policies to manage:
1. Local user access to local cluster resources
2. Local user access to grid resources
3. Outside grid user access to local cluster resources (general or specific policies)
Local Users
Outside Grid Users
1
2
3
Grid Administration Body
Grid Administration Body can apply policies to manage:
A. General grid policies (Sharing, Priority, Limits, etc.)
A
Administrator
Data Staging
Manages intra-cluster and inter-cluster job data staging requirements so as to minimize resource inefficiencies and maximize system utilization
Prevent the loss of compute resources due to data blocking and can significantly improve cluster performance.
©2006 anterio consult & research Moab Grid Suite
Administrator
©2006 anterio consult & research Moab Grid Suite
ClusterA
Moab
Local Grid Rules
4
Start job after the file is verified to be in the right location
1
StorageSystem
Data Manager
Job Submission
3
Benefits:Prevents non-staged jobs from blocking usage of nodes
Drawbacks:No job-centric prioritization takes place in the order of which data gets staged first
2
Verified Data Staging
Administrator
©2006 anterio consult & research Moab Grid Suite
ClusterA
Moab
Local Grid Rules
5
Priority order of data staging
1
StorageSystem
Data Manager
Job Submission
4
3
Priority Jobs First
2
Prioritized Data Staging
Administrator
©2006 anterio consult & research Moab Grid Suite
Priority order of data staging and data-staging centric scheduling
Benefits:Prevents non-staged jobs from blocking usage of nodes Provides soft prioritization of data staging requestsIntelligently schedule resources based on data staging information
Drawbacks:Prioritization is only softly provided
ClusterA
Moab
Local Grid Rules
6
StorageSystem
Data Manager
Job Submission
5Priority
Jobs First
32
4
1
Fully Scheduled Data Staging: External Storage
Administrator
©2006 anterio consult & research Moab Grid Suite
Priority order of data staging and data-staging centric scheduling
Benefits:Prevents non-staged jobs from blocking usage of nodes Provides soft prioritization of data staging requestsIntelligently reserves resources based on data staging information
Drawbacks:Prioritization is only softly provided
ClusterA
Moab
Local Grid Rules
Data Manager
Job Submission
52
4S
S S
Storage is on Local Compute Nodes
1
3Priority
Jobs First
6
Fully Scheduled Data Staging: Local Storage
Administrator
©2006 anterio consult & research Moab Grid Suite
Monitoring performance statistics of multiple independent clusters
Detecting and diagnosing failures from geographically distributed clusters
Tracking cluster, storage, network, service, and application resources
Generating load-balancing and resource state information for users and middleware services
Information Services
Administrator
©2006 anterio consult & research Moab Grid Suite
Administer Resources and Workload Policies Through an Easy-to-Use Graphical User Interface
Monitor, Diagnose and Report Resource Allocation and Usage
Moab Grid ManagerTM
Administrator
Moab Grid ManagerTM
©2006 anterio consult & research Moab Grid Suite
Service Monitoring and Management:
Administrator
Moab Grid ManagerTM
©2006 anterio consult & research Moab Grid Suite
RealTime Performance and Accounting Analysis:
End User
Moab Access PortalTM
Submit jobs from a web browser View and modify only your own workload Assist end users to self-manage behaviours
Assists users in better utilizing resources:
General info Job eval. Completed job failure post-mortem. Job start time estimates. Job control. Reservation control.
©2006 anterio consult & research Moab Grid Suite
Contact
Tassilo Eckerle
Sales Manager
Phone: +49 (6 21) 4 00 41-34Fax: +49 (6 21) 4 00 41-40Mobile: +49 (1 63) 4 00 41 34E-Mail: teckerle@anterio.com
Jianxin Duan, PhD
Research Scientist
Phone: +49 (6 21) 4 00 41-57Fax: +49 (6 21) 4 00 41-40
E-Mail: jduan@anterio.com
Thank you for your attention!
For further questions please do not hesitate to contact us.