Reflections on Production Grids Enterprise and
Research
Frederica Darema <[email protected]>Dynamic Data Driven Application Systems (DDDAS)Division of CCF&Fillia Makedon <[email protected]>Office of CyberinfrastructureNational Science FoundationGlobal Grid Forum, Athens,Greece
2
Nat
ion
al S
cien
ce F
ou
nd
atio
n Outline Production Grids for Science and Engineering
Research applications
Challenges
Enterprise applications
Non homogeneous architectures
Heterogeneous solutions
Interoperability
Servicing the user
Service oriented architectures
Balance optimality with flexibility
Enterprise applications driving grid computing Utility computing
Resource Brokers Coordinate distributed resources and users
3
Nat
ion
al S
cien
ce F
ou
nd
atio
n Production Grids for Science and Engineering
Primary task to enable large-scale scientific research Not just a test-bed for software development, or
experimental computer science. Severe constraints in construction because:
access to resources subject to stringent requirements of security and high quality of service.
Production grid resources may not be for Grid use by design.
Production Grid middleware cannot control policy on such Grids, it must co-operate with the site policy and resource management systems.
Challenge in dealing with non computational resources, (e.g., telescopes and experimental facilities) that do not have a full operating system, and may have policy and management requirements that are very different from each other and from a computational node.
4
Nat
ion
al S
cien
ce F
ou
nd
atio
n
Research applications Life science problems
E.g., protein folding requires enormous quantities of multiprocessing power
self-deploying multiprocessing support is essential (e.g. SDSC’s Rocks allow for affordable human resources).
5
Nat
ion
al S
cien
ce F
ou
nd
atio
n Challenges Unlike supercomputers, clusters, servers and P2P,
grids have heterogeneity, flexibility and reliability and not a single control
A grid virtualizes resources Resources may be quite different in character and
capabilities May come and go without warning as their
availability changes over time. Massive volumes of data and increasing…
in e-business Web site operations, customer relationships, management applications, financial services
Need to mine millions of rows of data. IT professionals can obtain serious computing power
from low-cost components.
6
Nat
ion
al S
cien
ce F
ou
nd
atio
n
Non homogeneous architectures Production Grids have multiple architectures, clusters of
workstations, specialized massively-parallel machines, clusters of shared-memory nodes, and machines with vector rather than scalar processors
Different operating systems and no common set of software
Grid middleware used to combine them in a virtual organization
Constraints exist in providing reliability and high quality of service and availability
Important to Monitor the Grid Establish a set of Core Grid Functions
7
Nat
ion
al S
cien
ce F
ou
nd
atio
n
Heterogeneous solutions Heterogeneous solutions in business force
users to remember many logins, such as, Oracle E-Business Suite for financial and order
management; Siebel for customer relationship management
(CRM); SAP for inventory management; PeopleSoft for Human Resources and multiple
applications.
8
Nat
ion
al S
cien
ce F
ou
nd
atio
n Interoperability Interoperability is Key to extending Grids across
organizational boundaries. Issues:
Diverse hardware resources (supercomputers, clusters, databases and devices)
Application software, services, files and data archives need to work with diverse hardware
Complex middleware for job submission, software Application projects that require access to resources in multiple
grid systems Data as important as computation : data-handling
and transfer issues must be part of core functionality. This is accomplished in three ways:
1) A set of uniform core software services that manage and provide access to heterogeneous, distributed resources,
2) a widely deployed infrastructure, and 3) higher level services like the Data Grid tools
9
Nat
ion
al S
cien
ce F
ou
nd
atio
n Servicing the user Information services
Help user discover, track and provide information on Grid resources
vital to Grid Infrastructure Example: How does a user identify and locate the information
that is important to her? Obstacles in extracting useful and meaningful information
Dynamically changing heterogeneous resources Large amounts of disparate information about the Grid
Enterprise Portal: a single source of interaction with corporate information and for day-to-day business.
key components of Business Process Management (BPM), Enterprise Application Integration (EAI), and Business Activity Monitoring (BAM) initiative
Single sign-on (SSO) for users to access multiple applications.
10
Nat
ion
al S
cien
ce F
ou
nd
atio
n Service oriented architectures Evolving applications
from monolithic, closed systems to modular, open systems with well-defined interfaces.
Middleware complexity remains an obstacle to meeting business demands (e.g., Oracle’s solution with the Application Server)
“service-oriented architectures” to design and integrate with existing legacy systems and business applications.
Like web services attract enterprise users and applications on demand, grids can deliver computational resources on demand.
11
Nat
ion
al S
cien
ce F
ou
nd
atio
n
Balance optimality with flexibility
Do not ask whether a grid is the right model for any particular task but what is the latency and bandwidth requirements of an application
Some problems require a large shared memory best provided by a multiprocessor server with the fastest possible interconnections.
Other problems, best handled on a cluster or grid using low-cost compute nodes with affordable Ethernet connections.
balance between optimality for one task and flexibility for many tasks
12
Nat
ion
al S
cien
ce F
ou
nd
atio
n Enterprise applications driving grid computing
Supercomputing on sale: cost-push of a commodity technology, rather than the
demand-pull of problems worth solving at any price. The bang for the buck : good price/performance ratio
in building a grid of high-density x86 blade servers running Linux-based operating systems.
Grid raw computing power Example off the shelf solutions entering the market
Oracle’s Real Application Clusters technology, with its Cache Fusion architecture
Globus Toolkit, an open-source "service factory" framework providing state maintenance and discovery tools
Utility grid computing
13
Nat
ion
al S
cien
ce F
ou
nd
atio
n Utility computing In the real world, grids provide utilities (e.g., gas,
electricity or water) on-demand to consumers who will pay for them.
"Utility computing is a model of how you pay for computing resources.
It's purchasing computer resources in a pay-per-use model. Grid and utility are complementary concepts. The grid is the infrastructure for sharing resources. Utility is the concept of paying for what you need.
Appeals to government agencies that experience seasonal spikes in demand, which require more power but may not justify purchasing -- or the agency simply can't afford -- new hardware.
14
Nat
ion
al S
cien
ce F
ou
nd
atio
n
Resource Brokers Dealing with dynamic, multi-institutional virtual
organizations :A set of individuals and/or institutions defined by set of sharing rules form a virtual organization.
Social and policy issues: Negotiate resource-sharing arrangements among a set of participating parties (providers and consumers) and then use the resulting resource pool for some purpose.
Resource Brokering Strategies Sharing not only file exchanges but also access to computers,
software, data, and other resources, as needed by a collaborative problem-solving application
Highly controlled sharing with resource providers and consumers defining clearly and carefully just what is shared, who is allowed to share, and the conditions under which sharing occurs.
15
Nat
ion
al S
cien
ce F
ou
nd
atio
n Coordinate distributed resources and users
Tools to integrate and coordinate resources and users that are not centrally controlled
Address distributed issues: security, policy, payment, membership, etc
Refine multi-purpose grid protocols and interfaces by addressing authentication, authorization, resource discovery, and resource access.
Make protocols and interfaces standard and open in order to deliver nontrivial qualities of service
relate to response time, throughput, availability, and security, and/or co-allocation of multiple resource types to meet complex user demands
Make the utility of the combined system significantly greater than that of the sum of its parts.
16
Enterprise Grid
17
Nat
ion
al S
cien
ce F
ou
nd
atio
n
What is EG What’s difference with GRID Open problems Future Issues
18
Nat
ion
al S
cien
ce F
ou
nd
atio
n
What is EG A GRID technology focuses on Enterprise level. Some samples:
Alchemi: a framework for EG
Entropia
Grid MP
SETI @ home
Condor in enterprise
19
Nat
ion
al S
cien
ce F
ou
nd
atio
n
What’s difference with GRID Different priorities to traditional Grid
community:
EG does not require best efforts as GRID
Still focus on Mission critical application Be economic
Better performance
20
Nat
ion
al S
cien
ce F
ou
nd
atio
n
Open Problems Privacy protection
How to share computation but not privacy
Security in Enterprise-level How to get high performance
21
Nat
ion
al S
cien
ce F
ou
nd
atio
n
Reference Enterprise Grid Alliance
www.gridalliance.org