Reflections on Production Grids Enterprise and

Research

Frederica Darema <fdarema@nsf.gov>Dynamic Data Driven Application Systems (DDDAS)Division of CCF&Fillia Makedon <fmakedon@nsf.gov>Office of CyberinfrastructureNational Science FoundationGlobal Grid Forum, Athens,Greece

2

Nat

ion

al S

cien

ce F

ou

nd

atio

n Outline Production Grids for Science and Engineering

Research applications

Challenges

Enterprise applications

Non homogeneous architectures

Heterogeneous solutions

Interoperability

Servicing the user

Service oriented architectures

Balance optimality with flexibility

Enterprise applications driving grid computing Utility computing

Resource Brokers Coordinate distributed resources and users

3

Nat

ion

al S

cien

ce F

ou

nd

atio

n Production Grids for Science and Engineering

Primary task to enable large-scale scientific research Not just a test-bed for software development, or

experimental computer science. Severe constraints in construction because:

access to resources subject to stringent requirements of security and high quality of service.

Production grid resources may not be for Grid use by design.

Production Grid middleware cannot control policy on such Grids, it must co-operate with the site policy and resource management systems.

Challenge in dealing with non computational resources, (e.g., telescopes and experimental facilities) that do not have a full operating system, and may have policy and management requirements that are very different from each other and from a computational node.

4

Nat

ion

al S

cien

ce F

ou

nd

atio

n

Research applications Life science problems

E.g., protein folding requires enormous quantities of multiprocessing power

self-deploying multiprocessing support is essential (e.g. SDSC’s Rocks allow for affordable human resources).

5

Nat

ion

al S

cien

ce F

ou

nd

atio

n Challenges Unlike supercomputers, clusters, servers and P2P,

grids have heterogeneity, flexibility and reliability and not a single control

A grid virtualizes resources Resources may be quite different in character and

capabilities May come and go without warning as their

availability changes over time. Massive volumes of data and increasing…

in e-business Web site operations, customer relationships, management applications, financial services

Need to mine millions of rows of data. IT professionals can obtain serious computing power

from low-cost components.

6

Nat

ion

al S

cien

ce F

ou

nd

atio

n

Non homogeneous architectures Production Grids have multiple architectures, clusters of

workstations, specialized massively-parallel machines, clusters of shared-memory nodes, and machines with vector rather than scalar processors

Different operating systems and no common set of software

Grid middleware used to combine them in a virtual organization

Constraints exist in providing reliability and high quality of service and availability

Important to Monitor the Grid Establish a set of Core Grid Functions

7

Nat

ion

al S

cien

ce F

ou

nd

atio

n

Heterogeneous solutions Heterogeneous solutions in business force

users to remember many logins, such as, Oracle E-Business Suite for financial and order

management; Siebel for customer relationship management

(CRM); SAP for inventory management; PeopleSoft for Human Resources and multiple

applications.

8

Nat

ion

al S

cien

ce F

ou

nd

atio

n Interoperability Interoperability is Key to extending Grids across

organizational boundaries. Issues:

Diverse hardware resources (supercomputers, clusters, databases and devices)

Application software, services, files and data archives need to work with diverse hardware

Complex middleware for job submission, software Application projects that require access to resources in multiple

grid systems Data as important as computation : data-handling

and transfer issues must be part of core functionality. This is accomplished in three ways:

1) A set of uniform core software services that manage and provide access to heterogeneous, distributed resources,

2) a widely deployed infrastructure, and 3) higher level services like the Data Grid tools

9

Nat

ion

al S

cien

ce F

ou

nd

atio

n Servicing the user Information services

Help user discover, track and provide information on Grid resources

vital to Grid Infrastructure Example: How does a user identify and locate the information

that is important to her? Obstacles in extracting useful and meaningful information

Dynamically changing heterogeneous resources Large amounts of disparate information about the Grid

Enterprise Portal: a single source of interaction with corporate information and for day-to-day business.

key components of Business Process Management (BPM), Enterprise Application Integration (EAI), and Business Activity Monitoring (BAM) initiative

Single sign-on (SSO) for users to access multiple applications.

10

Nat

ion

al S

cien

ce F

ou

nd

atio

n Service oriented architectures Evolving applications

from monolithic, closed systems to modular, open systems with well-defined interfaces.

Middleware complexity remains an obstacle to meeting business demands (e.g., Oracle’s solution with the Application Server)

“service-oriented architectures” to design and integrate with existing legacy systems and business applications.

Like web services attract enterprise users and applications on demand, grids can deliver computational resources on demand.

11

Nat

ion

al S

cien

ce F

ou

nd

atio

n

Balance optimality with flexibility

Do not ask whether a grid is the right model for any particular task but what is the latency and bandwidth requirements of an application

Some problems require a large shared memory best provided by a multiprocessor server with the fastest possible interconnections.

Other problems, best handled on a cluster or grid using low-cost compute nodes with affordable Ethernet connections.

balance between optimality for one task and flexibility for many tasks

12

Nat

ion

al S

cien

ce F

ou

nd

atio

n Enterprise applications driving grid computing

Supercomputing on sale: cost-push of a commodity technology, rather than the

demand-pull of problems worth solving at any price. The bang for the buck : good price/performance ratio

in building a grid of high-density x86 blade servers running Linux-based operating systems.

Grid raw computing power Example off the shelf solutions entering the market

Oracle’s Real Application Clusters technology, with its Cache Fusion architecture

Globus Toolkit, an open-source "service factory" framework providing state maintenance and discovery tools

Utility grid computing

13

Nat

ion

al S

cien

ce F

ou

nd

atio

n Utility computing In the real world, grids provide utilities (e.g., gas,

electricity or water) on-demand to consumers who will pay for them.

"Utility computing is a model of how you pay for computing resources.

It's purchasing computer resources in a pay-per-use model. Grid and utility are complementary concepts. The grid is the infrastructure for sharing resources. Utility is the concept of paying for what you need.

Appeals to government agencies that experience seasonal spikes in demand, which require more power but may not justify purchasing -- or the agency simply can't afford -- new hardware.

14

Nat

ion

al S

cien

ce F

ou

nd

atio

n

Resource Brokers Dealing with dynamic, multi-institutional virtual

organizations :A set of individuals and/or institutions defined by set of sharing rules form a virtual organization.

Social and policy issues: Negotiate resource-sharing arrangements among a set of participating parties (providers and consumers) and then use the resulting resource pool for some purpose.

Resource Brokering Strategies Sharing not only file exchanges but also access to computers,

software, data, and other resources, as needed by a collaborative problem-solving application

Highly controlled sharing with resource providers and consumers defining clearly and carefully just what is shared, who is allowed to share, and the conditions under which sharing occurs.

15

Nat

ion

al S

cien

ce F

ou

nd

atio

n Coordinate distributed resources and users

Tools to integrate and coordinate resources and users that are not centrally controlled

Address distributed issues: security, policy, payment, membership, etc

Refine multi-purpose grid protocols and interfaces by addressing authentication, authorization, resource discovery, and resource access.

Make protocols and interfaces standard and open in order to deliver nontrivial qualities of service

relate to response time, throughput, availability, and security, and/or co-allocation of multiple resource types to meet complex user demands

Make the utility of the combined system significantly greater than that of the sum of its parts.

16

Enterprise Grid

17

Nat

ion

al S

cien

ce F

ou

nd

atio

n

What is EG What’s difference with GRID Open problems Future Issues

18

Nat

ion

al S

cien

ce F

ou

nd

atio

n

What is EG A GRID technology focuses on Enterprise level. Some samples:

Alchemi: a framework for EG

Entropia

Grid MP

SETI @ home

Condor in enterprise

19

Nat

ion

al S

cien

ce F

ou

nd

atio

n

What’s difference with GRID Different priorities to traditional Grid

community:

EG does not require best efforts as GRID

Still focus on Mission critical application Be economic

Better performance

20

Nat

ion

al S

cien

ce F

ou

nd

atio

n

Open Problems Privacy protection

How to share computation but not privacy

Security in Enterprise-level How to get high performance

21

Nat

ion

al S

cien

ce F

ou

nd

atio

n

Reference Enterprise Grid Alliance

www.gridalliance.org