GridSphere’s Grid Portlets A Grid Portal Development Framework

transcript

2004 1Michael Russell <michael.russell@aei.mpg.de>

GridSphere’s Grid PortletsA Grid Portal Development Framework

Jason Novotny

Lawrence Berkeley National Laboratory

Jason Novotny, Michael Russell, Oliver WehrensThe GridSphere Project

www.gridsphere.org

The Purpose Of This Talk

The purpose of this talk is to provide a brief overview of the Grid Portlets web application.

We will describe the basic ideas behind Grid Portlets, what functionality it currently provides and development plans

GridSphere’s Grid Portlets

The GridSphere portlet container is designed to be web application independent. Indeed, one of the key advantages of the Portlet API is the reuse of web applications.Thus, the GridSphere portlet container does not contain any support for using Grid technologies.Instead, GridSphere’s Grid related functionality is contained in a web application we call Grid Portlets.Grid Portlets, together with the GridSphere portlet container, offers a generic Grid portal and can be used to develop application-specific Grid portal applications.

The GridLab Project

Funded by the EU (5+ M€), January 2002 – December 2004Application and Testbed oriented

Cactus Code, Triana Workflow, all the other applications that want to be Grid-enabled

Main goal: to develop a Grid Application Toolkit (GAT) and set of grid services and tools...:

resource management (GRMS), data management,monitoring, adaptive components, mobile user support, security services,portals,

... and test them on a real testbed with real applications

GridLab Is An Architecture

And A Global Effort

PSNC (Poznan) - coordination AEI (Potsdam) ZIB (Berlin) Univ. of Lecce Cardiff University Vrije Univ. (Amsterdam) SZTAKI (Budapest) Masaryk Univ. (Brno) NTUA (Athens)

Sun MicrosystemsCompaq (HP)

ANL (Chicago, I. Foster) ISI (LA, C.Kesselman) UoWisconsin (M. Livny)

collaborating with:Users!

EU Astrophysics Network,

DFN TiKSL/GriKSL

NSF ASC Project

other Grid projectsGlobus, Condor,

GrADS,

PROGRESS,

GriPhyn/iVDGL,

Most of the other European Grid Projects (GRIDSTART)

GridLab Testbed Snapshot

GridLab Goals

Get Computational Scientists using the “Grid” and Grid services for real, everyday, production work (AEI Relativists, EU Network, Grav Wave Data Analysis, Cactus User Community), all the other potential grid appsMake it easier for applications to make flexible, efficient, robust, use of the resources available to their virtual organizationsDream up, prototype, and test new application scenarios which make adaptive, dynamic, wild, and futuristic uses of resources.

What Do Our Users Need?

Application oriented environment

Flexible, easy-to-use, simple interfaces

Efficient and effective use of resources

Robustness, fail-safety, adapability

The ability to work in distributed teams

Support for mobile working environments

What Do Our Users Want?

Larger computational resources Memory/CPU

Faster throughputCleverer scheduling, configurable scheduling, co-scheduling, exploitation of un-used cycles

Easier use of resourcesPortals, grid application frameworks, information services, mobile devices

Remote interaction with simulations and dataNotification, steering, visualization, data management

Collaborative toolsNotification, visualization, video conferencing, portals

Dynamic applications, New scenariosGrid application frameworks connecting to services

Many Application Scenarios!

Dynamic Stagingmove to faster/cheaper/bigger machine

Multiple Universecreate clone to investigate steered parameter

Automatic Convergence Testingfrom initial data or initiated during simulation

Look Aheadspawn off and run coarser resolution to predict likely future

Spawn Independent/Asynchronous Taskssend to cheaper machine, main simulation carries on

Application Profilingbest machine/queuechoose resolution parameters based on queue

Dynamic Load Balancinginhomogeneous loads

multiple grids

Portal User/virtual organisation interface to the grid.

Intelligent Parameter Surveysfarm out to different machines

Make use ofRunning with management tools such as Condor, Entropia, etc.

Scripting thorns (management, launching new jobs, etc)Dynamic use of eg MDS for finding available resources

Where Grid Portals Fit In

GridLab needs an effective way to deliver its tools and services to its target users.

We want to avoid having to develop new client-side technologies and methods for deploying them.

Given the ubiquitous Web browser and the evolution of Grid standards and technologies, a Grid portal provides an obvious solution to this problem.

Developing Grid Portals

Grid web application development still remains a tedious task with little in the way of reusable components, forcing developers to constantly “re-invent” the wheel.

Often difficult and hard to maintain glue code must be written connecting the portal to Grid services, due to lack of/evolving standards.

Lack of real usability has made it difficult to test and evaluate user interfaces.

A portal is only as good as the underlying deployed infrastructure…. portal development often involves debugging underlying middleware

What is a Grid Portal?

“A portal is a web based application that commonly provides personalization, single sign on, content aggregation from different sources and hosts the presentation layer of Information Systems”(JSR 168)

Grid Portals build upon the familiar Web portal model, such as Yahoo or Amazon, to deliver the benefits of Grid computing to virtual communities of users, providing a single access point to Grid services and resources.

Early Grid Portal Projects

Grid-Port: Perl based framework developed by Mary Thomas and Steve Mock at San-Diego Supercomputing Center (SDSC)

Grid Portal Development Toolkit (GPDK):Developed by Jason Novotny at Lawrence Berkeley National Laboratories (LBNL)

Astrophysics Simulation Collaboratory (ASC):Developed by Michael Russell at University of Chicago

Grid Portals Work Package

Within the GridLab Project, the Grid Portals Work Package (WP4) is responsible for developing a Grid portal for the GridLab Project.

The GridSphere Project was created by WP4 as it saw a need in the general community for a Grid portal framework.

WP4 is both developing and deploying GridSphere to serve target user communities in the GridLab Virtual Organization.

GridSphere 2.0

Personalized Environment

Single Sign-On Capabilities

Submit Jobs

Perform File Transfers

Manage Resources

Grid Portal Design Requirements

We require a portal that is generic and not tied to any particular set of Grid technologies. The GridLab Testbed, for example, is Globus 2.X based although integartion with OGSA well under way

Develop a set of portlets and portlet services that provide a higher level model of Grid concepts.The Grid portlet services should provide a generic API to which the Grid portlets are written.

Grid Portlets API

Our Grid Portlets API attempts to provide a high-level model of Grid concepts that

Abstracts the developer from particular Grid technologies and infrastructure.

Is extensible, can build upon its concepts to provide more complex services, resources and tasks.

Supports multiple implementations of the API.

Configurable at run-time via administrative interfaces

The Grid Portlets API builds from basic concepts to more advanced ones…

Basic Concepts

UsersUsers should get a personalized a view of the Grid based upon their access rights to resources and their preferences.

CredentialsCredentials are required to authenticate users to resources and authorize their rights to utilize them.

ResourcesResources are anything that can be “utilized” (by users, portlet services and other resources).

TasksMany operations are quite naturally modeled as tasks. Workflow is not provided yet, but is desired in future releases.

More Advanced Concepts

JobsJob submission to LSF, PBS, etc…

Job migration from one resource to another

Files & Related TasksPhysical file browsing (list, transfer, etc.)

Logical and replica file browsing

Information ResourcesLDAP/MDS information providers

Web service based providers

Other types…

Simple Resource Diagram

Hardware Resource Model

Hardware resources are utilized for performing computations, storing files, etc….Hardware resources contain…

Service resources (such as a Grid FTP resource)Software resources (such as Globus client tools)Hardware accounts (allowing people to login and perform operations in a shell).

Job Resource Model

Job resources (such as Globus Gatekeepers) are service resources that enable the remote clients to submit jobs on computing resources.Job resources provide access to job schedulers (such as PBS).Job schedulers contain queues with different resource capabilities and usage policies.

Some Job Resources

In Grid Portlets, we’ve modeled two types of service resources as job resources, both of which have different capabilities for supporting job submission.GTX Gatekeepers provide the ability to submit jobs directly to job managers running on hardware resources using the GRAM protocol.GridLab Resource Manager Services (GRMS)s are web services that provide resource brokering capabilities and support job submission with GRAM as well as Condor.

Simple Task Diagram

How Portlets Are Developed

In Grid Portlets, the presentation layer is made up of portlets which contain reusable UI components.

The business logic layer is implemented as a collection of portlet services which provide some high-level capabilities and make use of external resources.

Job Submission Portlet

For example, the job submission portlet uses a number of UI components, including file browser components (i.e. for specifying locations of an executable).

These JSP components utilize a number of portlet services which provide access to job resources, file resources and so on…

Current Status

Grid Portlets is still in prototype development mode.

Most of what is their works, but API is still evolving.

Documentation is still poor… Mike promises to improve it

Source code is available via anonymous CVS.

Will move towards a regular development and release cycle in the coming months.

Mike in Brazil ;-) but plans on continuing to enhance grid portlets

Current Functionality

Credential Retrieval & ManagementSupport for retrieval & storing with MyProxy

Resource Registry & ManagementResources can be described in XML and online

Resource Information ProviderResource discovery and update with MDS2

Job Submission & MigrationGRAM job submission with Java CoG 1.1GRMS 1.9 job submission and migration

File Browsing & TransferringSupport for GridFTP with Java CoG 1.1

What’s Next?

Enhancements to UI component model… These enhancements may be incorporated into GridSphere portlet container, but we are still reviewing JSF.

Many enhancements planned to existing portlets, including better integration among portlets and reuse of UI components.

Progress already underway to make JSR 168 compliant

Please submit feature requests and additions to bugtracker and mail list

Planned Development

Support for logical files & replica managementSupport for GridLab Data Management Services

Support for Globus Replica Catalogue

Enhancements to information gatheringPortlets for managing use of information providers

More resource information providersSupport for GridLab Adaptive Information Service

Support for Network Weather Service

How To Obtain Grid Portlets

Grid Portlets can be obtained from CVS….

Required Software:GridSphere 2.0.2 or higher.

Tomcat 4.1.29 or higher (Tomcat 5.X supported)

OGSA 3.0.2 (Plan to move OGSA 3.2)

MyProxy

cvs -d :pserver:anonymous@portal.aei.mpg.de:/home/rep

ository co gridportlets

Configuring Grid Portlets

Configuration amounts to specifying what resources Grid Portlets makes available to users.

The first thing you should do is specify what MyProxy resource your portal will use for retrieving / storing Grid credentials.

If want to use our MDS2 information provider, then you should specify what GIIS resource you want to use.

Otherwise, you should make sure to specify the hardware and service resources that make up your Grid…

How To Specify Resources

There are 2 ways to add resources to Grid PortletsOnline with the Resource Registry Portlet

Offline with Resources.xml

How To Get Involved

Join one of gridsphere mail lists. Ask questions, provide feedback / comments.

Submit bugs and feature requests to http://bugs.gridsphere.org.

Because we are a small group, getting involved as a developer means demonstrating some knowledge / commitment to the project. Best way is to email ideas / patches to the list. If you’re loud enough, we’ll probably give you commit access and a few tasks!

Projects Using Grid Portlets

AEI Numerical Relativity Portal

CCT Numerical Relativity Portal

NRC Portal

Unversity of Melborne, Australia

SDSC…

Conclusion

Grid Portlets contains the Grid functionality in the GridSphere Project. It provides a framework for developing Grid portals.

It is still in prototype development mode, but we are already using it in the GridLab Project and in several other projects with which we are affiliated.

We will make announcements on our website at www.gridsphere.org as the status of Grid Portlets changes.

Feel free to try it out and send us comments.

GridSphere’s Grid Portlets A Grid Portal Development Framework

Documents