Portals and PortletsPortals and Portlets

Marlon PierceMarlon PierceCommunity Grids LabCommunity Grids Lab

Indiana UniversityIndiana UniversitySeptember 27, 2004September 27, 2004

What a Science Portal Is/Is NotWhat a Science Portal Is/Is Not It isIt is

• A tool for aggregating and managing web contentA tool for aggregating and managing web content• A user customizable view of these Web content pieces.A user customizable view of these Web content pieces.

You see what you want/can see.You see what you want/can see. But you must log in.But you must log in. The portal recognizes you.The portal recognizes you.

• Implemented on top of standard servicesImplemented on top of standard services Like login, authorization, customization.Like login, authorization, customization. May include collaboration, etc, that depend on login.May include collaboration, etc, that depend on login.

• A way to accomplish tasks through browsers:A way to accomplish tasks through browsers: Launch, monitor jobsLaunch, monitor jobs Move filesMove files Run science applications based on these services.Run science applications based on these services.

• Compatible with emerging standards and best practices (such Compatible with emerging standards and best practices (such as portlets, JSR 168 and WSRP).as portlets, JSR 168 and WSRP).

It is not (just)It is not (just)• A web pageA web page• A collection of linksA collection of links• An appletAn applet

Things I Hate About PortalsThings I Hate About Portals If you involved in portal efforts, be aware of the following:If you involved in portal efforts, be aware of the following: Browsers have Browsers have limited interactivitylimited interactivity..

• Desktop GUIs provide much better interactivity but have other Desktop GUIs provide much better interactivity but have other problems.problems.

• Applets are a solution, but they don’t interact with other parts of the Applets are a solution, but they don’t interact with other parts of the browser very well.browser very well.

• SolutionSolution: Service Oriented portals let you use services through both : Service Oriented portals let you use services through both portals and grid desktops.portals and grid desktops.

Developing Developing really useful user interfacesreally useful user interfaces to your set of services is a to your set of services is a time consuming, non-scaling process.time consuming, non-scaling process.• Get users involved early in design.Get users involved early in design.• If you can get users directly involved, even better.If you can get users directly involved, even better.

Browsers notoriously have Browsers notoriously have incompatibleincompatible features. features.• Things don’t work the same on IE, Mozilla, Konqueror, etc.Things don’t work the same on IE, Mozilla, Konqueror, etc.• Same browsers on Macs, Windows don’t work the same.Same browsers on Macs, Windows don’t work the same.• No substitute for lots of testing.No substitute for lots of testing.

Portals depend upon backend resources and security Portals depend upon backend resources and security • But these things are not under portal developers’ control.But these things are not under portal developers’ control.• When things change (and they do, constantly) things will break.When things change (and they do, constantly) things will break.• You will be blamed.You will be blamed.

Interoperable Portal Interoperable Portal Services (ET-03-011)Services (ET-03-011)

Marlon PierceMarlon Pierce

Community Grids LabCommunity Grids Lab

Indiana UniversityIndiana University

What 3What 3rdrd Party Technologies Did Party Technologies Did We Use?We Use?

Tomcat 3.x and 4.x, Jetspeed 1.4, JSP Tomcat 3.x and 4.x, Jetspeed 1.4, JSP 2.x, JavaBeans, Apache Axis 1.x2.x, JavaBeans, Apache Axis 1.x

We also used kerberized CORBA in We also used kerberized CORBA in previous projectsprevious projects• Unfashionable these daysUnfashionable these days

If I were starting today, I would useIf I were starting today, I would use• Gridsphere (for JSR168), JavaServer Gridsphere (for JSR168), JavaServer

Faces, Apache Axis, NaradaBrokeringFaces, Apache Axis, NaradaBrokering

Project: ET011Project: ET011 Goals of the project are to demonstrate Goals of the project are to demonstrate

interoperability between Portal/PSE interoperability between Portal/PSE projectsprojects• Mary Thomas (PI), TACC: HotPageMary Thomas (PI), TACC: HotPage• Tomasz Haupt, MSU: DMEFSTomasz Haupt, MSU: DMEFS• Marlon Pierce, IU: Gateway Marlon Pierce, IU: Gateway

We investigated building interoperability We investigated building interoperability at two levels:at two levels:• Web services provide standard interfacesWeb services provide standard interfaces• Portlets provide component-based interfacesPortlets provide component-based interfaces

I was responsible for deployment at ASC I was responsible for deployment at ASC and ARL.and ARL.

Project ended Oct 1, 2003.Project ended Oct 1, 2003.

Portal SecurityPortal Security We are building off We are building off

Gateway’s approach for Gateway’s approach for Web-based security for Web-based security for DOD portals.DOD portals.• Approved for ARL and ASCApproved for ARL and ASC

Users kinit to a web server Users kinit to a web server to get a ticket.to get a ticket.• SSL, MD’d sessions, SSL, MD’d sessions,

Certificates maintain Certificates maintain secure connection.secure connection.

• Web server typically Web server typically located in “DMZ”located in “DMZ”

Web server manages Web server manages session IDs, invokes session IDs, invokes backend requests with backend requests with Kerberos client utilities.Kerberos client utilities.

Probably would break Probably would break today.today.

Browser

WebServer

HPCHPC

HPC

DMZ

HTTPS

krcp, krsh

Portlets and ContainersPortlets and Containers One of the problems of previous portal One of the problems of previous portal

development is that there is no good way development is that there is no good way to share interface components.to share interface components.• How do developers share web interfaces?How do developers share web interfaces?

Also, how can we avoid constantly Also, how can we avoid constantly reinventing things like login services, reinventing things like login services, customization services, page organization, customization services, page organization, access controls.access controls.

Answer: use portlets and containers.Answer: use portlets and containers.• Becoming a recognized best practice for portal Becoming a recognized best practice for portal

development because it enables distributed development because it enables distributed portal development.portal development.

What Is a Portlet?What Is a Portlet? A portlet is a piece of Java code that runs in a A portlet is a piece of Java code that runs in a

Web server inside a container servlet.Web server inside a container servlet. Portlets can do two things:Portlets can do two things:

• Perform non-visual operations such as make connections Perform non-visual operations such as make connections to remote hosts, perform operations.to remote hosts, perform operations.

Example: get a list of local files.Example: get a list of local files.• Create their display Create their display

The portlet passes its display to its parent, which The portlet passes its display to its parent, which is responsible for constructing the entire display.is responsible for constructing the entire display.• Typically this is HTML, with tables uses to organize Typically this is HTML, with tables uses to organize

component displays.component displays.• Other displays are possible (VoiceXML, WML).Other displays are possible (VoiceXML, WML).

Portal ServicesPortal Services

We had several services that we We had several services that we portletizing as part of this project:portletizing as part of this project:• Job submissionJob submission• File TransferFile Transfer• Job MonitoringJob Monitoring

We develope DOD versions of TACC’s GPIR We develope DOD versions of TACC’s GPIR servicesservices

We extended Jetspeed login to support We extended Jetspeed login to support web kiniting (with SecurID).web kiniting (with SecurID).

Job SubmissionJob Submission

Primarily based at ARLPrimarily based at ARL• Support Fluent, ANSYS, ABAQUSSupport Fluent, ANSYS, ABAQUS• Services construct GRD scripts, allow users to Services construct GRD scripts, allow users to

run and archive jobs.run and archive jobs. We are extending this to support ANSYS at We are extending this to support ANSYS at

ASC, DMEFS codes at ARL.ASC, DMEFS codes at ARL. We need to extend script generators for We need to extend script generators for

other queuing systems.other queuing systems.• PBS, LoadLeveler, LSFPBS, LoadLeveler, LSF

File ManagementFile Management

File management services allow you toFile management services allow you to• Upload, download files between desktop and Upload, download files between desktop and

remote HPCremote HPC• Download entire directories as zipped files.Download entire directories as zipped files.• Delete remote files.Delete remote files.• Navigate remote directories.Navigate remote directories.• Unzip/untar remotely.Unzip/untar remotely.

Targetting ASC and ARL initiallyTargetting ASC and ARL initially• ARL is available in productionARL is available in production

Job MonitoringJob Monitoring

We have web interfaces that will allow to We have web interfaces that will allow to monitor your jobs on various hosts.monitor your jobs on various hosts.• Constructs an HTML table of your running jobs Constructs an HTML table of your running jobs

in a unified format.in a unified format.• Allows you to stop jobsAllows you to stop jobs

We support GRD in production portal at We support GRD in production portal at ARL.ARL.

Have ported this to PBS, LSF, and Have ported this to PBS, LSF, and LoadLeveler as part of this project.LoadLeveler as part of this project.

Access to PortletsAccess to Portlets Obviously not all users have accounts at Obviously not all users have accounts at

all centers.all centers.• An ASC file browser should be accessible only An ASC file browser should be accessible only

to users with an ASC account.to users with an ASC account. Jetspeed has role based accessed control Jetspeed has role based accessed control

to portlets.to portlets. Each user can be assigned to one or more Each user can be assigned to one or more

user roles (“ERDC”, “ASC”, etc).user roles (“ERDC”, “ASC”, etc). This controls which portlets a user can add This controls which portlets a user can add

to his or her display.to his or her display.

GridPort Information Repository GridPort Information Repository (GPIR)(GPIR)

Developed by TACC group for NPACI Developed by TACC group for NPACI resources.resources.

Porting this to DOD.Porting this to DOD. Aim is to aggregate and cache grid Aim is to aggregate and cache grid

and portal related data from multiple and portal related data from multiple sources in a uniform way. sources in a uniform way. • MDS, NWS, custom data providersMDS, NWS, custom data providers

GPIR ApproachGPIR Approach GPIR is implemented as a set of Java Web GPIR is implemented as a set of Java Web

Services, one to handle the input of GPIR data Services, one to handle the input of GPIR data (Ingester WS) and another to facilitate the (Ingester WS) and another to facilitate the querying of that data (Query WS) querying of that data (Query WS)

The Ingester WS accepts or "ingests" several The Ingester WS accepts or "ingests" several types of XML documents and stores them in a types of XML documents and stores them in a relational database (currently MySQL, Postgres).relational database (currently MySQL, Postgres).• These documents are created by a variety of means, These documents are created by a variety of means,

including Java Clients that exist on the resources including Java Clients that exist on the resources themselves, http "web scraping" of machine-specific flat-themselves, http "web scraping" of machine-specific flat-file formats, and queries of additional information file formats, and queries of additional information providers such and MDS, GMS (Grid Monitor Service), providers such and MDS, GMS (Grid Monitor Service), and NWS (Network Weather Service). and NWS (Network Weather Service).

Persistently stored data can then be queried via Persistently stored data can then be queried via the Query Web Service which uses the same XML the Query Web Service which uses the same XML resources used by the Ingester, in addition to resources used by the Ingester, in addition to some Query specific documents that can return some Query specific documents that can return XML such as Machine Summary data.XML such as Machine Summary data.

GPIR Schema TypesGPIR Schema Types Static: static data for a machine. Static: static data for a machine. Load: load data for a machine. Load: load data for a machine. Status: machine status (up, down, unavailable). Status: machine status (up, down, unavailable). Downtime: downtime data for a machine. Downtime: downtime data for a machine. Jobs: job data for a machine. Jobs: job data for a machine. MOTD: Message of the Day data for a machine. MOTD: Message of the Day data for a machine. Nodes: Nodes data for a machine. Nodes: Nodes data for a machine. Services: represents the status of grid software Services: represents the status of grid software

running on a system. running on a system. NWS: This returns bandwidth and latency NWS: This returns bandwidth and latency

measurements of the type returned by NWS. measurements of the type returned by NWS.

Current Developments in Current Developments in the Portal Worldthe Portal World

Where Are We Today?Where Are We Today? JSR 168 is an important new standard for JSR 168 is an important new standard for

Java portals.Java portals.• Standardizes portlet containersStandardizes portlet containers• Commercial products available from Sun, IBM, Commercial products available from Sun, IBM,

Oracle, BEAOracle, BEA• Open source implementations include uPortal, Open source implementations include uPortal,

GridSphere, eXo, Jetspeed2GridSphere, eXo, Jetspeed2 Web Services for Remote Portlets (WSRP) Web Services for Remote Portlets (WSRP)

also available.also available.• Compatible standard to JSR 168 that uses Compatible standard to JSR 168 that uses

SOAP and WSDL to communicate between SOAP and WSDL to communicate between portlet container and portlet.portlet container and portlet.

• Potentially allows containers and portlets to be Potentially allows containers and portlets to be from different languages (Java, C#, Python, from different languages (Java, C#, Python, PHP).PHP).

Some Portlet ProjectsSome Portlet ProjectsOpen Grid Computing Open Grid Computing Environments ConsortiumEnvironments Consortium

Portlet component providersPortlet component providers

SakaiSakai Education-focused portal Education-focused portal tools. UMich’s CHEF, uPortal, tools. UMich’s CHEF, uPortal, IU’s OneStart, etc.IU’s OneStart, etc.

GridSphereGridSphere EU project that develops both EU project that develops both portlet containers and Grid portlet containers and Grid portlets.portlets.

Portal Capabilities Description

Grid Proxy Certificate Manager Get MyProxy certs after logging in.

Schedule Interactive individual and group calendars

Discussion Persistent topic-based discussion for groups

Chat Live chat services and interfaces

Document managers WEBDav based document system for group file sharing

MDS/LDAP Browsers Basic Globus MDS browsing and navigating

GridContext Portlets Access context services for managing metadata

GRAM Job Submission Run simple executables on remote hosts

GridFTP Upload, download, crossload remote files.

GPIR Portlets View, interact with HPC status, job, etc information.

Anabas Access to Anabas shared display applets

Newsgroups and citation portlets Post topics to newsgroup, manage group references and citations with access controls

User Portlets from OGCEUser Portlets from OGCE

Portal Capabilities Description

CoG Workflow demonstration portlet Set up and run task graphs using the Java CoG

Job Scheduling and Sequencing Schedule sequences of jobs on several hosts using Community Scheduling Framework.

Condor Portlets Interact with Condor through browser.

Application Management Combine GridFTP and GRAM into application wizard forms.

Newsgroup Services Download and install server side of the OGCE newsgroup system.

OGRE Job Management Services Manage complicated grid tasks through an extensible, Apache Ant like task list.

XDirectory Services The backend for the Grid Context portlet

More User Portlets and ServicesMore User Portlets and Services

Why Is This Important?Why Is This Important? HPCMP developers generate extremely HPCMP developers generate extremely

sophisticated web interfaces and services to sophisticated web interfaces and services to science and engineering applications.science and engineering applications.• These should be portletsThese should be portlets

Things like login, user display layouts managers, Things like login, user display layouts managers, access controls to content, etc., should not be access controls to content, etc., should not be reinvented. reinvented. • There are plenty of portal projects that have done this.There are plenty of portal projects that have done this.

These are portlet containers.These are portlet containers.• You should not reinvent this.You should not reinvent this.

By adopting the portlet/container approach, you By adopting the portlet/container approach, you can also create portals that combine HPCMP can also create portals that combine HPCMP specific content as well as third party portlets.specific content as well as third party portlets.• Calendars, RSS news feeds, document managers, WIKI Calendars, RSS news feeds, document managers, WIKI

like features, etc.like features, etc.

What is JSR 168?What is JSR 168? Defines the (Java) standard for vendor Defines the (Java) standard for vendor

container-independent portlet components.container-independent portlet components.• Portlets can be developed independently of the Portlets can be developed independently of the

container.container. Many implementations:Many implementations:

• Gridsphere, uPortal, WebSphere, Jetspeed2, ….Gridsphere, uPortal, WebSphere, Jetspeed2, …. From the portlet development point of view, it From the portlet development point of view, it

is really very simple:is really very simple:• You write a java class that extends GenericPortlet.You write a java class that extends GenericPortlet.• You override/implement several methods inherited You override/implement several methods inherited

from GenericPortlet.from GenericPortlet.• You use some supporting classes/interfacesYou use some supporting classes/interfaces

Many are analogous to their servlet equivalents Many are analogous to their servlet equivalents Some (portletsession) actually seem to be trivial Some (portletsession) actually seem to be trivial

wrappers around servlet equivalents in Pluto.wrappers around servlet equivalents in Pluto.

The Infamous Big PictureThe Infamous Big Picture As a portlet developer, the previous set of classes are As a portlet developer, the previous set of classes are

all you normally touch.all you normally touch. The portlet container (such as Pluto or Gridsphere) is The portlet container (such as Pluto or Gridsphere) is

responsible for running your portlets.responsible for running your portlets.• Init, invoke methods, destroy. Init, invoke methods, destroy.

Portlets have a very limited way of interacting with the Portlets have a very limited way of interacting with the container.container.• It is a black box.It is a black box.• The API is basically one-way.The API is basically one-way.

Some SuggestionsSome Suggestions Java-based systems should adopt JSR 168 based Java-based systems should adopt JSR 168 based

portlet containers for Java based systems.portlet containers for Java based systems. All components should be at least Web Service All components should be at least Web Service

“ready”“ready”• Use WSDL to define APIs for services (and bind later to Use WSDL to define APIs for services (and bind later to

specific programming implementations)specific programming implementations) Make a clean separation between portals and Make a clean separation between portals and

services.services.• ““Service oriented approach” will allow both desktop GUIs Service oriented approach” will allow both desktop GUIs

and portals to use the same services.and portals to use the same services. Monitor WSRP as a way to build multilingual Monitor WSRP as a way to build multilingual

portals.portals.

More SuggestionsMore Suggestions Portal/service teams have to be first-Portal/service teams have to be first-

class members of the MSRC class members of the MSRC infrastructure.infrastructure.• Need to know about HPC turnover well Need to know about HPC turnover well

in advance.in advance.• Command line login changes need to be Command line login changes need to be

accompanied by simultaneously accompanied by simultaneously approved Web logins.approved Web logins.

Need more uniformity in MSRCsNeed more uniformity in MSRCs• TeraGrid’s CTSS “common software TeraGrid’s CTSS “common software

stack” as a possible model.stack” as a possible model.

Telescoping Portal ArchitectureTelescoping Portal Architecture

Job

Sub

mi

t

FileM

anag

e

Secu

rity