Flexible Service Composition

Adam BarkerSchool of InformaticsUniversity of Edinburgh

Robert G. MannInstitute for AstronomyUniversity of Edinburgh

Introduction

• Service Oriented Architectures– Distributed computing platform targeted at the web– Define a standard way to perform program to program

interaction– Can tie together any OS, application, data store,

programming language, device etc.– Defined using: XML, SOAP, WSDL etc.

• A Service Oriented World– Wide spread adoption and interest in simple, vanilla web

service standards– This standardised architecture is unlikely to change in

the forcible future• Technologies need to fit around and consume these

existing standards

Scientific Workflow

• Most workflow engines focus on Business Process Modelling

• A Scientific Workflow captures a series of analytical steps which describe a computational experiment– In order to verify a hypothesis etc.

• Scientific Workflow has an extra set of requirements:– Rapid prototyping of experiments– User Interaction with the scientist– Reliability and Fault Tolerant execution– Transparent access to resources– Repeatability, Smart re-runs and parameterisation– Provenance information, presentation of the results– Control Flow vs. Data Flow

Motivation

• Few languages which deal with the flexible, knowledge acquisition and discovery processes found in the sciences– myGrid, Kepler, ICENI frameworks allow

statically defined, pre-designed/pre-planned workflows to be executed by a centralised workflow engine

• The following science use-case serves as a counter example of coordination which is difficult to achieve by existing systems– Large Synoptic Survey Telescope (LSST)– Time-Domain astronomy

Science Use Case Background

• Current observatories are able to look very deeply at very small sections of sky, not likely to catch transient events:– Observatories always looking somewhere else– Small field of vision means that an impractically

large number of separate observations are required to map the entire night sky

– Observing facilities are scarce

• Observations of change in the universe are difficult to obtain

Science Use Case Background

• Automated Classification– For known classes of object– LSST is a first attempt at time-domain astronomy

• Likely to discover new (previously undetected) classes of object– Always data which the automated stage cannot classify

• Typically most of these will be junk, however this may only be revealed by comparisons with other detections made on the same night

Science Use Case

Reasoning

Web Service

Web Service

Web Service

Web Service

WorkFlow Requirements

• Complex coordination behaviour– Data is inherently distributed– Resources are scarce

• Requires selection and negotiation– Large quantities of data

• Requires Autonomous, Intelligent software• Human in the loop only when needed, e.g. specialist

– Workflow specification cannot be fully predicted at design-time

• Difficult to classify data type or quantity in advance• Component choice, e.g. web service, database etc.

– (semi) Flexible, Dynamic, runtime composition• In a decentralised, peer-to-peer environment

A convergence of Interests

Flexible

Autonomous

Cooperation, Negotiation

Run-Time Decisions

Uncertain Environments

Reliable

Secure

Middleware Support

Working Systems

Scaleable

Age

nts

SO

A

Workflow based on Interaction Protocols

• Interaction Protocols– Collection of conventions that allow cooperation

between agents in an open MultiAgent System– A recipe for communication– Defines ‘if and when’ agents communicate– Order and kind of messages relating to a certain domain

• Multi Agent Service Composition (MASC) – Agent-Based Workflow language and framework based

on Interaction Protocols– Decentralised, peer-to-peer architecture– Aimed at Scientific Workflow composition

• In order to solve the motivating science use-case– Discuss the language and framework in the following

slides

MASC: Scenes

• Scene– Bounded space in which a group of

agents interact on a single shared task– Divide a large, complex protocol into

manageable chunks– Scenes cannot begin execution until all

agents have reference to the protocolS := scene(ids, {ip}, {op},

{R}, {A})ip := inport(ids:idpin, T)

op := outport(ids:idpout,T )

MASC: Roles• Role

– A role type allows an engineer to specify a pattern of behaviour which an agent can adopt

• Many agents can adopt the same role type • Roles are defined as a set of methods {M}

– Methods are constructed from operations and actions

A := agent(ida, idr, Φ(k) )R := {idr, config(k) , {M}}M := method idm(Φ(k)) = op

Φ := v:T, _ , c:TT := XML Data Types

MASC: Operation Set

op := action

| op1 then op2

| op1 or op2

| op1 par op2

| waitfor op1 timeout op2

| invoke idm(Φ(k))

THEN OR

PAR RECURSION

MASC: Action Set

action := empty (empty)

| p(Φ(k)) → agent( ida, idr) (send)

| p(Φ(k)) → multicast(idr) (multicast) | p(Φ(k)) → user() (user send) | p(Φ(k)) ← agent(ida, idr) (receive) | p(Φ(k)) ← user() (user receive) | Φ(k) = portread(idpin) (port read)

| portwrite(idpout, Φ(k)) (port write)| Φ(k) = p(Φ(l)) fault Φ(m) (decision)

| Φ(k) = service(ws, Φ(l)) fault Φ(m) (service)

MASC: Decision Procedures

• Decision Procedures– Connect the protocol code (describing the interaction

model) to an agent’s internal reasoning model – Each agent interacting within a scene references a set

of decision procedures• Implemented as a set of methods inside a reasoning web

service• Throughout protocol execution agents can invoke methods

on its reasoning web service

• Does not sacrifice the autonomy of the agent• Each agent can subscribe to their own reasoning model

(BDI etc.)• Up to the agent engineer to specify these methods• Allowing a personalised strategy within the interaction

protocol

MASC: Service Enactment

• External Web Services– Agents consume external web services, acting

as a proxy to their execution• Extra level of abstraction

– Allows agents to consume the passive Service Oriented Architecture found in Internet and Grid Systems

– Can be hard coded in the interaction protocol or determined at runtime• e.g. from a broker or message exchange

MASC: Execution

P

P

P

P

R

R

R

R

W W

W

W

Web Service

MASC: Dataflow

P := protocol(idp,{S}, link(L)*)

L := source → sink+

• source/sink:– Web Service– Application– User interaction– File reading/writing

• High level experiment composition

A

A

A

A

A

A

A

A

A

A

A

A

A

A

A

A

A

A

A

A

MASC: Layers of Abstraction

Agents

Scientist: Has a Scientific Hypothesis

Engineer: Has an Engineering Aim

Reasoning Model

Dataflow MASC

Scene Definition

Port Definitions

Existing SOA

DataFlow Engineer

Interaction Engineer

Agent Engineer

The MASC Framework

• Implementation:– Full Java implementation

• Built using the Sun Java Web Services Development Pack (JWSDP) 2.0

– Agents can be executed locally: as a closely coupled system or as distributed processes

– SOAP messaging– XML Language specification– https://sourceforge.net/projects/multiagent

• Application– Requirements based on Real Science use cases– Applied to the UK e-Science project, AstroGrid

Advantages

• Standard MAS Arguments– Inherently a Distributed Peer-to-Peer System

• No centralised server• No central point of failure• Allows engineering in an open environment

• Interactions that are too complex, or simply cannot be specified at design-time– Agents act as a proxy to the services which are

being coordinated, extra level of abstraction– Reactive agents, runtime decision and

coordination

Advantages

• Inter-operability– Infrastructure independent: Interaction model

always remains a layer above any specific middleware or OS

– The web services being invoked require no modification before taking part in the interaction

– Low engineering requirement • Layer of s/w to translate protocol steps

• Reasoning Models– Protocols do not sacrifice the autonomy of the

agents– Each agent can subscribe to their own reasoning

model

Conclusions

• Internet and Grid Systems are filled with passive objects (services)

• Agency paradigm offers a way of programming autonomous, social and active components which consume this SOA

• Language provides a way of applying the principles and well understood concepts of agency to the web service composition problem

• Framework fits in with existing standards• AstroGrid provides a live science test bed

Questions

Thanks for listening