OCCF – The Realtime Grid

2

Characteristics of Current Grid Computing

• Static data sets

- Generally from fixed length experiments

- Statistical measurements of error

• Early bound computations

- Shared Memory

- Compiled into calculation

• Point to Point TCP messaging

- PVM

- MPI

3

Challenges of The Realtime Grid

• Data sampling is Non Stop

- Cannot turn off the markets

- Bad Prices and mistakes may have significant effect

- Do errors fit statistical models ?

- Volumes & Rates are growing – (OPRA etc)

- Cascade effects of derivatives

• Combination of time and event series

- News is stimulus rather than a value

4

Sponsors/Components

Unlikely Bedfellows ?

The Industry Standard Data Feed

Collection – The Real Time Loader

Storage - Object Relational Database

The Grid Hardware and Engine

Optimized Computational Functions

5

RMDS – The Industry Standard Feed Infrastructure

6

Collection – The Real Time Loader

Collection Server Database Server

coll # 1

coll # 2

coll # n

Rea

l-tim

e D

ata

Dis

trib

utio

n P

latfo

rm

RTLoader

MemoryStorage Historic

Storage

ifx Instance client # 1

client # 2

client # n

Sockets Sockets Sockets

TMIS - Schematic for RTL Configuration

IBM/Informix IDS.2000 Component

MIS TMIS Component

ifx Real Time LoaderDatabladeRTLoader.

bld

AnalyticalFunctions

Analytical Function,imbedded in DB Instance,

operating in real time -MIS or Client developed

ODBC

JDBC

IBM componentcustomised by MIS

7

Storage – Object Relational Database

8

Sun Grid Engine 1- The problem space

Basic workflow of compute Grids is mechanical

Problems exist around resource allocation/prioritization

Difficult to introspect tightly bound IPC

What about stalled/looping computations or failures ?

May have mixed resource platforms

Open Source for bespoke enhancements

9

Submit, monito

r,

delete, migrate jo

bs

Output files

User Environment

Launch job,

Manipulate job

Deliver re

sults

,

exit st

atus, acc

ounting..

Compute Resources

Where to ?When ?Choice

Load license

reports

Licenses ?Load ?Policy ?

Weekend ?Parallel ?

...

SGE Master

SGE Scheduler

Master/Scheduler Nodes

SGE components & compute resources transparent to users

Master/Scheduler failover provided if required

Master/Scheduler nodes can be sited on Compute Resource systems

Sun Grid Engine 2 – Key Components

10

Optimized Computational Functions

AnalysisAnalysis

Data from diskDatabase

Server

Data from diskDatabase

ServerClient

All the data passed to the

client

Client

All the data passed to the

client

AnalysisAnalysis

Data from diskDatabase

Server

Data from diskDatabase

ServerClient

Just pass the results to the

client(s)

Client

Just pass the results to the

client(s)

Data blade approach

•Extension of traditional RDBMS stored procedures•Analysis performed in DB memory •Reduced network traffic•Fan out results 1-N…

Traditional Client/Server mechanism

•Large raw data volumes passed across the network

11

Putting it all together

12

Build Process

• Offsite prototype at Reuters Integration Lab

- Full set of exchanges

- Most sponsors in London

• Benchmark tests for H/W and S/W validation

- Sponsored student projects

• Standard Reuters small site install at Oxford Comlab

• Remote build of multicast addressing

• Remote maintenance of collection S/W

13

Next Steps – 1: Other middleware

© 2003 IBM CorporationSW Industry Value | Financial Services | Confidential

IBM Software Group

RTL in Pub/Sub Environment

RTloader

Shared Memory

DB(IDS now but all

Later)

MessagesMessagesMessages Application

ApplicationApplication

Up to 100K messages/sec from many different message streams

i

Query msgs in DB and Shared mem

Aggregation, Analytics, Event Detection

Subscribe Publish

MQ, RMM, TCP, UDP Message Protocols

14

Next Steps – 2: Fast Forward to RAPID

• Reuters feed infrastructure and site equipment will change in 2004

• Growing exchange/derivatives tick volume

• Richer Object-Based API

• Extended Analytics

• Fundamental and Reference data built in

15

Summary

• We started this post Dot Com

• Sponsors have maintained their commitments despite downturn

• Has proven all sponsors commitments to Open Computing standards

• No competitive barriers to implementation

• A unique industry collaboration