+ All Categories
Home > Technology > EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

Date post: 12-May-2015
Category:
Upload: isvana
View: 3,962 times
Download: 3 times
Share this document with a friend
Description:
Presentation of functional design patterns for event processing with examples from TIBCO, Oracle, IBM and Prova event processing technologies. Superset of the Tutorial at DEBS2012. Editors Alex Alves, Catherine Moxey
Popular Tags:
282
epts event processing technical society epts event processing technical society Adrian Paschke (Freie Universitaet Berlin) Paul Vincent (TIBCO Software) Catherine Moxey, Martin Hirzel, Dave Rushall, Richard Jacks (IBM) Alex Alves (Oracle) Event Processing Reference Architecture - Design Patterns Patterns v2.04b Content by: members of the EPTS Reference Architecture Working Group
Transcript
Page 1: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

epts event processing technical society

Adrian Paschke (Freie Universitaet Berlin) Paul Vincent (TIBCO Software) Catherine Moxey, Martin Hirzel, Dave Rushall, Richard Jacks (IBM) Alex Alves (Oracle)

Event Processing

Reference Architecture

- Design Patterns

Patterns v2.04b

Content by: members of the

EPTS Reference Architecture Working Group

Page 2: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

2 2

Agenda

• EPTS Reference Architecture Introduction

• EP Patterns:

– EPTS RA patterns

– Some Use Cases

– EPIA patterns

Page 3: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

3

Event Processing, Complex Event Processing, Event Stream Processing

Event Abstraction, Event Pattern

Detection, Event Composition

etc. etc.

Introduction to Event Processing

Event

Managem

ent

Event Consumer

(Event sink, event handler, event

listener)

Run time Administration

Event

Type D

efinitio

ns,

Event

Pro

cessin

g R

ule

s,

etc

.

Design time

Events Derived

Events

Event Producer

(Event Source, Event Emitter)

Page 4: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

4

Reference Architecture: Functional View

Event Production Publication,

Retrieval

Even

t Pro

cess M

on

itorin

g, C

on

trol

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Complex Event Detection Consolidation, Composition,

Aggregation

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Event Consumption Dashboard, Apps, External Reaction

Run time Administration

Even

t a

nd

Co

mp

lex E

ven

t (P

att

ern

, Contr

ol, R

ule

, Q

uery

, RegEx.e

tc)

D

efi

nit

ion

, M

od

eli

ng

, (co

nti

nu

ou

s) I

mp

ro

vem

en

t

Design time

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

0..*

0..*

0..*

0..*

Sta

te M

an

ag

em

en

t

Page 5: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

5

Comparison of EPTS RA Fns and EPIA EP Agents

5

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Complex Event Detection Consolidation, Composition,

Aggregation

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

EPA

Filter Transformation

Translate

Enrich Project

Aggregate Split Compose

Pattern detect

Page 6: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

6 6

Agenda

• EP Meta Pattern / Pattern Structure

Page 7: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

7

What do we want to cover in an EP Pattern?

7

Event Filtering: a stream or list of events may be filtered

on some payload or metadata information

such that some subset is selected for further processing.

Type /

synonym

Operation /

Operators

Operands

Rationale /

business rule Mapping to

EPL template

Page 8: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

8

EP MetaPattern / Pattern Description (version 1)

• Name & Alternative Names

• Classification versus other references

– EPTS RA and other references e.g. “EP in Action” book

• Description

– Role, and Business Rule type specification / requirement

• Structure

– EPTS Glossary terms

• Implementations

8

Page 9: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

9

EP MetaPattern future versions?

• Name & Alternative Names

• Classification versus other references

– EPTS RA and other references e.g. “EP in Action” book

• Description

– Role, and Business Rule type specification / requirement

• Structure

– EPTS Glossary terms

• Implementations

9

Extend examples

pseudocode – real example

may revise/improve EP Fns

classification / ontology

compare/extend

other “event patterns”

Page 10: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

10

EP Pattern Implementations

• Differ in Implementation type (Event Processing

Language, Execution semantics, etc)

• Sampled systems we cover:

1.TIBCO BusinessEvents Rete-production-rule-engine

(with continuous query and event-expression options)

2.Oracle Event Processing Stream-processing engine

3.Prova Logic Programming Semantic CEP Rule Engine

4.IBM Decision Server Events component of WebSphere

Operational Decision Management

& IBM InfoSphere Streams stream processing engine

10

Page 11: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

Implementations:Sampled Systems: TIBCO BusinessEvents

• TIBCO BusinessEvents “event server” / event processing engines

• Multiple languages / engines in various packaging options

– UML-based event and concept (object) class hierarchical models

– Java-based language

– Eclipse-based IDE

• Distributed execution model:

– Inference agents, Query agents, Datagrid agents

– In-memory, memory+grid, explicit grid operational modes

– “Grid Options” of TIBCO ActiveSpaces / Oracle Coherence

– eXtreme Event Processing use cases: Fedex, US Govt, …

• Transactional model option:

– TIBCO AS Transactions

– eXtreme Event Processing use cases: Telco

• Extensible: eg OWL import, RDF import, etc

Page 12: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

Implementations:Sampled Systems: TIBCO BusinessEvents (ctd)

Event Processing

Events

Event Storage

Information Event and Data Structures

States and Transitions

Inference Rules

Sets and Queries Stored

Events and Data

via JMS, RV, MQ, TCP/IP, etc…

Continuously process events using procedural and declarative event processing elements

Temporal Patterns

Decision Tables

Page 13: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

13

Implementations:Sampled Systems: Oracle Event Processing (formerly Oracle CEP)

• Development platform for event processing applications

• Application model based on EPN (event processing network) abstraction running on top of OSGi-based Java container.

• Language is an extension of SQL with stream and pattern matching extensions

Page 14: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

14

Implementations:Sampled Systems: Oracle Event Processing (formerly Oracle CEP)

• CQL: Continuous Query Language

• Leverages SQL, extended with Stream, Pattern-matching, and Java.

• Continuous: driven by time, and events

• Highly-sophisticated push-down technology to RDBMS, Cache, Hadoop

Page 15: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

15

Implementations:Sampled Systems: Prova (http://prova.ws)

• Java JVM based, open source rule

language for reactive agents and event

processing

• Leverages declarative ISO Prolog standard

extended with (event,message) reaction

logic, type systems (Java, Ontologies),

query built-ins, dynamic Java integration.

• Combines declarative, imperative (object-

oriented) and functional programming

styles

• Designed to work in distributed Enterprise

Service Bus and OSGi environments

• Supports strong, loose and decoupled

interaction

• Compatible with rule interchange standards

such as Reaction RuleML

Event Cloud

(unordered events)

new auto pay account login

account login deposit

withdrawal

logout

account balance

transfer

deposit new auto pay

enquiry enquiry

logout

new auto pay account login

account login deposit

activity history

withdrawal

logout transfer deposit new auto pay

enquiry

enquiry

book

request incident

Event Streams (ordered events)

Agent / Service

Interface

Rule Service

Consumer

Message Interchange

0.0000

Request

0.0000

Response

Prova Agent

Prova Agent

Computer

Prova

direct API

implementation

Prova Agent

Prova Agent

Prova Agent Prova

Agent

Strong Coupling

Loose Coupling

Decoupled Prova Event

Processing

Network

Page 16: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

16

Implementations:Sampled Systems: IBM WODM Decision Server Events • Decision Server Events component of IBM WebSphere Operational

Decision Management (WODM)

• Manages business events flowing across systems and people to provide timely insight and responses

• Detect, evaluate, and respond to events

• Discover event patterns and initiate actions

Page 17: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

17

Implementations:Sampled Systems: IBM CICS Transaction Server for z/OS

Event Production Publication, Retrieva,

Identification, Selection, Filtering,

Enrichment, Transforms, Routing

Event Preparation Identification, Selection, Filtering, Monitoring,

Enrichment

Complex Event Detection Consolidation, Composition,

Aggregation

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Event Consumption Dashboard, Apps, External Reaction

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

CICS TS

Page 18: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

18

Implementations:Sampled Systems: IBM InfoSphere Streams – Stream Processing Language

• IBM InfoSphere Streams: platform for analyzing big data in motion i.e. high-volume, continuous data streamsSPL

• IBM Streams Processing Language (SPL)

– Programming language for InfoSphere Streams

– Exposes a simple graph-of-operators view

– Provides a powerful code-generation interface to C++ and Java

Streams Processing

Language and IDE

Runtime

Environment

Tools and

Technology

Integration

Highly Scalable

stream processing runtime

Streams Console & Monitoring,

Built-in Stream Relational Analytics,

Adapters, Toolkits

Streams Studio

Eclipse IDE for SPL

Front Office 3.0

Page 19: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

19 19

Agenda

• Functional Reference Architecture-based

EP Patterns

Page 20: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

20

Patterns Coverage

Event Production Application Time

Publication, Retrieval

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Complex Event Detection Consolidation, Composition,

Aggregation

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Event Consumption Dashboard, Apps, External Reaction

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

0..*

0..*

0..*

0..*

Sta

te M

an

ag

em

en

t

Page 21: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

21 21

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Event preparation is the process of preparing the event

and associated payload and metadata for further stages

of event processing.

For example, event preparation may involve the separation

and discarding of unused event payload data,

and the reformatting of that payload data

for downstream event processing.

EP Patterns:RA:Preparation

Page 22: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

22

RA:Preparation:Identification:

Identification: incoming events identified relative to prior

events and event types

– E.g. associating a SMS event with particular mobile phone

account

– E.g. recognizing an event from data (event entity recognition,

extraction) and identifying an event to be of a particular event

type

22

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 23: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

23

Preparation:Identification:Classification

• Alternative Names:

– Event Lookup, Event Entity Recognition, Event Extraction

• EPTS Reference Architecture:

– “Incoming events will need to be identified relative to prior

events and event types, such as associating events with

particular sources or sensors or recognizing / extracting

events from data relative to event type information (event

type systems, event ontologies)"

• EPIA:

– Transform, Translate, Enrich / Project

23

Event Preparation Identification, Selection,

Filtering, Monitoring, Enrichment

Page 24: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

24

Preparation:Identification:Description

• Role / Explanation

– Associating an event with some existing entity (data or past

event, event type)

• Associated Business Rule Specification

– a selection (to be enforced during the processing of events):

All <event entities> whose <attribute1> matches the

<attribute2> of <some other event or data or type>

is related by <relationship> to that <some other event or

data or type>.

24 event

Identification

f(event)

known

event

Other data

or event or type

Event Preparation Identification, Selection,

Filtering, Monitoring, Enrichment

Page 25: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

25

Preparation:Identification:Structure

• EPTS Glossary comparison

– Event type (event class, event definition, or event

schema)

• A class of event objects.

• Event types should be defined within some type definition

system ... will usually specify certain predefined data

(attributes), examples of which might be:

– A unique event identifier used to reference the event

25

Event Preparation Identification, Selection,

Filtering, Monitoring, Enrichment

Page 26: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

26

Preparation:Identification:Implementations

• TIBCO, Prova, Oracle and IBM implementations for

this pattern

26

event Identification

f(event)

known

event

Other data

or event

Event Preparation Identification, Selection,

Filtering, Monitoring, Enrichment

Page 27: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

27

Preparation:Identification:Implementations: TIBCO BusinessEvents: Rules

• General pattern: <declare> event, data

<if> event-data-match

<then> create-event-

association-with-data

• Effect: find appropriate data that

matches the incoming event*

and create a reference between

the event and the data

27

event

Identify

association

Event

Add

association

to event

Entity

community

* note: event object = concept: input events are immutable**

** note: except in PreProcessor

Event Preparation Identification, Selection,

Filtering, Monitoring, Enrichment

Page 28: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

28

Preparation:Identification:Implementations: Oracle EP: queries

• General pattern:

SELECT <property>

FROM <SOURCE>

WHERE <predicate-condition>

• Effect: matches events coming from

<SOURCE> with pre-defined event-type

28

event

SOURCE

event

Event-type

Event Preparation Identification, Selection,

Filtering, Monitoring, Enrichment

Page 29: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

29

Preparation:Identification:Implementations:Prova Semantic Identification

• General pattern: <rcvMsg / rcvMult> Event ^^ Semantic Type :-

• Effect: find appropriate event entity type that matches the

incoming event data and create a reference between the event

and the event type (from the semantic knowledge), so that the

syntactic (event) data becomes a (semantic) event, which has

relations with the semantic background knowledge base 29

Event Preparation Identification, Selection,

Filtering, Monitoring, Enrichment

Event

Data

Semantic

Identification

f(event)

Semantic

event

Semantic

Knowledge Base

Page 30: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

30

Preparation:Semantic Identification:Prova Semantic Rules

Event Stream {(Name, “OPEL”)(Price, 45)(Volume, 2000)(Time, 1) } {(Name, “SAP”)(Price, 65)(Volume, 1000) (Time, 2)}

{(OPEL, is_a, car_manufacturer), (car_manufacturer, build, Cars),

(Cars, are_build_from, Metall), (OPEL, hat_production_facilities_in, Germany),

(Germany, is_in, Europe) (OPEL, is_a, Major_corporation),

(Major_corporation, have, over_10,000_employees)}

Semantic Knowledge Base

rcvMult(SID,stream,“S&P500“, inform,

tick(Name^^car:Major_corporation,P^^currency:Dollar,

T^^time:Timepoint)) :- …<further processing of semantic event>.

Event Preparation Identification, Selection,

Filtering, Monitoring, Enrichment

Identify event and associate with

semantic background knowledge

Page 31: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

31

Preparation:Identification:Implementations: IBM WODM Decision Server Events

31

Event Preparation Identification, Selection,

Filtering, Monitoring, Enrichment

• Example: event rule relates to ‘new purchase order’ events,

and the events are processed in the context of the customer

number

• Effect: presence of ‘new purchase order’ in the rule

identifies that event type, and the events are processed in

the context of the customer number field – the events of

interest could be said to be identified by ‘new purchase

order’ type and customer number, or the latter could be

regarded as selection

Page 32: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

Preparation:Identification:Implementations: IBM InfoSphere Streams (SPL)

32

Event Preparation Identification, Selection,

Filtering, Monitoring, Enrichment

type Kind = enum { knownKind1, knownKind2, knownKind3 }; /*...*/ stream<rstring key, int32 payload> Events = PrepId_EventsSource() {} stream<rstring key, Kind kind> Mappings = PrepId_MappingsSource() {} stream<Kind kind, int32 payload> KnownEvents = Custom(Events; Mappings) { logic state: mutable map<rstring, Kind> _keyToKind = {}; onTuple Events: if (key in _keyToKind) submit({ kind=_keyToKind[key], payload=payload }, KnownEvents); onTuple Mappings: _keyToKind[key] = kind; }

Custom function

Known Events

Filter to map Event

key to known kind Identified

known events

• Example:

stream Events

stream Mapping

Page 33: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

33

RA:Preparation:Selection:

Selection: particular events selected for further analysis or

pattern matching

– E.g. Selecting the n’th event as a part of a sampling function

– E.g. Selecting events related to some existing event to allow

enrichment of that existing event

33

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 34: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

34

Preparation:Selection:Classification

• Alternative Names:

– Event Query

– See also Event Filtering and Event Identification

• EPTS Reference Architecture:

– “During event preparation, particular events may be

selected for further analysis. Different parts of event

processing may require different selections of events.

See also event filtering. "

• EPIA:

– Filter

34

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 35: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

35

Preparation:Selection:Description

• Role / Explanation

– Selecting or locating an event due to some criteria or rules

• Associated Business Rule Specification

– a selection (to be enforced during the processing of events):

All <event entities> whose <attribute1> matches the

<attribute2> of <some other event or data>

is selected for <processing>.

35 event

Selection

f(event)

selected

event

Other data

or event

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 36: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

36

Preparation:Selection:Structure

• EPTS Glossary comparison

– Event type (event class, event definition, or event

schema)

• A class of event objects.

• Event types should be defined within some type definition

system ... will usually specify certain predefined data

(attributes), examples of which might be:

– A unique event identifier used to reference the event

36

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 37: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

37

Preparation:Selection:Implementations

• TIBCO, Prova, Oracle and IBM implementations for

this pattern

37

event Selection

f(event)

selected

event

Other data

or event

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 38: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

38

Preparation:Selection:Implementations: TIBCO BusinessEvents: Rules

• General pattern: <declare> event, data

<if> event-data-match

<then> process-event

• Effect: filter event per selection criteria

(Note: action may be to

mark event as selected)

38

event

Compare event

vs selection

criteria

Event

Entity

community

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 39: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

39

Preparation:Selection:Implementations:Prova

• General pattern: <rcvMsg / rcvMult>

(CID,Protocol, Agent,Performativ,Event|Context)

:- <selection_processing> Event.

• Effect: select event according to message context (conversation

id, protocol, agent, pragmatic performativ, event context) (see

also monitoring) and further process selected events, e.g.

comparing them against existing events, data, guards, rules,

selection functions 39

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Event

Selection

f(message(event))

Selected

Event

Existing Event, Data,

Guards, Rules

Event

Compare Event vs.

selection criteria

((Events, Data, Guards)

Page 40: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

40

Preparation:Selection:Implementations:Prova Messaging Reaction Rules in Prova

• Send a message

sendMsg(CID,Protocol,Agent,Performative,[Predicate|Args]|Context)

• Receive a message

rcvMsg(CID,Protocol,Agent,Performative,[Predicate|Args]|Context)

• Receive multiple messages

rcvMult(CID,Protocol,Agent,Performative,[Predicate|Args]|Context)

Selection according to:

– CID is the conversation identifier

– Protocol: protocol e.g. self, jms, esb etc.

– Agent: denotes the target or sender of the message

– Performative: pragmatic context, e.g. FIPA ACL

– [Predicate|Args]|Context or Predicate(Arg1,..,Argn): Event

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 41: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

41

Preparation:Selection:Implementations:Prova Example % Select stock ticker events from stream "S&P500"

% Each received event starts a new subconversation (CID)

which further processes the selected event (select)

rcvMult(CID,stream,“S&P500“, inform, tick(S,P,T)) :-

select(CID,tick(S,P,T)).

% Indefinitely (count=-1) receive further ticker events from

other streams that follow the previous selected event in

event processing group (group=g1). If the price differs

for the same stock at the same time [T1=T2, P1!=P2] then …

select(CID,tick(S,P1,T1)) :-

@group(g1) @count(-1)

rcvMsg(CID,stream, StreamID ,inform, tick(S,P2,T2))

[T1=T2, P1!=P2] %guard: if at same time but different price,

println (["Suspicious:",StreamID, tick(S,P2,T2)]," ").

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 42: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

42

Preparation:Selection:Implementations: Oracle EP: queries

• General pattern:

SELECT <property>

FROM <SOURCE>

WHERE <predicate-condition>

• Effect: select certain properties of events

from <SOURCE>

42

event

select

event

Event

properties

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 43: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

43 43

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Preparation:Selection:Implementations: IBM WODM Decision Server Events

• Uses ‘business objects’ to select properties

of incoming events that will be used when

events are processed

• Business Objects allow for a level of

indirection between incoming events and

specifications made by the business about

how to respond to events

• Mapped via the Event Designer tool

Page 44: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

44 44

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

type Kind = enum { knownKind1, knownKind2, knownKind3 };

/*...*/

stream<int32 payload> Selected = Functor(KnownEvents) {

param filter : kind == knownKind1;

}

stream<int32 payload> ProcessedEvents = PrepSel_Process(Selected) {}

Preparation:Selection:Implementations: IBM InfoSphere Streams (SPL)

event stream Functor selected

events

Known

Events

Select events

which match

known kind

Kind 1 of

known events

• Example:

Page 45: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

45

RA:Preparation:Filter

Filter: filter out all events that have some property in their

payload / data

– E.g. customer purchases: filter out those with values < $100

45

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 46: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

46

Preparation:Filter:Classification

• EPTS Reference Architecture:

– "During event preparation, a stream or list of events may

be filtered on some payload or metadata information

such that some subset is selected for further processing."

• EPIA:

– Filter (EPA or Event Processing Agent) performs filtering

only and has no matching or derivation steps, so it does

not transform the input event.

46

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 47: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

47

Preparation:Filter:Description

• Role / Explanation

– Comparing some property of the event (an attribute or

metadata) with some other value (from some other event, or

data)

• Associated Business Rule Specification

– As a constraint: a selection (to be enforced during the

processing of events):

All <event entities>

that have

<filter expression>

must have

<some status>. 47

event Filter

f(event) event

discarded

event

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 48: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

48

Preparation:Filter:Structure

• EPTS Glossary comparison

– A filter pattern can relate to several terms in the EPTS

Glossary

• can be specified in an event pattern (A template containing

event templates, relational operators and variables...)

• .. by an event pattern constraint (A Boolean condition that

must be satisfied by the events observed in a system...)

• .. as part of an event processing rule (A prescribed method

for processing events.)

• .. or as part of event stream processing (Computing on inputs

that are event streams.)

48

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 49: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

49

Preparation:Filter:Implementations

• TIBCO, Oracle, Prova and IBM implementations for

this pattern

49

event Filter

f(event) event

discarded

event

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 50: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

50

Preparation:Filter:Implementations: TIBCO BusinessEvents: rules

• General pattern: <declare> event

<if> event-boolean-expression

<then> update-event-with-fact

OR

action-for-filtered-event

• Effect: ignore filtered events

on operations on non-filtered events

50

event

Filter

f(event)

event

Operation on

event

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 51: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

51

Preparation:Filter:Implementations: TIBCO BusinessEvents: event removal rules

• Implementation type: Rete-based rule-driven system

• Alternate pattern: <declare> event

<if>

NOT event-boolean-expression

<then>

consume-event

• Effect: removed filtered events from

system (and ALL subsequent operations!)

51

event

NOT Filter

!f(event)

event

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 52: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

52

Preparation:Filter:Implementations: TIBCO BusinessEvents: State transitions

• General pattern: <state models for concept C>

<route from State n to n+1>

<on> event

<where> event-filter

• Effect: change state for some concept

• Notes: this is a specialisation of the

general rule pattern!

52

event

Filter

query(event)

State

transition

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 53: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

53

Preparation:Filter:Implementations: TIBCO BusinessEvents: Queries

• General pattern: <select> output

<from> event

<where> event-filter

<policy> notification

• Effect: collect filtered events for

event operations per some policy

• Note: the pattern matching framework

has a similar notion

but a different surface syntax 53

event

Filter

query(event)

Event

subset

Operations on

event subset

on policy event

Policy

event

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 54: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

54

Preparation:Filter:Implementations: Oracle EP: queries

• General pattern:

SELECT <property>

FROM <SOURCE>

WHERE <predicate-condition>

• Effect: discard events whose predicate

evaluate to false

54

event

predicate

event

Operation on

event

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 55: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

55

Preparation:Filter:Implementations: Oracle EP: model

55

• Scenario:

– Select only stocks from the stock tick stream whose

symbols are ‘AAA’.

• Application Model

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 56: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

56

Preparation:Filter:Implementations: Oracle EP: query

56

Specify STREAM source

Define predicate for

filtering

SELECT *

FROM StockTickStream

WHERE symbol = ‘AAA’

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 57: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

57

Preparation:Filter:Implementations: Prova: rules

• Implementation type: backward reasoning logical filter

• General pattern template: <rcvMsg / rcvMult> Event :-

logical filter condition(s),

... .

logical derivation rule 1 :-

filter conditions.

logical derivation rule 2 :-

filter conditions.

...

• Effect: Unify received events with

event pattern definition and apply logical

filter rules (derivation rules with rule chaining) 57

event

Event pattern +

logical rule filter

event

Operation on

event

event

Backward

Resoning

Rule

chaning

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 58: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

58

Preparation:Filter:Implementations: Prova: Messaging Reaction Rules in Prova

• Send a message

sendMsg(XID,Protocol,Agent,Performative,[Predicate|Args]|Context)

• Receive a message

rcvMsg(XID,Protocol,Agent,Performative,[Predicate|Args]|Context)

• Receive multiple messages

rcvMult(XID,Protocol,Agent,Performative,[Predicate|Args]|Context)

Description:

– XID is the conversation identifier

– Protocol: protocol e.g. self, jms, esb etc.

– Agent: denotes the target or sender of the message

– Performative: pragmatic context, e.g. FIPA Agent Communication

– [Predicate|Args] or Predicate(Arg1,..,Argn): Message payload

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 59: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

59

Preparation:Filter:Implementations: Prova: Example

% Filter for stocks starting with „A“ and price > 100

rcvMult(SID,stream,“S&P500“, inform, tick(S,P,T)) :-

S = "A.*",

P > 100,

sendMsg(SID2,esb,"epa1", inform, happens(tick(S,P),T).

% Filter for stocks starting with „A“ and price > 100

rcvMult(SID,stream,“S&P500“, inform, tick(S,P,T)) :-

filter(S,P),

sendMsg(SID2,esb,"epa1", inform, happens(tick(S,P),T).

filter(Symbol,Price) :-

Price > 100,

Symbol = "A.*".

Example with rule chaining

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 60: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

60

Preparation:Filter:Implementations: Prova: Semantic Event Processing Example

Event Stream {(Name, “OPEL”)(Price, 45)(Volume, 2000)(Time, 1) } {(Name, “SAP”)(Price, 65)(Volume, 1000) (Time, 2)}

Semantic Query Filer: Stocks of companies, which have production facilities in Europe and produce products out of metal and Have more than 10,000 employees.

{(OPEL, is_a, car_manufacturer), (car_manufacturer, build, Cars),

(Cars, are_build_from, Metall), (OPEL, hat_production_facilities_in, Germany),

(Germany, is_in, Europe) (OPEL, is_a, Major_corporation),

(Major_corporation, have, over_10,000_employees)}

Semantic Knowledge Base

rcvMult(SID,stream,“S&P500“, inform,

tick(Name^^car:Major_corporation,P^^currency:Dollar,

T^^time:Timepoint)) :- … <semantic filter inference> .

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 61: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

61

Preparation:Filter:Implementations: IBM WODM Decision Server Events

61

event

Filter

event

Action

event

• Example:

Define a filter to only process events with a

purchase value greater than £100

• Effect: Only process purchase events

which match the filter, taking the action to

send a discount voucher

event

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 62: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

62

Preparation:Filter:Implementations: IBM CICS Transaction Server for z/OS application event capture (1)

62

event

Filter

event

Operation on

event

• Example:

Define a filter to only produce events for banking transactions with

a value greater than $5000

• Effect: Only produce events which match the filtering predicates

event

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 63: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

63

Preparation:Filter:Implementations: IBM CICS Transaction Server for z/OS application event capture (2) Details

63

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

<eventCaptureSpecification>

<name>LargeCashDepositCS</name>

<eventIdentifier>LargeCashDeposit</eventIdentifier>

<description>Capture details for event when large cash deposit is

made</description>

<filter>

<contextFilter>

. . .

</contextFilter>

<locationFilter filterType="PROGRAM_INIT">

<programInit>

<PROGRAM filterValue="CPPEPM22" filterOperator="EQ"

keyword="PROGRAM"/>

<CHANNEL filterValue="" filterOperator="OFF" keyword="CHANNEL"/>

</programInit>

</locationFilter>

<dataFilter>

<filterItem>

<dataFilter filterValue="5000" filterOperator="GT"

languageStructureFilename="" languageStructureName="" languageVariableName=""

dataPrecision="0" dataType="PACKED" length="8" offset="5" container="APP&quot;PX22"

source="CHANNEL"/>

</filterItem>

</dataFilter>

</filter>

<dataCapture>

. . .

</dataCapture>

</eventCaptureSpecification>

Page 64: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

64 64

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

stream<rstring name, uint32 age> People = PrepFil_Source() {}

stream<rstring name, uint32 age> YoungPeople = Filter(People) {

param filter : age < 30u;

}

Preparation:Filter:Implementations: IBM InfoSphere Streams (SPL)

event stream Filter event

stream

People Filter age < 30

Young People

• Example:

event stream

Un-Young People

Page 65: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

65

RA:Preparation:Monitoring:

Monitoring: particular event channels are monitored to identify

events of interest

– E.g. Selecting a particular channel and event selector in

some middleware subscription

65

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 66: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

66

Preparation:Monitoring:Classification

• Alternative Names:

– Event Subscribe

– See also Event Selection

• EPTS Reference Architecture:

– “During event preparation, particular types of events may

be monitored for selection for further processing. This

may utilise specific mechanisms external to the event

processing such as exploiting event production features."

• EPIA:

– Filter

66

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 67: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

67

Preparation:Monitoring:Description

• Role / Explanation

– Directing an external agent to select events

– Applying some monitor function over some event channel

• Associated Business Rule Specification

– a monitor (of an event source):

All <event entities> whose <attribute1> matches the

<attribute2> of <some monitoring criteria>

is selected for <processing>.

67 event

Monitor

f(event)

monitored

event

Monitor

criteria

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 68: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

68

Preparation:Monitoring:Structure

• EPTS Glossary comparison

– (Example mechanism): Publish-and-subscribe (pub-sub)

• A method of communication in which messages are

delivered according to subscriptions .

• Note 1. Subscriptions define which messages should flow to

which consumers.

• Note 2. Event processing applications may use publish-and

- subscribe communication for delivering events. However,

publish - and - subscribe is not definitional to event

processing – other communication styles may be used.

– Subscriber

• An agent that submits a subscription for publish- and –

subscribe communication.

68

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 69: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

69

Preparation:Monitoring:Implementations

• TIBCO, Prova, Oracle and IBM implementations for

this pattern

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

event Monitor

f(event)

monitored

event

Monitor

criteria

Page 70: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

70

Preparation:Monitoring:Implementations: TIBCO BusinessEvents: Event Destination

• General pattern: <define> channel,

destination, event

• Effect: subscribe to events on this

channel per this destination

(Note: part of Event definition)

70

Channel

Destination

Event

Entity

channel

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 71: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

71

Preparation:Monitoring:Implementations: Oracle EP: Event Destination

• General pattern: <channel>

<name>dest1</name>

<selector>query1</selector>

</channel>

• Effect: subscribe to events on this

channel per selected queries.

71

Channel

Selector

Event

Entity

channel

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 72: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

72

Preparation:Monitoring:Implementations: Prova: Selection

• General pattern: <rcvMsg / rcvMult>

(CID,Protocol, Agent,Performativ,Event|Context)

:- <selection_processing> Event .

• Effect: monitor message channels and select (see also

Selection Pattern) events according to message context

(conversation id, protocol, agent, pragmatic performativ, event

context); 72

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Event

Monitor + Selection

f(message(event))

Selected

Event

Event

Page 73: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

73

Preparation:Monitoring:Implementations: Prova: Messaging Reaction Rules in Prova

• Send a message

sendMsg(CID,Protocol,Agent,Performative,[Predicate|Args]|Context)

• Receive a message

rcvMsg(CID,Protocol,Agent,Performative,[Predicate|Args]|Context)

• Receive multiple messages

rcvMult(CID,Protocol,Agent,Performative,[Predicate|Args]|Context)

Selection according to:

– CID is the conversation identifier

– Protocol: protocol e.g. self, jms, esb etc.

– Agent: denotes the target or sender of the message

– Performative: pragmatic context, e.g. FIPA ACL

– [Predicate|Args]|Context or Predicate(Arg1,..,Argn): Event

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 74: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

74 74

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Preparation:Monitoring:Implementations: IBM WODM Decision Server Events

• Example: Use an event

connector that

subscribes for

aircraft arrivals

from a JMS topic

• Effect: A message

selector is used to

limit which messages

will trigger events,

and an XSL

stylesheet is used to

transform the

message text into an

event packet.

Page 75: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

75 75

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

type BargainType = rstring symbol, float32 price;

/* in exporting application */

() as ExportTech = Export(TechSectorBargains) {

param properties : { kind = "bargains", category = "tech" };

}

/* in importing application */

stream<BargainType> TechBargains = Import() {

param subscription : kind == "bargains" && category == "tech";

}

Preparation:Monitoring:Implementations: IBM InfoSphere Streams (SPL)

• Example:

• Effect: Stream of bargains in tech sector is sent (exported) from

current application, making it available to another application which

imports the stream to monitor bargains in various sectors

Page 76: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

76

RA:Preparation:Enrichment:

Enrichment: add some information based on prior events

– E.g. customer purchase: add the customer history to the

purchase event

– E.g. enrich the event with semantic background knowledge

76

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 77: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

77

Preparation:Enrichment:Classification

• Alternative Names:

– Event Annotation, Semantic Event

• EPTS Reference Architecture:

– "During event preparation, events may be "enriched"

through knowledge gained through previous events or

data (including semantic background knowledge). "

• EPIA:

– Enrich (EPA): a translate EPA that takes a single input

event, uses it to query data from a global state element,

and creates a derived event which includes the attributes

from the original event, possibly with modified values... 77

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 78: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

78

Preparation:Enrichment:Description

• Role / Explanation

– Updating some attribute or metadata of the event with some

other values, possibly computed, from some other event or

data

• Associated Business Rule Specification

– a selection (to be enforced during the processing of events):

All <event entities> that are also <enriched data

expression>

must have <some status>.

78

event Enrichment

f(event) event

Other data

or event

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 79: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

79

Preparation:Enrichment:Structure

• EPTS Glossary comparison

– Enrichment can be specified in an event pattern (pattern

definition containing additional meta data, semantic

knowledge)

– .. as part of an event processing rule (A method for

enriching events during processing, e.g., by analysing the

meta data and querying external data sources or

inferencing semantic knowledge bases)

– .. or as part of event stream processing (additional

additional information to event stream data, e.g. by

querying external data sources)

79

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 80: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

80

Preparation:Enrichment:Implementations

• TIBCO, Oracle, Prova and IBM implementations for

this pattern

80

event Enrichment

f(event) event

Other data

or event

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 81: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

81

Preparation:Enrichment:Implementations: TIBCO BusinessEvents: Rules

• General pattern: <declare> event, data

<if> event-data-match

<then> update-event-with-data

• Effect: find appropriate data for

events and apply enrichment

operations on them*

81

event

Identify

enrichment data

Event

Enrich

Operation

on event

Enrich

data

* note: event object: events cannot be modified

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 82: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

82

Preparation:Enrichment:Implementations: Oracle EP: queries

• General pattern template:

SELECT <property>

FROM <STREAM>[WIN],<RELATION>

WHERE <join-condition>

• Effect: enrich events from stream

with contextual data from relation

where condition applies.

82

event

join-condition

event

Operation on

event

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 83: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

83

Preparation:Enrichment:Implementations: Oracle EP: model

83

• Scenario:

– For every stock event from stock tick stream whose bid is

greater than 5.0, find its company’s location.

• Application Model

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 84: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

84

Preparation:Enrichment:Implementations: Oracle EP: queries

84

SELECT event.symbol, location

FROM

StockTickStream [NOW] AS event,

StockTable AS data

WHERE event.symbol = data.symbol AND

event.lastBid > 5.0

No need for window

operator on a ‘relation’

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 85: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

85

Preparation:Enrichment:Implementations: Oracle EP: table metaphor

85

Time Input (StockTickStream) Output (QUERY)

1

2

3

Symbol Full-name Location

AAA The AAA company San Francisco

BBB The BBB company San Jose

{“AAA”, 10.0, 10.5} {“AAA”, “San…”}

{“BBB”, 11.0, 12.5} {“BBB”, “San Jose”}

{“AAA”, 4.0, 4.5}

StockTable

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 86: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

86

Preparation:Enrichment:Implementations: Prova: rules

• General pattern template: rcvMsg / rcvMult <Event Msg Pattern(s)> :-

<query built-in(s)>,

<enrichment condition(s)>

... .

<logical enrichment rule 1> :-

<enrichment operations>.

• Effect: query data or knowledge from

external source and enrich the event with it

86

event

Query

event*

Operation on

enriched event

External

data

Enrichment

rules

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 87: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

87

• Java ..., L=java.util.ArrayList(),, ..., java.lang.String(“Hello”)

• File Input / Output ..., fopen(File,Reader), ...

• XML (DOM) document(DomTree,DocumentReader) :- XML(DocumenReader),

• SQL …,sql_select(DB,cla,[pdb_id,“1alx“],[px,Domain]).

• RDF ...,rdf(http://...,"rdfs",Subject,"rdf_type","gene1_Gene"),

• XQuery ..., XQuery = ' for $name in StatisticsURL//Author[0]/@name/text() return $name', xquery_select(XQuery,name(ExpertName)),

• SPARQL ...,sparql_select(SparqlQuery,name(Name),class(Class), definition(Def)),

Preparation:Enrichment:Implementations: Prova: External Data and Object Integration

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 88: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

88

Preparation:Enrichment:Implementations: Prova: Example with SPARQL Query % Filter for car manufacturer stocks and enrich the stock tick

event with data from Wikipedia (DBPedia) about the manufacturer

and the luxury cars

rcvMult(SID,stream,“S&P500“, inform, tick(S,P,T)) :-

carManufacturer(S,Man), % filter car manufacturers

findall([luxuryCar|Data],luxuryCar(Man,Name,Car),Data), % query

EnrichedData = [S,Data], % enrich with additional data

sendMsg(SID2,esb,"epa1", inform, happens(tick(EnrichedData,P),T).

% rule implementing the query on DBPedia using SPARQL query

luxuryCar(Manufacturer,Name,Car) :-

Query="SELECT ?manufacturer ?name ?car % SPARQL RDF Query

WHERE {?car <http://purl.org/dc/terms/subject>

<http://dbpedia.org/resource/Category:Luxury_vehicles> .

?car foaf:name ?name .

?car dbo:manufacturer ?man .

?man foaf:name ?manufacturer.

} ORDER by ?manufacturer ?name“,

sparql_select(Query,manufacturer(Manufacturer),name(Name),car(Car)).

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Page 89: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

89

Preparation:Enrichment:Implementations: IBM WODM Decision Server Events

89

event

Enriched

event

• Example: Define a Mapped Key expression, where a

field in the event is used to look up objects in a

database table

• Effect: Required enrichment data is obtained from the

data base and added to the event

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

Mapped Key

look up

Page 90: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

90 90

Event Preparation Identification, Selection, Filtering,

Monitoring, Enrichment

/* VWAP = volume-weighted average price */

stream<rstring sym, float32 vwap> Vwaps = PrepEnr_VwapsSource() {}

stream<rstring sym, float32 price> Trades = PrepEnr_TradesSource() {}

stream<rstring sym, float32 vwap, float32 price> E = Join(Vwaps; Trades) {

window

Vwaps : sliding, count(0);

Trades : sliding, count(100), count(1), partitioned;

param

equalityLHS : Vwaps.sym;

equalityRHS : Trades.sym;

partitionByRHS : Trades.sym;

}

Preparation:Enrichment:Implementations: IBM InfoSphere Streams (SPL)

stream

Join Enriched Stream

Trades Trades

enriched with

VWAPs

• Example:

stream VWAPs

Page 91: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

91

Event Analysis is the process of analysing suitably

prepared events and their payloads and metadata for

useful information.

For example, event analysis may involve the

identification of events against existing entities.

91

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

EP Patterns:RA:Analysis

Page 92: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

92

RA:Analysis:Analytics

Analytics: use of mathematical methods like statistics to

derive additional information about an event or set of

events

– E.g. standard deviation of the response time of a system,

used to give a quantitative comparison of current with past

performance of a system

92

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

Page 93: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

93

Analysis:Analytics:Classification

• Alternative Names:

– Event Statistics

• EPTS Reference Architecture:

– “Event Analytics are the use of statistical methods to

derive additional information about an event or set of

events.”

• EPIA:

– Aggregate (EPA): a transformation EPA that takes as

input a collection of events and creates a single derived

event by applying a function over the input events.

93

Event Analysis Analytics, Transforms, Tracking,

Scoring, Rating, Classification

Page 94: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

94 event

event

Analysis:Analytics:Description

• Role / Explanation

– Analytics is where multiple data items (in a single event, or

more commonly across multiple events), are analysed

mathematically to define some additional information

– For example: an average or mean value of event data can

be used to compare against subsequent events to indicate

some trend (see Learning)

• Associated Business Rule Specification

– an event analytics (event processing) process:

The <analytic fact> is provided by <analytic function>

applied to <events>

94

event Analytics

f(events) info

Event Analysis Analytics, Transforms, Tracking,

Scoring, Rating, Classification

Page 95: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

95

Analysis:Analytics:Structure

• EPTS Glossary comparison

– Cause: An event A is a cause of another event B, if A had to happen

in order for B to happen.

– Event abstraction: The relationship between a complex event and the

other events that it denotes, summarizes, or otherwise represents.

– Derived event (synthesized event, synthetic event): An event that is

generated as a result of applying a method or process to one or

more other events.

– Relationships between events: Events are related by time,

causality, abstraction, and other relationships. Time and causality

impose partial orderings upon events.

– Event pattern: A template containing event templates, relational

operators and variables.

– Constraint (event pattern constraint): A Boolean condition that must

be satisfied by the events observed in a system. 95

Event Analysis Analytics, Transforms, Tracking,

Scoring, Rating, Classification

Page 96: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

96

Analysis:Analytics:Implementations

• TIBCO, Prova, Oracle, IBM implementations for this

pattern

96

event Analytics

f(events,data) info

Event Analysis Analytics, Transforms, Tracking,

Scoring, Rating, Classification

past

event data

Page 97: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

97

Analysis:Analytics:Implementations: TIBCO BusinessEvents: Rules

• General pattern: <declare> event,

historic-event-object

<if>

event-matches-statistical-

requirement

<then>

apply-update-function for

event, historical-event-object

• Effect: event-driven statistic function (eg update of mean

for a given event type or other condition) 97

event

Statistical

operation

or process

Information

past data

Event Analysis Analytics, Transforms, Tracking,

Scoring, Rating, Classification

Page 98: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

98

Analysis:Analytics:Implementations: Prova: Rules

• General pattern:

rcvMsg / rcvMult <Event Msg Pattern(s)> :-

<query (past) data>,

<analytics functions(s)

with data>

... .

<analytics rule 1> :-

<Java call to analytics API>,

<or Prova functional programming>.

• Effect: event-driven statistic function (eg update of mean

for a given event type or other condition) 98

event

Analytics

function(s)

Information

(past) data

Event Analysis Analytics, Transforms, Tracking,

Scoring, Rating, Classification

Page 99: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

99

Analysis:Analytics:Implementations: Prova: Java Programming and Functional Programming

Event Analysis Analytics, Transforms, Tracking,

Scoring, Rating, Classification

Prova Dynamic Java Programming

• Ability to embed Java calls … :- X = java.lang.Math.abs(2.99), X+1, …

• Java constructors, object instance + methods and static methods, and

public field access; (+ Java exception handling)

Prova Functional Programming

• single‐ and multi‐valued functions,

• functional composition with the extended derive built‐in;

• partial evaluation;

• lambda functions;

• monadic functions;

• monadic bind using a composition of map and join;

• maybe, list, state, tree and fact monads

Page 100: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

100

Analysis:Analytics:Implementations: Oracle EP: queries

• General pattern:

SELECT analytical-

function(<property>)

FROM <SOURCE>

WHERE <predicate-condition>

• Effect: performs analytical function on

selected properties, such as std-dev,

regressions, etc.

100

event

analytics

info

Event Analysis Analytics, Transforms, Tracking,

Scoring, Rating, Classification

Page 101: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

101 101

Analysis:Analytics:Implementations: IBM WODM Decision Server Events

Event Analysis Analytics, Transforms, Tracking,

Scoring, Rating, Classification

• General Pattern: Filters in Decision Server

Events allow analytic expressions to be applied

Page 102: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

102 102

Analysis:Analytics:Implementations: IBM InfoSphere Streams (SPL)

Event Analysis Analytics, Transforms, Tracking,

Scoring, Rating, Classification

stream<rstring sym, float32 vwap, float32 price> E = Join(Vwaps; Trades) {

/* (see Preparation:Enrichment) */

}

/* bargain if price lower than volume-weighted average price */

stream<rstring sym, float32 index> Bargains = Functor(E) {

output Bargains : index = (vwap <= price)

? 0f : price * exp(vwap – price);

}

event stream Functor event

stream

Trades with ave price

Analyze trades to

detect bargains Bargains

• Example:

Page 103: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

103

RA:Analysis:Transforms

Transforms: processes carried out on events’ payloads or

data, such for as standardising schema or information

types

– E.g. an insurance application event from a custom IT system

may have the customer information translated into an

ACORD-type structure for onward processing

103

Event Analysis Analytics, Transforms, Tracking,

Scoring, Rating, Classification

Page 104: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

104

Analysis:Transforms:Classification

• Alternative Names:

– Event Processes, Event Operations

• EPTS Reference Architecture:

– “Event Transforms are processes carried out on event

payloads or data, either related to event preparation,

analysis or processing.”

• EPIA:

– Transformation (EPA): an EPA that includes a derivation

step and optionally also a filtering step.

• Derived event: an event generated as a result of EP in EPA

104

Event Analysis Analytics, Transforms, Tracking,

Scoring, Rating, Classification

Page 105: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

105

Analysis:Transforms:Description

• Role / Explanation

– Transforms are applied to events (event data) to process

into new, derived events or event data

– For example: a transform may reformat an input data

schema into an output schema format

• Associated Business Rule Specification

– an event transform:

The <derived fact> is derived from <input event> when

<transform fn> is applied

105

event Transform

f(event) derivation

Event Analysis Analytics, Transforms, Tracking,

Scoring, Rating, Classification

Page 106: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

106

Analysis:Transforms:Structure

• EPTS Glossary comparison

– Derived event (synthesized event, synthetic event): An event that is

generated as a result of applying a method or process to one or

more other events.

106

Event Analysis Analytics, Transforms, Tracking,

Scoring, Rating, Classification

Page 107: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

107

Analysis:Transforms:Implementations

• TIBCO, Prova, Oracle, and IBM implementations for

this pattern

107

event Transform

f(events,data) info

Event Analysis Analytics, Transforms, Tracking,

Scoring, Rating, Classification

Page 108: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

108

Analysis:Transforms:Implementations: TIBCO BusinessEvents: Rules

• General pattern: <declare> event

<if>

event-matches-selection

<then>

apply-derivation-function

for event

• Effect: (usually identified, filtered) events have some

transformation applied to their data, used to update some

event object (or create a new event) 108

event

Event operation

Information

Event Analysis Analytics, Transforms, Tracking,

Scoring, Rating, Classification

Page 109: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

109

Analysis:Transforms:Implementations: Prova: Rules

• General pattern:

rcvMsg / rcvMult <Event Msg

Pattern(s)> :-

<transformation function

(Output, Input)>,

... .

<transformation rule 1>(Output,

Input) :-

<transformation logic >

• Effect: transforms selected properties into

new properties, applying function if needed. 109

event

transformation

info

Event Analysis Analytics, Transforms, Tracking,

Scoring, Rating, Classification

Page 110: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

110

Analysis:Transforms:Implementations: Oracle EP: queries

• General pattern:

SELECT f(<property>) as <derived>

FROM <SOURCE>

WHERE <predicate-condition>

• Effect: transforms selected properties

into new properties, applying function if

needed.

110

event

transformation

info

Event Analysis Analytics, Transforms, Tracking,

Scoring, Rating, Classification

Page 111: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

111 111

Analysis:Transforms:Implementations: IBM WODM Decision Server Events

Event Analysis Analytics, Transforms, Tracking,

Scoring, Rating, Classification

• Example: Use

Javascript to

transform an

event e.g.

transform order

total from number

of units to overall

price in euros

• Transforming in

the event runtime

simplifies coding

in the connectors

Page 112: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

112 112

Analysis:Transforms:Implementations: IBM CICS Transaction Server for z/OS

Event Analysis Analytics, Transforms, Tracking,

Scoring, Rating, Classification

• Example: Define an EP

adapter for a CICS Event

Binding (i.e. group of related

events) which will transform

events into a specified format

Page 113: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

113 113

stream<rstring sym,float32 price,float32 vol> PreVwaps=Aggregate(Trades) {

/* (see Detection:Aggregation) */

}

/* VWAP = volume-weighted average price */

stream<rstring sym, float32 vwap> Vwaps = Functor(PreVwaps) {

output Vwaps : vwap = priceVol / vol;

}

Analysis:Transforms:Implementations: IBM InfoSphere Streams (SPL)

Event Analysis Analytics, Transforms, Tracking,

Scoring, Rating, Classification

event stream Functor event

stream

Trades Weighting

transformation Trades with price

transformed to volume-

weighted average price

• Example:

Page 114: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

114

RA:Analysis:Tracking

Tracking: use of events to follow some related entity’s state

(such as in space, time or process status)

– E.g. “track and trace” of the location of airline baggage

114

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

Page 115: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

115

Analysis:Tracking:Classification

• Alternative Names

• EPTS Reference Architecture:

– “Event Tracking is where events related to some entity are

used to identify state changes in that entity.”

• Typically state is related to geolocation, business process, etc

• EPIA:

– State-oriented context: see also spatial context,

segmentation-orientated context, temporal context

115

Event Analysis Analytics, Transforms, Tracking,

Scoring, Rating, Classification

Page 116: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

116

Analysis:Tracking:Description

• Role / Explanation

– Tracking is applied to entities via events (event data) to

maintain information about the entity state (eg location).

– For example: a patient may be tracked by active and

passive sensors creating movement and activity events in a

care facility

– Usually associated with the term “track and trace”

• Associated Business Rule Specification

– an track operation:

The <entity> is tracked to <state>

whenever

<event> occurs 116

event Tracking

f(event)

Event Analysis Analytics, Transforms, Tracking,

Scoring, Rating, Classification

entity

Updated

entity

Page 117: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

117

Analysis:Tracking:Structure

• EPTS Glossary comparison

– Derived event (synthesized event, synthetic event): An event that is

generated as a result of applying a method or process to one or

more other events.

117

Event Analysis Analytics, Transforms, Tracking,

Scoring, Rating, Classification

Page 118: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

118

Analysis:Tracking:Implementations

• TIBCO, Prova, Oracle, IBM implementations for this

pattern

118

Event Analysis Analytics, Transforms, Tracking,

Scoring, Rating, Classification

event Tracking

f(event)

entity

Updated

entity

Page 119: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

119

Analysis:Tracking:Implementations: TIBCO BusinessEvents:StateModel

• General pattern: <state models for concept C>

<route TrackingState n to n+1>

<on> event

<where> event-filter

• Effect: change tracking state

for some concept / entity

• Notes:

1. this is best for explicit, unchanging states

(eg “in range” vs “out of range”)

2. “Tracing” can be added with a simple additional action… 119

Event Analysis Analytics, Transforms, Tracking,

Scoring, Rating, Classification

event

Track(event)

New State

entity

Page 120: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

120

Analysis:Tracking:Implementations: TIBCO BusinessEvents: Rules

• General pattern: <declare> event, entity

<if>

event-matches-entity-selection

<then>

apply-event-tracking-data

to entity

• Effect: tracking events are associated

to the relevant entity (eg by Rete algorithm)

and the rule updates the entity track

• Note: also updateable to “trace” with an additional action

120

event

Event tracking

conditions

Update entity track

Event Analysis Analytics, Transforms, Tracking,

Scoring, Rating, Classification

entity

Page 121: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

121

Analysis:Tracking:Implementations Prova: Rules

121

Event Analysis Analytics, Transforms, Tracking,

Scoring, Rating, Classification

event Tracking Rule

f(event)

entity

Updated

entity

Java

Type Semantic

Type

Page 122: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

Prova: Semantic Type System Top-Level Ontologies

General concepts such as space, time, event, action and their properties and relations

Temporal Ontology

Action Ontology Process

Ontology Agent

Ontology Situation Ontology

Domain Ontologies

Vocabularies related to specific domains by specializing the concepts introduced in the top-level ontology

Task Activities Ontologies

Vocabularies related to generic tasks or

activities by specializing the

concepts introduced in the top-level ontology

Application Ontologies

Specific user/application ontologies

E.g. ontologies describing roles played by domain entities while perfoming application activities

Spatio Ontology

Event Ontology

Event Analysis Analytics, Transforms, Tracking,

Scoring, Rating, Classification

Page 123: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

Prova: Situation/State Top Ontology Model

Situation

Heterogeneous Situation

Homogenous Situation

Dynamic Change

Situation

Time Frame

Situation

Frequency Situation

State Situation

Process Situation

Iterative Situation

Habitual Situation

Situation Properties

(relations to time, location, participants, …)

Situation Content

hasContent hasProperties

Situation Types

Situation Descriptions

LaysOn TheFloor

TrafficLightChanges

Within5Minutes

He Runs

Rings3 Times

He Coughs

He Smokes

Situation Individuals

use the other top ontologies

Event Analysis Analytics, Transforms, Tracking,

Scoring, Rating, Classification

Page 124: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

124

Analysis:Tracking:Implementations: Oracle EP: EPN

124

• Effect:

– Queries select events to be tracked by putting them in

cache. Events can be retrieved by using cache as source

to processor.

• Application Model

Event Analysis Analytics, Transforms, Tracking,

Scoring, Rating, Classification

Page 125: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

125

type UpdateKind = enum { MOVE, ENTER }; /* ... */ stream<int32 vid, UpdateKind kind> Updates = Custom(Tracked as Curr) { logic state : { mutable map<int32, Position> _vid2pos = { }; } onTuple Tracked : { if (Curr.vid in _vid2pos) { if (Curr.pos != _vid2pos[vid]) submit({ vid=Curr.vid, kind=MOVE }, Updates); } else { submit({ vid=Curr.vid, kind=ENTER }, Updates); } _vid2pos[Curr.vid] = Curr.pos; } }

Analysis:Tracking:Implementations: IBM InfoSphere Streams (SPL)

Event Analysis Analytics, Transforms, Tracking,

Scoring, Rating, Classification

• Example:

• Effect: Custom operator used to Track position of vehicles

(by id) in transportation application

Page 126: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

126

RA:Analysis:Scoring

Scoring: ranking events or event data based on some

predefined criteria

– E.g. scoring a customer account application based on

weighting specific aspects of the application to give an

overall “score”

– See Score Modeling and Predictive Analytics

126

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

Page 127: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

127

Analysis:Scoring:Classification

• Alternative Names

• EPTS Reference Architecture:

– “Event Scoring is the process by which events are ranked

using a score, usually as a part of a statistical analysis of a

set of events”.

– Note: Possibly this should be subsumed into Event Analytics

• EPIA:

• Aggregate (EPA): a transformation EPA that takes as input a

collection of events and creates a single derived event by

applying a function over the input events.

127

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

Page 128: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

128

Analysis:Scoring:Description

• Role / Explanation

– Scoring is applied to events or entities by associating score

values to attributes of the event or entity.

– For example: an intelligence report event may be scored

against its attributes (source, time, subject etc) to give an

overall indication (score, for example as a %) of its reliability

– Usually associated with predictive analytics

• Associated Business Rule Specification

– an score operation on an event:

The <entity> is given a score <s> by applying

transform

model <m>

128

event Score model

f(event)

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

Score

Page 129: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

129

Analysis:Scoring:Structure

• EPTS Glossary comparison

– Derived event (synthesized event, synthetic event): An event that is

generated as a result of applying a method or process to one or

more other events.

– Relationships between events: Events are related by time,

causality, abstraction, and other relationships. Time and causality

impose partial orderings upon events.

– Event pattern: A template containing event templates, relational

operators and variables.

– Constraint (event pattern constraint): A Boolean condition that must

be satisfied by the events observed in a system.

129

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

Page 130: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

130

Analysis:Scoring:Implementations

• TIBCO, Prova, Oracle, and IBM implementations for

this pattern

130

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

event Score model

f(event) Score

Page 131: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

131

Analysis:Scoring:Implementations: TIBCO BusinessEvents: Rules

• General pattern: <declare> event

<if>

event-matches-entity-selection

<then>

apply-score-model(eventdata),

combine-score-criteria

• Effect: events’ criteria are applied to

score functions and then combined into a

single score via an event object

• Note: score typically applied in a separate rule

(if score>50% then …) 131

event

Score model

(event)

Update score

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

Page 132: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

132

Analysis:Scoring:Implementations: Prova: Rules

• General pattern: rcvMsg / rcvMult <event> :-

@score(Old) <event>,

<score function (event, Old, New),

<update> @score(New) <event>.

<score function rule>(E,O,N) :-

<compute new score>.

• Effect: events’ metadata score (and other

metadata or event data properties) are

applied to score functions and then combined into a

single updated score as metadata of the event object

132

event

Score model

(event)

Update

event

score

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

Page 133: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

133

Analysis:Scoring:Implementations: Prova: Example – Metadata Scoped Reasoning

% event instances with metadata @key(value,[,value]*)

@score(1) @src(stream1) @label(e1) tick("Opel",10,t1).

@score(3) @src(stream2) @label(e2) tick("Opel",15,t1).

happens(tick(S,P),T):-

% scope defined over @score metadata bound to "Value"

@score(Value) tick(S,P,T) [Value>2]. %guard Value > 2

@score(1) @src(stream1) @label(e1) tick("Opel",10,t1).

@score(3) @src(stream2) @label(e2) tick("Opel",15,t1).

happens(tick(S,P),T):-

% select ticks from „stream1“

@score(Value) @src(stream1) @label(E) tick(S,P,T),

NewValue = Value + 3, %add 3 to score „Value“

%update selected events from „stream1“ with „NewValue“

update(@label(E), @score(NewValue) tick(S,P,T)),

@score(S) tick(S,P,T) [S>3]. %guard Value > 3

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

Page 134: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

134

Analysis:Scoring:Implementations: Oracle EP: queries

• General pattern:

SELECT score@model(<property>)

FROM <SOURCE>

WHERE <predicate-condition>

• Effect: scores properties of events using

model.

134

event

score

info

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

Page 135: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

135

RA:Analysis:Rating

Rating: comparison of events or event data with some metric

to provide an ordering

– E.g. trade transactions may be rated with respect to overall

achieved profit

135

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

Page 136: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

136

Analysis:Rating:Classification

• Alternative Names

• EPTS Reference Architecture:

– “Event Rating is where events are compared to others to

associate some importance or other, possibly relative,

measurement to the event”.

• EPIA:

– Transformation (EPA): an EPA that includes a derivation

step and optionally also a filtering step.

• Derived event: an event generated as a result of EP in EPA

136

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

Page 137: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

137

Analysis:Rating:Description

• Role / Explanation

– Rating provides a quantitative or ordering.

– For example: an customer order shipment may be rated as

“courier” based on the customer order details

– Could be part of “classification”

• Dictionary definition: classification according to grade or rank

• Associated Business Rule Specification

– an rating operation on an event:

The <event> is rated to <rating> by <rating criteria>

137

event Rating

f(event)

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

Rating

Page 138: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

138

Analysis:Rating:Structure

• EPTS Glossary comparison

– Derived event (synthesized event, synthetic event): An event that is

generated as a result of applying a method or process to one or

more other events.

– Relationships between events: Events are related by time,

causality, abstraction, and other relationships. Time and causality

impose partial orderings upon events.

– Event pattern: A template containing event templates, relational

operators and variables.

– Constraint (event pattern constraint): A Boolean condition that must

be satisfied by the events observed in a system.

138

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

Page 139: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

139

Analysis:Rating:Implementations

• TIBCO, Prova, Oracle, IBM implementations for this

pattern

139

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

event Rating

f(event) Rating

Page 140: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

140

Analysis:Rating:Implementations: TIBCO BusinessEvents: Rules

• General pattern: <declare> event

<if>

event-matches-entity-selection

<then>

rating = rating-model(event)

• Effect: events’ criteria

(and possibly enrichment data)

are applied to the rating function

to create a rating classification or ordering

• Note: rating typically applied in a separate rule

(if rating = “First” then …) 140

event

Rating

(event)

Rating

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

Page 141: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

141

Analysis:Scoring:Implementations: Prova: Rules

• General pattern:

see scoring

• Effect: rate events and select rated events

141

event

rating

(event)

Rated Event

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

Page 142: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

142

Analysis:Rating:Implementations: Prova: Metadata Scoped Reasoning and Guards % stream1 is trusted but stream2 is not, so one

solution is found: X=e1

@src(stream1) event(e1).

@src(stream2) event(e2).

%note, for simplicity this is just a simple fact, but

more complicated rating, trust, reputation policies

could be defined

trusted(stream1).%only event from „stream1“ are trusted

ratedEvent(X):-

@src(Source) %scoped reasoning on @src

event(X) [trusted(Source)]. %guard on trusted sources

:-solve(ratedEvent(X)). % => X=e1 (but not e2)

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

Page 143: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

143

Analysis:Rating:Implementations: Oracle EP: queries

• General pattern:

SELECT rate@model(<property>)

FROM <SOURCE>

WHERE <predicate-condition>

• Effect: rates properties of events using

model.

143

event

rate

info

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

Page 144: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

144 144

Analysis:Rating:Implementations: IBM WODM Decision Server Events

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

• General Pattern: Rating in WODM is typically

carried out by the Decision Server Rules

component – the output events indicate something

interesting happened, and then business rules rate

the type of response depending on details in the

events

Page 145: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

145

type Bracket = enum { YOUNG, OLD, OTHER };

stream<rstring name, int32 age> Folks = /*...*/

stream<rstring name, Bracket bracket> Rated = Functor(Folks) {

output Rated : bracket = age < 30 ? YOUNG : age > 65 ? OLD : OTHER;

}

Analysis:Rating:Implementations: IBM InfoSphere Streams (SPL)

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

event stream Functor

event stream

Folks Rating functor

to rate by age

• Example:

Events rated as

YOUNG, OLD or

OTHER

Page 146: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

146

RA:Analysis:Classification

Classification: comparison with and association of events with

some classification scheme to which the event is applied

– E.g. classifying a credit card user as a “gold shopper” for

marketing purposes

146

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

Page 147: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

147

Analysis:Classification:Classification

• Alternative Names

• EPTS Reference Architecture:

– “Event Classification is where events are associated with

some classification scheme (for use in downstream

processing)”.

• EPIA:

– Transformation (EPA): an EPA that includes a derivation

step and optionally also a filtering step.

• Derived event: an event generated as a result of EP in EPA

147

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

Page 148: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

148

Analysis:Classification:Description

• Role / Explanation

– Classification provides associates an event to a class or

category.

– For example: an customer may be classified as “Gold

customer” based on the customer order details

– Discovery / learning may be involved in the definition of

classifications

• Associated Business Rule Specification

– an event is classified:

The <event> is classified as <class>

by <classification criteria>

148 event

Classification

f(event)

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

Classification

Page 149: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

149

Analysis:Classification:Structure

• EPTS Glossary comparison

– Derived event (synthesized event, synthetic event): An event that is

generated as a result of applying a method or process to one or

more other events.

– Relationships between events: Events are related by time,

causality, abstraction, and other relationships. Time and causality

impose partial orderings upon events.

– Event pattern: A template containing event templates, relational

operators and variables.

– Constraint (event pattern constraint): A Boolean condition that must

be satisfied by the events observed in a system.

149

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

Page 150: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

150

Analysis:Classification:Implementations

• TIBCO, Prova, Oracle, IBM implementations for this

pattern

150

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

event Classification

f(event) Classification

Page 151: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

151

Analysis:Classification:Implementations: TIBCO BusinessEvents: Rules

• General pattern: <declare> event

<if>

event-matches-classification

<then>

apply-classification(event)

• Effect: 1 rule per criteria + classification,

and then classification is updated as an

event object / concept attribute

• Note: classification typically used in separate rules

(if class = “Gold customer” then …)

151

event

Classification

(event)

Classification

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

Page 152: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

152

Analysis:Classification:Implementations: Prova: Rules

• General pattern: <rcvMsg / rcvMult> Event ^^ Semantic Type :-

<conditional type discovery>,

derivedEvent ^^ Semantic Type.

• Effect: classify event according to a (semantic) type system.

Discover event type (may include learning). 152

Event (Semantic)

Classification

f(event)

Typed

Event

(Semantic)

Type System

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

Page 153: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

153

Analysis:Classification:Implementations: Prova: Rules - Example

Event Stream {(Name, “OPEL”)(Price, 45)(Volume, 2000)(Time, 1) }

{(OPEL, is_a, Car_Manufacturer), (Car_Manufacturer, sameAs, Motor_Vehicel_Manufacturer),

(Car_Manufacturer, subClassOf, Automotive_Company), (Automotive_Company, sameAs, Automotive_Corporation), (Automotive_Company, subClassOf, Automotive_Industry),

(Automotive_Corporation, subClassOf, Corporation) (Major_Corporation, have, over_10,000_employees),

(Major_Corporation, subClassOf, Corporation)}

Semantic Knowledge Base

(T-Box Model is used as Type

System)

rcvMult(SID,stream,“S&P500“, inform,

tick(Name^^Car:Manufacturer,P,T)) :-

size(Name, Eymployees), Employees > 10000, %type discovery

sendMsg(SID,self,0,inform,tick(Name^^Major_Cooperation,P,T)

.

Classification according to

Type System

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

Classification by Type Discovery

(may also include Type Learning)

Page 154: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

154

Analysis:Classification:Implementations: Oracle EP: queries

• General pattern:

SELECT classify@model(<property>)

AS <classification>

FROM <SOURCE>

WHERE <predicate-condition>

• Effect: classifies properties of events

using model, and applies to result.

154

event

classification

info

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

Page 155: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

155

Analysis:Classification:Implementations: IBM WODM Decision Server Events

155

event

condition

output event (action)

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

classification rules

decision

• General Pattern: Decision Server

Events component produces actions

(or output events) and then

business rules in Decision Server

Rules apply classification to these to

decide on the final outcome

Page 156: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

156 156

/* (see Analysis:Rating) */

Analysis:Classification:Implementations: IBM InfoSphere Streams (SPL)

Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

Page 157: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

157 157

Complex Event Detection Consolidation, Composition,

Aggregation

Complex Event Detection is the process by which

event analysis results in the creation of new event information,

or the update of existing complex events.

For example, complex event detection may result

from identifying a pattern of event occurences

that indicate some complex event has taken or is taking place.

EP Patterns:RA:Detection

Page 158: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

158

RA:Detection:Discussion

• Are consolidation, composition and aggregation

sufficiently defined or different?

• Original EPTS RA definitions

– Consolidation: combining disparate events together into a

"main" or "primary" event.

– Composition: composing new, complex events from existing,

possibly source, events

– Aggregation: combining events to provide new or useful

information, such as trend information and event statistics.

158

Complex Event Detection Consolidation, Composition,

Aggregation

Page 159: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

159

RA:Detection:Discussion(2)

• EPTS Glossary (v2) Terms

– Derived event (synthesized event, synthetic event)

• An event that is generated as a result of applying a method or

process to one or more other events.

– Composite event

• A (type of) derived event that is created by combining a set of

other simple or complex events (known as its members) using a

specific set of event constructors such as disjunction, conjunction,

and sequence. A composite event always includes the member

(base)events from which it is derived.

• A derived event is not a composite if its method of derivation lies

outside a specified set of allowed constructors.

159

Complex Event Detection Consolidation, Composition,

Aggregation

Page 160: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

160

RA:Detection:Discussion(3)

• Dictionary definitions

– Consolidation: bring together (separate parts) into a single

or unified whole

– Composition: combining parts or elements to form a whole

– Aggregation: a group or mass of distinct or varied things

• From these we can infer:

composition is a generic term,

consolidation joins events of a similar or related type,

and

aggregation joins events of differing types

160

Complex Event Detection Consolidation, Composition,

Aggregation

Page 161: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

161

RA:Detection:Discussion(4)

• Other interesting pattern classifications?

– Composition from an ordered or partially ordered sequence of

events

– Composition from missing events or non-events

– Composition from out of order events in some sequence of

events

• Other types of complex event detection?

– Situations or States, indicating some prior sequence of

events, significant to be classified or specified

161

Complex Event Detection Consolidation, Composition,

Aggregation

Page 162: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

162

RA:Detection:Discussion(5)

• Summary

– Original EPTS RA classification can be safely revised /

extended …

– Here we look at

Consolidation and Aggregation (i.e. composition of similar

and different event types)

+ Composition of events including a non-event

162

Complex Event Detection Consolidation, Composition,

Aggregation

Page 163: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

163

RA:Detection:Discussion(6)

• Possible new EP RA definitions:

– Consolidation: reinforcing a primary event with evidence

from new events of the same type

– Composition: composing a new event from existing events

of similar types + all contributing events are included

– Aggregation: combining event data from disparate event

types to create a new event

163

Complex Event Detection Consolidation, Composition,

Aggregation

Page 164: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

164

RA:Detection:Consolidation

Consolidation: combine events of similar type together into an new event

or combine disparate events together into a main or primary event

– E.g. multiple events occur to indicate a complex event; the complex

event is given a new consolidate identity as derived primary event or

main event

– E.g. several messages occur that together form a complex event

indicating a business transaction

– E.g. multiple events of the same type occur within a time period to

indicate an event called “a high rate of events”

– E.g. abstraction of events into a situation which is initiated or terminated

by the (derived complex) event as effect of the detection and

consolidation of the complex event, e.g. the complex event “plane take-

off” has the effect initiating the situation “flying”

Complex Event Detection Consolidation, Composition,

Aggregation

Page 165: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

165

Detection:Consolidation:Classification • Alternative Names:

– Event Reinforcement from Subevents, Event / Situation Reasoning

• EPTS Reference Architecture:

– "During complex event detection, combining events together into a

"main" or "primary" event. Similar to event aggregation and

composition", but with a new explicit derived event from the

aggregated/composed events (the consolidated derived event might

not contain all events from which it was derived);

– a special abstraction of the effect of a (derived) event is a situation

which is initiated or terminated by events

• EPIA:

– Aggregate (EPA): a transformation EPA that takes as input a

collection of events and creates a single derived event by applying a

function over the input events. 165

Complex Event Detection Consolidation, Composition,

Aggregation

Page 166: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

166 event

event

Detection:Consolidation:Description

• Role / Explanation

– Consolidation is used to describe several events supporting

the creation of a single composite event, generally where

component events together provide evidence.

– For example: a “conflict” event (effect = situation “in

conflict”) is considered to take place when both a “conflict

declaration” occurs and an “act of violence” occurs.

• Associated Business Rule Specification

– a consolidation (to be enforced during event processing):

The <collection of events> supporting <event> that

are <related by

some relationship

constraint>. 166

event Consolidation

f(events) event

Complex Event Detection Consolidation, Composition,

Aggregation

Page 167: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

167

Detection:Consolidation:Structure

• EPTS Glossary comparison

– The term consolidated event is sometimes used for some

forms of composite or derived event.

– Composite event: a derived, complex event

• is created by combining base events using a specific set of

event constructors such as disjunction, conjunction,

sequence, etc.

• always includes the base (member) events from which it is

derived.

– Derived event (/synthesized event): an event that is

generated as a result of applying a method or process to

one or more other events.

167

Complex Event Detection Consolidation, Composition,

Aggregation

Page 168: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

168

Detection:Consolidation:Implementations

• TIBCO, Prova, Oracle, IBM implementations for this

pattern

168

Complex Event Detection Consolidation, Composition,

Aggregation

event event

event Consolidation

f(events)

derived

event

Page 169: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

169

Detection:Consolidation:Implementations: TIBCO BusinessEvents: Rules

• General pattern: <declare> event,

consolidated-event-object

<if>

event-supports-event-object

<then>

update-consolidated-object

with event-data

• Effect: construct evidence for

(main event) object for n incoming events 169

event

Associate

event for

consolidation

Updated

Main Event

Object

Consolidate

/ reinforce

event

Main event

object

Complex Event Detection Consolidation, Composition,

Aggregation

Page 170: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

170

Detection:Consolidation:Implementations: Prova: Consolidation

• General pattern:

<rcvMult> Event :-

DerivedEvent = <consolidate> (

<compose/aggregate> Events ),

<initiate / terminate> (

DerivedEvent, Situation).

• Effect: a detected complex event (see

composition, aggregation) is given a new

derived event identity. The effect of the

event occurrence / detection might initiate of

terminate a situation. 170

Event

Composition/Aggregation

f(Events)

derived

event

Event

Complex Event Detection Consolidation, Composition,

Aggregation

Consolidate

f(Events)

Situation Situation

initiate terminate

Page 171: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

171 171

– Interval-based Event Calculus semantics (model-theory + proof theory) based on

time intervals modeled as fluents

I: Tinterval x Fl {true, false}

– Example: D = A;(B;C) (consolidation: derive event D from sequence composition)

– Example: derived situation from complex event detection (consolidation: initiate situation1 by event D)

initiates(D,situation1,T). holdsAt(situation1, t5)? => yes

T1 T2 T3 T4

A B C

[B,C] [A,A]

[[A,A],[B,C]]

Complex Event Detection Consolidation, Composition,

Aggregation

Detection:Consolidation:Implementations: Prova: Consolidation Example

D = [A,C]

Composition

Consolidation

Page 172: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

172

Dection:Consolidation:Implementations: Oracle EP: pattern matching

• General pattern:

SELECT <property>

FROM <SOURCE>

MATCH_RECOGNIZE (

MEASURES <property>

PATTERN (<regular-expression>)

DEFINE <expression>

)

• Effect: consolidates events from source

into measures. 172

Complex Event Detection Consolidation, Composition,

Aggregation

Event

Pattern match

derived

event

Event

Page 173: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

173

Dection:Consolidation:Implementations: Oracle EP: pattern matching

• Example:

SELECT M.goingUpPrice

FROM marketFeed

MATCH_RECOGNIZE (

MEASURES B.price as goingUpPrice

PATTERN (A B)

DEFINEA as price < 29.0,B as price > 30.0

) as M

• Description: consolidates stock ticks

into a single ‘going up’ alert event. 173

Complex Event Detection Consolidation, Composition,

Aggregation

Event

Pattern match

derived

event

Event

Page 174: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

174 174

Detection:Consolidation:Implementations: IBM WODM Decision Server Events

Complex Event Detection Consolidation, Composition,

Aggregation

• Example: LargeTransaction events generated for any

type of large banking transaction (deposit, withdrawal,

transfer)

• RecentSignificantAccountActivity is consolidation of

several recent LargeTransaction events, each such

event further reinforces the consolidated event

• Effect of the example: Detect customers with

recent significant account activity

Consolidate

Consolidated

Event

event event

Large Check

Deposit Large Cash

withdrawal

Significant

Account

Activity

Note: this screenshot is from an older version of the product

Page 175: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

175 175

type Trade = tuple<rstring sym, int32 price, int32 vol>;

type Quote = tuple<rstring sym, int32 offer>;

type TradeOrQuote =

tuple<rstring sym, int32 price, int32 vol, enum{TRADE, QUOTE} kind>;

/*...*/

stream<Trade> Trades = /*...*/

stream<Quote> Quotes = /*...*/

stream<TradeOrQuote> TQs = Custom(Trades; Quotes) {

logic

onTuple Trades:

submit({sym=sym, price=price, vol=vol, kind=TRADE}, TQs);

onTuple Quotes:

submit({sym=sym, price=offer, vol=-1, kind=QUOTE}, TQs);

}

Detection:Consolidation:Implementations: IBM InfoSphere Streams (SPL)

Complex Event Detection Consolidation, Composition,

Aggregation

stream Custom Stream

Trades

TQs stream Quotes

Page 176: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

176

RA:Detection:Composition

Composition: composing new complex events from existing,

possibly source, events

– E.g. deduce a complex event from a history of past events

Interesting compositions

1. By type (see consolidation and aggregation)

2. By sequence, order, and missing events

3. By combination of operators

(e.g. sequence of increasing values until stops increasing) 176

Complex Event Detection Consolidation, Composition,

Aggregation

Page 177: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

177

Detection:Composition:Classification

• EPTS Reference Architecture:

• "Event composition is about composing new, complex events

from existing, possibly source, events.”

• For example, event composition may take information from a

range of past events to determine that a new, complex, event has

occurred and needs to be composed.

• EPIA:

– Aggregate (EPA): a transformation EPA that takes as input a

collection of events and creates a single derived event by

applying a function over the input events.

177

Complex Event Detection Consolidation, Composition,

Aggregation

Page 178: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

178 event

event

Detection:Composition:Description

• Role / Explanation

– Composition is used to describe combining multiple events

into a composite event, generally where the component

events justify the existence of the complex event.

– For example: a telephone call event is considered to take

place when a call end event occurs after a call start event.

• Associated Business Rule Specification

– a composition (to be enforced during event processing):

The <collection> of <event entities> that

are <related by

some relationship

constraint>.

178

event Composition

f(events) event

Complex Event Detection Consolidation, Composition,

Aggregation

Page 179: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

179

Detection:Composition:Structure

• EPTS Glossary comparison:

– Composite event: A derived event that is created by combining a

set of other simple or complex events (known as its members)

using a specific set of event constructors such as disjunction,

conjunction, and sequence.

– Event pattern: A template containing event templates, relational

operators and variables. An event pattern can match sets of

related events by replacing variables with values.

– Event pattern detection: Finding instances of an event pattern.

– Event template: An event form or descriptor some of whose

parameters are variables. An event template matches single

events by replacing the variables with values. 179

Complex Event Detection Consolidation, Composition,

Aggregation

Page 180: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

180

Detection:Composition:Implementations: Missing Event in a Series

• TIBCO, Prova, Oracle, IBM implementations for this

pattern

180

event event

event

Composition

f(events)

over some

pattern

sequence

Missing event

indicator

Complex Event Detection Consolidation, Composition,

Aggregation

Page 181: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

181

Detection:Composition:Implementations: Missing Event in a Series :TIBCO BusinessEvents: Rules

• General pattern:

• Rules to

1. indicate matching process

is continuing to succeed

2. Indicate end of sequence and therefore an

event is missing

• Effect: construct evidence for the

missing event object based on

n incoming events / target pattern

181

event

Compare

event to

Target Pattern

Missing event

Match end of

sequence

for missing

event

Target pattern

Complex Event Detection Consolidation, Composition,

Aggregation

Page 182: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

182

Detection:Composition:Implementations: Missing Event in a Series :TIBCO BusinessEvents: Rules(2)

• Rule 1

<declare> event, target-event-pattern

<if>

not event-matches-target-event-pattern

<then>

reset-target

• Rule 2

<declare> event, target-event-pattern

<if>

event-matches-target-event-pattern-end

<then>

reset-target, create missing-event-object

182

Complex Event Detection Consolidation, Composition,

Aggregation

Page 183: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

183

Detection:Composition:Implementations: Prova Complex Event Processing

• General pattern: <declare> event,

event-object

<if>

event-relates-to-event-object

<then>

define-composed-object-event

with event-data

• Effect: construct evidence for

(new event) object for n incoming events 183

event

Relate

events

Complex Event

Object

Compose

complex event

Event object

Complex Event Detection Consolidation, Composition,

Aggregation

Page 184: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

184 184

– Interval-based Event Calculus semantics (model-theory + proof theory) based on

time intervals modeled as fluents

I: Ti x Fl {true, false}

– Example: D = A;(B;C) (consolidation: derive event D from sequence composition)

– Example: derived situation from complex event detection (consolidation: initiate situation1 by event D)

initiates(D,situation1,T). holdsAt(situation1, t5). => yes

T1 T2 T3 T4

A B C

[B,C] [A,A]

[[A,A],[B,C]]

Complex Event Detection Consolidation, Composition,

Aggregation

Detection:Consolidation:Implementations: Prova: Composition Example

D = [A,C]

Composition

Consolidation

Page 185: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

185

Detection:Composition:Implementations: Prova: Event Composition as Event-Driven Workflow

rcvMsg(XID,Process,From,event,["A"]) :-

fork_b_c(XID, Process).

fork_b_c(XID, Process) :-

@group(p1) rcvMsg(XID,Process,From,event,["B"]), … .

fork_b_c(XID, Process) :-

@group(p1) rcvMsg(XID,Process,From,event,["C"]), … .

fork_b_c(XID, Process) :-

% OR reaction group "p1" waits for either of the two

event message handlers "B" or "C" and terminates the

alternative reaction if one arrives

@or(p1) rcvMsg(XID,Process,From,or,_).

Complex Event Detection Consolidation, Composition,

Aggregation

A

B

C

185

Page 186: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

186

Detection:Composition:Implementations: Oracle EP: missing event

• General pattern:

SELECT <property>

FROM <SOURCE>

MATCH_RECOGNIZE (

MEASURES <property>

PATTERN (<regular-expression>)

DURATION <time-interval>

DEFINE <expression>

)

• Effect: detects if pattern has not

matched after duration expires. 186

Complex Event Detection Consolidation, Composition,

Aggregation

Event

Pattern match

derived

event

Event

Page 187: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

187

Detection:Composition:Implementations: Oracle EP: missing event

• Example:

SELECT M.maxUpPrice FROM marketFeed

MATCH_RECOGNIZE (

PARTITION BY symbol

MEASURESMAX(B.price) as maxUpPrice

INCLUDE TIMER EVENTS

PATTERN (A B+? C) DURATION 10 MINUTES

DEFINEB as B.price > A.price,

C as C.price < MAX(B.price)

) as M

• Description: send max up price even if trend is

not going down after 10 minutes. In other words,

even if going “down event” is missing. 187

Complex Event Detection Consolidation, Composition,

Aggregation

Event

Pattern match

derived

event

Event

Page 188: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

188 188

Detection:Composition:Implementations: IBM WODM Decision Server Events

Complex Event Detection Consolidation, Composition,

Aggregation

• General Pattern: Share data across multiple events

by appending the data to be composed into a business

object array... then when processing the events, use

relevant composition

• Effect of the example: sum up the value of a

customer’s purchases over a week

Add to array

business

Object Array

Compose

Operation

event event

Page 189: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

189 189

/* research prototype, see DEBS 2012 paper */

stream<MatchT> Matches = MatchRegex(Quotes) {

param

pattern : ". rise+ drop+ rise+ drop* deep";

partitionBy : symbol;

predicates : {

rise = price>First(price) && price>=Last(price),

drop = price>=First(price) && price<Last(price),

deep = price<First(price) && price<Last(price) };

output

Matches : symbol=symbol, seqNum=First(seqNum),

count=Count(), maxPrice=Max(price);

}

Detection:Composition:Implementations: IBM InfoSphere Streams (SPL)

Complex Event Detection Consolidation, Composition,

Aggregation

• Example:

• Effect: Compose quotes based on price behaviour

Page 190: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

190

Detection:Aggregation

Aggregation: combine events of dissimilar type together into

an event

– E.g. combine credit card purchase event with competitor

store nearby not-busy event, and generate competitor offer

event for a complementary product

190

Complex Event Detection Consolidation, Composition,

Aggregation

Page 191: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

191

Detection:Aggregation:Classification

• Alternative Names:

– Event Summarization

• EPTS Reference Architecture:

– "During complex event detection, combining events to

provide new or useful information, such as trend

information and event statistics. Similar to event

consolidation "

• EPIA:

– Aggregate (EPA): a transformation EPA that takes as

input a collection of events and creates a single derived

event by applying a function over the input events. 191

Complex Event Detection Consolidation, Composition,

Aggregation

Page 192: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

192

event event

Detection:Aggregation:Description

• Role / Explanation

– Aggregation is where multiple events support the creation of a

composite event, either as components or evidence, and are of

different type.

– For example: a “start call” event aggregated with an “end call” event

indicates a “telephone call event”

– For example: generate a (composite) event that contains the medium

price of a stock over a 10 minute stream of events.

• Associated Business Rule Specification

– a summarisation (to be enforced during event processing):

The <aggregation fn> of <event entities> that

are <selection constraint>

must have

<some constraint>. 192

event Aggregation

f(events) event

Complex Event Detection Consolidation, Composition,

Aggregation

Page 193: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

193

Detection:Aggregation:Structure

• EPTS Glossary comparison: see previous

193

Complex Event Detection Consolidation, Composition,

Aggregation

Page 194: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

194

Detection:Aggregation:Implementations

• TIBCO, Oracle, Prova and IBM implementations for

this pattern

194

event event

event Aggregation

f(events) event

Complex Event Detection Consolidation, Composition,

Aggregation

Page 195: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

195

Detection:Aggregation:Implementations: TIBCO BusinessEvents: Rules

• General pattern: <declare> event,

aggregation-object

<if>

event-member-of-aggregation

<then>

update-aggregation-object

with event-data

• Effect: construct aggregation

(event) object for each incoming event 195

event

Identify

event for

aggregation

Updated

Aggregation

Object

Aggregate

Operation

on event

Aggregation

object

Complex Event Detection Consolidation, Composition,

Aggregation

Page 196: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

196

Detection:Aggregation:Implementations: Oracle EP: queries

• General pattern template:

SELECT <summary-property>

FROM <STREAM>[WIN]

WHERE <predicate>

GROUP BY <aggregation-identity>

• Effect: summarizes the properties of

several simple event into a new

complex event

196

event

Identify

event for

aggregation

Updated

Aggregation

Object

Aggregate

Operation

on event

Aggregation

object

Complex Event Detection Consolidation, Composition,

Aggregation

Page 197: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

197

Detection:Aggregation:Implementations: Oracle EP: model

197

• Scenario:

– Output the average bid and ask price of a stock in the

last 10 seconds.

• Application Model

Complex Event Detection Consolidation, Composition,

Aggregation

Page 198: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

198

Detection:Aggregation:Implementations: Oracle EP: queries

198

SELECT symbol, AVG(bid), AVG(ask)

FROM

StockTickStream [RANGE 10 SECONDS]

GROUP BY

symbol

Complex Event Detection Consolidation, Composition,

Aggregation

Page 199: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

199

Detection:Aggregation:Implementations: Oracle EP: table metaphor

199

Time Input Output

1s

4s

9s

15s

20s

{“AAA”, 10.0, 12.0} {“AAA”, 10.0, 12.0}

{“AAA”, 12.0, 14.0} {“AAA”, 11.0, 13.0}

{“BBB”, 4.0, 5.0} {“AAA”, 11.0, 13},

{“BBB”, 4.0, 5.0}

{“BBB”, 8.0, 10.0} {“BBB”, 6.0, 7.5}

{“BBB”, 8.0, 10.0}

> 10s

Complex Event Detection Consolidation, Composition,

Aggregation

Page 200: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

200

Detection:Aggregation:Implementations: Prova: rules

• General pattern template:

<rule_head> :-

<Create Aggregator>,

@group(<reaction group>)

@timer|size(Start,Interval,Aggregator)

rcvMsg <EventPattern> [<Aggregate Operation>].

<rule_head> :-

@or(<reaction group>)

rcvMsg <Aggregator>,

... <consume Aggregator>.

• Effect: Repeated incremental

aggregations over new events

200

event

Check & Update

Timer / Size Count

Updated

Aggregation

Object

Aggregate

Operation

on event

Aggregation

object

Identify event

for aggregation

Consume

Aggregation

Object

or

Complex Event Detection Consolidation, Composition,

Aggregation

Page 201: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

201

Detection:Aggregation:Implementations: Prova: Example with Time Counter

% This reaction operates indefinitely. When the timer

elapses (after 25 ms), the groupby map Counter is sent

as part of the aggregation event and consumed in or

group, and the timer is reset back to the second

argument of @timer.

groupby_rate() :-

Counter = ws.prova.eventing.MapCounter(), % Aggr. Obj.

@group(g1) @timer(25,25,Counter) % timer every 25 ms

rcvMsg(XID,stream,From,inform,tick(S,P,T)) % event

[IM=T,Counter.incrementAt(IM)]. % aggr. operation

groupby_rate() :-

% receive the aggregation counter in the or reaction

@or(g1) rcvMsg(XID,self,From,or,[Counter]),

... <consume the Counter aggreation object>.

Complex Event Detection Consolidation, Composition,

Aggregation

Page 202: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

202

Detection:Aggregation:Implementations: IBM WODM Decision Server Events

• General Pattern: Share data across multiple events

by appending the data to be aggregated into a business

object array... then when processing the events, use

relevant aggregator

• Effect of the example: sum up the value of a

customer’s purchases over a week

Add to array

business

Object Array

Operation

on aggregate

event event

Complex Event Detection Consolidation, Composition,

Aggregation

Page 203: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

203 203

type Trade = rstring sym, timestamp ts, decimal64 price, decimal64 vol;

/* define PreVwap to subsume attributes of Trade */

type PreVwap = Trade, tuple<decimal64 priceVol>;

/* ... */

/* VWAP = volume-weighted average price */

stream<PreVwap> PreVwaps = Aggregate(Trades) {

window Trades : sliding, delta(ts, 60.0), count(1), partitioned;

param partitionBy : sym;

output PreVwaps : priceVol = Sum(price*vol), vol = Sum(vol);

}

Detection:Aggregation:Implementations: IBM InfoSphere Streams (SPL)

Complex Event Detection Consolidation, Composition,

Aggregation

Aggregate stream

Trades Summed trades

• Example:

Sliding Window

Page 204: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

204 204

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Event Reaction is the process subsequent

to event analysis and complex event detection

to handle the results of analysis and detection.

For example, an event reaction could be

the invocation of some service to process the event

in some particular way.

EP Patterns:RA:Reaction

Page 205: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

205

RA:Reaction:Assessment:

Assessment: evaluate the event for inclusion in some

process, collection, classification or complex event

– E.g. assess a cash withdrawal event for signs of it being a

fraud event

205

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 206: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

206

Reaction:Assessment:Classification

• Alternative Names:

– Event Classification

• EPTS Reference Architecture:

– Event assessment is the process by which an event is

assessed for inclusion in some process, incorporation in

some other event, etc.

• EPIA: no direct analogy, possibly maps to translate

– Translate (EPA): a stateless Transformation EPA that

takes a single event as its input, and generates a single

derived event which is a function of the input event, using

a derivation formula 206

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 207: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

207

Reaction:Assessment:Description

• Role / Explanation

– Assessment is used to describe the process of evaluating an

event for the purposes of inclusion in some process, collection

or classification.

– For example: assess a parcel scan event to check whether it is

classed as “in order” or “out of order” relative to some

presumed parcel process

• Associated Business Rule Specification

– a constraint (to be enforced during event processing):

The <event> that

satisfies <assessment constraint>

achieves a

<boolean result>

207

event Assessment

= f(event+data) result

Other data or event

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 208: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

208

Reaction:Assessment:Structure

• EPTS Glossary comparison

– Constraint (also event pattern constraint): A Boolean

condition that must be satisfied by the events observed in

a system.

– Event sink (event consumer) is an entity that receives

events.

• Examples:

• ƒ Software module

• ƒ Database

• ƒ Dashboard

• ƒ Person

208

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 209: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

209

Reaction:Assessment:Implementations

• TIBCO, Prova, Oracle, IBM implementations for this

pattern

209

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

event Assessment

= f(event+data) result

Other data or event

Page 210: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

210

Reaction:Assessment:Implementations: TIBCO BusinessEvents: Rules

• General pattern: <declare> event,

assessment-data

<if>

event-matches-assessment-data

<then>

assign <assessment result>

to <event object>

and/or <other>

• Effect: match event to assessment criteria

and store result 210

event

Assess

event

against

data

event

Store/act on

result

Past event

/ data

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 211: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

211

Reaction:Assessment:Implementations: Prova: Rules

• General pattern: <rcvMsg / rcvMult> Event :-

assessment condition(s),

....

assessment rule 1 :-

conditions.

assessment rule 2 :-

filter conditions.

...

• Effect: match event to assessment criteria

and react result

211

event

Assess

event

against

data

event

react on

result

Past event

/ data

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 212: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

212

Reaction:Assesment:Implementations: Prova: Example – Detect Suspicious Logins

% detect suspicious logins by assessing the IP numbers

of the login events from the same user login

rcvMsg(XID,Protocol,From,request,login(User,IP)) :-

% if the next follow up event (@count(1)) that follows

the previous login is send from another IP (IP2!=IP)

@group(g1) @count(1)

rcvMsg(XID,Protocol,From,request,login(User,IP2))

[IP2!=IP],

println(["Suspicious login",User,IP,IP2]," ").

212

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 213: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

213

Reaction:Assessment:Implementations: Oracle EP: queries

• General pattern:

SELECT <property>)

FROM <SOURCE>,

<ASSESSMENT-RELATION>

WHERE <assessment-condition>

• Effect: join with assessment relation, and

verify assessment conditions.

213

data

join and assess

info

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

event

Page 214: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

214 214

Reaction:Assessment:Implementations: IBM WODM Decision Server Events

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

• Example: Assess customer enters branch event

against whether this is a customer of interest

(customer has carried out large banking transactions

recently)

Page 215: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

215 215

/* (see Selection:Filtering) */

Reaction:Assessment:Implementations: IBM InfoSphere Streams (SPL)

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 216: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

216

RA:Reaction:Routing:

Routing: based on the event type and data pass the event on

to the appropriate service

– E.g. customer purchase event: pass on to a provisioning

service based on the type of product / product classification

216

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 217: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

217

Reaction:Routing:Classification

• Alternative Names:

– Event Gateway

• EPTS Reference Architecture:

– "During event reaction, event routing is the process by

which an event is redirected to some process,

computation element, or other event sink. "

• EPIA: no direct analogy, possibly maps to split /

compose / project

– Project (EPA): a translate EPA that takes an input event

and creates a single derived event containing a subset

of the attributes of the input event. 217

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 218: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

218

Reaction:Routing:Description

• Role / Explanation

– Routing is used to describe the process of adding one or

more new destinations to an event.

– For example: route an input event to the appropriate

specialist agent for that event type.

• Associated Business Rule Specification

– a constraint (to be enforced during event processing):

The <event> that

satisfies <selection constraint>

must be assigned to

<destination>.

218

event Routing

dest = f(event+data) event

Other data

or event

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 219: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

219

Reaction:Routing:Structure

• EPTS Glossary comparison

– Routing (a process on events) is not defined, but is

related to associating an event to an event sink.

– Event sink (event consumer) is an entity that receives

events.

• Examples:

• ƒ Software module

• ƒ Database

• ƒ Dashboard

• ƒ Person

219

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 220: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

220

Reaction:Routing:Implementations

• TIBCO, Oracle, Prova and IBM implementations for

this pattern

220

event Routing

dest = f(event+data) event

Other data

or event

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 221: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

221

Reaction:Routing:Implementations: TIBCO BusinessEvents: Rules

• General pattern: <declare> event,

routing-data

<if>

event-matches-routing-data

<then>

create-new-routed-event and

send-new-routed-event

• Effect: match events to routing rule

and construct / send new event 221

event

Identify

event for

routing

event

Create routed

event

Routing

data

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 222: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

222

Reaction:Routing:Implementations: Oracle EP: query pattern

• General pattern template:

Route 1: <filtering pattern> query

Route 2: <filtering pattern> query

Route n: <filtering pattern> query

• Effect: Different predicates

associated to different queries

evaluate and select appropriate

destinations

222

event

Identify

event for

routing

event

Select destination

Routing

data

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 223: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

223

Reaction:Routing:Implementations: Oracle EP: model

223

• Scenario:

– Output stocks whose symbols start with ‘a-m’ to

destination 1, and whose symbols start with ‘n-z’ to

destination 2.

• Application Model

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 224: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

224

Reaction:Routing:Implementations: Oracle EP: queries

224

Destination-1:

SELECT *

FROM StockTickStream

WHERE symbol.matches(“^[a-m]”)

Destination-2:

SELECT *

FROM StockTickStream

WHERE symbol.matches(“^[n-z]”)

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 225: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

225

Reaction:Routing:Implementations: Prova: rules

• General pattern template:

rcvMsg <Event Msg Pattern> :-

<routing decisions>,

sendMsg <route event >.

<routing decision rule 1> :-

<decision conditions>.

<routing decision rule 2> :-

<decision conditions>.

...

• Effect: Events are routed

according to the routing decision

rules 225

event

Identify

event for

routing

event

Create routed

event

Routing

data

Routing

Decisions

Route

event

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 226: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

226

Reaction:Routing:Implementations: Prova: Example with Agent (Sub-) Conversations

rcvMsg(XID,esb,From,query-ref,buy(Product) :-

routeTo(Agent,Product), % derive processing agent

% send order to Agent in new subconversation SID2

sendMsg(SID2,esb,Agent,query-ref,order(From, Product)),

% receive confirmation from Agent for Product order

rcvMsg(SID2,esb,Agent,inform-ref,oder(From, Product)).

% route to event processing agent 1 if Product is luxury

routeTo(epa1,Product) :- luxury(Product).

% route to epa 2 if Product is regular

routeTo(epa2,Product) :- regular(Product).

% a Product is luxury if the Product has a value over …

luxury(Product) :- price(Product,Value), Value >= 10000.

% a Product is regular if the Product ha a value below …

regular(Product) :- price(Product,Value), Value < 10000.

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 227: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

227

Reaction:Routing:Implementations: Prova: Event Routing in Event-Driven Workflow

rcvMsg(XID,Process,From,event,["A"]) :-

fork_b_c(XID, Process).

fork_b_c(XID, Process) :-

@group(p1) rcvMsg(XID,Process,From,event,["B"]),

execute(Task1), sendMsg(XID,self,0,event,["D"]).

fork_b_c(XID, Process) :-

@group(p1) rcvMsg(XID,Process,From,event,["C"]),

execute(Task2), sendMsg(XID,self,0,event,["E"]).

fork_b_c(XID, Process) :-

% OR reaction group "p1" waits for either of the two

event message handlers "B" or "C" and terminates the

alternative reaction if one arrives

@or(p1) rcvMsg(XID,Process,From,or,_).

227

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

A

B

C

Task1

Task2

D

E

Page 228: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

228

Reaction:Routing:Implementations: IBM WODM Decision Server Events

• Example: Define an event rule

which will send the request to supplier A

when the purchase indicates supply

from Supplier A

• Effect of example:

Events are routed as required

228

event

event

Routing

information

Consume

event

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Identify

event

routing

Page 229: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

229

Reaction:Routing:Implementations: IBM CICS Transaction Server for z/OS

• Example: Define a EP adapter for a CICS

Event Binding (i.e. group of related events)

which will route (and format) the event for

delivery to a consumer

• Effect of example: Events routed as required 229

event

Identify

event

routing

event

Routing

information

Consume

event

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 230: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

230 230

type Bracket = enum { YOUNG, OLD, OTHER }; /*...*/ stream<rstring name, int32 age> Folks = /*...*/ stream<rstring name, Bracket bracket> Rated = /*...*/ (stream<Rated> Young; stream<Rated> Old) = Custom(Rated) { logic onTuple Rated: { if (bracket == YOUNG) submit(Rated, Young); else if (bracket == OLD) submit(Rated, Old); } }

Reaction:Routing:Implementations: IBM InfoSphere Streams (SPL)

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

• Example:

• Effect: Route based on age rating

Page 231: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

231

RA:Reaction:Prediction:

Prediction: an output of some event is the prediction that a

new event will occur (or will have occurred)

– E.g. based on some sequence of sensor events, an

earthquake event is predicted in location L and within time

period P

231

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 232: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

232

Reaction:Prediction:Classification

• Alternative Names:

– Event Anticipation; subtype of Event Assessment

• EPTS Reference Architecture:

– Event Prediction is where the reaction to some event

processing is that some new event is predicted to occur.

• Example, a complex event detection may predict some future

event that should then be assessed and processed further.

• EPIA: no direct analogy, possibly maps to split /

compose / project

– Project (EPA): a translate EPA that takes an input event

and creates a single derived event containing a subset

of the attributes of the input event. 232

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 233: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

233

Reaction:Prediction:Description

• Role / Explanation

– Prediction is used to describe the process of evaluating the

state of a set of events to predict some future event.

– For example: a new purchase event of an airline ticket may

cause a prediction that a business trip event will occur

• Associated Business Rule Specification

– a constraint (to be enforced during event processing):

The <events> that satisfies <event relationships>

causes a prediction of

<anticipated event>

233

event Prediction

= f(event+data)

predicted

event

Other data or events

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 234: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

234

Reaction:Prediction:Structure

• EPTS Glossary comparison

– Virtual event: An event that does not happen in the

physical world but is imagined, modeled or simulated.

• Example: Events predicted by a weather simulation

• Note: A virtual event can refer to either an event object or a

thing that happens.

234

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 235: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

235

Reaction:Prediction:Implementations

• TIBCO, IBM, Oracle implementations for this pattern (Prova – Prova Abductive and Inductive Logic Programming or

Java API Call to external functionality)

235

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

event Prediction

= f(event+data)

predicted

event

Other data or events

Page 236: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

236

Reaction:Prediction:Implementations: TIBCO BusinessEvents: Rules

• General pattern: <declare> event,

prediction-data

<if>

event-matches-prediction-data

<then>

create <predicted event object>

• Effect: match event to prediction criteria

and create predicted event or object

236

event

Assess

event

against

data

event

Compute

prediction

Past event

/ data

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 237: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

237

Reaction:Prediction:Implementations: TIBCO BusinessEvents: State Model

• General pattern: <on> event

<condition>

event-matches-prediction-data

<then change state to>

predicted-event-state

• Effect: match event to prediction criteria

and change to predicted state

• Note that there may be a set of prior states that form part

of the state model for predicting events

237

event

Assess

event

against

data

Past event

/ data

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Prior states

Predicted

state

Page 238: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

238

Reaction:Prediction:Implementations: Oracle EP: queries

• Effect: integration with ODM (Oracle

Data Mining) to import model and do

predictive score in an online fashion.

238

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 239: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

239 239

Reaction:Prediction:Implementations: IBM WODM Decision Server Events

239

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

• General Pattern: Prediction in WODM is provided

by the Decision Server Rules component – the Rule

Designer API provides classes and interfaces for a

BRL prediction engine

Page 240: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

240 240

use com.ibm.streams.mining.scoring::*; type Person = tuple<rstring clientId, int32 age, rstring gender>; type Predict = Person, tuple<float64 predictedVal, float64 predictedStdDev>; /*...*/ stream<Person> People = /*...*/ stream<Predict> Predictions = Regression(People) { param //PMML (predictive model markup language) generated e.g. by SPSS model : "../models/linreg.pmml"; clientId : "CLIENT_ID"; age : "AGE"; gender : "GENDER"; }

Reaction:Prediction:Implementations: IBM InfoSphere Streams (SPL)

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

• Example:

• Effect: Regression operator calculates predicted values and standard

deviation for each tuple in the input stream of People, and assigns to the

output stream (giving predictions about age, gender)

Page 241: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

241

RA:Reaction:Discovery:

Discovery: in event processing some new type, schema, state

or classification of events is discovered

– An event pattern instance may be detected, where as an

event pattern class is discovered

– E.g. a customer query event may indicate a combination of

prior transaction events are a new discovered potential fraud

event

241

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 242: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

242

Reaction:Discovery:Classification

• Alternative Names:

– Event Mining (c.f. Data Mining)

– subtype of Event Assessment,

parent type of Event Prediction

• EPTS Reference Architecture:

– Event Discovery is where the reaction to some event

processing is the disclosure of a new, typically complex,

event type.

• For example, a complex event detection may be a new event

discovery, or it may simply be detection of an existing event type.

– Note: event prediction is predicting some future event, usually

of a known type, whereas event discovery is the uncovering

of a new event type. See also event-based learning. 242

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 243: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

243

Reaction:Discovery:Classification(ctd)

• EPTS Reference Architecture:Note

– Event prediction is predicting some future event, usually

of a known type, whereas event discovery is the

uncovering of a new event type. See also event-based

learning.

• EPIA: no direct analogy, possibly maps to split /

compose / project

– Project (EPA): a translate EPA that takes an input event

and creates a single derived event containing a subset

of the attributes of the input event.

243

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 244: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

244

Reaction:Discovery:Description

• Role / Explanation

– Discovery is used to describe the process of evaluating the

state of a set of events to determine some useful information.

– For example: a history of shop purchases may be used to

classify the type of shopper

• Associated Business Rule Specification

– a constraint (to be enforced during event processing):

The <events> that satisfies <event relationships>

causes a discovery of

<related fact>

244

event Discovery

= f(event+data)

discovered

data

Other data or events

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 245: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

245

Reaction:Discovery:Structure

• EPTS Glossary comparison

– Event pattern discovery: Finding new event patterns

245

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 246: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

246

Reaction:Discovery:Implementations

• TIBCO, Oracle, IBM implementations for this pattern (Prova – Prova Abductive and Inductive Logic Programming or

Java API Call to external functionality)

246

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

event Discovery

= f(event+data)

discovered

data

Other data or events

Page 247: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

247

Reaction:Discovery:Implementations: TIBCO BusinessEvents: Rules

• General pattern: <declare> event,

past-data

<if>

event-matches-discovery-data

<then>

update <event object>

• Effect: match event to discovery criteria

and add new data to event object

247

event

Assess

event

against

data

event

Compute

discovered

data

Past event

/ data

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 248: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

248

Reaction:Discovery:Implementations: Oracle EP: queries

• General pattern:

SELECT <property>

FROM <SOURCE>,

<DISCOVERY-RELATION>

WHERE <discovery-condition>

• Effect: join with discovery relation, and

verify discovery conditions.

248

data

discover

info

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

event

Page 249: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

249 249

/* SPL allows you to write your own operator */

Reaction:Discovery:Implementations: IBM InfoSphere Streams (SPL)

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 250: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

250

RA:Reaction:Learning:

Learning: uses new event information to refine the knowledge

of event patterns, for example using statistics

– E.g. a neural network for fraud detection uses a learning

mode using historic events versus results in order to train a

new neural net for predicting possible fraud events

250

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 251: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

251

Reaction:Learning:Classification

• Alternative Names:

– Event Analytics (c.f. Predictive Analytics)

– Event System Training

– Subtype of Event Assessment

• EPTS Reference Architecture:

– Event learning, or event-based learning, is the reaction to

some event processing that uses new event information

to add to some, typically statistical-based, understanding

of events.

– For example, the use of new event information to provide

closed-loop feedback to adjust event detection or

prediction thresholds. 251

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 252: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

252

Reaction:Learning:Classification(ctd)

• EPTS Reference Architecture:Notes

– Event-based learning is a specialisation of general

machine learning and predictive analytics.

– c.f. Event-driven learning is where some (complex) event

determines that some conventional data-driven analytics

requires re-training or re-modelling

• EPIA: no direct analogy, possibly maps to split /

compose / project

– Project (EPA): a translate EPA that takes an input event

and creates a single derived event containing a subset

of the attributes of the input event.

252

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 253: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

253

Reaction:Learning:Description

• Role / Explanation

– Learning is used to describe the process of evaluating the state

of a set of events to identify or refine event processes.

– For example: evaluating history of shop purchases may

determine the classification rule criteria for the type of shopper

• Associated Business Rule Specification

– a constraint (to be enforced during event processing):

The <events> that satisfies <event relationships>

causes the learning of

<new event relationship>

253

event Learning

= f(event+data)

New

f(event)

Other data or events

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 254: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

254

Reaction:Learning:Structure

• EPTS Glossary comparison

– See Discovery?

254

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 255: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

255

Reaction:Learning:Implementations

• TIBCO, IBM implementations for this pattern (Prova – Prova Abductive and Inductive Logic Programming or

Java API Call to external functionality)

255

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

event Learning

= f(event+data)

New

f(event)

Other data or events

Page 256: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

256

Reaction:Learning:Implementations: TIBCO BusinessEvents: Rules

• General pattern: <declare> event,

past-data

<if>

event-matches-learning-data

<then>

update <event pattern process>

• Effect: match event to learning criteria

and update some rule condition criteria

256

event

Assess

event

against

data

Manage new

process

Compute

learnt

process

Past event

/ data

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 257: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

257 257

/* typically done by using Hadoop from SPL via HDFS adaptors */

Example of HDFS: namespace com.ibm.streams.bigdata.hdfs;

composite SourceExample {

graph

stream<rstring filename> filenames = HDFSDirectoryScan() {

param directory : "test";

}

stream<rstring writer, int32 number> tupleStream

= HDFSFileSource(filenames) {

param

format: txt:

}

// Test reading and the lines of a file

stream<rstring lines> lines = HDFSFileSource(){

param

file : "test/tuple_counter.txt";

format : line ;

}

}

Reaction:Learning:Implementations: IBM InfoSphere Streams (SPL)

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning

Page 258: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

258 258

Agenda

• Non-RA EP Patterns

– Other Patterns

– Use Cases using Patterns

Page 259: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

259

Other:Application-Time

• Problem:

– Need to use application’s view of time, instead of CPU wall-clock.

– This is particularly useful when events originate from different machines and need some way of synchronizing.

• Scenario:

– Again, consider bid and ask stream…

– However, bid and ask requests are time-stamped on trader at the time that the order is placed.

259

Page 260: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

260

Other:Application-Time

• Scenario:

– Seller places two ask requests respectively at time

8:00:00 and 8:00:12

– Buyer places one bid request at time 8:00:11

260

Page 261: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

261

Other:Application-Time

• Scenario:

– Remember that we want to correlate using a 10 seconds

window, as anything older is stale…

– Hence first event from seller should not be considered,

and we should correlate the ask price of 11.0 with the bid

price of 9.5

261

Page 262: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

262

Other:Application-Time

• Scenario:

– However, the reality is that the first two events could

arrive together in a burst in the exchange, and the third

could be delayed in the cloud…

262

Page 263: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

263

Other:Application-Time

• Scenario:

– In this case, bid price of 9.5 would correlate to ask price of 10.0

and the exchange would loose money as the spread is lower…

263

Page 264: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

264

Other:Application-Time:Implementations: Oracle EP

• Solution:

– The query does not change…

– However STREAMS must be configured to use

application time-stamps based upon some event

property, instead of having events being system-

timestamped…

264

Page 265: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

265

Other:Application-Time:Implementations: IBM Business Monitor

• Example: Define a correlation expression

• Effect of example: The duration of a customer order can be obtained from

the timestamps on start and end events

265

‘Order Complete’

event

Correlate

Calculate

Duration

‘Order Placed’

event

Page 266: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

266

Other:Application-Time:Implementations: IBM WODM Decision Server Events

• Example: Events in WODM carry both the time the event

occurred (a timestamp in the event payload) and the time

when the event was received into Decision Server Events

• The example shows event XML received by Decision Server

Events (in one of the supported XML formats) with the

application-time time stamp

<?xml version="1.0" standalone="yes"?>

<connector name="OrderingPatterns" version="2.2">

<connector-bundle name="About_to_place_order" type="Event">

. . .

</connector-bundle>

<system>GBIBMIYACICSAOR1</system>

<timestamp>2012-04-19T15:41:12+00:00</timestamp>

</connector>

266

Page 267: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

267

Other:Application-Time:Implementations: Prova: rules

• General pattern template:

rcvMsg <Event Msg Pattern(s)> :-

<add application timestamp to event>,

sendMsg <Adjusted Event>.

• Effect: Any timestamp can be adjusted

(for time computations)

267

Event

adjust event

Updated

Event

Page 268: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

268

Other:Application-Time:Implementations: TIBCO BusinessEvents: PreProcessor Pattern

• General Pattern: adjust Event metadata (e.g.

Timestamp) in preprocessor for that event

– events are generally immutable!

– PreProcessor applies regardless of engine used!

• Effect: Any custom timestamp can be adjusted (for time

computations)

268

Event

Preprocess:

adjust event

Updated

Event

Note: If the problem is “out of order” events

(rather than synchronizing clocks) a

different pattern may be better!

Page 269: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

269 269

Agenda

• EP Patterns in Use Cases

Page 270: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

270

ATM/Branch

IBM Business

Monitor

SMS System

Transactions

Decision Server Events

CICS TS

Actions

Internet

Banking

Customer

Profile

WebSphere MQ

Customer Deposit and

Withdrawal

Transactions from CICS

1

Process Customer

Deposit and

Withdrawal Events

and identify event

patterns

2

Determine

appropriate

customer

promotion

4

Customer/RM SMS

5

Collect

Customer’s

profile

3 Decision

Server Rules

Measure success of

promotions 6

WebSphere

Operational

Decision

Management

IBM CICS and WebSphere Operational Decision Management Use Case

Page 271: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

271

Oracle Use-case: Capital Markets

• Low-Latency Trading

– Algorithm Trading

• Trade Matching

– High throughput

– In-flight fraud

• Position Capture and Aggregation

• Risk and Analytics

– Pre/post trade analytics

– On-demand risk management

271

Page 272: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

272

Oracle Use-case: Telemetry and Geo-fencing

• Spatial extension in CQL

– Geometry objects (point, polygon)

– Operations: within, inside, touches,

etc.

• Allows queries such as:

– Is streaming location of vehicle within

pre-defined areas?

– Is streaming customer close to shop?

272

Page 273: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

273

TIBCO Use Case: Application Service Gateway Telco high performance service policies using event processing

273

Mobile Services

difficult to monitor

end to end

Auto detect and

fixing of issues

for better

user service

Excellent service quality

needed to avoid

customer churn

Service quality affects

take-up of new services

Page 274: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

274

• Standardized channel for monitoring subscribers

• Standardized access to services (e.g. Facebook, Google, etc)

• Consolidation of existing multiple gateways

• Allows analysis of service usage, subscriber behavior, service performance, marketing campaign success, etc

• Creates events for revenue generation and reconciliation

• Handles Common Telco Notifications - change-of-device, contract, service- allowing UpSell/Cross sell etc

Service Gateway architecture exploits event processing

Page 275: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

275

• Standardized channel for monitoring subscribers

• Standardized access to services (e.g. Facebook, Google, etc)

• Consolidation of existing multiple gateways

• Allows analysis of service usage, subscriber behavior, service performance, marketing campaign success, etc

• Creates events for revenue generation and reconciliation

• Handles Common Telco Notifications - change-of-device, contract, service- allowing UpSell/Cross sell etc

Service Gateway architecture exploits event processing

Routing Pattern

Transformation

Patterns

Decision / Policy

Pattern

Security Pattern

Page 276: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

276

TIBCO Use Case: Service Performance Manager SOA metrics generation using event processing

276 TIBCO BW Domain

ESB

Java

Matrix

.NET

BW

Matrix

Instrumentation & Management Bus (EMS/JMX)

TIBCO ActiveMatrix Enterprise

Alert &

Assure SLA

Analyze

TIBCO ActiveMatrix SPM Monitoring Dashboard

SLA Dashboard

Rules

Dash

boa

rd M

essa

ge B

us

Page 277: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

277 277

Agenda

• EP Pattern Classifications

• EP Patterns @EPIA

Page 278: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

278

Categorization of Event Processing Patterns

Source:Paschke, A.: Design Patterns for Complex Event Processing, DEBS'08, Rome, Italy, 2008

http://arxiv.org/abs/0806.1100v1

Page 279: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

279

Categorization of Patterns

• Categorization according to Good and Bad Solutions – CEP Patterns

– CEP Anti-Patterns

• Categorization according to the Abstraction Level – Guidelines and Best Practices

– Management patterns

– Architecture patterns

– Design patterns

– Mapping patterns

– Idioms / Realization patterns

– Smells / Refactoring patterns

• Categorization according to the Intended Goal – Adoption patterns

– Business patterns

– Integration patterns

– Composite patterns:

– …

• Categorization according to the Management Level – Strategic patterns

– Tactical patterns

– Operational patterns Source:Paschke, A.: A Semantic Design Pattern Language for Complex Event Processing,

Intelligent Event Processing, Papers from the 2009 AAAI Spring Symposium (2009) , p. 54-60.

http://www.aaai.org/Papers/Symposia/Spring/2009/SS-09-05/SS09-05-010.pdf

Page 280: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

280

Pattern categorization from ‘Event Processing in Action’

1. Filtering

– stateless or stateful

2. Transformation

– Project

– Translate

– Enrich

– Split

– Aggregate

– Compose

3. Pattern Detection - the types of pattern that can be detected

– Basic patterns

• Logical Operator patterns

– ALL

– ANY

– ABSENCE

– Threshold patterns

– COUNT

– VALUE MAX

– VALUE MIN

– VALUE AVERAGE

– FUNCTOR

• Subset selection patterns

– RELATIVE n HIGHEST VALUES

– RELATIVE n LOWEST VALUES

– Modal patterns

• ALWAYS

• SOMETIMES

– Dimensional patterns

• Temporal order patterns

– SEQUENCE

– FIRST n

– LAST n

• Temporal Trend patterns

– INCREASING

– DECREASING

– STABLE

– NON-INCREASING

– NON-DECREASING

– MIXED

• Spatial patterns

– MAX DISTANCE

– AVERAGE DISTANCE

– RELATIVE MIN DISTANCE

– RELATIVE MAX DISTANCE

– RELATIVE AVERAGE DISTANCE

• Spatiotemporal patterns

– MOVING IN A CONSTANT DIRECTION

– MOVING IN A MIXED DIRECTION

– STATIONARY

– MOVING TOWARD

Page 281: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

Summary

• Functional Patterns v1.0 are

“complete”

• Next: further analysis and refinement

terminology

+ more examples

+ drill-down into operation level

281

Page 282: EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts event processing technical society

282

References

• EPTS Glossary

http://www.ep-ts.com/component/option,com_docman/task,doc_download/gid,66/Itemid,84/

• Event Processing in Action

– Opher Etzion and Peter Niblett, Manning Publications Co. (2011)

– http://www.manning.com/etzion/


Recommended