Complex Event Processing

Bulut Altıntaş, Sybase, Teknik Danışman

16.03.2012

SAP ANALYTICS & INNOVATION FORUM '12

© 2012 SAP AG. All rights reserved. 2

SYBASE: THE UNWIRED ENTERPRISE

INFORMATIONMANAGEMENT

Any Application

Any Server Platform

Any Data Source

INFORMATIONMOBILITY

Always Available

Any Person

Any Device

INTEGRATION COLLABORATION

DECISION-READYINFORMATION

MOBILIZEANALYZEMANAGE

Creating Value at the Point of Decision

© 2012 SAP AG. All rights reserved. 3

Complex Event Processing Overview

What is Complex Event Processing?

© 2012 SAP AG. All rights reserved. 4

Need For Complex Event Processing

© 2012 SAP AG. All rights reserved. 5

INSIGHT FROM FAST-MOVING DATACONTINUOUS INTELLIGENCE™

• Situational Awareness

• Immediate Response

• Better decision: complete, timely insight

© 2012 SAP AG. All rights reserved. 6

A Simple CEP Example

Music Detected

+Truck Approaching

=+Children running across street Ice cream truck coming

Event correlation, pattern matchingDerived result

© 2012 SAP AG. All rights reserved. 7

CEP is used in four different ways – often in combination

Situation DetectionAnalyze live event streams for patterns

which identify business critical situations

Examples:

• Fraud Detection

• Customer Behavior Patterns

indicate a problem

Automated responseContinuously adjust business

processes to reflect current conditions

Examples:

• Automated pricing based on the market

• Real-time Resource allocation

Stream Transformation

Cleanse and enrich live data streams, turning raw data into actionable info

Examples:

• Streaming alternative to ETL

• Filter and aggregate into summary information

Continuous Intelligence

Live dashboards that show actionable information for better decision making

Examples:

• Real-time P&L, Limits/Exposure Mgmt

• Online marketing and offer performance

• System monitoring

Real-time Business Intelligence System Automation

© 2012 SAP AG. All rights reserved. 8

The SYBASE Event Stream Processor (ESP)

Unlimited number of input streams

Incoming data is processed as it arrives, according to the business logic defined using high level authoring tools

Send output to apps, dashboards

Query on demand

Interactive analysis

© 2012 SAP AG. All rights reserved. 9

Key Components

ESP Server

– Manages the run-time components; starts and stops Projects

– A “project” is like an application: it contains a set of stream, window, schema and query definitions

– ESP server is designed for speed, scalability, and resilience

ESP Studio

– Eclipse® based environment used to build and test ESP Projects

Adapters

– Input adapters connect to an external data source and stream the data into an ESP stream or window

– Output adapters connect to external consumers and stream ESP output to them

Application Programming Interfaces (API)

– Available for C/C++, Java and .NET

Sybase RAP

– Complementary product that integrates seamlessly with CEP engine for data capture, providing long term history that can be used for historical analysis, tick capture, ad-hoc analysis and compliance

Panopticon EX Dashboard

– Optional real-time dashboard providing visually rich, interactive data analysis across both streaming CEP data and historical data in RAP

© 2012 SAP AG. All rights reserved. 10

Sybase ESP Architecture

10I/O

Gate

way

Security

Log Store Mem Store

Data Streams

Applications,

Third party clients,

Query Client

- Ad-hoc queries- Alerts via call-out functions

Adapte

rs

Config

Stream Processor

• Multi-threaded

• 64 bit

• High message rates

• High retention capacity

• Ultra-low latency

• Optional persistence

Security (PAM, SSL, Audits)

C&C

(XML-RPC) SQL

Connection Mgmt (ODBC, JDBC)

Authoring Tools

Security

I/O G

ate

way

Data Streams

Command Interface,

Stream Viewer,

Resource Monitor

- Platform management- Platform monitoring

Publish

Data Streams

Adapte

rs

Data Streams

Subscribe

© 2012 SAP AG. All rights reserved. 11

Sybase ESP Engine Performance

Source: STAC Research Sep 08

Only CEP vendor to publish STAC benchmarks

STAC Certified measurement of throughput and latency

Linear scaling with addition of

more cores:

− 200,000 messages/sec on 16

cores

− 300,000 messages/sec on 24

cores

Consistent/predicable low latency.

Only minor increase in latency

from 20,000 to 300,000

messages/sec:

− Mean: 1.296 - 1.568 ms

− Medium: 1.177 - 1.502 ms

− Std Dev: 0.567 - 1.104 ms

− Min: 0.091 – 0.115 ms

© 2012 SAP AG. All rights reserved. 12

ESP STUDIO: Both Visual & Texual Authoring

Visual dataflow authoring

based on Eclipse®

© 2012 SAP AG. All rights reserved. 13

Key Concepts

Input Streams

Receive events from adaptersDerived Streams

Apply continuous query

operators to one or more

input streams to produce a

new stream

Database Analogy

• Streams are like tables

• Events are rows in the table

• Fields in an event message are

columns in the table

• Continuous Queries produce

result sets/streams

Streams can Have State

• Retention windows define how many or

how long events are kept

• Opcodes in events can indicate

insert/update/delete and can be

automatically applied to the window

© 2012 SAP AG. All rights reserved. 14

CCL: CONTINUOUS COMPUTATION LANGUAGE

• Leverage familiarity and simplicity of SQL

• Instead of ‘snapshot’ queries, continuous queries

Filter (WHERE), Aggregate (GROUP BY), Join, Compute

• Extension for event stream

Windows

Patterns

SQL-BASED EVENT PROCESSING LANGUAGE

© 2012 SAP AG. All rights reserved. 15

Adapters - Rich Integration Options

© 2012 SAP AG. All rights reserved. 16

Panopticon: Real Time Data Analysis

© 2012 SAP AG. All rights reserved. 17

DATA MOVEMENT COMPONENTS

Replicate

Queues

Rep Agent

Primary

Rules Engine

Message Repository

Change Detection

Catalog/Control DB

RSSD

Stable Devices

ECDA Gateway

Sybase ASE

Oracle

MS SQL

IBM DB2/OS390

IBM DB2/UDB

RepConnector

Sybase ASE

Sybase IQ

Oracle

MS SQL

IBM DB2

Any ODBC

Sybase Control Center

Monitoring

Route

Replication Server/ECO

Replication Server

SYBASE

ESP

© 2012 SAP AG. All rights reserved. 18

CEP Industry Mapping

Capital Markets

Retail GovernmentDefense/Aerospace

Banking Energy Telecom

CEP Industry Mapping

Algorithmic Trading Smart Order Routing Fraud Detection Energy Grid Management

Real-time Profit and Loss Market Data Enrichment Retail – Online Commerce Intrusion Detection/Network Monitoring

Real-time Risk Analysis Real-time Surveillance/ Compliance

Customer Management Real-time Data Management

© 2012 SAP AG. All rights reserved. 19

Using CEP and RAP together

© 2012 SAP AG. All rights reserved. 20

Capital Markets Data Management

WHERE WE PARTICIPATE…

Sources

CEP

Sybase Products

Partner Products

TIBCO RV

IBM MQ Series

Microsoft MSMQ

Reuters RMDS

WOMBAT

Bloomberg

DATA

Panopticon

MS Excel

VISUALIZATION

RAP

• Competitive trading strategy

• Real time risk management

RogueWave

MATH TOOLS & LIBRARIES

Exchanges & Consolidators

Message Bus

Ticker Plants

IBM DB2

MicrosoftSQL Server

Oracle

Sybase ASE

Mar

ket

Dat

a C

on

nec

tivi

ty

Even

t P

roce

ssin

g

In-M

emo

ry D

atab

ase

His

tori

cal R

epo

sito

ry

Tim

e Se

ries

+ F

inan

cial

A

nal

ytic

s

Mat

hem

atic

al T

oo

ls

Vis

ual

izat

ion

& R

epo

rtin

g

Trad

ing

& R

isk

Ap

ps

© 2012 SAP AG. All rights reserved. 21

National Security

Rules(Authoring)

Reporting Tools?

Input Streams

GeographicData

HistoricalProfile

Vessel / Vehicle Location

Events

Dashboards

Alerts

© 2012 SAP AG. All rights reserved. 22

Telecom: BICS

BICS is a joint-venture between Belgacom, Swisscom and MTN

CHALLENGE

• Act proactively on possible future events that would threaten their customers’ quality of service

• Immediate visibility into service problems

• Flexible real-time aggregation of data

SOLUTION

• Sybase CEP (Complex Event Processing)

RESULTS

• Quality guarantee thanks to real-time, proactive response to emergency signals in periods of peak traffic

• Real-time voice network monitoring and analytics for high QoS

• Comprehensive network monitoring and visibility

• Rapid, flexible analysis of real-time data on demand

The gain in storage space

generated by Sybase

CEP’s data aggregation

saves us from having to

make substantial

investments in additional

servers, CPU, power, a

more extensive database

infrastructure, making

major investments that

weren’t very future-proof

would have meant going

through the same process

again a few years later.Laurent Missa,

OSS Manager at BICS

© 2012 SAP AG. All rights reserved. 23

CHALLENGE

• Gain real-time insight on customer activity and traffic trends threaten their customers’ quality of service

• improve everyday business operations and marketing campaigns

SOLUTION

• Sybase CEP (Complex Event Processing)

RESULTS

• Reduces costs by enabling more effective and targeted marketing and advertising programs

• Allows for the management of marketing and advertising programs in real-time

• Enables immediate response to new trends and internet traffic patterns

• Produces a better product for advertisers

The Sybase Aleri CEP

engine will allow us to

monitor user behavior,

traffic, and trends in real-

time, allowing us to

provide a more targeted

approach to our

marketing and advertising

programs in order to

provide a more unique

experience.

Steve Ernst, Chief Technology

Officer

BANKS.com, Inc. - financial services portal

Finance: BANKS.COM

© 2012 SAP AG. All rights reserved. 24

Finance: TURKDEX

Turkish Derivatives Exchange, Inc.

The combination of

simplicity and power

provided by the

Continuous Computation

Language and the rapid

integration capabilities of

Sybase CEP, gave our

developers the critical

tools to create the

sophisticated applications

in extremely challenging

schedules to meet

regulatory and corporate

needs.Abdullah Akoglu,

Lead Developer

CHALLENGE

• Managing market and trading risk effectively

• Establish a comprehensive trading system to detect abusive trading patterns at high-speed market rates

• Speed the development and deployment of their surveillance applications.

SOLUTION

• Sybase CEP (Complex Event Processing)

RESULTS

• Creates a reliable, secure trading environment

• Supports the increase in derivative market investments and trading volume

• Delivers simplified surveillance mechanism to identify market abnormalities

© 2012 SAP AG. All rights reserved. 25

Just A Few of Our CEP Customers

Thank you

İletişim Bilgileri

Bulut Altıntaş

Teknik Danışman

bulut.altintas@sybase.com.tr

+90 (0) 212 351 27 30