Business Event Processing –Beyond the HorizonKyle Brown, Opher Etzion

�

Our Vision: Event Processing in 2019

� Event Processing repeats (in 30-something years offset) the success of “Data Management”

� Part of main stream computing� Wide coverage in term of applications that are doing some type of

event processing� Broadly accepted standards� Event Processing extensions to programming languages� Large amount of developers are familiar with the concepts� Widely taught in universities with popular textbooks� Well-established Research community that contribute to the concepts

and the engineering aspects � Other disciplines focused on extracting events and event patterns

(image processing, information retrieval, search engines, data mining).

�

An 2009 View

In recent Years Event Processing has become one of the fastest growing segments ofenterprise integration middlewareThere have been many talks in this conferenceabout IBM’s Products in this space, and it was

Mentioned a lot as a key enabler of thesmarter planet

However, event processing as a discipline is still in the relatively early phases; many more developments to this technologyare expected beyond the horizon

�

Main challenges

����

�������

���������

� ������������������������� ���������� ��� ��������� ���������������� �������� ����� ������������������������������������

������������������� ��������������������� �� !"�������#������������ ��� �$����������������� �$�����

��������

��������� �

���� ������$������ ������������� ���� ����

�

Platform Oriented ChallengesMove from engines to platforms. Each platform can host a

variety of specialized agents optimized for a specific task. The same platform will be embeddable in various higher level platforms – as event processing is typically a part of a b i g g e r p i c t u r e

%� $����&"�� ����

%� $����&"�� ����

'���#������ �����

'���#������ �����

(��� �����

(��� �����

��)�&�%��������� ������

��)�&�%��������� ������

����������������

�

Engineering Oriented Challenges

EPNEPNEvent

ProducerEvent

Consumer

EventProducer

EventProducer

Event Consumer

Event Consumer

EPA

Pattern

EPA

Pattern

EPA

Pattern

EPAEPA

*������ �"����+�� �������������� �������

������ �$����������

,����������������

"����������"�����������������

�������� ��

QoSDriven

�

User Oriented Challenges

����������� �'��� ������

-���� �.����������

'�������� ������

)� �$����� �+�

�� ����/����� ����/���������

,��$����� �+�

�������/������

�

Functional challenges – the focus of our talk

Geo-Spatial Event Processing

Automatically generating events

and patterns

Processing the past and the future

Uncertain event processing

�

How is this presentation structured?

Dr. Opher Etzion, IBM Senior Technical Staff Member, Event Processing Scientific Leader in IBM Research will focus on the technology side

Kyle Brown, IBM Distinguished Engineer in IBM Software Servicesand Support for Websphere will focus on use cases to explain theTechnologies

Automatically generating events and patterns

Background: Events

EurekaAn Event is something that happens. Event representation in a computerized system answers questions like: • What happened ?• When did it happen ?• Where did it happen ?• Who was involved ?• What other information is relevant to

understand this event?

Current event processing systems process events that are typicallystructured and obtained by instrumentation (e.g. state observers), sensors

(e.g. RFID tag readers), and adapters from various sources

�

More event sources

Video Streams

Audio Streams

Internet goodies

Using various techniques(such as: image Processing, voice analysis, information retrieval, natural language Processing) to understand the event and its details:What happened ?When did it happen ?Where did it happen ?Who was involved ?What other information is relevant to understand this event?

�

Examples: Extracting events frommulti-media streams

Many toll roads and traffic lights use video cameras to take pictures of license plates as a car passes by

Allows the picture and license plate # to be extracted and used for billing or ticketing

Can also extract sounds from a continuous audio stream Used by law enforcement to detect gunshots and determine both time and location

�

Examples: Extracting events from textsThere have been many examples of people using Twitter feeds to represent events:

NYU’s Botanicals group has made it possible for your plants to tweet you when they need watering

����������� ����� ����������� �������������������� ��� ��������� ��������� ����������

One West coast Krispy Kreme donut franchise tweets their “Hot Donuts Now” sign

����� ��� �� ��� ��� ����������� ���������������������� ��� ������ ����� �� �

�

Background: Event Patterns

PatternMatching

One of the main function of business event Processing is “pattern matching”: find if a Certain combination of events happened.For example: • Find if the same customer already made product inquiry about the sameproduct recently (see below)• Find if a customer issued three complains already recently

Event Processing

The result of a pattern detection maybe interpreted as “situation” –an occurrence in the user’s domainthat requires notification / reaction

�

Extracting patterns from higher level abstractions

In some cases the patterns can be extracted from legaldocuments, regulations, policies

In other cases the pattern can be extracted from decision modeling

The idea is to enable automated creation of patterns and ingeneral the business logic behind BEP, to enable agility andreduce the long IT life-cycles.

�

Extracting patterns by machine learning techniques

In some cases the patterns to be watched can be obtained by looking at past event and determine causalities among events using machine learning techniques. This can be static (off-line) or dynamic(on-line) learning.

�

Example: Automatic extraction of pattern and business logic

• Analyze the event flows in money inflow and outflow and in declared investment strategies in hedge funds or mutual funds shown to be Ponzi schemes (like Madoff’s investment fund)

• The stated Madoff investment strategy, called "split-strike conversion," is known to be very volatile; it involves trading huge positions around options expirations.• Despite that, the fund’s returns over the past decade were a stable 8-10 percent.

• These patterns can then be applied to existing money flows and detect Ponzi schemes currently in progress

�

Geo-Spatial Event Processing

�

Geo-spatial events and patterns

�� �� �� ��

�� �� �� ��

�

�

����������������

������� �!������

"!#����"

�� �$

�� �� �� �� �� ��

����������������

������� �!������

"!#����"��

�

�� �� �� �$

�� �% �� ��

����������������

������� �!������

"!#����"�

�� �&

�� �'

�

�

"!#����"

"!#����"�

"!#����"�

GPS and other location sensors enablelocating events and moving objects

Patterns can be based on locations, for example: observe traffic patterns on highway, andtrack individual moving objects,

�

Geospatial Customer Examples

Healthcare Event Processing: Tracking of medical equipment within a hospital campus – knowing that certain equipment needs to be within a certain room at a certain time

Manufacturing Event ProcessingTracking the arrival of parts into a work stationTracking the creation of parts as they are createdTracking the disposition of shared resources in a factory (such as pallets or forklifts)

Shipping and Tracking event processing

When does something arrive at a freight terminalWhen does the same object move on to the next stage in its journey

��

Processing the past and future

��

Retrospective Event Processing

Situation Reinforcement: An event pattern designates the possibility that

a business situation has occurred; in order to provide positive or negative reinforcement, as part of the on-line pattern detection, there is a need to find complementary pattern (which is typically not traced) in order to assert or refute the occurrence of the situation.

Patterns for observations on past events

Event Patterns can be used to find periodic observations about past events

��

Predictive Event Processing

Processing events that have not yet happened:

Event are predicted by causality relationships with other events or using predictive analysis tools

Alerting, mitigating, adaption or eliminating the occurrence of the predictive events

Alerts, and in some cases autonomous actions to decide how to mitigate past events.

��

Retrospective Customer Example I• on-line situation:.

– A person that has deposited (in aggregate) more than $20,000 within a single working day is a SUSPECT in money laundering

• Reinforcement situation (conjunction of…)– There has been a period of week within the last year

in which the same person has deposited (in aggregate) $50,000 or more and has withdrawn (in aggregate) at least $50,000 within the same week.

– The same person has already been a "suspect" according to this definition within the last 30 business days.

• If the on-line situation occurs then look for the reinforcement situation – if it satisfied then the event "confirmed suspect" is derived.

Money Laundering

��

Retrospective Customer Example II

• An electronic trade site provides the opportunity to customers to offer items for sale, but letting them conduct a bid, and provide bid management system (using a CEP system, of course). One of the services it provides to the customer is "alert that you are over-estimating the price you can get”

• On-line Situation: – There has been at least two bidders, however

none of them have matched the minimum price of the seller then this may be an indication of "too expensive bid".

• Reinforcement Situation:– at least 2/3 of the past bids of the same sellers

have also resulted in a "too expensive bid" situation,

– If the on-line situation occurs then look for the reinforcement situation – if it satisfied then the event then send the seller a notification "you are too greedy".

The Greedy Seller

��

Predictive Customer Examples

Simple transportation example:• Departure of a large number of rail cars from a shipping port is always followed up within 12 hours by arrival of a large (but smaller) number of rail cars at a major routing depot• This physically indicates the arrival of one or more container ships that have been unloaded and the containers shipped out• By analyzing this recurring traffic pattern the rail company could plan to reschedule track maintenance activities to reduce congestion

��

Uncertain Event Processing

��

Uncertain Events

�Uncertainty IF the event happened

� Uncertainty WHEN the event happened

� Uncertainty about the event content (exactly WHAT happened)

Using techniques for representing and process uncertainInformation, and adapt them to event processing

�

Uncertain Situations and Event Patterns

Recall: Situation is something that requires reaction from the user’s point of view: it can be either a raw event, or a result of a detected pattern.

The event or pattern may just approximate the conditions where the situation occurs, but may not have a complete match – example: a collection of medical symptoms may indicate a differential diagnosis (with some certainty measurement).

Using techniques for uncertainty inference and reasoning, suchas: fuzzy reasoning, Bayesian networks, evidential reasoning…

�

Customer Uncertainty Example

• The traffic jam example: – Consider a truck freight routing system that

takes as one input reports of traffic jams– If manual data entry is required then an event

(such as a report of a traffic backup) may be reported within an uncertainty of several minutes – the backup may be cleared by the time it is reported

– Also people may misreport a traffic jam (was that accident at the Corner of Main and 5th or Main and 4th?)

– Likewise if the reporting of an event requires individual judgment then the existence of the event itself may be in doubt (what determines if it is a “Major” traffic backup)

��

Pattern Uncertainty example

In many cases the pattern itself has an uncertainty figure attached to the result

This also applies to events derived from multimedia streams (e.g. handwriting or character recognition, voice recognition)

Example: Diagnostics rules – A diagnosis may be within a level of uncertainty (e.g. an 80% chance patient has a staph infection)

��

Summary

The area of Event Processing just scratchedthe surface of its potential and is spreadingto different directions, all based on customer applications we already identified in the present

��

Summary (II)

IBM Research is actively involved in driving the IBM products in theBusiness Event Processing spaceto advance beyond the currentstate of the art

IBM is also leading the event processingcommunity to form the “Event ProcessingTechnical Society” which is engagedin a community effort to advance the2019 vision

��

We love your Feedback!

• Don’t forget to submit your Impact session and speaker feedback! Your feedback is very important to us, we use it to improve our conference for you next year.

• Go to www.impact09guide.com on a smartphone device or a loaner device

• From the Impact 2009 Online Conference Guide;– Select Agenda– Navigate to the session you want to give feedback on– Select the session or speaker feedback links– Submit your feedback

��

© IBM Corporation 2009. All Rights Reserved.

The workshops, sessions and materials have been prepared by IBM or the session speakers and reflect their own views. They are provided for informational purposes only, and are neither intended to, nor shall have the effect of being, legal or other guidance or advice to any participant. While efforts were made to verify the completeness and accuracy of the information contained in this presentation, it is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this presentation or any other materials. Nothing contained in this presentation is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software.

References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer.

The following are trademarks of the International Business Machines Corporation in the United States and/or other countries: ibm.com/legal/copytrade.shtmlAIX, CICS, CICSPlex, DataPower, DB2, DB2 Universal Database, i5/OS, IBM, the IBM logo, IMS/ESA, Power Systems, Lotus, OMEGAMON, OS/390, Parallel Sysplex, pureXML, Rational, Redbooks, Sametime, SMART SOA, System z , Tivoli, WebSphere, and z/OS.

A current list of IBM trademarks is available on the Web at “Copyright and trademark information” at ibm.com/legal/copytrade.shtml.

Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency which is now part of the Office of Government CommerceJava and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both.ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark OfficeIntel and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.UNIX is a registered trademark of The Open Group in the United States and other countries.Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.