SA2011 Event Driven Systems - artechra.com...•A simple idea, using a standard set of concepts •...

Eoin Woods www.eoinwoods.info

Starting out with Event Driven Systems

Software Architect, London

October 2011

• Software architect & dev manager at UBS Investment Bank •  responsible for equity swaps systems within Equity Derivatives

• Software architect for ~10 years

• Author of “Software Systems Architecture” book with Nick Rozanski

• 2nd Edition published in October 2011

• IASA and BCS Fellow, MIET, CEng

20111020

About Me

2 © Eoin Woods 2011

• Difficulties of Distributed Systems

• Event Driven Architecture

• Building Event Driven Systems

• Case Study

• Extending the Event Driven Approach

• Conclusion

20111020

Agenda


20111020 4 © Eoin Woods 2011

1.  The network is not reliable

2.  Latency is not zero

3.  Bandwidth is not infinite

4.  The network is not secure

5.  Topology changes

6.  There are many administrators

7.  Transport cost is not zero

8.  The network is not homogeneous

Peter Deutsche and others, from 1994

20111020

Difficulties of Distributed Systems


•  The point of RPCs and services is to try to hide the difficulties of distributed systems

•  This doesn’t work and never has •  fragile connections

•  difficult to evolve

•  unexpected system qualities

•  Better to accept the difficulties and pick an architectural style that can help

•  event driven architecture is one of those styles

20111020

RPCs and Services


20111020 7 © Eoin Woods 2011

• According to the Event Processing Technical Society … [EDA is] An architectural style in which some of the components are event driven and communicate by means of events. [Event Driven means] The behaviour of a device, software module or other entity whose execution is in response to the arrival of events from external or internal sources [An Event is] An object that represents encodes or records an event, generally for the purpose of computer processing

Event Processing Glossary – v1.1 The Event Processing Technical Society

20111020

Event Driven Architecture (EDA)


• Events are a record of activity in a system or outside

• Events are characterised by their • form – e.g. a tuple, a header/body structure, an XML block … • significance – what activity they signify • relativity – how they relate to other events (by time, causality or aggregation)

David Luckham (Power of Events)

(Events are usually state changes in entities of interest to the system or record the passing of time)

20111020

Finding Events


• A simple idea, using a standard set of concepts • Event producers, consumers and event processing agents

• Events •  raw events and derived events • events contain a set of header fields and a body (structured or

opaque)

• Event channels • Shared state

• These concepts form a standard reference model • an Event Processing Network (EPN)

• An EPN exists to detect, decide and respond

20111020

EDA - Overview


20111020

Real World Event Processing


Taxi please!

I’m coming on duty

I’m going home

You’re booked

Taxi Dispatcher (Event Processor)

• Broadcast Communication • event producers broadcast events rather than send them

• Timeliness • events are dispatched when ready, not in batches

• Asynchrony • event producers do not wait for events to be processed

• Fine Grained • events are published individually

• Ontology •  clear classification hierarchy and semantics for events

• Complex Event Processing • events can be related (aggregation, causality, …)

(Gregor Holpe) 20111020

Characteristics of Event Driven Architecture


20111020

EDA – Types of Event Processing Agents


Event Processing

Agent

Filter

Transform

Translate

Enrich

Project

Aggregate

Split

Compose

Pattern Detect

20111020

EDA – Example Network


Order Processing

System

Shipping System

Billing System

Fulfillment System

Order Emitter

Order Consumer

Fulfillment Emitter

Shipping Combiner

Customer Details

Billing Consumer

Shipping Consumer

Shipping Enricher

“order created”

“order packed”

“order addressed”

Billing Status Emitter

“ready to ship”

“order billed”

20111020

EDA Example State Machine for Orders


• Simple event processing • events (typically) created for state changes or external occurrences, usually drive state machines

• often all that is needed for enterprise applications

• Event Stream Processing (ESP) • filtering and processing of streams of events (e.g. prices)

• Complex Event Processing (CEP) • complex events are those derived from other events • CEP is the process of creating & processing complex events

20111020

Types of Event Processing


• Very strong decoupling (+) • when done well and the rules are obeyed

• Very good scalability (+) •  inherently scalable and distributed architectural style

• Very flexible extension and evolution (+) • again, if you stick to the rules

20111020

EDA – Strengths and Weaknesses


Can be complicated to implement (-) • a lot of state, a lot of pieces

• unfamiliar, needs tight semantics

• Causality and traceability difficult to achieve (-) • makes tracking problems difficult

• needs to be built in from start

• Monitoring is crucial to provide visibility (-) • otherwise failure isn’t evident, things just “don’t happen”

20111020

EDA – Strengths and Weaknesses


• Inadvertent Coupling • events sometimes morph into requests … coupling emerges

• Mixing Activities and Events • activity records aren’t events (can’t drive state changes)

• Failure Recovery • recovering from node or system failure can be complicated • what is the state of the system? how about in flight events? • design recovery early and test it, use monotonic consumers, assume replay needed

• Operational Complexity • operational independence can affect backup, recovery, DR, security (operate a “system of systems”)

20111020

EDA – Common Problems (1)


• Latency • processing a piece of work can take longer than with a monolithic system

• Event Semantics •  just defining fields is fatal … what does the event signify?

• Accessing Reference Data •  can become a new bottleneck or consistency problem

20111020

EDA – Common Problems (2)


20111020 21 © Eoin Woods 2011

• Define a domain model •  identify state-oriented, data-oriented and reference classes

• Design state models for state-oriented classes • ensure entry and exit states exist (create and destroy) • add error conditions to state models (nested states?)

•  identify actions needed on transitions as well as events

• Define the set of event processors & connect them •  interaction design needs careful analysis to avoid problems

• Add the non-event driven parts of the system • services, reference data, batch processing, …

20111020

Design Approach for Event Based Systems


20111020

Reference Design for an Event Driven System


Entity Store

Event Processors

Reference Data

State Store Application Services

Event Channel(s)

• Central Event Store • build everything, use database or cache as an event store

• more a blackboard style than “real” EDA

• Use Messaging • use messaging system as event transmission / routing / filtering mechanism, possibly still use db as event store

• Use an ESB • ESB services (transformation, routing, …) can support EDA • commercial (e.g. Sonic) or open source (Mule, Service Mix)

• Use CEP / Stream Processors • adding complex events • open source or commercial (e.g. Esper, Streambase)

20111020

Technology for Building Event Based Systems


Simple EDA from Scratch

20111020 25 © Eoin Woods 2011

Event Store Table(s)

Event Emitters

event records

Event Transformers

Event Consumers (Processors)

event records

event records

event records

Evaluating Our Simple Implementation

20111020 26 © Eoin Woods 2011

Strengths Weaknesses

Achieves loose coupling Poor scalability due to db contention

Simple, easy to understand & build

Lots of custom event handling code

Flexible processing model No visibility of event processing network structure

No special technology needed Implementation easily diverges from the EDA style

Simple EDA Using Messaging

20111020 27 © Eoin Woods 2011

Pub/Sub Messaging

message_topic_3

message_topic_1

message_topic_2

Event Emitters Event Consumers

Event Transformers

Message Replay

Idempotent Transactional Message Consumption

Message Model and Representation

State Machines

Routing

Event Processing Core – the Event Processor

20111020 28 © Eoin Woods 2011

• Event model and representation

• Event processing model • state-machine based processing (events trigger state transitions)

• activity-model based processing (events trigger activities) • often one can be translated to the other

• Event routing

• Idempotent, transactional message consumption

• Reliable, standardised message replay

• Publish/subscribe message transport •  ideally with message header filtering

20111020

Concerns for Message Based EDA


Event Model and Representation

20111020 30 © Eoin Woods 2011

<event> <event-header> <event-id>…</event-id> <event-time>…</event-time> <event-type>….</event-type> … </event-header> <event-body/> </event>

<event> <event-header> … <event-type>com.x.orders.dispatched</event-type> … </event-header> <event-body> <order-id>123</order-id> <dispatch-time>20110921T120413</dispatch-time> … </event-body>> </event>

• Define a clear hierarchy for the events • and make sure they are events

•  clearly define what they mean!

• Define a standard set of “header” (or metadata) fields which every event will carry

•  id, timestamp, fully qualified type, type of business entity it relates to, id of business entity it relates to, event source

• Define one or more clear concrete encodings • e.g. an XML schema or JSON encoding

• Strongly type the event header and body fields

• Allow for extension but control its use • e.g. XML schema’s <xs:any>, JSON schema’s any

• Don’t rely on ordering in the encoding if possible

20111020

Event Model


• Events need to trigger business logic • Custom code is sometimes used

• Better via a standard model (visibility, reliability, reuse)

• State Machine and Process based models are both used

20111020

Event Processing Model


Events

Your Code Here

State Machines

20111020 33 © Eoin Woods 2011

•  State machine options: • An FSM library • Create a simple interpreter (representation + execution)

• Custom event processing business logic (custom code in each event consumer)

•  Open source examples: • SCXML, Tungsten, Jolt.NET, …

• Event routing can easily become its own project • keep it simple and avoid building complicated routing software

• Simple message-topic based approach usually sufficient • events typically relate to a business object type (e.g. “new order”) • use a message topic per business object type •  include the message type and business object type and key in the

message header fields

• encode a simple naming convention in your event emitter and consumer library code

• use message selectors (filters) to subscribe to events of interest for particular event types

20111020

Event Routing


20111020

Event Routing Using Message Topics


Reliable Message Consumption

20111020 36 © Eoin Woods 2011

To receive without 2PC: •  receive the message •  store to db if not there •  then commit message read

Implement carefully, test thoroughly, log everything - create a library and reuse it!

Message Replay

20111020 37 © Eoin Woods 2011

• Reliable message replay is key to system recovery

• Replay requires: • a log of sent messages • the ability to resend messages from the log

• “Dispatcher” is used for sending events

• “Replay Dispatcher” used in recovery situations to resend missing events

Implementing Simple Message Based EDA

20111020 38 © Eoin Woods 2011

Evaluating A Message-Based Implementation

20111020 39 © Eoin Woods 2011

Strengths Weaknesses

Loose coupling Debugging and monitoring can be difficult

Good scalability High volumes can require complicated messaging configuration

Uses straightforward, well understood technology

No standard event processing agents (e.g. splitting) for “free”

Flexible, easy to extend, faithful implementation of EDA

Can be difficult to visualise the event processing network

20111020

Using an Event Bus (ESB)


• Enterprise Service Bus products provide useful EDA features … • message transformation services

•  routing services •  service hosting • multiple transports (WS-*, RMI, CORBA, .NET Remoting, MQ, …) • management tools

• … at a price •  literally if you buy commercial • development & deployment complexity wherever you acquire one

from

• Look carefully before committing …

20111020

Using an ESB


HTTP Listener

XSLT Translator

JMS Sender

JMS topic

JMS Receiver

JMS Receiver

Custom Code Call

Custom Code Call

Database Listener

Email Sender

App DB

MSMQ Sender

Workflow Engine

• Complex Event Processing is processing event groups and patterns as well as individual events

• detection of event patterns • recognising relationships between events • creating new inferred events as a result (the “complex” events)

• Many enterprise systems don’t need CEP • but CEP adds many new interesting possibilities

• CEP is usually achieved using a “CEP Engine” • no shortage of vendors who’d like to sell you one • open source (e.g. Esper) and commercial (e.g. Apama)

20111020

Using a CEP Product


20111020

Using a CEP Product


Event Sources

CEP Engine Systems

CEP Definitions

Events Actions

Notifications Derived (“Complex”) Events

20111020 44 © Eoin Woods 2011

Case Study Starting Point

20111020 45 © Eoin Woods 2011

Front Office

Systems

Middle Office System

Confirmations Processing

Settlement Processing

Payments System

Operations Users

Trade Store

• Capital Markets domain (Fixed Income Middle Office) • existing monolithic system had served well but become difficult to

scale and evolve • a new middle office environment for FI rates products needed • didn’t want a single large system for the replacement

• Future State Architecture and Build project • decision made to build 5 independent systems

•  but all 5 must cooperate to implement the trade lifecycle

• very strong desire to avoid technical or organisational coupling •  avoid shared release cycle, no shared database, no point to point links

• no special purpose EAI or EDA software used •  but ended up using events for integration •  and created a custom UML state chart evaluation library

20111020

Case Study Background


• Messaging identified as a key technology • desire for independence, lack of coupling, scalability

• Previous experience pointed to this not being enough • coupling and complexity can quickly occur with ad-hoc messaging

• Event based messaging identified as the solution • the systems all broadcast trade lifecycle events • message subscriptions route events to the right processor(s)

20111020

Case Study Architectural Decisions


• No request/response interactions

• event emitters do not know of the existence of event consumer(s)

• No shared entity database that everything reads and writes

• Systems use state charts to track trade lifecycle

• Very large events need to use hybrid approach • events are state refs into a shared event store

20111020

Case Study Architectural Implications


The Replacement

20111020 49 © Eoin Woods 2011

Front Office

Systems

event-based message bus

Confirmations System

Settlements System

Payments System

End User Workbench

Trade Store

• An EDA approach resulted in • Smaller systems, better defined responsibilities

• Well defined neutral (constantly evolving) event models • No direct dependencies between systems

•  Avoided coupling by using the common event models

• No global shared entity stores

• Development team independence (relatively)

• Has allowed much easier change • Systems can have own development & release cycles

• The architecture has enabled overall agility • Overall result is more effective delivery and change

20111020

Case Study Results


• Don’t ask for something, announce something • use events don’t make requests

• Don’t send business entities, send lifecycle events • one of the major mental shifts people need to make

• Use “failsafe” processing to spot missing/extra events • essential for troubleshooting and monitoring

• Monitoring is key and needs to be built in from the start •  there is no one place to look any more

20111020

EDA’s Affect on Design


• Event model is key to reliability and evolution • needs strong semantics and support for event model evolution

• Monitoring and fail-safe processing need from the start • don’t drop unrecognised events, flag up unexpected events • alert when events are likely to have gone missing

• Asynchronous processing can cause problems •  race conditions can occur if assumptions made about propagation

• System-wide view is necessary • no one system knows the whole story – “end to end” view needed

• Partitioning needs to be considered carefully • what information needs to be where? how to partition cheaply

• Designing and developing using events needs some practice •  request/response is very ingrained in most of us

20111020

Case Study Lessons We Learned


20111020 53 © Eoin Woods 2011

• EDA is rarely the whole answer

• EDA has been both extended and combined with other architectural styles successfully

• Common combinations of EDA and other styles • SOA • EAI

• EDA has been specialised and extended in a number of ways • CEP • Staged Event Driven Architecture (SEDA) • Event Staging • CQRS (where it is an option for the command side of the system)

20111020

Extending the Event Driven Approach


• SOA is a popular architectural style, aiming to encourage loose coupling and flexible systems composed of services

• SOA and EDA can work quite well together •  services can generate events •  services can be called as part of processing an event

• EDA can act as the “choreography” for SOA’s services

• With a number of specific complications to address • who owns business object state? or does it need to be shared?

(breaking service encapsulation)

• does the EDA system perform any business logic? or is it all in the services?

•  some services implicitly manage the state of business objects so this needs to be integrated into the EDA view

• Ad-hoc EDA/SOA integration is likely to be very complicated

20111020

SOA and EDA


20111020

SOA and EDA


• SEDA is a specific specialisation of EDA

• Designed to address the needs of Internet scale systems •  scalability, reliability, runtime adaptation to load, efficiency

• Uses event channels as the architectural connectors between parts of a system

•  finer grained use of EDA than typical enterprise system

20111020

Staged Event Driven Architecture (SEDA)


(Adapted from “SEDA: An Architecture for Well-Conditioned Scalable Internet Services” Matt Welsh et al – 2001)

20111020 58 © Eoin Woods 2011

1.  The network is not reliable

2.  Latency is not zero

3.  Bandwidth is not infinite

4.  The network is not secure

5.  Topology changes

6.  There are many administrators

7.  Transport cost is not zero

8.  The network is not homogeneous

20111020

Reviewing the Fallacies of Distributed Systems


(Doesn’t affect this one much)

EDA helps to avoid (most of) these traps

• Event driven approach solves many traditional problems • builds on many of the strengths of other approaches

• drives coupling out of the system • allows scalability across the integration infrastructure • provides flexible options for change and extension • provides timely consistency while avoiding shared data store

• Naturally brings its own challenges too • more sophisticated approach, needing careful design

•  stick to the rules!

• event model needs to be done early and to be extensible

• asynchronous nature means that race conditions are possible • monitoring and failsafe processing must be built in • event causality and tracing can be difficult (harder debugging)

20111020

Summary


• EDA can be adopted in stages •  simple event processing

• event stream processing •  complex event processing

• Simple event processing is good for many enterprise systems

• Implementing EDA doesn’t need complicated middleware • databases, messaging and markup provide most of what you need • ESBs and CEP engines can add sophistication, but also complication

• EDA can integrate well into other architectural styles

• EDA helps to meet some of the key challenges of creating resilient, adaptable distributed systems

20111020

Summary (ii)


•  Event Processing Technical Society •  articles, Event Processing Glossary, event listings, (quiet) forums •  http://www.ep-ts.com

•  David Luckham’s site •  articles, news, pointers to research, event listings

•  http://www.complexevents.com

•  Opher Etzion’s blog and presentations •  http://epthinking.blogspot.com

•  http://www.slideshare.net/opher.etzion

•  Tim Bass’ blog and presentations •  http://www.thecepblog.com

•  http://www.slideshare.net/TimBassCEP

•  Jack van Hoof’s blog •  http://soa-eda.blogspot.com

•  Brenda Michelson’s EDA Overview •  http://tinyurl.com/eda-overview

20111020

More Information


• Some of the vendor people’s blogs •  Tibco (Paul Vincent) - http://tibcoblogs.com/cep •  Oracle CEP - http://blogs.oracle.com/CEP •  Apama (Progress) - http://apama.typepad.com •  Streambase – http://streambase.typepad.com

• Some representative books on the topic

20111020

More Information


20111020

Software Architecture More Generally


Eoin Woods www.eoinwoods.info

[email protected]

Questions and Comments?

65 20111020 © Eoin Woods 2011

Date post:	14-Sep-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

SA2011 Event Driven Systems - artechra.com...•A simple idea, using a standard set of concepts •...

Documents