Esper

• Language,compilerandruntime forDSM&CEP• BasicversionisalibrarythatcanbeimportedinJavaor.NET• EventProcessingLanguage(EPL)enablesrichexpressionsovereventsandtime

• ThemandatoryassignmentusesEsperandwillrequireyoutowriteEPLqueriestogetdatafromastream!

Documentation: http://www.espertech.com/esper/esper-documentationTry online: https://esper-epl-tryout.appspot.com/epltryout/mainform.html

Documentation: http://www.espertech.com/esper/esper-documentationTry online: https://esper-epl-tryout.appspot.com/epltryout/mainform.html

EPLSyntax• Create schema

• DataStreamManagement:

• CEP:

Documentation: http://www.espertech.com/esper/esper-documentationTry online: https://esper-epl-tryout.appspot.com/epltryout/mainform.html

select ID as sensorId, sum(countTags) as numTagsPerSensorfrom AutoIdRFIDExample#win:time(60 seconds) where Observation[0].Command = 'READ_PALLET_TAGS_ONLY’ group by ID

select a.custId, sum(a.price + b.price) from pattern [every a=ServiceOrder ->

b=ProductOrder(custId = a.custId) where timer:within(1 min)]#win:time(2 hour)

where a.name in ('Repair', b.name) group by a.custIdhaving sum(a.price + b.price) > 100

create schema SchemaName(field_name1 type, field_name2 type)

Window• Called aview inEsper• Choose correctwindowwhen writingEPLqueries!• Examples of windows/views:

View Syntax Description

Timewindow win:time(time period) Sliding timewindow

Externally-timed window win:ext_timed(timestampexpression, timeperiod)

Sliding timewindow,based on the millisecond timevalue supplied byanexpression

Timebatchwindow win:time_batch(timeperiod[,optional reference point][,flow control])

Tumbling window that batches events andreleasesthem every specified time interval,with flow controloptions

Externally-timed batchwindow

win:ext_timed_batch(timestamp_expression,time_period[,optional_reference_point])

Tumbling window,based on the millisecond timevalue supplied byanexpression

Documentation: http://www.espertech.com/esper/esper-documentationTry online: https://esper-epl-tryout.appspot.com/epltryout/mainform.html

CEPsystemoverview

CEPsystemoverview

Programming languages used

ApacheFlink• Stream processing systemwith many features• Librarieson top of batchandstream processing• CEPuses the DataStream API• CEPispatternmatchingoverevent streams

ApacheApex

• Stream processingplatform• Nativehadoopsupport• Dependsonhadoop

ApacheKafka

• OriginallybyLinkedIn,open-sourced in2011• 1stversion:High-throughputpublish subscribe• 2ndversion:High-availabilityanddurability• 3rdversion:Stream processing• Then kafkastreams-cep came(unofficial),enablingCEP

Siddhi CEP

define stream TempStream (deviceID long,roomNo int,tempdouble);@info(name ='5minAvgQuery’)fromTempStream#window.time(5min)select roomNo,avg(temp)as avgTempgroup by roomNoinsert into OutputStream;

• Stream processinglibrary,similar toEsper.• Also offersaSQL-likeinterface forquery processing.

SQLandObject-oriented interfaces

val revenue = orders .filter('cCountry === "FRANCE") .groupBy('cID, 'cName) .select('cID, 'cName, 'revenue.sum AS 'revSum)

orders = table_env.scan("Orders") evenue = (orders.filter("cCountry === 'FRANCE'").group_by("cID, cName").select("cID, cName, revenue.sum AS revSum"))

Table orders = tableEnv.scan("Orders");Table revenue = orders.filter("cCountry === 'FRANCE'").groupBy("cID, cName").select("cID, cName, revenue.sum AS

revSum");

Table revenue = tableEnv.sqlQuery("SELECT cID, cName, SUM(revenue) AS

revSum " +"FROM Orders " +"WHERE cCountry = 'FRANCE' " +"GROUP BY cID, cName");

Flink's JavaTableAPI Flink's JavaSQL

Flink’s ScalaTableAPI Flink’s PythonTableAPI

Benchmark

• Few exist ....andtheymay notbegeneralenough• Difficulties inbenchmarking:biases, lack of knowledge,unfaircomparisons

Feature comparison of DSMS/CEPlibraries

Feature comparisonof EPplatforms• Additional cloud features

• Scalability• Highavailability (HA)• Fault tolerance

Internet of ThingsUse Case

Assign 10 => Temp, 11 => Humidity, 12 => FireDefine Fire(area:string,temp: float)From Temp(value > 45) andlast Humidity([string]area=Temp.area,perc < 25)within 5000 from TempWhere area := Temp.area, temp := Temp.valueConsuming Temp, Humidity;

DistributedComplex Event Processing

• Splitqueries into subqueries• Placethem on bestnode

• Which isbestnode?• Minimize latency?• Minimize energy consumption?

DistributedComplex Event Processing

Streamprocessingvs.CEP– Stillconfused?

• Recommendedreading:• What'sthedifferencebetweenESPandCEP?ByDavidLuckham,originatorofComplexEventProcessing(CEP),authorof"ThePowerofEvents". https://en.wikipedia.org/wiki/David_Luckham• http://www.complexevents.com/2019/07/15/whats-the-difference-between-esp-and-cep-2/