Post on 29-Nov-2014
description
transcript
epts event processing technical society
eptsevent processing technical society
Pedro Bizarro on behalf of theUse Case Working Group
Results of the Survey on Event Processing Use CasesMarch 24, 2011
epts event processing technical society
2
How are event processing technologiesbeing used?
Classify scenarios,help othersselect solutions
Inspire use ofevent processing
Food for thought forresearchers & engineers
epts event processing technical society
3
Earlier versions
• 2007: Kick-off – Problem: world of superficial use cases
• 2008: v1: 6 use cases, 54-questions (@4th epts)
• 2009: v2: 5 use cases 9 lessons learned (@debs2009)
• 2011: v3: 30 use cases plenty of statistics (today!)
epts event processing technical society
4
24 questions~13 minutes
30 use casesEPTS, DEBS
community
epts event processing technical society
Large Variety of Use Cases
5
GridDocument workflow
Patient discharge
Revenues and expensesETL in Telcos
NYC transportation
Content authoring
Gas station networks
Emergency management
Home energy
Testing algorithms…
epts event processing technical society
6
Banking/Financial
Energy/Utilities
Aerospace/Defense
Education
Telecom
Transportation/Logistics
Other
0% 5% 10% 15% 20% 25%25%
18%
11%
7%
7%
7%
25%
Industry background
Computer SoftwareHealthcareManufacturingRetail and distributionetc<4% each
epts event processing technical society
7
Functional area
Operations
Research and Development
Finance
IT Support / Administration
Customer Service and Support
Human Resources
Sales
Marketing
Other
0% 5% 10% 15% 20% 25% 30% 35%
33%
18%
12%
12%
9%
6%
3%
0%
6%
epts event processing technical society
8
Maturity level
A concept
Prototyping - any stage
Pilot
In developement
In production
0% 5% 10% 15% 20% 25% 30% 35%
4%
17%
26%
22%
30%
52%
epts event processing technical society
9
Primary project drivers
Enhance/improve user services
Add new features
Reduce production cost
Faster user service
Reduce deployment cost
Compliance
Other
0% 5% 10% 15% 20% 25% 30% 35% 40%
35%
16%
14%
11%
8%
5%
11%
epts event processing technical society
10
Data sources
Databases
Files
ESB, JMS, other msg protocols
RSS, twitter, stock, news, other feeds
Application or OS events
Video, sound, multimedia streams
Other
0% 5% 10% 15% 20% 25% 30% 35%
31%
18%
18%
10%
5%
0%
18%
Non“streaming”
“streaming”
epts event processing technical society
11
Destinations and actions
Msg people (E-Mail, SMS, MMS)
Trigger other applications
Databases
Dashboards
ESB, JMS, other msg protocols.
RSS, twitter, stock, news, other feeds
Other
0% 5% 10% 15% 20% 25% 30%
24%
24%
20%
17%
7%
4%
4%
someautomation
but peoplestill stronglyin the loop
epts event processing technical society
12
Desired features
Correlations or joins
Summaries or aggregations
Sliding windows
Tumbling windows
Forecasting and predictive functions
Probability or statistical models
Other
0% 5% 10% 15% 20% 25% 30%
24%
24%
17%
16%
9%
9%
2%
simplerbeats
complex
epts event processing technical society
13
Data models and data types
Relational
Column-Oriented
Generic XML
Objects
RDF
XML (industry specific schema)
Extensibility
Other
0% 5% 10% 15% 20% 25% 30% 35%
32%
18%
12%
9%
9%
9%
3%
9%
epts event processing technical society
14
Performance – input events per second
NONE
'1-10
'10-100
'100-1.000
'10.000-100.000
'100.000-1.000.000
1.000.000 or more
No answer
0% 5% 10% 15% 20% 25% 30% 35% 40%
5%
11%
16%
37%
11%
5%
0%
16%
Not that much!
epts event processing technical society
15
Number of data sources
'1-10
'10-100
'100-1.000
'10.000-100.000
'100.000-1.000.000
1.000.000 or more
No answer
0% 5% 10% 15% 20% 25% 30% 35% 40% 45%
42%
26%
11%
0%
5%
0%
16%
Or maybe not! Typo?!
Data comes fromfew sources
epts event processing technical society
16
Number of “AI” models
NONE
'1-10
'10-100
'100-1.000
'10.000-100.000
'100.000-1.000.000
1.000.000 or more
No answer
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50%
47%
16%
11%
0%
0%
0%
0%
26%
3 in 4 don’t answeror don’t use
epts event processing technical society
17
NONE
'1-10
'10-100
'100-1.000
'1.000-10.000
'10.000-more
No answer
0% 5% 10% 15% 20% 25% 30% 35% 40%
5%
11%
16%
11%
5%
16%
37%
Expected data size growth rate/year (in MB)
or not growing much
Hard to forecast
47% increase lessthan 10Gb/year
epts event processing technical society
18
Why did we use log scales?
Because we didn’t knowwhat to expect
epts event processing technical society
19
Enterprise capabilities
High availability
Online system maintenance
Provenance
Fault tolerance
Recovery of any data or meta-data
Security
Encryption
0% 5% 10% 15% 20% 25% 30% 35% 40% 45%
39%
24%
24%
21%
21%
12%
0%
Cannot stop!
Not a big concern
epts event processing technical society
20
Implementation constraints
Specific CEP engine
Specific file or database system
Specific operating system
Specific application server
Specific development environment
Specific hardware
Other
0% 5% 10% 15% 20% 25% 30%
27%
19%
15%
12%
12%
8%
8%
surprise
epts event processing technical society
21
The typical use case
• In production to improve banking/utilities user services
• Gets data from databases, files and message queues
• Notifies people and other applications
• Does correlations, joins and aggregations
• Handles less than 1000 events/second (<10 sources)
• Must run a specific CEP engine
• Cannot stop
epts event processing technical society
22
The surprises (personal take)
• Few telcos, transportation and logistics
• Not from IT department
• Not about lower TCO, faster deployments
• “Non-streaming” dominate
• Low throughput, few sources, little growth
• Not about forecasting, predictions
• Very few AI models
• Not about security, encryption
epts event processing technical society
23
Members and acknowledgements
• Pedro Bizarro <bizarro@dei.uc.pt>
• Christoph Emmersberger <christoph.emmersberger@citt-online.com>
• Thomas Ertlmaier <thomas.ertlmaier@citt-online.com>
• Matthew Cooper <M.Cooper@eventzero.com>
• Tina Groves <Tina.Groves@ca.ibm.com>
• Dieter Gawlick <dieter.gawlick@oracle.com>
• Brian Connell <brian@westglobal.com>
epts event processing technical society
eptsevent processing technical society
Q&A?
24