1 Confidential Information of Talksum, Inc.

I’m Brian!

Who Is This Guy?

2 Confidential Information of Talksum, Inc.

• Principal Architect at Talksum

•Open Source Contributor ZeroMQ Rsyslog

Who Is This Guy?

3 Confidential Information of Talksum, Inc.

Talksum Data Stream Router

Next Age of Data Management

November 2013

4 Confidential Information of Talksum, Inc.

The Talksum Data Stream Router

Apache Common Logging – FilesSNMP - UDP

Unix Logs – RFC3164 UDP/TCPNetflow – UDP – NG v.5, 8, 9, 10

Patient Records (HL7) XML/ASN.1Transportation (BSM) SAE J2735

I2C, CAN, SNMP, Serial

XML, JSON, File, HTTP REST

Twitter, RSS, CAP (Weather Alerts)

Refined Data Stream

Refined Data Stream

Refined Data Stream

Indexed, Mapped, ReducedOrdered, Sorted Data Streams

Bulk Data Streams(Lightly Ordered

and Filtered)

TalksumData Stream

Router(TDSR)

• Data Normalization• Parsers• Filters• Metrics and

Counts• Inline ETL/PTL• Asynchronous

Outputs• Protocol

Verification

Customer A:Summarized Data

SystemLogs

ApplicationData

Sensor andIndustrial

Data

3rd Party DataB2B/M2M

Social andPublic Data

Customer B:Aggregated Data

Customer C:Dynamic Stream

ApplicationLogs

• SQL Warehouse• Bulk Data Stores• File Storage

• Object Data Stores• Indexed Data Caches• NoSQL Data

Warehouses

5 Confidential Information of Talksum, Inc.

We turn alphabet soup into coherent streams of information so that you can get to work solving problems.

In Other Words…

6 Confidential Information of Talksum, Inc.

Cross Domain Routing with ZeroMQ

November 2013

7 Confidential Information of Talksum, Inc.

“A territory over which rule or control is exercised”

“A sphere of activity, concern, or function”

- freedictionary.com

Domains

8 Confidential Information of Talksum, Inc.

•Energy

•Transportation

•Biomedical

•Information Systems

Some Domains!

9 Confidential Information of Talksum, Inc.

•Application Threads

•Application Processes

•Systems on a network

•Networks themselves

Some More Domains!

10 Confidential Information of Talksum, Inc.

ØMQ \zeromq\:

• A network socket library that acts as a concurreny framework

• Carries messages across multiple transports

• Enables networking patterns

• Provides Asyncronous I/O

• Large and active open source community

• Bindings for 40+ programming languages

• Works on most operating systems

What is ZeroMQ

11 Confidential Information of Talksum, Inc.

What is ZeroMQ

WHAT IF I TOLD YOU

WE CAN HAVE LOCK-FREE IN MEMORYMESSAGE PASSING BETWEEN THREADS?

Here there be dragons

12 Confidential Information of Talksum, Inc.

What Is ZeroMQ To Us

ZeroMQ is a tool which facilitates the construction of cross domain communication patterns

13 Confidential Information of Talksum, Inc.

When we can apply a pattern from one domain to a different domain, we can do more with less work.

Patterns!

14 Confidential Information of Talksum, Inc.

Network Monitoring & Optimization

Unix Logs – RFC3164 UDP/TCPNetflow – UDP – NG v.5, 8, 9, 10

Refined Data Stream

Refined Data Stream

Indexed, Mapped, ReducedOrdered, Sorted Data Streams

Bulk Data Streams(Lightly Ordered

and Filtered)

TalksumData Stream

Router(TDSR)

• Data Normalization• Parsers• Filters• Metrics and

Counts• Inline ETL/PTL• Asynchronous

Outputs• Protocol

Verification

Existing BI Tools

SystemLogs

NOC Alerting

• SQL Warehouse• Bulk Data Stores• File Storage

• Object Data Stores• Indexed Data Caches• NoSQL Data

Warehouses

Customer: Large European ISP/Email Communications ProviderUse Case: Ingest Netflow data, parse and aggregate in real time, monitors and alerts, optimize network topology Status: Deploying beta appliance

15 Confidential Information of Talksum, Inc.

Financial

XML, JSON, File, HTTP REST

Twitter, RSS, CAP (Weather Alerts)

Refined Data Stream

Refined Data Stream

Indexed, Mapped, ReducedOrdered, Sorted Data Streams

Bulk Data Streams(Lightly Ordered

and Filtered)

TalksumData Stream

Router(TDSR)

• Data Normalization• Parsers• Filters• Metrics and

Counts• Inline ETL/PTL• Asynchronous

Outputs• Protocol

Verification

Alerting & Notification

3rd Party DataTrading Desks

Social andPublic Data

Market Dashboard

• SQL Warehouse• Bulk Data Stores• File Storage

• Object Data Stores• Indexed Data Caches• NoSQL Data

Warehouses

Customer: Major Financial Stock ExchangeUse Case: Ingest unstructured financial market data, parse and filter for quality, aggregate, integrate with existing data warehouseStatus: Acquiring data sample for POC

16 Confidential Information of Talksum, Inc.

Request Reply

17 Confidential Information of Talksum, Inc.

Round Robin Load Balancing

18 Confidential Information of Talksum, Inc.

Unicast

19 Confidential Information of Talksum, Inc.

Multicast

20 Confidential Information of Talksum, Inc.

Fair Queuing

21 Confidential Information of Talksum, Inc.

Explicit Routing

22 Confidential Information of Talksum, Inc.

•Within a thread

• Between threads

• Between processes on the same system

• Between processes on different systems

Inter Domain Messaging

23 Confidential Information of Talksum, Inc.

Why Talksum Contributes to Zeromq

• We are all facing a lot of problems. • We believe in the value of empirical evidence.• Solutions emerge from all of us working together

across our various domain boundaries.• Participate!

24 Confidential Information of Talksum, Inc.

• http://www.zeromq.org/ - Main ZeroMQ Web Site

• http://zguide.zeromq.org/ - ZeroMQ User Guide

• http://github.com/zeromq - ZeroMQ Git Repository

• http://api.zeromq.org/ - ZeroMQ Reference Manual

• http://zeromq.org/whitepapers:architecture – Architecture Whitepaper

• http://czmq.zeromq.org/ - ZeroMQ C Binding

• http://zeromq.github.io/pyzmq/ - ZeroMQ Python Binding

Resources