Streaming Integration and Intelligence For Automating · MQTT Messaging Kafka JMS AMQP Big Data...

© 2018 Striim, Inc. All rights reserved.

Streaming Integration and Intelligence

For Automating

“Time Sensitive” Events Ted Fish

Director Sales, Midwest

[email protected]

312-330-4929

mailto:[email protected]


Founded Founded in 2012 by leaders of GoldenGate Software and BEA/WebLogic

Lead investors Backed by leading investors: Summit Partners, Intel Capital, Atlantic Bridge, & Dell

Customers Deployments in financial services, telco, healthcare, retail, IoT

Striim Executive Summary

Delivering Data for “Time Sensitive” Processes & Decisions


Sample Customers

Transportation & Logistics

Telco, Manufacturing

Financial Services

Retail, High Tech/IoT


Striim Awards @ 2017 Strata Data Conference

“Best Big Data Technology for Real-Time Analytics” / “Top 5 Vendors to Watch” / “Best IoT Platform”

https://www.datanami.com/this-just-in/striim-wins-two-datanami-readers-editors-choice-awards/






















“Time Sensitive” Processes Across All Industries – What are Yours?

Financial Services - Anti-money laundering - Fraud prevention - Risk management - VIP customer service

Healthcare - Proactive illness detection - Staff allocation optimization - Point of care compliance - Eligibility verification

Manufacturing - Quality management - Predictive maintenance - Equipment monitoring - Capacity optimization

Retail - Security Threats Detection - Real-time offers - Geo-targeted marketing - Dynamic pricing

Communications - Network health monitoring, - Predict network failures - Proactive service outreach - Location-based advertising

Transportation/Logistics - Connected car - Predictive maintenance - Asset tracking - Route optimization

Public Sector - Crime detection and prevention - Cyber security - Traffic management - Connect City

IT - Cyber security - Replication validation - API usage monitoring - SLA monitoring

Insurance - Claim fraud detection - Agent fraud detection - Risk-based policy pricing - Agency performance - Usage-based insurance


Challenge: Supporting Time-Sensitive Decisions, Processes, Insights

Devices, Mobile, IoT

Data is

New

Websites

Network

Applications & Databases

Data is

Old

Messaging

Big Data

Cloud

Files

Data

Warehouse

ETL & Batch

Legacy

Replication & CDC

Messaging

Integration

Options

Customers

Orders / Sales

Inventory

Security

Process Automation

Lack of Data

for

Real-Time Events


How Striim Transforms Your Organization

Devices, Mobile, IoT

Data is

New

Websites

Network

Applications & Databases

Data is

Old

Messaging

Big Data

Cloud

Files

Data

Warehouse

End-To-End Solution

7/24 Enterprise Grade Platform

In-Memory Streaming

Integration & Intelligence

Customers

Orders

Inventory

Security

Safety

Time

Sensitive

Events

“Before the Data Lands”

Automate Processes, Deliver Insight,

Make Timely Decisions


Striim: Fast Data, In-Flight Intelligence, Fast Decisions

Streaming

Integration

External

Context

Filtering

Transformation

Enrichment

Aggregation

Databases

Log files

Sensors

IoT

Any Messaging,

Built-In Kafka

Data is

New

Customers

Orders / Sales

Inventory

Security

Process Automation

Time Sensitive

Decisions & Processes

Fast Deliver to

Anywhere

Messaging

Big Data

& NOSQL

Cloud

Files

Data

Warehouse

Str

ea

min

g D

ata

De

live

ry

Real-Time

Insights &

Action Alerts Triggers

Real-Time Visualization

Machine

Learning/ AI

Models

Streaming

Intelligence

Anomaly

Detection

Pattern

Matching

Multi-Stream

Correlation

Complex Event

Rules

CDC

Str

ea

min

g E

ve

nts

Edge


Visualization & Drilldowns Through Streaming Dashboards


Real-Time Treat/Fraud Security Monitoring – Largest Credit Card Company

Correlate Logs From Multiple Security Products to Identify Cross-Domain

Issues or Exploits that are not Obvious from a Single Security Product

• Source Logs In Real-Time from Multiple Security Products

• Ingests and Analyzes security log & session data, capturing all events from 50+ siloed security

solutions

• Looks for Patterns of Activity Across Logs that Indicate Exploits or Anomalies

• Provide Real-Time Monitoring Dashboard and Immediate Alerts

blacklist locations


Continuous Data Collection

Stream Processing

Streaming Analytics

Continuous Results Delivery

End-to-End: In-Memory – Integration – Intelligence – Kafka - Hybrid

Parsers Delimited

JSON

XML

Free Text

Binary

Name/Value

Zipped

AVRO

OGG Trail

Apache Log

Sys Log

MS Event Log

Mail Log

SNMP

CollectD

CEF

DHCP Log

WCF

+Others

Databases

JDBC/SQL

Oracle CDC

MS/SQL CDC

MySQL CDC

HPE NSK

Salesforce

Files

Log Files

System Files

Batch Files

Network

TCP

UDP

HTTP

MQTT

Netflow

PCAP

Messaging

Kafka

Flume

JMS

AMQP

Big Data

HDFS

Hbase

Hive

RESTful API

Sources

Formats Delimited

JSON

XML

Template

AVRO

Targets

Databases

JDBC/SQL

Oracle

MS/SQL

MySQL

Teradata

Files

Network

MQTT

Messaging

Kafka

JMS

AMQP

Big Data

HDFS

Hbase

Hive

Hazelcast

Cloud

Azure Blob

Azure SQL DB

Amazon S3

Amazon Redshift

Google Big Query

Alerting Email

SMS

Operating Systems

Virtual Machines Big Data

Cloud

IoT Gateways


UD

Fs

Striim “Next Generation” Architecture

RDBMS Generic JDBC/SQL

Oracle CDC MS/SQL CDC

HPE NonStop CDC

Network TCP

UDP HTTP

Files CSV/TSV

JSON XML

Apache Avro, Free-form

BigData HDFS Hive

Message Queues Kafka/ Flume

JMS

Sources & Parsers Applications

DB Persistence JDBC/SQL

Oracle MS/SQL, Teradata

HPE NonStop

File Persistence CSV/TSV

JSON XML

Message Queues JMS

Kafka

BigData HDFS

Hbase, Hive

Alerting Email

SMS

So

urc

es

Str

eam

s

Win

do

ws

Queries

Ca

ch

es

Ta

rge

ts

Targets & Formatters

Business-Level Logic

With (extended) SQL

Real-time

Dashboards

Continuous Query Processor

External Context

Scalable IMC Cluster

Cloud Google BigQuery

MS Azure SQL, AWS Redshift

Distributed In-Memory Store

Kafka Streams (optional)

Distributed Indexed Store (ES)

Distributed In-Memory Cache

Node n Node 1 Node 2 Node 3


Kafka Enhancements – Integration, Performance, Support

• Kafka built-in - for persisting data and performance

• Continuous collection - data and deliver to Kafka

• Intelligence - on streaming Kafka data

• Visualization and Drill Downs - on streaming data

• SQL Queries against Kafka, without coding

• Performance - each writer dynamically processes

24 parallel threads

• Enterprise Grade - Scalability, Reliability, Security

• Exactly-Once-Processing (E1P) from sources to targets

Performance, Scalability, Security, Easier Manageability


Built-In “CDC” Change Data Capture / Replication Database to Database

Oracle MS SQL Server HPE NonStop MySQL

CDC

• On-Premise Database to On or Off Premise Databases • Read current Base Table to perform Initial Load to Target Data Tables

• Start change data capture on Source database to read transaction logs

• Striim Transforms Change Records to DML Operation and Applies changes to DBMS destinations

• Add Kafka Persistent Streams to add Mission Critical E1P reliability for fault tolerant replication

“Striim Leveraged their Golden Gate Development Expertise”


IoT Projects – Exploits Striim’s Platform

Cloud

HU

B

Machine Learning

Protocol Translation

Exported ML Model

Protocol Translation

Exported ML Model

Continuous Real-Time Processing Intelligence at the EDGE

Visualization & Automation


Machine Learning and AI – Operationalize with Seamless Integration

Collect Data to Train Model and Use Model in Striim to Perform Real-Time Anomaly Detection or Predictions • Source Data From Databases, Files, Kafka, etc.

• Process and Prepare the Data and Write to Disk to Train Machine Learning

• Export Trained Model and Use in Streaming Analytics Real-Time Scoring

• Present Results on Dashboard and Alert on Anomalies

Server

Real-Time

Dashboard

Training Files

Process & Prepare Data

Machine Learning

Exported Model

Wrapper Function

Streaming Analytics

CQ Real-Time

Scoring


Application Development Environment - Robust & Flexible

• End-to-end integrated application development

• Visual dashboard designer

• Visual application designer (Flow Designer)

• Application templates

• SQL-like programming interface

• Source data preview

• Live / ad-hoc query & parameterized query interfaces

• Predictive analytics

Applications Visual designer

Business-Level Logic

With Tungsten QL (extended SQL)


• Clustering - Multiple node use cases – Architecture

• Hybrid / Public / Private Cloud

• Striim Agents

• Mesh network

• Democratic

– Scale out

• Memory on one node not enough for holding windows/caches

• Processing power on one node not enough to process the number of incoming events

• Consistent hashing and distribution of data stream, caches, results caches,

• Co-location of data and processing

• Dynamic resizing

– Availability

• Guarding against node outages

• Guarding against errors on nodes

• Dynamic resizing

Built-In Enterprise Operations / Clustering, HA, Fault Tolerant

…

Collection Agents

Processing Cluster

Events Partitioned Over Cluster


Built-In Enterprise Operations / Scalability

• Cluster scales horizontally on commodity hardware

• Lightweight agents for edge collection of data

• Consistent data partitioning

• Scalable indexed results store

0

10

20

2 12 24 32 48

Event Rate Per Servers

Rate M Events/s

…

Collection Agents

Processing Cluster

Events Partitioned Over Cluster


THANK YOU

Date post:	28-Jun-2018
Category:	Documents
Upload:	lyxuyen
View:	220 times
Download:	0 times

Streaming Integration and Intelligence For Automating · MQTT Messaging Kafka JMS AMQP Big Data...

Documents