+ All Categories
Home > Documents > Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka....

Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka....

Date post: 20-May-2020
Category:
Upload: others
View: 22 times
Download: 0 times
Share this document with a friend
45
Journey to a Real-Time Enterprise Neha Narkhede, Co-founder/CTO at Confluent, Co-Creator Apache Kafka
Transcript
Page 1: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

Journey to a Real-Time EnterpriseNeha Narkhede, Co-founder/CTO at Confluent, Co-Creator Apache Kafka

Page 2: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

Relational Database

Management Systems

DataWarehousing

Infrastructure Technology

?

Page 3: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

Adoption in Silicon Valley

Page 4: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

44

Adoption in Silicon Valley

Page 5: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

55

Adoption in the Enterprise

Page 6: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

Fortune 500’s are using Apache KafkaTM

Global Banks

Insurance

Telecom

Travel Companies

Page 7: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

Emergence of the Streaming Platform

Page 8: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

Pre-Streaming

Page 9: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

Request-Response Applications

Deterministic

Rigid

Tight coupling

App

Service

Service

Service Service

Service

Service

Service

Service

App

Page 10: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

App

App

Developer APIs

Service

Service

Event-Driven Applications

Responsive

Flexible

Extensible

Service

Service

Service

Streaming Platform

Page 11: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

Pre-Streaming -> Event-Driven

Request-Response Event-Driven

Page 12: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

1212

Message-Oriented Middlewhere

No persistence

Single point of failure

Not fault tolerant

Cannot order messages

Cannot process messaging in flight

Order of magnitude lower throughput

No “Replay” functionality

EAI & ESBs

Not event-oriented

Fragile and bespoke

Weak transformation capabilities

ETL

Often slow, batch oriented, and

non-scalable

Point-to-point not publish subscribe

Not a true infrastructure platform

Why Didn’t It Work Before? Past Solutions Are Insufficient

Page 13: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

Microservices Mobile Machine Learning

Internet of Things

The World has Changed

Page 14: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

What’s Needed? Event Centric Thinking

Page 15: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

Events

What is an event?

Page 16: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

Events

Page 17: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

Events

A Sale An Invoice A Trade A Customer Experience

Page 18: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

All Your Data is Streams of Events

Page 19: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

What is a Company?

A business is a series of events and reacting to those events.

Page 20: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

5.2 Million Citizens

Event-Driven Government Norwegian Work and Welfare Administration

Life is a Stream of Events

Page 21: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

The Future of the Automotive Industry is a Real Time Data Cluster

Front, rear and top view cameras

Ultrasonic SensorsCrash Sensors

Front Camera Infrared Camera

Front and Rear Radar Sensors

Traffic Alerts

Hazard Alerts

Personalization

Anomaly Detection

MQTT MQTTMQTT

MQTT MQTTMQTT

Page 22: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

Royal Bank of Canada Event-Driven Banking

30+ Use-cases

50+ apps

10+ different lines of businesses

Digital Marketing Security

Consumer Credit Services

SaaS

Corporate Real Estate

Investor Services

Treasury Services

….

FraudData Warehouse

Microservices

Page 23: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

Internet of Things

Page 24: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

Banking

Page 25: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

Retail

Page 26: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

What is a Streaming Platform?

Page 27: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

The Streaming Platform

Technical Capabilities

Store ProcessPublish & Subscribe

Page 28: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

Three Lenses

Page 29: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

Messaging done right.

Lens 01

Page 30: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

Way More Than Message QueueLens 01

True Storage Real-time ProcessingScalability

Messaging done right.

Page 31: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

Hadoop made fast.

Lens 02

Page 33: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

Lens 02

Applications are different

Hadoop made fast.

Page 34: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

ETL and Data Integrationas a platform.

Lens 03

Page 35: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

Lens 03

Scalable Streaming Data Pipelines

Page 36: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

Lens 03

Stream Processing is for more than data pipelines

ETL and Data Integrationas a platform.

Page 37: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

Streaming Platform

Page 38: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

Journey to an Event-Driven Enterprise

Page 39: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

Streaming Adoption Journey

Pre-StreamingStreaming Awareness and Pilot

Early Production Streaming

Mission Critical,

Integrated Streaming

Global Streaming

Central Nervous System

Page 40: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

What does the Event-Driven Architecture look like in its end state?

Page 41: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

An Event-Driven Enterprise

What are the possibilities?

● Everything is an event

● Available instantly to all applications in a company

● Ability to query data as it arrives vs when it is too late

● Simplifying the data architecture by deploying a single platform

Page 42: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

Management

RepresentingData

Connectors

Support

Apps

RDBMS

K/V

Monitoring

Search

DWH

HADOOP

StreamProcessing

Real-TimeAnalytics

Page 43: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

Management

RepresentingData

Connectors

Support

Apps

RDBMS

K/V

Monitoring

Search

DWH

HADOOP

StreamProcessing

Real-TimeAnalytics

An open streaming platform around Kafka and it’s ecosystem

Page 44: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

Apps

RDBMS

K/V

Monitoring

Search

DWH

HADOOP

StreamProcessing

Real-TimeAnalytics

Page 45: Co-Creator Apache Kafka Co-founder/CTO at Confluent, Neha ... · Co-Creator Apache Kafka. Relational Database Management Systems Data Warehousing ... Detection MQTT MQTT MQTT MQTT

Thank You


Recommended