BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF
HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH
Ereignis-getriebene Microservices mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018
@gschmutz guidoschmutz.wordpress.com
Guido Schmutz
Working at Trivadis for more than 21 years
Oracle ACE Director for Fusion Middleware and SOA
Consultant, Trainer Software Architect for Java, Oracle, SOA and
Big Data / Fast Data
Head of Trivadis Architecture Board
Technology Manager @ Trivadis
More than 30 years of software development experience
Contact: [email protected]
Blog: http://guidoschmutz.wordpress.com
Slideshare: http://www.slideshare.net/gschmutz
Twitter: gschmutz
Building event-driven Microservices with Kafka Ecosystem
COPENHAGEN
MUNICH
LAUSANNE
BERN
ZURICH BRUGG
GENEVA
HAMBURG
DÜSSELDORF
FRANKFURT
STUTTGART
FREIBURG
BASEL
VIENNA
With over 600 specialists and IT experts in your region.
14 Trivadis branches and more than
600 employees
200 Service Level Agreements
Over 4,000 training participants
Research and development budget:
CHF 5.0 million
Financially self-supporting and
sustainably profitable
Experience from more than 1,900
projects per year at over 800
customers
Building event-driven Microservices with Kafka Ecosystem
Agenda
1. Where do we come from?
2. What are Microservices?
3. Why not more Event Driven?
4. Why Kafka for Event-Driven Microservices?
5. What about streaming sources?
6. What about integrating legacy applications?
7. What about (historical) data analytics?
8. Summary
Building event-driven Microservices with Kafka Ecosystem
Building event-driven Microservices with Kafka Ecosystem
Where do we come from?
Shop Rich UI
Shop Backend Application
Traditional Approach
Search Facade
Customer DAO
Order DAO
Order Facade
Shop UI
Product DAO
UI Logic
Data Business
GUI
Customer Fat Client App
Customer BO Customer UI
Data GUI
Data Storage
Shared Database
Building event-driven Microservices with Kafka Ecosystem
sync request/response
Shop UI App
Business Activity Service
SOA Approach
Building event-driven Microservices with Kafka Ecosystem
Contract-first
Web Services
Technical layers
offer their own
interfaces
Reuse on each
level
Lower layer
often wraps
legacy code
Search BAS
Customer DAO
Order DAO
Order BAS
Shop UI
Product DAO
UI Logic
GUI
Business Entity Service Shop Web App
Shop UI UI Logic
GUI
Data Storage
Customer Database
Customer BES
Payment BES
Product BES
Order BES
Custer BAS
Order and Product DB
SOAP
SOAP
SOAP SOAP
SOAP
SOAP
SOAP
Shop UI App
Business Activity Service
Virtualized SOA Approach
Search BAS Customer DAO
Order DAO
Order BAS
Shop UI UI Logic
GUI
Business Entity Service
Shop Web App
Shop UI UI Logic
GUI
Data Storage
Customer Database
Customer BES
Payment BES
Product BES
Order BES
Custer BAS
Order and Product DB
Service Virtualization Layer
Service Bus
SOAP SOAP
SOAP
SOAP
SOAP
SOAP
SOAP
Building event-driven Microservices with Kafka Ecosystem
Building event-driven Microservices with Kafka Ecosystem
What are Microservices?
Microservice Approach
Building event-driven Microservices with Kafka Ecosystem
Tightly Scoped behind clear
interfaces
Responsible for managing their
own data (not necessarily the
infrastructure)
Should be highly decoupled
Independently deployable, self-
contained and autonomous
SOA done right ?!
Tightly Scoped Responsible for managing their own data Highly decoupled Independently deployable, self-contained and autonomous
{ }
Customer API Customer
Customer Logic
Order Microservice
{ }
Order API Order
Order Logic
Product Microservice
{ }
Product API Product
Product Logic
Stock Microservice
{ }
Stock API Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST
Microservice Approach
with API Gateway
Customer Microservice
{ }
Customer API Customer
Customer Logic
Order Microservice
{ }
Order API Order
Order Logic
Product Microservice
{ }
Product API Product
Product Logic
Stock Microservice
{ }
Stock API Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST
API Gateway
Building event-driven Microservices with Kafka Ecosystem
Synchronous World of Request-Response leads to tight,
point-to-point couplings
Building event-driven Microservices with Kafka Ecosystem
problem in lower end of chain have a ripple
effect on other service • crash of service
• overloaded service / slow response time
• change of interface
Service 2 Service 1
{ }
API Logic
{ }
API Logic
State State
Service 3
{ }
API Logic
State
Service 4
{ }
API Logic
State
Service 5
{ }
API Logic
State
Service 7
{ }
API Logic
State
Service 6
{ }
API Logic
State
Building event-driven Microservices with Kafka Ecosystem
Why not more Event-Driven?
Three mechanisms through which services can interact
Request-Driven (Imperative) Event Driven (Functional)
Service
Logic
State
Consume Event “Event Ordered”
OrderEvent
Command ”Order IPad”
boolean order(IPad)
Publish Event ”Event Validated”
OrderValidatedEvent Query
”Retrieve my Orders List<Orders> getAllOrders(for)
Building event-driven Microservices with Kafka Ecosystem
Event Broker
Service
Logic
State
Event-Driven (Async)
Microservice Approach
Customer Microservice
{ }
Customer API Customer
Customer Logic
Order Microservice
{ }
Order API Order
Order Logic
Product Microservice
{ }
Product API Product
Product Logic
Stock Microservice
{ }
Stock API Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST
API Gateway
Building event-driven Microservices with Kafka Ecosystem
Event Store
sync request/response
async request/response
async, event pub/sub
Why Kafka for Event-Driven Microservices?
Building event-driven Microservices with Kafka Ecosystem
Apache Kafka – A Streaming Platform
Building event-driven Microservices with Kafka Ecosystem
High-Level Architecture
Distributed Log at the Core
Scale-Out Architecture
Logs do not (necessarily) forget
Hold Data for Long-Term – Data Retention
Producer 1
Broker 1
Broker 2
Broker 3
1. Never
2. Time based (TTL) log.retention.{ms | minutes | hours}
3. Size based log.retention.bytes
4. Log compaction based
(entries with same key are removed):
kafka-topics.sh --zookeeper zk:2181 \
--create --topic customers \
--replication-factor 1 \
--partitions 1 \
--config cleanup.policy=compact
Building event-driven Microservices with Kafka Ecosystem
Keep Topics in Compacted Form
0 1 2 3 4 5 6 7 8 9 10 11
K1 K2 K1 K1 K3 K2 K4 K5 K5 K2 K6 K2
V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11
Offset
Key
Value
3 4 6 8 9 10
K1 K3 K4 K5 K2 K6
V4 V5 V7 V9 V10 V11
Offset
Key
Value
Compaction
Building event-driven Microservices with Kafka Ecosystem
V1
V2
V3 V4
V5
V6
V7
V8 V9
V10
V11
K1
K3
K4
K5
K2
K6
Topic Viewed as Event Stream or State Stream (Change
Log)
Event Stream State Stream (Change Log Stream)
2015-10-02
11,Peter,Muster,3010,Berne
2016-10-04
12,Paul,Steffen,8001,Zurich
2016-12-02
21,Lisa,Meier,3043,Ittigen
2017-05-03
11,Peter,Muster,3015,Berne
2017-05-03
21,Lisa,Steffen,8001,Zurich
2017-07-03
11,Peter,Muster,3052,Zollikofen
Building event-driven Microservices with Kafka Ecosystem
2015-10-02
11,Peter,Muster,3010,Berne
2016-10-04
12,Paul,Steffen,8001,Zurich
2016-12-02
21,Lisa,Meier,3043,Ittigen
2017-05-03
11,Peter,Muster,3015,Berne
2017-05-03
21,Lisa,Steffen,8001,Zurich
2017-07-03
11,Peter,Muster,3052,Zollikofen
Keep Topics both in Original and Compacted Form
Building event-driven Microservices with Kafka Ecosystem
Kafka Broker A B C B A E K A E A
C K
B E A C K
Producer
Kafka Broker A B C B A E K A E A
C K
B E A C K
Producer
Consume and Produce
OR
TX
What about streaming sources?
Building event-driven Microservices with Kafka Ecosystem
How to work with
streaming data sources
Customer Microservice
{ }
Customer API Customer
Customer Logic
Order Microservice
{ }
Order API Order
Order Logic
Product Microservice
{ }
Product API Product
Product Logic
Stock Microservice
{ }
Stock API Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST
Event Store
Location
Social
Click stream
Sensor Data
Mobile Apps
Weather Data
Event Stream
Building event-driven Microservices with Kafka Ecosystem
Hadoop Clusterd Hadoop Cluster
Stream Processing Cluster
Streaming Processing & Microservices Architecture BI Tools
Search / Explore
Online & Mobile Apps
Search
Event Stream
Results Stream Analytics
Reference / Models
Dashboard
Location
Social
Click stream
Sensor Data
Mobile Apps
Weather Data
Microservice Cluster
Microservice State
{ }
API
Event Stream
Event Stream
Event Store
Building event-driven Microservices with Kafka Ecosystem
Stream Processing Capability - Windowing
Introduction to Stream Processing
Computations over events done using
windows of data
Due to size and never-ending nature of it,
it’s not feasible to keep entire stream of data in memory
A window of data represents a certain
amount of data where we can perform
computations on
Windows give the power to keep a
working memory and look back at recent
data efficiently
Time
Stream of Data Window of Data
Sliding Window (aka Hopping Window) - uses eviction and trigger policies that are based on time: window length and sliding interval length
Fixed Window (aka Tumbling Window) - eviction policy always based on the window being full and trigger policy based on either the count of items in the window or time
Session Window – composed of sequences of temporarily related events terminated by a gap of inactivity greater than some timeout
Stream Processing Capability - Windowing
Introduction to Stream Processing
Time Time Time
Stream Processing Capability - Joining
Introduction to Stream Processing
Challenges of joining streams
1. Data streams need to be aligned as they
come because they have different timestamps
2. since streams are never-ending, the joins
must be limited; otherwise join will never end
3. join needs to produce results continuously as
there is no end to the data
Stream to Static (Table) Join
Stream to Stream Join (one window join)
Stream to Stream Join (two window join)
Stream-to-Static
Join
Stream-to-Stream
Join
Stream-to-Stream
Join
Time
Time
Time
Stream Processing Capability - State Management
Introduction to Stream Processing
Necessary if stream processing use case
is dependent on previously seen data or
external data
Windowing, Joining and Pattern Detection use State Management behind
the scenes
State Management services can be made
available for custom state handling logic
State needs to be managed as close to
the stream processor as possible
Options for State Management
How does it handle failures? If a machine
crashes and the/some state is lost?
In-Memory Replicated, Distributed
Store
Local, Embedded
Store
Operational Complexity and Features
Low high
Apache Kafka – scalable message processing and
more!
Building event-driven Microservices with Kafka Ecosystem
Source Connector
trucking_ driver
Kafka Broker
Sink Connector
Stream Processing
Schema Registry
Kafka Kafka
Kafka Connect - Overview
Source Connector
Sink Connector
Introduction to Stream Processing
Kafka Streams - Overview
Introduction to Stream Processing
Designed as a simple and lightweight library in Apache Kafka
no other dependencies than Kafka
Supports fault-tolerant local state
Supports Windowing (Fixed, Sliding and
Session) and Stream-Stream / Stream-Table Joins
Millisecond processing latency, no
micro-batching
At-least-once and exactly-once
processing guarantees
KTable<Integer, Customer> customers = builder.stream(”customer"); KStream<Integer, Order> orders = builder.stream(”order"); KStream<Integer, String> enriched = orders.leftJoin(customers, …); joined.to(”orderEnriched");
trucking_ driver
Kafka Broker
Java Application
Kafka Streams
KSQL - Overview
Introduction to Stream Processing
STREAM and TABLE as first-class
citizens
• STREAM = data in motion
• TABLE = collected state of a stream
Stream Processing with zero coding
using SQL-like language
Built on top of Kafka Streams
Interactive (CLI) and headless (command
file)
ksql> CREATE STREAM customer_s \ WITH (kafka_topic=‘customer', \ value_format=‘AVRO'); Message ---------------- Stream created ksql> SELECT * FROM customer_s \ WHERE address->country = ‘Switzerland’; ...
trucking_ driver
Kafka Broker
KSQL Engine
Kafka Streams
KSQL CLI Commands
What about integrating legacy applications?
Building event-driven Microservices with Kafka Ecosystem
Data Store
Integrate existing systems through CDC
Customer
Event Store
Integration
Microservice
State Logic
CDC CDC Connector
Customer Fat Client App
Customer BO Customer UI
Stream Processing
Results Stream Processor
Reference / Models
Dashboard
Capture changes directly on database
Change Data Capture (CDC) => think like a
global database trigger
Transform existing systems to event
producer
Building event-driven Microservices with Kafka Ecosystem
Legacy Microservice
Change Data Capture (CDC) with Kafka
Building event-driven Microservices with Kafka Ecosystem
RDBMS cdc-
source trucking_
driver
Customer Topic elasticsearch
-sink NoSQL
Hadoop Clusterd Hadoop Cluster
Stream Processing Cluster
Integrate existing systems through CDC BI Tools
Search / Explore
Online & Mobile Apps
Search
Event Stream
Results Stream Processor
Reference / Models
Dashboard Location
Social
Click stream
Sensor Data
Mobile Apps
Weather Data
Microservice Cluster
Microservice State
{ }
API
Event Stream
Event Stream
Event Store
Billing & Ordering
CRM / Profile
Marketing Campaigns
Building event-driven Microservices with Kafka Ecosystem
What about (historical) data analytics?
Building event-driven Microservices with Kafka Ecosystem
Streaming & (Big) Data Analytics Architecture
Event Stream
Hadoop Clusterd Hadoop Cluster
Big Data Cluster
Parallel Processing
Storage
Storage
Ra
w
Re
fin
ed
Results
Microservice Cluster
Microservice State
{ }
API
Stream Processing Cluster
Stream Processor
State
{ }
API
Event Stream
Event Stream
SQL
Search
BI Tools
Enterprise Data Warehouse
Search / Explore
Online & Mobile Apps
Event Store
Service
Location
Social
Click stream
Sensor Data
Mobile Apps
Weather Data
Billing & Ordering
CRM / Profile
Marketing Campaigns
File Import / SQL Import
Building event-driven Microservices with Kafka Ecosystem
Summary
Building event-driven Microservices with Kafka Ecosystem
Hadoop Clusterd Hadoop Cluster
Big Data
Summary
Billing & Ordering
CRM / Profile
Marketing Campaigns
SQL
Search
BI Tools
Enterprise Data Warehouse
Search / Explore
Online & Mobile Apps
File Import / SQL Import
Event Hub
Parallel Processing
Storage
Storage
Ra
w
Re
fin
ed
Results
Microservice State
{ }
API
Stream Processor
State
{ }
API
Event Stream
Event Stream
Service
Location
Social
Click stream
Sensor Data
Mobile Apps
Weather Data
Stream Processing
Microservices
Summary
Building event-driven Microservices with Kafka Ecosystem
• service autonomy is key in a Microservices Architecture!
• not all communication need to be synchronous => separate into
• commands
• events
• queries
• Kafka is well suited as an event broker / event store
• brings many more interesting features beyond just “message passing”
References
Building event-driven Microservices with Kafka Ecosystem
Microservices Blog Series, Ben Stopford, Confluent:
• https://www.confluent.io/blog/tag/microservices
Apache Kafka for Microservices: A Confluent Online Talk Series:
• https://www.confluent.io/landing-page/microservices-online-talk-series/
Turning the database inside-out with Apache Samza, Martin Kleppmann, Con
• https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/
Event sourcing, CQRS, stream processing and Apache Kafka: What’s the connection?, Neha Narkhede, Confluent: • https://www.confluent.io/blog/event-sourcing-cqrs-stream-processing-apache-kafka-whats-connection/
Immutability Changes Everything, Pat Helland, Salesforce:
• http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf
Commander: Better Distributed Applications through CQRS and Event Sourcing, Bobby Calderwood:
• https://www.youtube.com/watch?v=B1-gS0oEtYc
Technology on its own won't help you. You need to know how to use it properly.
Building event-driven Microservices with Kafka Ecosystem