+ All Categories
Home > Data & Analytics > Kafka & Couchbase Integration Patterns

Kafka & Couchbase Integration Patterns

Date post: 06-Apr-2017
Category:
Upload: manuel-hurtado
View: 223 times
Download: 1 times
Share this document with a friend
30
Kafka & Couchbase Integration Patterns Manuel Hurtado Couchbase Solutions Engineer
Transcript
Page 1: Kafka & Couchbase Integration Patterns

Kafka & Couchbase Integration Patterns

Manuel HurtadoCouchbase Solutions Engineer

Page 2: Kafka & Couchbase Integration Patterns

©2016 Couchbase Inc. 2

Agenda

• Couchbase Introduction• Kafka Introduction• Kafka Connect & Couchbase Kafka Connector• Use cases

Page 3: Kafka & Couchbase Integration Patterns

©2016 Couchbase Inc. 3

Agenda

• Couchbase Introduction• Kafka Introduction• Kafka Connect & Couchbase Kafka Connector• Use cases

Page 4: Kafka & Couchbase Integration Patterns

©2016 Couchbase Inc. 4

What is Couchbase?

Couchbase delivers the Data Platform for the Digital Economy• Products: Couchbase Server & Couchbase Mobile• Open source NoSQL, JSON document database• Founded 2010• 500+ enterprise customers, including 20+ Fortune 100

UNIFIED ADMINISTRATION

UNIFIED PROGRAMMING INTERFACE

Data Query Index SearchMobileReplication Analytics

{N1QL}

Page 5: Kafka & Couchbase Integration Patterns

©2016 Couchbase Inc. 5

Growing number of use cases

5

Catalog Metadata

Operational Dashboarding

User Profile Database Session Database Inventory &

AvailabilityEntitlement

Management

Field Service EnablementCustomer 360

Asset/ResourceManagement

Device User Data Management

Endpoint Data Management

Page 6: Kafka & Couchbase Integration Patterns

©2016 Couchbase Inc. 6

Why customers choose Couchbase?

6

Memory-first Architecture

Full SQL Query Language

Active-Active Global Data Replication

Multi-dimensional scaling

Mobile

Page 7: Kafka & Couchbase Integration Patterns

©2016 Couchbase Inc. 7

Agenda

• Couchbase Introduction• Kafka Introduction• Kafka Connect & Couchbase Kafka Connector• Use cases

Page 8: Kafka & Couchbase Integration Patterns

©2016 Couchbase Inc. 8

Kafka Introduction

What is Kafka?

• Kafka is a distributed, partitioned, replicated, log service developed by LinkedIn and open sourced in 2011.

• Basically it is a massively scalable pub/sub message queue architected as a distributed transaction log. It was created to provide “a unified platform for handling all the real-time data feeds a large company might have”.

• It powers a large number of high-profile companies including LinkedIn, Yahoo and Netflix.

Page 9: Kafka & Couchbase Integration Patterns

©2016 Couchbase Inc. 9

Kafka Architecture

Page 10: Kafka & Couchbase Integration Patterns

©2016 Couchbase Inc. 10

Kafka Architecture

Page 11: Kafka & Couchbase Integration Patterns

©2016 Couchbase Inc. 11

Agenda

• Couchbase Introduction• Kafka Introduction• Kafka Connect & Couchbase Kafka Connector• Use cases

Page 12: Kafka & Couchbase Integration Patterns

©2016 Couchbase Inc. 12

The challenge: Stream Data Platform with Kafka

Source: https://www.confluent.io/blog/announcing-kafka-connect-building-large-scale-low-latency-data-pipelines/

Page 13: Kafka & Couchbase Integration Patterns

©2016 Couchbase Inc. 13

Kafka Connect

Source: https://www.confluent.io/blog/announcing-kafka-connect-building-large-scale-low-latency-data-pipelines/

Page 14: Kafka & Couchbase Integration Patterns

©2016 Couchbase Inc. 14

Kafka Connect

Source: https://www.confluent.io/blog/announcing-kafka-connect-building-large-scale-low-latency-data-pipelines/

Page 15: Kafka & Couchbase Integration Patterns

©2016 Couchbase Inc. 15

Couchbase Connector in Kafka

Page 16: Kafka & Couchbase Integration Patterns

©2016 Couchbase Inc. 16

Database Change Protocol (DCP)

Page 17: Kafka & Couchbase Integration Patterns

©2016 Couchbase Inc. 17

Agenda

• Couchbase Introduction• Kafka Introduction• Kafka Connect & Couchbase Kafka Connector• Use cases

Page 18: Kafka & Couchbase Integration Patterns

©2016 Couchbase Inc. 18

Couchbase & Kafka Use Cases

Page 19: Kafka & Couchbase Integration Patterns

©2016 Couchbase Inc. 19

Couchbase & Kafka Use Cases

Currently using Kafka, but not Couchbase

Reading from Couchbase with almost no code (Source)○ Adopt default message format given by connector○ Write custom converter○ Use Kafka Streams to process events from Couchbase, and write

them back.

Writing to Couchbase without touching SDK (Sink)○ Provide document in the generic form

(it will be converted into JSON)

Page 20: Kafka & Couchbase Integration Patterns

©2016 Couchbase Inc. 20

Couchbase & Kafka Use Cases

Currently using Couchbase, but not Kafka

Backup of the data○ Need to be filtered/analyzed/transformed by the business logic○ Checking integrity

Turn realtime data into history○ e.g. Couchbase keeps the exchange rates, but it is necessary to emit

stream of the deltas, or log of the changes.

Page 21: Kafka & Couchbase Integration Patterns

©2016 Couchbase Inc. 21

Couchbase & Kafka Use Cases

Currently using Kafka and Couchbase

Integrate more naturally with Kafka infrastructure.A lot of the custom code may be moved into connector.

○ Filtering○ Transformation○ Process/Task management

Page 22: Kafka & Couchbase Integration Patterns

©2016 Couchbase Inc. 22

Page 23: Kafka & Couchbase Integration Patterns

©2016 Couchbase Inc. 23

Demo: Couchbase as Data Source Basic Scenario

Console C

onsumer

• OOTB Couchbase Connector• Bucket “travel-sample” • Output all-content• Consume messages to console• Insert/Update/Delete• JMX Monitor

Page 24: Kafka & Couchbase Integration Patterns

©2016 Couchbase Inc. 24

Demo: Couchbase as Data Source + Data Sink Basic Scenario

• OOTB Couchbase Connector• Bucket “travel-sample” to “receiver”• Raw content• Write messages to Couchbase• JMX Monitor

Page 25: Kafka & Couchbase Integration Patterns

©2016 Couchbase Inc. 25

Demo: Couchbase as Data Source + Data Sink Custom Scenario

• Custom Schema Converter• Custom Filter• “travel-sample” to “receiver”: only “airlines”• JSON content• Write messages to Couchbase• JMX Monitor

AirlineConverter

AirlineFilter

Page 26: Kafka & Couchbase Integration Patterns

©2016 Couchbase Inc. 26

Demo: Twitter Data Source + Couchbase as Data Sink

• Custom Kafka Producer & Consumer• Bucket “twitter”• Write messages to Couchbase• JMX Monitor

Twitter Producer

Couchbase

Consum

er

Page 27: Kafka & Couchbase Integration Patterns

©2016 Couchbase Inc. 27

Couchbase Connector in Confluent Platform

Page 28: Kafka & Couchbase Integration Patterns

©2016 Couchbase Inc. 28

Confluent Platform

Page 29: Kafka & Couchbase Integration Patterns

©2016 Couchbase Inc. 29

Resources

• Couchbase Kafka Connectorhttps://developer.couchbase.com/documentation/server/current/connectors/kafka-3.1/kafka-intro.html

• Couchbase Kafka Connector (code)https://github.com/couchbase/kafka-connect-couchbase

• Demo code repositorieshttps://github.com/mahurtado/KafkaCouchbaseConnectorSamplehttps://github.com/mahurtado/TwitterKafkaCouchbasePipeline

• Kafka Connect Confluenthttps://www.confluent.io/product/connectors/

Page 30: Kafka & Couchbase Integration Patterns

Q & A


Recommended