Date post: | 06-Apr-2017 |
Category: |
Data & Analytics |
Upload: | manuel-hurtado |
View: | 223 times |
Download: | 1 times |
Kafka & Couchbase Integration Patterns
Manuel HurtadoCouchbase Solutions Engineer
©2016 Couchbase Inc. 2
Agenda
• Couchbase Introduction• Kafka Introduction• Kafka Connect & Couchbase Kafka Connector• Use cases
©2016 Couchbase Inc. 3
Agenda
• Couchbase Introduction• Kafka Introduction• Kafka Connect & Couchbase Kafka Connector• Use cases
©2016 Couchbase Inc. 4
What is Couchbase?
Couchbase delivers the Data Platform for the Digital Economy• Products: Couchbase Server & Couchbase Mobile• Open source NoSQL, JSON document database• Founded 2010• 500+ enterprise customers, including 20+ Fortune 100
UNIFIED ADMINISTRATION
UNIFIED PROGRAMMING INTERFACE
Data Query Index SearchMobileReplication Analytics
{N1QL}
©2016 Couchbase Inc. 5
Growing number of use cases
5
Catalog Metadata
Operational Dashboarding
User Profile Database Session Database Inventory &
AvailabilityEntitlement
Management
Field Service EnablementCustomer 360
Asset/ResourceManagement
Device User Data Management
Endpoint Data Management
©2016 Couchbase Inc. 6
Why customers choose Couchbase?
6
Memory-first Architecture
Full SQL Query Language
Active-Active Global Data Replication
Multi-dimensional scaling
Mobile
©2016 Couchbase Inc. 7
Agenda
• Couchbase Introduction• Kafka Introduction• Kafka Connect & Couchbase Kafka Connector• Use cases
©2016 Couchbase Inc. 8
Kafka Introduction
What is Kafka?
• Kafka is a distributed, partitioned, replicated, log service developed by LinkedIn and open sourced in 2011.
• Basically it is a massively scalable pub/sub message queue architected as a distributed transaction log. It was created to provide “a unified platform for handling all the real-time data feeds a large company might have”.
• It powers a large number of high-profile companies including LinkedIn, Yahoo and Netflix.
©2016 Couchbase Inc. 9
Kafka Architecture
©2016 Couchbase Inc. 10
Kafka Architecture
©2016 Couchbase Inc. 11
Agenda
• Couchbase Introduction• Kafka Introduction• Kafka Connect & Couchbase Kafka Connector• Use cases
©2016 Couchbase Inc. 12
The challenge: Stream Data Platform with Kafka
Source: https://www.confluent.io/blog/announcing-kafka-connect-building-large-scale-low-latency-data-pipelines/
©2016 Couchbase Inc. 13
Kafka Connect
Source: https://www.confluent.io/blog/announcing-kafka-connect-building-large-scale-low-latency-data-pipelines/
©2016 Couchbase Inc. 14
Kafka Connect
Source: https://www.confluent.io/blog/announcing-kafka-connect-building-large-scale-low-latency-data-pipelines/
©2016 Couchbase Inc. 15
Couchbase Connector in Kafka
©2016 Couchbase Inc. 16
Database Change Protocol (DCP)
©2016 Couchbase Inc. 17
Agenda
• Couchbase Introduction• Kafka Introduction• Kafka Connect & Couchbase Kafka Connector• Use cases
©2016 Couchbase Inc. 18
Couchbase & Kafka Use Cases
©2016 Couchbase Inc. 19
Couchbase & Kafka Use Cases
Currently using Kafka, but not Couchbase
Reading from Couchbase with almost no code (Source)○ Adopt default message format given by connector○ Write custom converter○ Use Kafka Streams to process events from Couchbase, and write
them back.
Writing to Couchbase without touching SDK (Sink)○ Provide document in the generic form
(it will be converted into JSON)
©2016 Couchbase Inc. 20
Couchbase & Kafka Use Cases
Currently using Couchbase, but not Kafka
Backup of the data○ Need to be filtered/analyzed/transformed by the business logic○ Checking integrity
Turn realtime data into history○ e.g. Couchbase keeps the exchange rates, but it is necessary to emit
stream of the deltas, or log of the changes.
©2016 Couchbase Inc. 21
Couchbase & Kafka Use Cases
Currently using Kafka and Couchbase
Integrate more naturally with Kafka infrastructure.A lot of the custom code may be moved into connector.
○ Filtering○ Transformation○ Process/Task management
©2016 Couchbase Inc. 22
©2016 Couchbase Inc. 23
Demo: Couchbase as Data Source Basic Scenario
Console C
onsumer
• OOTB Couchbase Connector• Bucket “travel-sample” • Output all-content• Consume messages to console• Insert/Update/Delete• JMX Monitor
©2016 Couchbase Inc. 24
Demo: Couchbase as Data Source + Data Sink Basic Scenario
• OOTB Couchbase Connector• Bucket “travel-sample” to “receiver”• Raw content• Write messages to Couchbase• JMX Monitor
©2016 Couchbase Inc. 25
Demo: Couchbase as Data Source + Data Sink Custom Scenario
• Custom Schema Converter• Custom Filter• “travel-sample” to “receiver”: only “airlines”• JSON content• Write messages to Couchbase• JMX Monitor
AirlineConverter
AirlineFilter
©2016 Couchbase Inc. 26
Demo: Twitter Data Source + Couchbase as Data Sink
• Custom Kafka Producer & Consumer• Bucket “twitter”• Write messages to Couchbase• JMX Monitor
Twitter Producer
Couchbase
Consum
er
©2016 Couchbase Inc. 27
Couchbase Connector in Confluent Platform
©2016 Couchbase Inc. 28
Confluent Platform
©2016 Couchbase Inc. 29
Resources
• Couchbase Kafka Connectorhttps://developer.couchbase.com/documentation/server/current/connectors/kafka-3.1/kafka-intro.html
• Couchbase Kafka Connector (code)https://github.com/couchbase/kafka-connect-couchbase
• Demo code repositorieshttps://github.com/mahurtado/KafkaCouchbaseConnectorSamplehttps://github.com/mahurtado/TwitterKafkaCouchbasePipeline
• Kafka Connect Confluenthttps://www.confluent.io/product/connectors/
Q & A