Introducing IBM Message Hub

Oliver Deakin20/04/2016

Apache Kafka as a Service

2

To do…

What is IBM Bluemix? What is Message Hub? What is Message Hub… really? What else have we done?

3

What is IBM Bluemix?

Bluemix is an open-standards, cloud-based platform for building, running, and managing applications

Less managed

More managed

4

What is IBM Bluemix?

5

To do…

What is IBM Bluemix? What is Message Hub? What is Message Hub… really? What else have we done?

6

What is Message Hub?

7

What is Message Hub?

8

To do…

What is IBM Bluemix? What is Message Hub? What is Message Hub… really? What else have we done?

9

What is Message Hub… really?

Kafka 0.9

Some additions– Multi-tenancy– Security– Confluent REST API– Administration REST API– Monitoring

10

Message Hub is… Kafka

Kafka 0.9– Took early releases– Built, tested and deployed– Deployed as Docker containers– Continuous delivery pipeline

Community involvement– Active submitting bug fixes– KIP-43 – extend SASL mechanism support

11

Message Hub is… Kafka

Kafka

Zookeeper

Kafka

Zookeeper

Kafka

Zookeeper

Kafka

Zookeeper

Kafka

Zookeeper

User App

12

Message Hub is… Multi-tenant

Single Kafka clusterSupport up to 500 tenantsRestrictions on each tenant’s resources

– Up to 100 partitions per tenant (3-way replicated)– Max partition size 1G– Max message age 24 hours– Clean-up policy is deletion– Max message size 1M

13

Message Hub is… Multi-tenant

Kafka

Zookeeper

User1 App

Send to mytopic

User2 App

Send to mytopic

User1 clientLookup User1

tenant prefix, 1234

Add tenant prefix to mytopic

Continue with topic name 1234mytopic

User2 clientLookup User2

tenant prefix, ABCD

Add tenant prefix to mytopic

Continue with topic name ABCDmytopic

14

Message Hub is… Secure

TLS introduced in Kafka 0.9Clients authenticated with SASL Plain IBM cloud security standards

– Customer message data encrypted everywhere– Data storage must be separated from public interface

15

Message Hub is… Secure

Kafka

Zookeeper

Kafka

Zookeeper

Kafka

Zookeeper

Kafka

Zookeeper

Kafka

Zookeeper

User App

16

Message Hub is… Secure

Kafka

Zookeeper

Kafka

Zookeeper

Kafka

Zookeeper

Kafka

Zookeeper

Kafka

Zookeeper

User App

TLS

TLS

TLS TLS

TLS

17

Message Hub is… Secure

Kafka

Zookeeper

Kafka

Zookeeper

Kafka

Zookeeper

Kafka

Zookeeper

Kafka

Zookeeper

User App

TLSTLS TL

S TLS TLS

HA Proxy HA Proxy HA Proxy HA Proxy HA Proxy

18

Message Hub is… Secure

Kafka

Zookeeper

Kafka

Zookeeper

Kafka

Zookeeper

Kafka

Zookeeper

Kafka

Zookeeper

User App

TLSTLS TL

S TLS TLS

HA Proxy HA Proxy HA Proxy HA Proxy HA Proxy

TLS

TLS

TLS

TLS

TLS

19

Message Hub is… Secure

Kafka

Zookeeper

Kafka

Zookeeper

Kafka

Zookeeper

Kafka

Zookeeper

Kafka

Zookeeper

User App

TLSTLS TL

S TLS TLS

HA Proxy HA Proxy HA Proxy HA Proxy HA Proxy

TLS

TLS

TLS

TLS

TLS

20

Message Hub is… Secure

Throughput of 900k msgs/s– 100 byte messages– 6 clients– 1000 topics @1 partition

Latency ranges from 10 – 70ms typically

21

Message Hub is… Flexible

Kafka REST Proxy– Confluent platform 2.0– Binary message format only– No schema registry

22

Message Hub is… Flexible

Kafka

Zookeeper

Kafka

Zookeeper

Kafka

Zookeeper

Kafka

Zookeeper

Kafka

Zookeeper

User App

RESTREST

HA Proxy HA Proxy HA Proxy HA Proxy HA ProxyHA Proxy

23

Message Hub is… Managed

Kafka Administration REST interface– Used to manage topic lifecycle– Topics not auto-created– Web UI also available for manual management

Topics can be created and deleted– Topics can have 1+ partitions– 100 partition limit per tenant across all topics

24

Message Hub is… Managed

Monitoring… and lots of it!– collectd containers deployed on every host– metrics pushed locally– forwarded to central graphite + grafana dashboards– continuous testing

Automated alerts– thresholds trigger support callouts– Slack integration to notify whole team

25

Message Hub is… Managed

26

To do…

What is IBM Bluemix? What is Message Hub? What is Message Hub… really? What else have we done?

27

Message Connect

Connect Enterprise and Cloud– Experimental Bluemix service– Messaging pipeline from MQ to Message Hub– Bluemix apps have visibility of on-prem messages– Allows connectivity with cloud analytics and other services

28

Message Hub Incubator

AMQP proxy layer– Experimental Bluemix service– Kafka is backing messaging engine– Horizontally scalable

Target is to provide interoperability with Kafka clients

29

Bluemix Dedicated and Local

Bluemix Dedicated– IBM cloud and IBM managed– Customer gets dedicated Bluemix environment– Full Message Hub cluster– Spec defined by customer requirements– Available now

Bluemix Local– Customer’s cloud but IBM managed– TBD…

30

Twitter: @IBMmessaging

DeveloperWorks:https://developer.ibm.com/messaging/message-hub/

Bluemix blogs:https://developer.ibm.com/bluemix/blog/

LinkedIn: ibm.biz/ibmmessaging

Message Hub service:https://console.ng.bluemix.net/catalog/services/message-hub