Reliable System Integration and Scaling with WSO2 Message Broker

Ramith JayasingheSenior Technical Lead

WSO2

Messaging Models/Concepts• RPC style

- Request/Response, Synchronous, Non-persistent - e.g. Web Services, Thrift, REST

• Point to Point (Queues)

- A message is delivered only once to a single consumer

• Publish/Subscribe (Topics)

- Broadcast a message to all the subscribers

Messaging Models/Concepts• Durable Subscriptions

- Same as topic subscriptions- Subscriber offline - Messages get queued- Subscriber back online - Messages get delivered- Un-subscription needed to remove queuing

Messaging Models/Concepts• Transactional ( send/consume -> rollback, reject, commit)• Guaranteed Delivery, QOS ( MQTT, AMQP), Persistence• Dead letter channel• Hierarchical topics

- Games.FootBall.UK, Games.FootBall.USA, - Games.Cricket.SriLanka, Games.Football.UK

Distributed Queues

• Provides strict or best effort support for inorder delivery• There are no guarantee about the global order seen

across subscribers

WSO2 Message Broker

• Design Objectives:- Higher performance (by not routing messages over the

network multiple hops)- High availability and fault tolerance.- Leaner deployment layout ( and clustering story)

• v 3.0.0 - Significant architectural changes.• v 3.1.0 - Latest version

- Minor improvements - PostgreSQL support

WSO2 MB v 3.x.x Design Objectives

• Avoid message routing over the network multiple times.• Use storage to share messages between nodes

Allow database to scale as separate layer.

• Use distributed coordination to control the behavior (i.e. Hazelcast)

WSO2 MB v 3.x.x Design Objectives

• Content and metadata are saved separately into a database shared between

all cluster nodes.

• All nodes knows about all subscribers ( - via Hazelcast coordination)

• Each node will logically divide messages into group (known as Slots).

WSO2 Message Broker Design

More information about disruptor: https://lmax-exchange.github.io/disruptor/

WSO2 Message Broker Design - Inbound

All events coming to broker are added to ‘inbound disruptor’ such as:

● Messages● Message acknowledgements ● Create/Delete queues topics● Subscribe/Unsubscribe● UI Events

Inbound disruptor:● Sequence inbound events ( - disruptor

behaviour)● Generate internal Message IDs ( length =

java - long)● persist/delete messages● Execute other control functions.

Note: #1. Internal message Ids are time based. Therefore, MB cluster needs to be time synced.

#2. Inbound path can be tuned for best performance ( broker.xml).

WSO2 Message Broker Design - Outbound

All message going out of broker are added to ‘outbound disruptor”

Slot Delivery Worker pool:1) Connects to coordinator node ( - get

messages to deliver)2) Fetches metadata of messages and add

to outbound disruptor

Outbound disruptor:1) Fetch message payloads (in parallel)2) Forward fetched messages to transport.

Note: Slot delivery thread pool and outbound disruptors can be tuned for best performance (broker.xml).

WSO2 Message Broker Design - Distributed Queues

• Content and metadata are saved separately into a database shared between

all cluster nodes.

• All nodes knows about all subscribers ( - via Hazelcast coordination)

• Each node will logically divide messages into group (known as Slots).

WSO2 Message Broker Design - Distributed Queues

• Cluster elects a leader ( known as Coordinator); Usual the first node to start in cluster.

• Coordinator keeps track of all the slots available (via Hazelcast)

• Nodes with subscriptions acquire a slot (from Coordinator) and deliver them to

subscribers.

WSO2 Message Broker v 3.1.0 - Features

● Message Queuing Telemetry Transport (MQTT)

● Shared Subscriptions

● RDBMS based message store (MSSQL, Oracle, H2, MySQL, PostgreSQL)

● Real time statistics/graphs (on each server) via carbon-matrices

● Improved internals

MQTT support in WSO2 MB• Lightweight, pub-sub protocol designed for IOT use-cases.

• Hierarchical topic structures and wildcard subscriptions

• All quality of service levels and retained messages

Shared Durable Subscriptions

Extension to JMS 1.1, until JMS 2.0 support is released. Purpose: Ability to load balance.

broker.xml: <allowSharedTopicSubscriptions>true</allowSharedTopicSubscriptions>

Back pressure / Flow Control - Inbound• Every system has its limits.

• Safeguards server from sudden bursts of message loads

• Flow control based on:

– Total inbound events

– Inbound events per connection/channel

– Global error conditions (e.g. offline databases)

<flowControl> <!-- This is the global buffer limits which enable/disable the flow control globally --> <global> <lowLimit>800</lowLimit> <highLimit>8000</highLimit> </global>

<!-- This is the channel specific buffer limits which enable/disable the flow control locally.

--> <bufferBased> <lowLimit>100</lowLimit> <highLimit>1000</highLimit> </bufferBased> </flowControl>

General Improvements• Real time statistics/graphs

– Quickly figure out bottlenecks in a production deployments.

– Database read/write latencies– Inbound/Outbound event rates– Number of Connections/Channels etc.

• Performance Improvements – Disruptor based thread model– Improved internals ( topic matching, slot based delivery,

caching)

Improvements in MB v 3.1.0• Support for PostgreSQL as the message store.• LZ4 Compression for storage

– Dramatic performance improvement large/text oriented message payloads

• Forcefully disconnect Consumers– If a subscriber misbehaves, admin can disconnect the client

forcefully.

Roadmap• Support for AMQP 1.0 and JMS 2.0

• Move to Carbon 5

• Improved GUI

• Improvements to Storage Scheme

• Performance improvements.

• Websockets + Stomp

What makes WSO2 MB different● Simplified clustering story (- Queues/Topics)

- Users don’t have to worry about how brokers transmit messages in the cluster.

● RDBMS for persistence →Easy to manage and use

● Simpler Cluster deployments → Minimum deployment effort + maintenance.

Use Case – Guaranteed Delivery with WSO2 ESB

• Guarantees In-Order Delivery.• Throttle messages ( for slow consumers/systems).• Ensure reliable messaging

An Extended Use case - Where MB fits

CONTACT US !