+ All Categories
Home > Technology > DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

Date post: 15-Apr-2017
Category:
Upload: hakka-labs
View: 227 times
Download: 0 times
Share this document with a friend
37
BYOMQ: Why We [re]Built IronMQ Reed Allman @rdallman10
Transcript
Page 1: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

BYOMQ: Why We [re]Built IronMQ

Reed Allman

@rdallman10

Page 2: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

Making [another!] Case for NIHReed Allman

@rdallman10

Page 3: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

Beating A Dead Horse [again!]Reed Allman

@rdallman10

Page 4: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

There

Page 5: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

Are

Page 6: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

Too

Page 7: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

Many

Page 8: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

Options

Page 9: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

Me Showing .gifs for 30 MinutesReed Allman

@rdallman10

Page 10: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ
Page 11: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

NIH: Not Invented Here Syndrome

A quick refresher:

...the tendency towards reinventing the wheel (reimplementing something that is

already available) based on the belief that in-house developments are inherently better

suited, more secure, more controlled, quicker to develop, and incur lower overall cost

(including maintenance cost) than using existing implementations.

Page 12: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

Proof by Contradiction

Page 13: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

Throw this at ${USE_CASE}

Page 14: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

Out of the Box

● Massive scale on commodity hardware in hours of easy set up

● Battle tested, many bugs worked out, lots of docs, tools, guides, etc.

● Operate JVM, hard not to code on JVM

● Any and all problems associated with any of the boxes or how they fit together

● And these are open source! Many boxes are not and suffer vendor lock-in

● At mercy of those who understand it / time for fixing bugs, making improvements

Page 15: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

Could you do better?

Page 16: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

Different people come from different backgrounds and, based on that, find different

tools useful. Ultimately you need to use the tools that let you get your job done most

effectively, however that is defined in your particular case.

- Russ Cox

Go

Page 17: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

If it's a core business function -- do it yourself, no matter what.

- Joel Spolsky

Excel

Page 18: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

I felt that my team, which was supposed to be made up of distributed systems

engineers, was really acting more as distributed system plumbers.

- Jay Kreps

Kafka

Page 19: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ
Page 20: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

Pain meow or later?

Page 21: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

To the Queues!

Page 22: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

Our Requirements:

● Easy to run (single binary ideal) in any environment & not hosted

● Persisted, consistent and performant

● Highly available -- fast recovery, no data loss, minimal downtime

● Horizontally scalable

● Timing based

● FIFO

● >= 1 time delivery

● Multi-Tenant capable+

● Preferably built in Go, HTTP interface

Page 23: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

...Scalable, Low-Latency, Persistent MQ for Job Processing

Page 24: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ
Page 25: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ
Page 26: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

What, briefly● Single Go binary per server

● RocksDB embedded to store all metadata and messages

● Built the distributed database - viewstamped replication, multi-master, gossip

membership - scales across queues across nodes, auto-balanced

● Linear scaling factor of .98

● Recovery time < 300ms

● Simple HTTP interface, stupidly simple clients

● Enqueue / Dequeue avg latency < 10ms for < 4KB messages

Page 27: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ
Page 28: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

boom

Page 29: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

Results

Page 30: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

Full understanding and control makes for quick fixes, easy to add features

Page 31: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

Don’t have to build a daemon to add features, can bake everything in

Page 32: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

Takes a long time to get to stable

Page 33: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

Make all your own tools, plumb any data you want out

Page 34: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

Build exactly what you need, optimized for what you are doing

Page 35: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

Use the language(s) you know and love

Page 36: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

My attempt to say profound shit:

Be a builder, not a plumber.

Page 37: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

Questions?


Recommended