Cassandra Day NY 2014: Message Architectures in Distributed Systems at SimpleReach

transcript

Eric Lubow

@elubow

elubow@simplereach.com

Message

Architectures in Distributed

Systems

Message Architectures in Distributed Systems Eric Lubow @elubow

Overview

• SimpleReach

• Why is messaging important

• Goals

• Explanations

• Questions

Personal Vanity

• CTO of SimpleReach

• Co-author of Practical Cassandra

• Skydiver, Mixed Martial Artist,

Motorcyclist, Dog dad, NY Giants fan

• IronMatt Foundation for Pediatric Brian

Tumors (ironmatt.org)

• Millions of URLs per day

• Over 3.75 billion page views per month

• 7b events per day (~80k events/second)

• Auto-scale 175-190 machines depending on traffic

• Built a predictive measurement algorithm for the social web

SimpleReach

Why is Messaging Important?

• Most large scale systems discussions only talk about storage

• Direct high volumes of data around your infrastructure

• Control flow of data through your infrastructure

• Decouple important systems

• Scalability, Elasticity, Deliverability, and Redundancy

• Buffering and Asynchronous communication

The database is NOT a transport layer

incoming request

sync persist data

send response

async queue message

Data Flow

Message Architectures in Distributed Systems Eric Lubow @elubow #ddtx14

Goals• Consistent interfaces between systems

• Allow access to many toolsets

• Minimize downtime/Minimize cost of downtime

• High availability

• Clients should have minimal architecture knowledge

• Horizontal Scaling

• Controlled Data Flow Patterns

• Enrichment/In-stream Modification Schemes

• Monitoring and Instrumentation

Messaging Systems

• RabbitMQ

• ZeroMQ

• Kafka

• Amazon SQS

• NSQ

• ActiveMQ

• Resque

• Custom

What Did SimpleReach Choose?

NSQ• Distributed and de-centralized topology

• At least once delivery guaranteed

• Multicast style message routing

• Simple to configure and deploy

• Allow for maintenance windows with no downtime

• Ephemeral channels for testing

• Channel sampling

github.com/bitly/nsq

separate hosts

• a topic is a distinct stream of messages (a single nsqd instance can have multiple topics)

• a channel is an independent queue for a topic (a topic can have multiple channels)

• consumers discover producers by querying nsqlookupd (a discovery service for topics)

• topics and channels are created at runtime (just start publishing/subscribing)

“metrics”

Channels

“event”

Topics

“enrichment”

“writer”

Consumers

AAABBB

Topics and Channels

Everyone Speaks The Same Language

http:// + {“content-type”: “application/json”}

• nsqadmin provides a web interface to administrate and introspect an NSQ cluster at runtime (and empty, pause, or delete topics/channels)

• nsq_to_http - utility that helps transport an aggregate stream over HTTP

• nsq_to_file - utility that safely persists an aggregated stream to disk

• nsq_stat - iostat like utility for a topic/channel

• nsq_tail - tail like utility for a topic/channel

NSQ Tools

Right Tool For The Job

NSQNSQD

consumer

NSQNSQD

consumer

nsqlookupd

PUBLISH

REGISTER

DISCOVER

SUBSCRIBE

How Does It Work?

The Schrute of the Problem

Simple Deployment & Automation

• Chef cookbook - github.com/simplereach/chef-nsq

• Written in Go

• Easily distributable binaries

• Deploy lookup nodes

• Nsqd’s installed locally

nsqlookupd nsqlookupd

consumer➊ regularly poll for topic producers

➋ connect to all producers

HTTP requests

Runtime Discovery

Path of a Packet

Internet

Vertica

Fire Hose

Controlled Data Flow

Social Event

CollectorSocial Data

Batch & Write

Processed Data

Batch & Write

Raw Data

Calculate Score Write

NSQ Broadcast NSQ

Social Event

Batch & Write

Processed Data

Batch & Write

Raw Data

NSQ Broadcast NSQ

Broadcast Importance for Polyglottany

Aggregator

Mongo Writer

Broadcast

Redis Writer

Cassandra Writer

Solr Writer

Calculator

Vertica Writer

Social Event

Batch & Write

Processed Data

Batch & Write

Raw Data

NSQ Broadcast NSQ

• Controlled Data Flow

What Is Enrichment?

A mechanism to add value to a message to enhance processing in

your system

How Do We Enrich

Raw EventEnriched

Consumer A

Consumer B

Consumer C

NSQ Broadcast

• Enrichment

Monitoring / Instrumentation

• Comes with statsd support built-in

• Statsd talks to both Graphite and nsqadmin

• Nsqadmin comes with graphs for message processing stats

• Nagios plugins available for monitoring topic/channel depth

• Average end to end latency calculations are done on a per-channel basis

• Enrichment

• Monitoring and Instrumentation

Summary• Large Systems are more than just storage

• Abstraction

• Highly Available

• Controlled Data Flow Patterns

• Monitoring & Automation

We’re

Hiring

Questions are guaranteed in life.

Answers aren’t.

Eric Lubow

@elubow

elubow@simplereach.com

Cassandra Day, New York

Thank you.

Cassandra Day NY 2014: Message Architectures in Distributed Systems at SimpleReach

Technology