Monitoring with Syslog and

EventMachine

"It’s up doesn’t mean it’s working"

Monitoring

dashboards/monitoring

sketching

dashboards/monitoring

sketching

dashboards/monitoring

building

sketching

dashboards/monitoring

building

outlook

Dashboards

Motivation

Moving Target

Failure is always an option

“Since the last deploy, the number of signup errors has gone up by 300 %. We might have

broken something.”

Counting Things

Concurrent Users

Signups

Logins

Errors

Sketch it

Application Server

Application Server

Application Server

Event Aggregation

Dashboard

Criteria

Simple

Polyglot

Fire And Forget

Simple/Lightweight

Polyglot

Fire and Forget

Criteria

Background story

Not Supported

node.js NewRelic RPM

erlang NewRelic RPM

haproxy

node.js node.js node.js

Load Balancer

haproxy logs

haproxy[674]: 127.0.0.1:33320 [15/Oct/2003:08:32:17.654] px-http \  px-http/srv1 9/0/7/14/+30 200 +243 - - ---- 3/3/3/1/0 0/0 \  "GET /image.iso HTTP/1.0"

haproxycustom syslog

server

New Relic

send logs

NewRelic Ruby Agent

haproxy!rpm

http://github.com/wooga/haproxy2rpm

Syslog

Syslog• Standard logging solution for Unix/Linux

• Facility (daemon, cron, user, local0, etc.)

• Priority/Level (Alert, Critical, Error, Warning, etc.)

• Client and server

• Since 1980

... It also provides devices which would otherwise be unable to communicate a

means to notify administrators of problems or performance.

http://en.wikipedia.org/wiki/Syslog

Syslog Format

Date Hostname Program Message

Jan 1 12:12:12 10.245.3.99 foo[421] : this is a message

:

Syslog client

# man loggerlogger -p local0.notice -t HOSTIDM My Message

UDP

Fire And Forget

Simple/Lightweight

Polyglot

Fire and Forget

Simple/Lightweight

Polyglot

Fire and Forget

✔

Simple/Lightweight

Polyglot

Fire and Forget

✔

✔

Simple/Lightweight

Polyglot

Fire and Forget

✔

✔

✔

Build it

Event Emitter

Dashboard

Event Aggregatorsend message

UDP : 514

Event Emitter Syslog

Dashboard Event Aggregator

send message

UDP : 514

forward

Emitting Eventsrequire 'syslog'

def log(message, level = :warning)  script_name = $0  syslog_option = Syslog::LOG_PID | Syslog::LOG_CONS  Syslog.open($0, syslog_option) do |s|    s.send(level, message)  endend

Server

• Listen on UDP

• receive and parse syslog messages

• update a counter

• push it out to dashboards

Serverclass UdpServer

HOST = '127.0.0.1' PORT = '3333'

def self.run EM::open_datagram_socket(HOST, PORT, Handler) endend

Handlerrequire 'eventmachine'

class Handler < EM::Connection def receive_data(data) puts "Received event: #{data.inspect}" puts SyslogParser.parse(data) endend

Syslog Parser

https://github.com/jordansissel/experiments/blob/master/ruby/eventmachine-speed/basic.rb

Web App

Event Source

Event Source Format

data: Hello dashboard\n\nevent: userlogon\ndata: {"username": "John123"}\n\nevent: update\ndata: {"username": "John123", "foo": "bar"}\n\n

Sinatrarequire 'sinatra'

get '/stream' do content_type 'text/event-stream' stream(:keep_open) do |out| endend

class WebApp < Sinatra::Base

def self.connections @connections ||= [] end

get '/stream' do content_type 'text/event-stream' stream(:keep_open) do |out| WebApp.connections << out end endend

class WebApp < Sinatra::Base # send the message to all clients def self.send_event(event = {}) connections.each do |c| if c.closed connections.delete(c) else c << "event: #{event[:event]}\n" c << "data: #{event[:data]}\n\n" end end end # here go sinatra routes ...end

# disable sinatra's autorundisable :run

EM.run do WebApp.run! UdpServer.run Signal.trap("INT") { EventMachine.stop } Signal.trap("TERM") { EventMachine.stop }end

View

// Open up an event source socketvar source = new EventSource('/stream');

source.addEventListener('message', function(e) { console.log("message:" + e.data); }, false);

source.addEventListener('open', function(e) { console.log('event source opened:' + e); }, false);

source.addEventListener('error', function(e) { console.log('received an error:' + e); }, false);

EventSource Basics

Custom Eventssource.addEventListener('login', function(e) { console.log("login: " + e.data);}, false);

source.addEventListener('loginError', function(e) { console.log("login error: " + e.data);}, false);

Testingrequire 'socket'

desc 'send test data'task :send_test_data do socket = UDPSocket.new host = "127.0.0.1" port = "3333" events = ["message", "login", "loginError"]

while true do random_index = random_index(0..2) random_value = rand(0..1000) message = "#{events[random_index]}:#{random_value}" socket.send(message, 0, host, port sleep(1) endend

https://github.com/phuesler/yada

Missing features

• Persistency

• Querying

• More complex aggregations (avg, timers, etc.)

• Fancy visualization

Existing Solutions

• https://github.com/etsy/statsd

• https://github.com/noahhl/batsd

• https://github.com/eric/metriks

• http://square.github.com/cube/

Graphitehttp://graphite.wikidot.com/screen-shots

Librato Metricshttps://metrics.librato.com/

Monitor all the things!!!!

wooga.com/jobs

WANTEDdead or alive

REWARD

www.wooga.com/ jobs/

BACKEND DEVELOPER

Further Reading

• http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/

• http://code.flickr.com/blog/2008/10/27/counting-timing

• https://speakerdeck.com/u/roidrage/p/metrics-monitoring-logging

Slideshttp://www.slideshare.net/wooga

CreditsCliff (Flickr)

!"#K (Flickr)

Scott Kidder (Flickr)