/

Monitoring and Observability

in Complex Architectures

Tuesday, October 2, 12

Hi! I’m @postwait

I founded @OmniTI and @MessageSystems and @Circonus

Tuesday, October 2, 12

Hi! I’m @postwait

I am very active in @TheOfficialACMparticipating in @ACMQueueand the practitioners board.

Tuesday, October 2, 12

Hi! I’m @postwait

I (regrettably) build complex systems.

Tuesday, October 2, 12

Why we are here

We’re here to talk aboutcoping with breakage

Tuesday, October 2, 12

Rule #1

Direct observation of failureleads to quicker rectification.

Tuesday, October 2, 12

Rule #2

You cannot correctwhat you cannot measure.

Tuesday, October 2, 12

Solution Approach #1

Debugging failures requires eithervisibility into theprecipitating state

Tuesday, October 2, 12

Precipitating State

Single threaded applications

✓ Easy

Tuesday, October 2, 12

Precipitating State

Multi-threaded applications

✓ Challenging

Tuesday, October 2, 12

Precipitating State

Distributed applications

here there be dragons

Tuesday, October 2, 12

Solution Approach #2

ordirect observation of a(and likely very many)failing transaction

Tuesday, October 2, 12

Direct Observation

Observing something fail...is priceless.

Tuesday, October 2, 12

Direct Observation

Observation leads tointelligent questioning.

Tuesday, October 2, 12

Direct Observation

Questioning leads to answers...but only through more observation.

Tuesday, October 2, 12

Direct Observation

Questioning leads to answers...but only through more observation.

and herein lies the rub.

Tuesday, October 2, 12

Leaning Towards Scientific Process

In production you don’t have• repeatability• control groups• external verification

Tuesday, October 2, 12

Leaning Towards Scientific Process

In production you don’t have• repeatability• control groups• external verification

... or do you?

Tuesday, October 2, 12

What’s monitoring got to do with it?

Monitoring is all about thepassive observation oftelemetry data.

Tuesday, October 2, 12

Monitoring Telemetry

cannot pinpoint problems

can provides evidence ofthe existence of a problem

Tuesday, October 2, 12

Monitoring

Gives you evidence thatthere is a problem

Tuesday, October 2, 12

Monitoring

Gives you evidence thatyou have fixed a problem(or at least the symptoms)

Tuesday, October 2, 12

Monitoring Tactically

If it could be of interest,measure it andexpose the measurement

Tuesday, October 2, 12

Monitoring: embedded

statsdhttps://github.com/etsy/statsd

resmonhttp://labs.omniti.com/labs/resmon

metricshttps://github.com/codahale/metrics

folsomhttps://github.com/boundary/folsom

metrics.jshttps://github.com/mikejihbe/metrics

metrics-nethttps://github.com/danielcrenna/metrics-net

Tuesday, October 2, 12

Monitoring: collection

reconnoiterhttp://labs.omniti.com/labs/reconnoiter

graphitehttp://graphite.wikidot.com/

OpenTSDBhttp://opentsdb.net/

circonushttp://circonus.com/

libratohttps://metrics.librato.com/

Tuesday, October 2, 12

Monitoring: Bling

visualizing an architecture rollout

Tuesday, October 2, 12

Monitoring: Bling

visualizing the impact on service times

Tuesday, October 2, 12

average API service time latency

Tuesday, October 2, 12

actual API service time latency

http://www.slideshare.net/postwait/atldevops

Tuesday, October 2, 12

Monitoring: Bling

Tuesday, October 2, 12

Repeatability is a Pipe Dream

You production problem is a(hopefully pathological)outcome of circumstance.

A circumstance which oftencannot be repeated.

Tuesday, October 2, 12

Control Groups

Control groups cancompensate for theinability toprecisely repeat an experiment.

Tuesday, October 2, 12

Control Groups

Most architectures have redundancy.

Tuesday, October 2, 12

Control Groups

With the right design,you can turn that redundancyinto a debugging environment.

[1] http://omniti.com/surge/2012/sessions/xtreme-deployment

Tuesday, October 2, 12

Control Groups: Simple Example

I have 10 web serversI fix 1I verify 1 is fixedI verify 9 are still broken

Tuesday, October 2, 12

Control Groups: Seems Easy

Web servers tend to be:• homogeneous• share-(nothing|little)• independent

Tuesday, October 2, 12

Control Groups: Not So Easy

Most other services aren’t so homogeneous and equal:databases, batch processes (think billings), orchestration middleware, message queues

Tuesday, October 2, 12

Observability

Some might claim thatseeing telemetry data isobservation...

It is doubly indirect at best.

Tuesday, October 2, 12

Observability

I want todirectly seetheerrant behaviour

Tuesday, October 2, 12

Observability is forgiving

In complex, multi-component architectures, errors can be observed as errant behaviour in many junction points.

Tuesday, October 2, 12

Observing the network

tcpdump / snoopwireshark

Tuesday, October 2, 12

Observing the network

Looking at just thearrival of new connections

tcpdump -nnq -tttt -s384'tcp port 80 and (tcp[13] & (2|16) == 2)'

Tuesday, October 2, 12

Observing the network

Looking at just the dataarrival and departure timestcpdump -nnq -tt-s 384 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)*4)) - ((tcp[12]&0xf0)/4)) != 0)'

snoop -rq -ta-s 384 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)*4)) - ((tcp[12]&0xf0)/4)) != 0)'

Tuesday, October 2, 12

Observing the network

Finding the difference betweena client’s question anda server’s answer(tcpdump | awk filter).{ gsub(".[0-9]+(: | >)"," \& "); gsub("[:=]"," "); EP=sprintf("%s%s", ($4==".80")?$6:$3, ($4==".80")?$7:$4);

if(S[EP] == "C" && $4 == ".80") { printf("%f %s\n", $1 - L[EP], EP); }

S[EP]= ($4==".80")?"S":"C"; L[EP]= $1;}

Tuesday, October 2, 12

Observing the network

Tuesday, October 2, 12

Observing the network

Tuesday, October 2, 12

Observing user-space

strace[1] / trussgstack / pstackgcore + gdb / dbx / mdb[2]

[1] http://www.cli.di.unipi.it/~gadducci/SOL-11/Local/referenceCards/LINUX_System_Call_Quick_Reference.pdf[2] http://hub.opensolaris.org/bin/download/Community+Group+mdb/tips/mdb-cheatsheet.pdf

Tuesday, October 2, 12

System call tracing

Watching sshdis a good way to get familiar.truss -f -p `pgrep sshd`

Tuesday, October 2, 12

System call tracing

An active web server is going to belike a firehose.truss -f -p `pgrep httpd`

Tuesday, October 2, 12

Observing the system

DTrace

Live production demo or GTFO.

Tuesday, October 2, 12

Thank You

Questions?

Tuesday, October 2, 12