How to ditch the debugger and use logging instead

Matthew Skelton & Rob Thatcher, Skelton Thatcher Consulting

/dev/winter 2015,

Saturday 24th January, Cambridge, UK

#devwinter

Event tracingVagrant + ELK

Demo: no debugging

Skelton ThatcherIntroduction and Services Overview

‘Cloud’ changes the way we mustdesign, deliver, and operate

our software systems

Changes for ‘cloud’

Event tracing

Event type IDsTransaction tracing

Adjusting logging levels

Event type IDs

How many event types in your application?

Event type IDs

Event type IDs

enum

Human-readable sets: unique values, sparse, immutable

C#, Java, Python, node(Ruby, PHP, …)

public enum EventID

{

// Badly-initialised logging data

NotSet = 0,

// An unrecognised event has occurred

UnexpectedError = 10000,

ApplicationStarted = 20000,

ApplicationShutdownNoticeReceived = 20001,

PageGenerationStarted = 30000,

PageGenerationCompleted = 30001,

MessageQueued = 40000,

MessagePeeked = 40001,

ItemAddedToBasket = 60001,

ItemRemovedFromBasket = 60002,

CreditCardDetailsSubmitted = 70001,

// ...

}

Technical

Domain

public enum EventID

{

// Badly-initialised logging data

NotSet = 0,

// An unrecognised event has occurred

UnexpectedError = 10000,

ApplicationStarted = 20000,

ApplicationShutdownNoticeReceived = 20001,

PageGenerationStarted = 30000,

PageGenerationCompleted = 30001,

MessageQueued = 40000,

MessagePeeked = 40001,

ItemAddedToBasket = 60001,

ItemRemovedFromBasket = 60002,

CreditCardDetailsSubmitted = 70001,

// ...

}

ItemAddedToBasket = 60001

ItemAddedToBasket = 60001

ItemAddedToBasket = 60001

Transaction tracing

Monolith to microservices:debugger does not have the full view

Distributed systems

Even with remote debugger, it’s boring to attach and detach

Remote debugging?

Transaction tracing

Unique-ish identifier assigned to web request

Passed through downstream layers

What about APM?

APM gives us application insightBUT

How much do we learn? Is APM available on the Dev box?

It’s not just ‘an Ops problem’!

Minimal viable tracing?

Helps us to understand how the software really works

Small overhead is worth it

Which log level is right?

DEBUG, INFO, WARNING, ERROR, CRITICAL

Which log level is right?

Log level should *not* be fixed at compile or build time!

<?xml version="1.0" encoding="utf-8" ?>

<Application.EventMappings>

<Events>

<Event Type="CacheServiceStarted">

<Severity Type="Information"/>

</Event>

<Event Type="PageCachePurged">

<Severity Type="Debug"/>

<Enabled="false"/>

</Event>

<Event Type="DatabaseConnectionTimeOut">

<Severity Type="Error"/>

</Event>

</Events>

</Application.EventMappings>

Tune log levels

Tune log levels

Tune log levels

Tune log levels

http://bit.ly/TuneLoggingLevels

Dev and Ops collaboration*

* and testers too!

Event tracing

Use enumerations (or closest thing)

Technical and Domain event types

Distributed systems: debuggers less useful

Trace calls with ‘unique-enough’ handles

Tune log levels via config

Vagrant + ELK

Log AggregationElasticsearch, Logstash & Kibana

Vagrant

Log Aggregation

Multiple log sourcesMany logs in one place

SearchableTime Sequenced

What is ELK?

http://www.exploringnature.org/db/detail.php?dbID=43&detID=956

Vagrant

Ruby CLI to deploy Virtual environments

Deploy locally to AWS or Azure etc.

Four Simple Steps

$ vagrant box add ubuntu/trusty64$ vagrant init trusty64$ vagrant up$ vagrant ssh

RapidRepeatable

Provisioning via scripts ++

• Listening to: • Local nginx logs• Local Syslog• (Application logs)

http://bit.ly/velk-demo

Vagrant + ELK Demo VM

Elasticsearch, Logstash, Kibana

Recap

Aggregated Logging:Vagrant + ELK

Centralised logs, time sequenced, searchableVagrant :quickly spin up & manage machines

ELK : collect, store, display & search logsDeveloper-friendly!

Demo: ditching the debugger

Vagrant on developer laptopELK for log aggregationWeb app: PHP (Slim)

Event type IDs

Demo: background

Use VirtualBoxshared folders

Browse to Kibana UI via NAT-ed interface

Event type IDsLog aggregation with ELK

Demo

Demo: summary

Super-easy to hook application logs into ELK

Sequences of events emerge

Compare timings of events: past and present

Recapitulation

How to ditch the debuggerUse Vagrant + ELK on development VM

Use a set of event type IDs (enums)Log distinct states of execution

Both technical and domain eventsConsistent event naming helps with searchCompare previous and current executions New logging-fu: make friends with Ops!

LinksTune logging levels in Production without recompiling code: http://bit.ly/TuneLoggingLevels

VELK (Vagrant + ELK) demo VM: http://bit.ly/velk-demo

GOOS book (section on logging: p.233-5):http://www.growing-object-oriented-software.com/

Thank you!

http://skeltonthatcher.com/

enquiries@skeltonthatcher.com

@SkeltonThatcher

+44 (0)20 8242 4103