Metrics for Faster Metrics for Faster FeedbackFeedback

Kris Buytaert

@krisbuytaert

Kris BuytaertKris Buytaert● I used to be a Dev,I used to be a Dev,● Then Became an OpThen Became an Op● Chief Trolling Officer and Open Source Chief Trolling Officer and Open Source

Consultant @inuits.euConsultant @inuits.eu● Everything is an effing DNS ProblemEverything is an effing DNS Problem● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore● Some books, some papers, some blogsSome books, some papers, some blogs● Evangelizing devopsEvangelizing devops

What's this devops thing What's this devops thing anyhow ? anyhow ?

● Devops is a growing professional and cultural Devops is a growing professional and cultural movementmovement

● We don't have all the answers yetWe don't have all the answers yet

● We are reaching out to different communities We are reaching out to different communities

● We will point out problems we see..We will point out problems we see..

● Only the name is new Only the name is new

While we are still working out the solutionsWhile we are still working out the solutions

devops, a definition:devops, a definition:

● Adopt the new philosophy. We are in a new economic age. Western management Adopt the new philosophy. We are in a new economic age. Western management must awaken to the challenge, must learn their responsibilities, and take on must awaken to the challenge, must learn their responsibilities, and take on leadership for change.leadership for change.

● Cease dependence on inspection to achieve quality. Eliminate the need for massive Cease dependence on inspection to achieve quality. Eliminate the need for massive inspection by building quality into the product in the first place.inspection by building quality into the product in the first place.

● Improve constantly and forever the system of production and service, to improve Improve constantly and forever the system of production and service, to improve quality and productivity, and thus constantly decrease costs.quality and productivity, and thus constantly decrease costs.

● Institute training on the job.Institute training on the job.● Institute leadership The aim of supervision should be to help people and machines Institute leadership The aim of supervision should be to help people and machines

and gadgets do a better job. and gadgets do a better job. ● Drive out fear, so that everyone may work effectively for the company. Drive out fear, so that everyone may work effectively for the company. ● Break down barriers between departments. People in research, design, sales, and Break down barriers between departments. People in research, design, sales, and

production must work as a team, in order to foresee problems of production and production must work as a team, in order to foresee problems of production and usage that may be encountered with the product or service.usage that may be encountered with the product or service.

● Eliminate slogans, exhortations, and targets for the work force asking for zero defects Eliminate slogans, exhortations, and targets for the work force asking for zero defects and new levels of productivity. Such exhortations only create adversarial and new levels of productivity. Such exhortations only create adversarial relationships, as the bulk of the causes of low quality and low productivity belong to relationships, as the bulk of the causes of low quality and low productivity belong to the system and thus lie beyond the power of the work force.the system and thus lie beyond the power of the work force.

● Eliminate management by objective. Eliminate management by numbers Eliminate management by objective. Eliminate management by numbers and numerical goals. Instead substitute with leadership.and numerical goals. Instead substitute with leadership.

● Remove barriers that rob the hourly worker of his right to pride of Remove barriers that rob the hourly worker of his right to pride of workmanship. The responsibility of supervisors must be changed from workmanship. The responsibility of supervisors must be changed from sheer numbers to quality.sheer numbers to quality.

● Remove barriers that rob people in management and in engineering of Remove barriers that rob people in management and in engineering of their right to pride of workmanship. their right to pride of workmanship.

● Institute a vigorous program of education and self-improvement.Institute a vigorous program of education and self-improvement.● Put everybody in the company to work to accomplish the transformation. The Put everybody in the company to work to accomplish the transformation. The

transformation is everybody's job.transformation is everybody's job.

William Edwards William Edwards Deming Deming

1986, Out of the Crisis. 1986, Out of the Crisis.

http://en.wikipedia.org/wiki/W._Edwards_Deminghttp://en.wikipedia.org/wiki/W._Edwards_Deming

C(L)AMSC(L)AMS● CultureCulture

● (Lean)(Lean)

● AutomationAutomation

● MeasurementMeasurement

● SharingSharing

Damon Edwards and John WillisDamon Edwards and John Willis

Gene KimGene Kim

Culture,Culture,automation, automation,

Measturement,Measturement,

sharingsharing

Breaking the SilosBreaking the Silos

OpsOpsDevs Devs

Getting AlongGetting Along

Self Service MetricsSelf Service Metrics

● Being able to add new metrics Being able to add new metrics

● Build your own dashboardsBuild your own dashboards

● Look at metrics / logs on all platformsLook at metrics / logs on all platforms

● Learn from the platforLearn from the platfor

Business ValueBusiness Value

Making Making UsageUsage

RevenueRevenue

IssuesIssues

GrowthGrowth

......

Visible Visible

Culture, Culture,

Automation,Automation,Measurement,Measurement,

SharingSharing

Automate all the thingsAutomate all the things● BuildBuild

• reproducable builds are undiscussablereproducable builds are undiscussable

● TestTest

• testing reduces risk testing reduces risk

• automate deployments of your test infra automate deployments of your test infra

● DeployDeploy

• Infrastructure as CodeInfrastructure as Code

• 100% automation 100% automation

• Can you rebuild your infrastructure ?Can you rebuild your infrastructure ?

•

Culture, Culture,

Automation,Automation,

Measurement :Measurement :

measure all the thingsmeasure all the thingsSharingSharing

IssuesIssues

Source CommitsSource Commits

Build/Test StatisticsBuild/Test Statistics

Code AnalyticsCode Analytics

Deploy StatisticsDeploy Statistics● Time To DeployTime To Deploy

● Deploy FrequencyDeploy Frequency

● Lifecycle frequencyLifecycle frequency

● Map to Map to

Operating System StatsOperating System Stats● Memory Memory

● DiskDisk

● CPUCPU

● Per hostPer host

● Per applicationPer application

e.g Collectd for systeme.g Collectd for system

DB / MiddleWare StatsDB / MiddleWare Stats● Transactions Transactions

● Queue lengthQueue length

● API Calls API Calls

● ConnectionsConnections

● Aborted ConnectionsAborted Connections

Application MetricsApplication Metrics● Number of current usersNumber of current users

● Number of sign upsNumber of sign ups

● Response timesResponse times

● Troughput Troughput

● XYZ UsageXYZ Usage

● # restarts# restarts

● Insert your specific valuable stuff here.Insert your specific valuable stuff here.

New feature impactNew feature impact● Deploy Metrics as part of the stackDeploy Metrics as part of the stack

● Canary TestingCanary Testing

● Silent launchesSilent launches

Availability MetricsAvailability Metrics

Collect & ShipCollect & Ship● One month laterOne month later

• You CAN delete metrics You CAN delete metrics

• You CANNOT create metrics You CANNOT create metrics

● Time Series Database Time Series Database

• RRD RRD

• Cacti Cacti

• GangliaGanglia

• GraphiteGraphite

● #monitoringlove#monitoringlove

Graphite Graphite ● Graphing at ScaleGraphing at Scale

● Graphing at EaseGraphing at Ease

● Any metric is a graphAny metric is a graph

● echo "somestring $somevalue $timestamp" | echo "somestring $somevalue $timestamp" | nc <%= graphitehost %> 2003nc <%= graphitehost %> 2003

Graphite ComposerGraphite Composer

x

Math 101Math 101

● f(x)f(x)

● f'(x)f'(x)

● f''(x)f''(x)

● ... statistics 101... statistics 101

Graphite APIGraphite API

Metrics from LogsMetrics from Logs● Sensible Log contentSensible Log content

● Ship logsShip logs

LogstashLogstash● Not your average centralized logging toolNot your average centralized logging tool

● Elasticsearch backedElasticsearch backed

● ShipperShipper

● IndexerIndexer

● WebWeb

● Collect from anywhereCollect from anywhere

● FilterFilter

● Send anywhereSend anywhere

● Queuing supportQueuing support

Logstash in ActionLogstash in Action

Kibana in ActionKibana in Action

StatsdStatsd

● A network daemon that runs on a zillion A network daemon that runs on a zillion platform and listens for statistics, like counters platform and listens for statistics, like counters and timers, sent over UDP and sends and timers, sent over UDP and sends aggregates to one or more pluggable backend aggregates to one or more pluggable backend services (e.g., Graphite).services (e.g., Graphite).

DashboardsDashboards

Self ServiceSelf Service

Culture, Culture,

Automation, Automation,

Measurement,Measurement,

SharingSharing

Visualize Business MetricsVisualize Business Metrics● $revenue$revenue

● #sales#sales

● signups signups

● conversionsconversions

● Api callsApi calls

● Application useApplication use

● Open SourceOpen Source

● Talk about ExperiencesTalk about Experiences

● Publish the codePublish the code

● No proprietary software was used in these No proprietary software was used in these slidesslides

Don't get Locked In !Don't get Locked In !

ContactContactKris.Buytaert@inuits.euKris.Buytaert@inuits.eu

Further ReadingFurther Reading@krisbuytaert @krisbuytaert http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/http://www.inuits.eu/http://www.inuits.eu/

InuitsInuits

Duboistraat 50Duboistraat 502060 Antwerpen2060 AntwerpenBelgiumBelgium891.514.231891.514.231

+32 475 961221+32 475 961221