Everyone has a plan until…Automation Evolution While Scaling

@petecheslock

Pete Cheslock @petecheslock

Tech Ops & Support at Threat Stack https://pete.wtf

@petecheslock

get.threatstack.com/automacon-2016

@petecheslock

@petecheslock

Cassandra

Mongodb

PostgreSQL

HadoopElasticsearch

Zookeeper

KafkaRabbitmq

Chef

Puppet

AWS

AzureMySQL

Golang

Python

RubyLOL

Docker

Docker

DockerNode.js

@petecheslock

@petecheslock

@petecheslock

@petecheslock

@petecheslock

Soooooooo……..

@petecheslock

Ummmmm……..

@petecheslock

@petecheslock

Now Launch

@petecheslock

@petecheslock

@petecheslock

@petecheslock

@petecheslock

@petecheslock

PREMATURE OPTIMIZATION IS THE

ROOT OF ALL EVIL

@petecheslock

Its just a DNS change

@petecheslock

Commit to master couple min elapsed 😭=+

Commit to master target canary nodes 😃=+

sign & release to repo

@petecheslock

@petecheslock

@petecheslock

@petecheslock

\

@petecheslock

# knife ec2 server create < 8 billion args>

@petecheslock

But what about your AWS bits?

@petecheslock

@petecheslock

@petecheslock

@petecheslock

@petecheslock

!!!!

@petecheslock

@petecheslock

# knife ec2 server create <args>

# sfn create ntp-server

@petecheslock

Its YAML now!https://aws.amazon.com/blogs/aws/aws-cloudformation-update-yaml-cross-stack-

references-simplified-substitution/

@petecheslock

@petecheslock

Did this work?

• I was paged (PD actually called my phone) 2 (two) times while on call.

• One time was 4pm (fixed during business hours)

• One time was 7:30pm (logged in to fix a known issue)

• In the first 6 months post launch - avg was once every other night.

@petecheslock

Did this work?

• I was paged (PD actually called my phone) 2 (two) times while on call.

• One time was 4pm (fixed during business hours)

• One time was 7:30pm (logged in to fix a known issue)

• In the first 6 months post launch - avg was once every other night.

@petecheslock

• We are currently ingesting over 100,000 events per second

• Represents a total ingest rate of about 10 TB of raw events PER DAY

• Sales hasn’t stopped selling.

• The diameter and the pressure of the firehose continues to grow

@petecheslock

Its OK to hate your stack

@petecheslock

There are two types of tools

The ones I don’t like

The ones I haven’t used yet

@petecheslock

Stop comparing your environment to others

@petecheslock

Everything CAN and WILL change

@petecheslock

Continuous Improvement is still a thing.

@petecheslock

And that’s OK

Thank You!

get.threatstack.com/automacon-2016