Monitoring with StackdriverOr: How I Learned How to Stop Worrying and Dig Hosted Monitoring.

denise stockman - denise@stockmans.org

Mad PropsThanks to Coupa for hosting and to Stackdriver for sponsoring and to Advanced AWS for organizing.

Hi, I’m denise● System Architect @ Maverix Biomics● 18 years button pushing● Musical Chair Champion● Data center to cloud● Early Stackdriver user &

puppet module maintainer

What is Stackdriver?“Intelligent Monitoring for your AWS Infrastructure, Systems and App”

Agenda● The Environment● Selection Process● Implementation● Day to Day● TODO● Questions

A Diverse Stack

Further Context● Green field!● We treat instances like cattle*● Modern and crusty backend services● “Quaint” team● Monitor & report system, service, event and application

statistics in one place.

* As much as possible

What to do for monitoring?Evaluate & Select● In-house or hosted?● Modern vs proven?● Specialized vs general?● Paid vs free?

Let’s Be Honest● Not enough time● Small environment● More important work to do● Everyone else is doing it

Winner, Winner, Chicken Dinner

● System & service metrics● Alerting● Custom metrics & annotations● collectd agent● Anomaly detection● AWS integration & cloud friendly● Cost per resource

Implementation

Consistent Resource NamingResource name format

env-function-id.location.domain.com

Tags for resources

env, owner, purpose, version

Agent Deployment

node /^dev-redis.*\.stockmans\.org$/ { class { "stackdriver::redis": stackdriver_redis_port => 6379, stackdriver_redis_host => localhost, stackdriver_redis_timeout => 2000, }}

● Installs base agent on all hosts● Configures API key● Includes plugins for supported services● Auto-upgrades agent

● Create groups based on filtering criteria● Dashboards auto-created for each group● Groups used in custom dashboarding and alerting policies● Set it once and fuggetaboutit*

Groups

Alerting

Day to Day Use

Custom Dashboards

Anomaly Detection

Event Correlation

Event Logging

Annotations

curl -X POST -H "x-stackdriver-apikey: DOOOOWEEEEOOOO" -H "Content-Type: application/json" -d '{"message":"Mind taking!", "annotated_by":"Mentok"}' https://event-gateway.stackdriver.com/v1/annotationevent

Custom Metrics

Ruby examplecustomer_id = 31337api_key = "DOOOOWEEEEOOOO"StackDriver.init customer_id, api_key

redis = Redis.newmindsTaken = redis.zcard("minds:taken")

StackDriver.send_metric "mentok.minds.taken", mindsTaken, Time.now.to_i

TODO.txt● Expand app metrics with statsd● Automate group creation & management● Annotate orchestrated events● Cloudwatch custom metrics● Stackdriver @ our colo

Thanks!

Questions?

denise stockman denise@stockmans.org

github.com/maverixbio/puppet-stackdriver-agentforge.puppetlabs.com/dstockman/stackdriver