SCALABLEMONITORING & ALERTING
NEW YORK CITY
NEW YORK CITY
EXPERIENCE
THE BEGINNING
NEW YORK CITY
NEW YORK CITY
DEPLOYMENTS
+ =
NEW YORK CITY
DDOS ATTACKS
NEW YORK CITY
THE VISION● Easy publishing of metrics
● Single source of truth
● Democratized access
NEW YORK CITY
CARBON-AGGREGATOR
CARBON-CACHE
WHISPER
FILESYSTEM (rotating disks)
SEPTEMBER 2013
GRAPHITE WEBAPP
NEW YORK CITY
16 CARBON-RELAY
10 CARBON-CACHE
WHISPER
FILESYSTEM (rotating)
LOAD BALANCER
WEBAPP
JANUARY 2014
16 CARBON-RELAY
10 CARBON-CACHE
WHISPER
FILESYSTEM (rotating)
WEBAPP
16 CARBON-RELAY
10 CARBON-CACHE
WHISPER
FILESYSTEM (rotating)
WEBAPP
NEW YORK CITY
CULTURE
NEW YORK CITY
CULTURE
SUPER BOWL 2014
NEW YORK CITY
NEW YORK CITY
NEW YORK CITY
THE PROBLEMS● Reads didn’t scale
● No usage insight
● Dumb dashboards
NEW YORK CITY
8 CARBON-RELAY
LOAD BALANCER
APRIL 2014
8 CARBON-RELAY 8 CARBON-RELAY
10 CARBON-CACHE
WHISPER
FILESYSTEM (SSDs)
WEBAPP
NEW YORK CITY
8 CARBON-RELAY
LOAD BALANCER
MAY 2014
8 CARBON-RELAY 8 CARBON-RELAY
10 CARBON-CACHE
WHISPER
FILESYSTEM (SSDs)
GRAPHITE API
NEW YORK CITY
8 CARBON-RELAY
LOAD BALANCER
JUNE 2014
8 CARBON-RELAY
8 CARBON-RELAY
10 CARBON-CACHE
WHISPER
FILESYSTEM (SSDs)
GRAPHITE API
10 CARBON-AGGREGATOR 10 CARBON-AGGREGATOR
10 CARBON-AGGREGATOR
NEW YORK CITY
USAGE INFORMATION
+
=
GRAPHITE API + +
NEW YORK CITY
NEW YORK CITYGRAFANA
ALERTING
NEW YORK CITY
Core InfrastructureTHRESHOLDS
CRITICAL THRESHOLD
WARNING THRESHOLD
Core InfrastructureALERTING MATH
Core InfrastructureALERTING MATH
SkipRead30 5
Alarm9
Core InfrastructureALERTING MATH
SkipRead30 5
Frequency10 sec
300 secs 50 secs
Alarm9
90 secs
Core InfrastructureNAGIOS INTEGRATION
NEW YORK CITY
NEXT STEPS● Replication
● Scale even more
● More robust alerting logic
NEW YORK CITY
8 CARBON-RELAY
LOAD BALANCER
REPLICATION
8 CARBON-RELAY
8 CARBON-RELAY
10 CARBON-CACHE
WHISPER
FILESYSTEM (SSDs)
GRAPHITE API
10 CARBON-AGGREGATOR 10 CARBON-AGGREGATOR
10 CARBON-AGGREGATOR
Core InfrastructureALERTING LOGIC
= Google Forms Throttling
Core InfrastructureALERTING LOGIC
Trial subscription signups
Internal subscription API calls
Database host offline
User
Application
System
NEW YORK CITYLEARN MORE
QUESTIONS
NEW YORK CITY
