@molly_struve
Building A Scalable Monitoring System
�1
@molly_struve
Monitoring Mistakes Overhauling the System The Payoff
�2
@molly_struve
Monitoring Mistakes
�3
Overhauling the System The Payoff
@molly_struve
�4
@molly_struve
Monitoring Mistakes
�5
Overhauling the System The Payoff
@molly_struve
Monitoring Mistakes Overhauling the System
�6
Overhauling the System The Payoff
@molly_struve
Monitoring Mistakes Overhauling the System The Payoff
�7
@molly_struve
Monitoring Mistakes Overhauling the System The Payoff
�8
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�9
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�10
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�11
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�12
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�13
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�14
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�15
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�16
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�17
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�18
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�19
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�20
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�21
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�22
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�23
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�24
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�25
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
Incredibly Inconsistent
�26
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
Inconsistent Alerts
Required no actionReported data
�27
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
Inconsistent Alerts
Required no actionReported data
Immediate action required
�28
@molly_struve
make on-call devs miserable
Monitoring Mistakes Overhauling the System The Payoff
�29
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�30
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�31
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�32
@molly_struve
Monitoring Mistakes Overhauling the System The Payoff
�33
@molly_struve
Monitoring Mistakes Overhauling the System The Payoff
�34
@molly_struve
Monitoring Must Haves
1
2
3
4
Monitoring Mistakes Overhauling the System The Payoff
Consolidate Monitoring To a Single Place
�35
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
!" #$
%
�36
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�37
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�38
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�39
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�40
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�41
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�42
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�43
@molly_struve
Make ALL Alerts Actionable
Monitoring Must Haves
1
2
3
4
Monitoring Mistakes Overhauling the System The Payoff
Consolidate Monitoring To a Single Place
�44
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�45
Action
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�46
Action
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
Action Required No action Needed
�47
@molly_struve
#ops_alerts
Monitoring Mistakes Overhauling the System The Payoff
#dev_alerts
Action Required No action Needed
�48
@molly_struve
#ops_alerts
Monitoring Mistakes Overhauling the System The Payoff
#ops_reporting#dev_alerts
Action Required No action Needed
#dev_reporting
�49
@molly_struve
Make Alerts MutableMake ALL Alerts ActionableConsolidate Monitoring To a Single Place
Monitoring Must Haves
1
2
3
4
Monitoring Mistakes Overhauling the System The Payoff
�50
@molly_struve
Make Alerts MutableMake ALL Alerts ActionableConsolidate Monitoring To a Single Place
Monitoring Must Haves
1
2
3
4
Monitoring Mistakes Overhauling the System The Payoff
�51
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
30 60 90minutes
�52
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�53
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�54
Miss new alerts
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�55
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�56
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�57
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�58
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�59
@molly_struve
Track Alert HistoryMake Sure Alerts Are MutableMake ALL Alerts Actionable
Monitoring Must Haves
1
2
3
4
Monitoring Mistakes Overhauling the System The Payoff
Consolidate Monitoring To a Single Place
�60
@molly_struve
Tracking Alert History
Monitoring Mistakes Overhauling the System The Payoff
�61
@molly_struve
Tracking Alert History
Monitoring Mistakes Overhauling the System The Payoff
�62
@molly_struve
Tracking Alert History
Monitoring Mistakes Overhauling the System The Payoff
�63
@molly_struve
Track Alert HistoryMake Sure Alerts Are MutableMake ALL Alerts ActionableConsolidate Monitoring To a Single Place
Monitoring Must Haves
1
2
3
4
Monitoring Mistakes Overhauling the System The Payoff
�64
@molly_struve
Monitoring Mistakes Overhauling the System The Payoff
�65
@molly_struve
Monitoring Mistakes Overhauling the System The Payoff
�66
@molly_struve
On boarding is a breeze
Monitoring Mistakes Overhauling the System The Payoff
�67
@molly_struve
3 On-boarding steps:
1
2
3
Monitoring Mistakes Overhauling the System The Payoff
�68
@molly_struve
Show them the monitoring setup1
2
3
Monitoring Mistakes Overhauling the System The Payoff
3 On-boarding steps:
�69
@molly_struve
Show them the monitoring setup1
2
3
Monitoring Mistakes Overhauling the System The Payoff
If an alert goes off you have to address it
3 On-boarding steps:
�70
@molly_struve
Show them the monitoring setup1
2
3
Monitoring Mistakes Overhauling the System The Payoff
How to mute a triggered alert
3 On-boarding steps:
�71
If an alert goes off you have to address it
@molly_struve
On boarding is a breeze
Monitoring Mistakes Overhauling the System The Payoff
�72
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
Happier on-call developers
�73
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
All alerts must be actionable
�74
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�75
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
No more noise
�76
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�77
@molly_struve
Developers began helping to improve our monitoring system
Monitoring Mistakes Overhauling the System The Payoff
�78
@molly_struve
30-40 Alerts
Monitoring Mistakes Overhauling the System The Payoff
�79
@molly_struve
>90 Alerts
Monitoring Mistakes Overhauling the System The Payoff
�80
@molly_struveMonitoring Mistakes Overhauling the System The Payoff
�81
@molly_struve
If you want…
1
2
3
Monitoring Mistakes Overhauling the System The Payoff
�82
@molly_struve
On boarding to be a breeze.1
2
3
Monitoring Mistakes Overhauling the System The Payoff
If you want…
�83
@molly_struve
On-call developers to be a lot happier
1
2
3
Monitoring Mistakes Overhauling the System The Payoff
If you want…
On boarding to be a breeze.
�84
@molly_struve
Developers to help improve your monitoring systems
1
2
3
Monitoring Mistakes Overhauling the System The Payoff
If you want…
On boarding to be a breeze.
�85
On-call developers to be a lot happier
@molly_struve
Track Alert HistoryMake Sure Alerts Are MutableMake ALL Alerts ActionableConsolidate Monitoring To a Single Place
Monitoring Must Haves
1
2
3
4
�86
@molly_struve
❤❤❤
�87
@molly_struve
Questions?
�88