Date post: | 08-Sep-2014 |
Category: |
Technology |
Upload: | andy-sykes |
View: | 84,907 times |
Download: | 5 times |
(so it can die peacefully)
Please stop using Nagios
Andy Sykes Devops @ Forward3D @supersheep [email protected]
Do you use Nagios? Tell me why you picked it. Go on. If you don't, why don't you?
Reasons for choosing Nagios
• stupid simple plugin system • billions* of existing plugins • years of development behind it • you can hire people who know it "Everybody uses it."**
* may not actually be true ** except me. and maybe you. and that guy at the back, who really likes Zabbix. you know who you are.
Reasons for choosing Nagios
• stupid simple plugin system • billions* of existing plugins • years of development behind it • you can hire people who know it "Everybody uses it."**
* may not actually be true ** except me. and maybe you. and that guy at the back, who really likes Zabbix. you know who you are.
So why did you pick Nagios?
Because it's the "safe", default choice. Because we've grown accustomed to the things that really, really suck about it. It's a little like we've all got Stockholm Syndrome.
What Nagios gets right
Incredibly simple plugin model. Fairly secure (SSL between agents + master). Very simple conceptually. Reliable.
Nagios, I hate thee; let me count thy ways
Doesn't scale. At all. World's second most horrible configuration*. Horrendous interface**. Assumes a static infrastructure. No decent programmatic interfaces***. Throws away perfdata. Stupid wire format for clients (NRPE/NSCA).
* the world's most horrible configuration is, obviously, Sendmail. ** even the paid Nagios XI one is ugly as sin and unusable. *** if I catch you parsing status.dat, I will beat your ass.
Expansion about config
Configuration has to be in two places:
Server has to know what checks to invoke via NRPE.
Client has to know what checks it will be asked to invoke with NRPE.
THIS IS MADNESS.
Scaling, or lack of it
No such thing as a Nagios cluster. More checks = more work = longer before you know something's happened! Every check increases your master's load average.
Okay, yes, there’s mod_gearman
But it’s a hack at best. No redundancy for the machine that distributes the checks, so it’s not a real cluster.
API poverty
Can't easily integrate with other systems. Can't easily write custom dashboards. Can't get information out again!
Assumes a static infra Master has to be told about a client before things can happen.
The bandaids we make Interface:
Opsview, Icinga, Shinken, others
API: Parsing status.dat, NDO
Client wire format: Opsview's NRPE, NRD
Config management: Puppet types, Chef cookbooks
None of it is good enough.
The take-home point:
"If we keep using Nagios, we'll never get anything better." (Writing monitoring systems is hard, and needs community involvement and real world adoption. Nagios steals mindshare by being just good enough. It's the monitoring system we deserve, but not the one we need right now.)
So, smart guy. What do we do?
Steal all the things that are great about Nagios. (existing plugin investment, simplicity, security, reliability)
Strap them to something more awesome. (scalable, API-ready, config management friendly, modern!)
THIS DOESN’T MEAN WRITING YOUR OWN MONITORING SYSTEM
Points for thought:
● What else are people using? ● Should we greenfield or lift existing tools? ● What tools could we go with?
My suggestion:
Like OMD, but better. Wrap up a series of “best in breed” tools to make one kickass monitoring tool.
What we need:
Core Agent Graphing Anomaly detection Alerting UI
Core:
Holds configuration about hosts / services Distributed across X masters Check execution (poke) Results queue (poke response)
There’s something we can use for this.
Sensu! Sensu is often described as the “monitoring router”.
{ "checks": { "chef_client": { "command": "check-chef-client.rb", "subscribers": [ "production" ], "interval": 60, "handlers": [ "pagerduty", "irc" ] } } }
Only on the server
Client requires no registration for the server to know about it Uses Nagios status return codes Doesn’t talk to the server - talks to RabbitMQ
Core:
Holds configuration about hosts / services Distributed across X masters Check execution (poke) Results queue (poke response)
What we need:
Core - Sensu-server Agent - Sensu-client Graphing Anomaly detection Alerting UI
Graphing is easy now.
If you’re not using Graphite, you should be. Sensu “metric” checks can pump data to it.
What we need:
Core - Sensu-server Agent - Sensu-client Graphing - Graphite Anomaly detection Alerting UI
Anomaly detection is hard.
We’ve got all this metric data, but how do we check it? - Skyline/Oculus (Etsy) - Grok (very early days) - ???
What we need:
Core - Sensu-server Agent - Sensu-client Graphing - Graphite Anomaly detection - ??? Alerting UI
Alerting is tricky, but mostly solved.
Flapjack! - flapjack.io Alerting is not the concern of your monitoring tool. Push all alerts at Flapjack - define gateways (PagerDuty, email) - create relationships between checks and gateways
What we need:
Core - Sensu-server Agent - Sensu-client Graphing - Graphite Anomaly detection - ??? Alerting - Flapjack UI
User interfaces are hard.
What do we need from it? - What’s broken - When it broke, when it broke in the past - Say “OK, I know it’s broken” - View graphs to see how quickly it broke - See every check everywhere, and filter the list
The Sensu Dashboard sucks.
No history! Acknowledgements aren’t easy to do. No graphing. Can’t see anything that’s reporting an OK status. This won’t do.
I’m going to have to write a UI. Sigh.
What we need:
Core - Sensu-server Agent - Sensu-client Graphing - Graphite Anomaly detection - ??? Alerting - Flapjack UI - ???
In Summary
Nagios sucks. There are good tools for each concern of monitoring. If we can package them together, we can have something that rocks.