The devops approach to monitoring, Open Source and Infrastructure as Code Style

Post on 26-Jan-2015

106 views 0 download

Tags:

description

Monitoring is critical for every decent application that runs on production. Many of the monitoring tools widely used show their limits at the age of Infrastructure as Code and Cloud computing. Let's investigate how monitoring can face the new challenges: scalability, reproducability and automation

transcript

;

The devops approach to monitoring,The devops approach to monitoring,

Open Source and IAC StyleOpen Source and IAC Style

Julien PivottoJulien Pivotto

Open World ForumOpen World ForumOctober 4, 2013October 4, 2013

;

whoamiwhoami• sysadmin @ inuitssysadmin @ inuits• open-source defender for 7+ yearsopen-source defender for 7+ years• devops believerdevops believer• @roidelapluie on twitter/github@roidelapluie on twitter/github

Julien Pivotto DevOps and monitoring

;

DevOpsDevOps

• CultureCulture• AutomationAutomation• MeasurementMeasurement• SharingSharing

Damon Edwards and John WillisDamon Edwards and John Willis

Julien Pivotto DevOps and monitoring

;

Monitoring is usuallyMonitoring is usuallyan afterthoughtan afterthought

ENOTIME, ENOBUDGETENOTIME, ENOBUDGET

Julien Pivotto DevOps and monitoring

;

#monitoringsucks#monitoringsucks

• A movement started in 2011A movement started in 2011• http://github.com/monitoringsuckshttp://github.com/monitoringsucks• A lot of tools and informationA lot of tools and information

Julien Pivotto DevOps and monitoring

;

GoalsGoals• Find when a service is unavailableFind when a service is unavailable• Understand failure post-mortemUnderstand failure post-mortem

• Learn from your infrastructureLearn from your infrastructure• AnticipateAnticipate

Julien Pivotto DevOps and monitoring

;

GoalsGoals• Find when a service is unavailableFind when a service is unavailable• Understand failure post-mortemUnderstand failure post-mortem• Learn from your infrastructureLearn from your infrastructure• AnticipateAnticipate

Julien Pivotto DevOps and monitoring

;

Monitor everythingMonitor everything

• ServersServers• ServicesServices• UsageUsage• HardwareHardware• SoftwareSoftware• PeoplePeople

Julien Pivotto DevOps and monitoring

;

Monitor every environmentMonitor every environment

• See performance changes in devSee performance changes in dev• Fix them before it hits productionFix them before it hits production

Julien Pivotto DevOps and monitoring

;

MetricMetric

• Time + name + value = metricTime + name + value = metric• Can be anythingCan be anything

Julien Pivotto DevOps and monitoring

;

EventEvent

• Time + fields = metricTime + fields = metric• Logs become usable dataLogs become usable data• Can be transformed into metricsCan be transformed into metrics

Julien Pivotto DevOps and monitoring

;

Metrics + eventsMetrics + events• Overview of your infrastructureOverview of your infrastructure• Usage and state of the servicesUsage and state of the services• Combine several metricsCombine several metrics• Extract business valuesExtract business values

Julien Pivotto DevOps and monitoring

;

AutomationAutomation

• Infrastructure as CodeInfrastructure as Code• Automate everythingAutomate everything⇒ One source of truthOne source of truth

Julien Pivotto DevOps and monitoring

;

DeploymentDeployment• Definitions of a service includes monitoringDefinitions of a service includes monitoring• Deployed ⇔ monitoredDeployed ⇔ monitored

Julien Pivotto DevOps and monitoring

;

ToolsTools

• No all-in-one toolNo all-in-one tool• No autodiscovery toolNo autodiscovery tool• Text-based configurationText-based configuration• ScalableScalable⇒ The Unix philosphyThe Unix philosphy

Julien Pivotto DevOps and monitoring

;

IcingaIcinga

• Fork of nagiosFork of nagios• Large and vibrant communityLarge and vibrant community• Configuration compatible with nagiosConfiguration compatible with nagios• User-friendly interfaceUser-friendly interface• Use Icinga Classic!Use Icinga Classic!

Julien Pivotto DevOps and monitoring

;

IcingaIcingahttps://icinga.org

Julien Pivotto DevOps and monitoring

;

SensuSensu• FlexibilityFlexibility• Compatible with nagios pluginsCompatible with nagios plugins• Connects to your source of trustConnects to your source of trust• Relies on RabbitMQRelies on RabbitMQ

Julien Pivotto DevOps and monitoring

;

CollectdCollectd

• Statistics collection daemonStatistics collection daemon• A lot of plugins available. . .A lot of plugins available. . .• Can send data to graphiteCan send data to graphite• Simple configurationSimple configuration

Julien Pivotto DevOps and monitoring

;

Collectd pluginsCollectd pluginshttp://www.flickr.com/photos/juhansonin/3141561416/

Julien Pivotto DevOps and monitoring

;

Collectd pluginsCollectd pluginsAMQP Apache APC_UPS Apple_Sensors Ascent Battery BIND CarbonAMQP Apache APC_UPS Apple_Sensors Ascent Battery BIND CarbonConnTrack ContextSwitch CPU CPUFreq CSV cURL cURL-JSON cURL-XMLConnTrack ContextSwitch CPU CPUFreq CSV cURL cURL-JSON cURL-XMLDBI DF Disk DNS E-Mail Entropy Exec FileCount FSCache GenericJMXDBI DF Disk DNS E-Mail Entropy Exec FileCount FSCache GenericJMXgmond HDDTemp Interface IPMI IPTables IPVS IRQ Java libvirt Loadgmond HDDTemp Interface IPMI IPTables IPVS IRQ Java libvirt LoadLogFile LPAR MadWifi MBMon memcachec memcached Memory ModbusLogFile LPAR MadWifi MBMon memcachec memcached Memory Modbus

Monitorus Multimeter MySQL NetApp Netlink Network NFS nginxMonitorus Multimeter MySQL NetApp Netlink Network NFS nginxNotify_Desktop Notify_Email NTPd NUT olsrd OneWire OpenVPN OpenVZNotify_Desktop Notify_Email NTPd NUT olsrd OneWire OpenVPN OpenVZOracle Perl Pinba Ping PostgreSQL PowerDNS Processes Protocols PythonOracle Perl Pinba Ping PostgreSQL PowerDNS Processes Protocols PythonRedis RouterOS RRDCacheD RRDtool Sensors Serial SNMP Swap SysLogRedis RouterOS RRDCacheD RRDtool Sensors Serial SNMP Swap SysLogTable Tail Tape TCPConns TeamSpeak2 TED thermal TokyoTyrant UnixSockTable Tail Tape TCPConns TeamSpeak2 TED thermal TokyoTyrant UnixSock

Uptime Users UUID Varnish vmem VServer Wireless XMMSUptime Users UUID Varnish vmem VServer Wireless XMMSWrite_Graphite Write_HTTP Write_MongoDBWrite_Graphite Write_HTTP Write_MongoDB

Write_Redis Write_Riemann ZFS_ARCWrite_Redis Write_Riemann ZFS_ARC

Julien Pivotto DevOps and monitoring

;

LogstashLogstash

• Ship logs from any sourceShip logs from any source• Filter themFilter them• Index themIndex them• Search themSearch them• Backed with elasticsearchBacked with elasticsearch

Julien Pivotto DevOps and monitoring

;

LogstashLogstash

Julien Pivotto DevOps and monitoring

;

GraphiteGraphite

• GraphingGraphing• Accept any metricAccept any metric• Store data in files (whisper)Store data in files (whisper)• A lot of helpers functionsA lot of helpers functions• Listen on UDP and TCPListen on UDP and TCP

Julien Pivotto DevOps and monitoring

;

Send data to graphiteSend data to graphiteecho "stats.sshd.login 1 $(date +%s)" | nc -u graphite.example.com 2003echo "stats.sshd.login 1 $(date +%s)" | nc -u graphite.example.com 2003

Julien Pivotto DevOps and monitoring

;

Graphite APIGraphite API

Julien Pivotto DevOps and monitoring

;

StatsdStatsd

• Graphite friendGraphite friend• Stats aggregationStats aggregation• Simple countersSimple counters• Flushes every XX seconds to graphiteFlushes every XX seconds to graphite• UDPUDP

Julien Pivotto DevOps and monitoring

;

Feeding statsdFeeding statsdecho "stats.sshd.login:1|c" |nc -u statsd.example.com 8125

Julien Pivotto DevOps and monitoring

;

gdashgdashhttps://github.com/ripienaar/gdash

Julien Pivotto DevOps and monitoring

;

giraffegiraffe

Alternative to gdash

Julien Pivotto DevOps and monitoring

;

KibanaKibana• Kibana is a web interface for Logstash/ESKibana is a web interface for Logstash/ES• Kibana 1 was written in PHPKibana 1 was written in PHP• Kibana 2 was written in RubyKibana 2 was written in Ruby• Kibana 3 is written in AngularJSKibana 3 is written in AngularJS

Julien Pivotto DevOps and monitoring

;

Kibana 3Kibana 3• Everything happens in the browserEverything happens in the browser• The browser is connected to ElasticsearchThe browser is connected to Elasticsearch• You can save dashboards into ESYou can save dashboards into ES• You can write/template dashboards to filesYou can write/template dashboards to files

Julien Pivotto DevOps and monitoring

;

Kibana queriesKibana queriesExample of a kibana query@fields.syslog_program:"httpd" AND@fields.http_host:"test.example.com" AND@fields.response:"404"

• LLucene query syntax• SSimple and effective• PPoint & click web interface

Julien Pivotto DevOps and monitoring

;Julien Pivotto DevOps and monitoring

;Julien Pivotto DevOps and monitoring

;Julien Pivotto DevOps and monitoring

;

Toolchain exampleToolchain example• Apache ships logs to rsyslogApache ships logs to rsyslog• Rsyslog ships logs to logstashRsyslog ships logs to logstash• Logstash ships metrics to statsdLogstash ships metrics to statsd• Statsd ships metrics to GraphiteStatsd ships metrics to Graphite• Icinga query metric from graphiteIcinga query metric from graphite• https://github.com/etsy/nagios_toolshttps://github.com/etsy/nagios_tools

Julien Pivotto DevOps and monitoring

;

Reusing Icinga/Nagios perfdataReusing Icinga/Nagios perfdata• Icinga performs various checksIcinga performs various checks• Icinga sends perfdata to graphiteIcinga sends perfdata to graphite• Graphite stores the dataGraphite stores the data• Gdash serves them inside dashboardsGdash serves them inside dashboards• https://github.com/roidelapluie/icinga-to-graphitehttps://github.com/roidelapluie/icinga-to-graphite

Julien Pivotto DevOps and monitoring

;

SharingSharing• Build dashboard: dashing, teamdashBuild dashboard: dashing, teamdash• Share with developersShare with developers• Share with managersShare with managers

Julien Pivotto DevOps and monitoring

;

Try them yourselfTry them yourselfhttps://github.com/KrisBuytaert/vagrant-graphitehttps://github.com/KrisBuytaert/vagrant-graphite

Julien Pivotto DevOps and monitoring

;

Thank youThank youAny question?Any question?

Julien Pivotto DevOps and monitoring

;

ContactContactJulien PivottoJulien Pivottojulien@inuits.eujulien@inuits.eu@roidelapluie@roidelapluie

INUITS bvbaINUITS bvbaDuboisstraat 50Duboisstraat 502060 Antwerp2060 AntwerpBelgiumBelgium+32 473 441 636+32 473 441 636https://inuits.euhttps://inuits.eu

Julien Pivotto DevOps and monitoring