Date post: | 16-Apr-2017 |
Category: |
Technology |
Upload: | nederlandstalige-zabbix-gebruikersgroep |
View: | 2,953 times |
Download: | 8 times |
Xifeo ICT | Dennis Kanbier
Monitoring the ELK stack using Zabbix (and
Grafana)Dennis Kanbier
26 Nov 2015 - NLZGG
Xifeo ICT | Dennis Kanbier
Overview• Introduction
• What is ELK and why do I want it?
• How to monitor ELK with Zabbix
• Impress people with pretty graphs
• Demo time ( please tweet using #nlzgg! )
Xifeo ICT | Dennis Kanbier
Who am I?
• Dennis Kanbier
• Linux Consultant with Xifeo ICT
• “dkanbier” on Zabbix forums and IRC
Xifeo ICT | Dennis Kanbier
What is the ELK stack• Elasticsearch
• Search server
• Based on Apache Lucene
• Logstash
• Data pipeline
• Processes logs and other data
• Plugins
• Kibana
• Web frontend for Elasticsearch
Xifeo ICT | Dennis Kanbier
Basic Logstash Pipeline
Xifeo ICT | Dennis Kanbier
Apache log example• 83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] "GET /presentations/
logstash-monitorama-2013/images/ahiruyaki.png HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36”
• A few interesting fields:
• client_ip: 83.149.9.216
• status code: 200
• timestamp: 04/Jan/2015:05:13:42 +0000
• and much more
Xifeo ICT | Dennis Kanbier
{ "clientip" : "83.149.9.216", "ident" : , "auth" : , "timestamp" : "04/Jan/2015:05:13:42 +0000", "verb" : "GET", "request" : “…/a-2013/images/ahiruyaki.png”, "httpversion" : "HTTP/1.1", "response" : "200", "bytes" : "203023", "referrer" : “http://semicomplete.com …“, "agent" : "Mozilla/5.0 (Macintosh; Intel Mac OS X…” }
Logstash processes the log message and produces output:
Xifeo ICT | Dennis Kanbier
Now what?• Use the processed data to do fun things like:
• plot all entries using the [timestamp] field to see how busy the web server is during the day
• which part of your website gets requested the most using the [request] field
• monitor the status of your web server using the [response] field
• Or do even more fun things using plugins…
Xifeo ICT | Dennis Kanbier
logstash-filter-geoip• Use the geo-ip filter plugin while logstash is processing the
message
• Looks up the geographical location of an IP address and adds this information to the message
• Use Kibana to create a heat map based on the geographical information in each message to see from which part in the world your website gets visited
• Very nice to have in case of DDOS attacks originating from places you don’t expect visitors from
• Or to determine where to up your advertising game
Xifeo ICT | Dennis Kanbier
Example heat map
Xifeo ICT | Dennis Kanbier
Developer issues
Xifeo ICT | Dennis Kanbier
Not always the developers fault
Xifeo ICT | Dennis Kanbier
Why developers love ELK• Logs from all components and servers are available in one central place, easily searchable using
the Kibana web interface
• No more begging the scary Unix administrator for log files
• Log message are enriched with meaningful fields which enables users to:
• query on stack traces
• query on specific thread id’s
• query on log_level
• query on customer_id
• list faults per server/application/cluster/log_type/etc..
• You can easily plot logs, see how fast requests are handled
• Timestamps are normalised, even log entries from different time zones are searchable chronologically
Xifeo ICT | Dennis Kanbier
Example Kibana screen
Xifeo ICT | Dennis Kanbier
Monitoring Logstash with Zabbix
• Used 3 components to generate data and send it to the Zabbix server:
• logstash-input-heartbeat
• logstash-filter-metrics
• logstash-output-zabbix
Xifeo ICT | Dennis Kanbier
logstash-input-heartbeat• Plugin to generate a heartbeat message at a
configurable interval
• The message contains a field called “clock” with the epoch time of the moment it was send
• Assign a type of “heartbeat” to the message so we can use that to filter out the message further in the pipeline and send it to the Zabbix server
Xifeo ICT | Dennis Kanbier
logstash.confinput { heartbeat { message => "epoch" interval => 20 type => "heartbeat" }}
filter {}
output {}
Xifeo ICT | Dennis Kanbier
logstash-output-zabbix
• Used to send data to Zabbix
• Requires a configured Zabbix host and trapper item to receive the data
• These fields must be added to the [@metadata] field of the message you want to send to the Zabbix server
Xifeo ICT | Dennis Kanbier
logstash.confinput { heartbeat { message => "epoch" interval => 20 type => "heartbeat" }}
filter { if [type] == "heartbeat" { mutate { add_field => { "[@metadata][zabbix_key]" => "logstash_heartbeat" } add_field => { "[@metadata][zabbix_host]" => "logstash-indexer-1" } } }}
output { zabbix { zabbix_server_host => “zabbix-server.example.com” zabbix_host => “[@metadata][zabbix_host]” zabbix_key => “[@metadata][zabbix_key]" zabbix_value => clock}
Xifeo ICT | Dennis Kanbier
Results in Zabbix• Zabbix host logstash-indexer-1 should already have the item
with key “logstash_heartbeat” configured as a Zabbix trapper item.
• Logstash now sends the epoch value every 20 seconds to the logstash_heartbeat trapper item
• You can use this to further process the data in Zabbix:
• trigger when there is too much time between epoch values
• trigger when you do not receive any values for a period of time
Xifeo ICT | Dennis Kanbier
Example item in Zabbix
Xifeo ICT | Dennis Kanbier
logstash-filter-metrics• Plugin to count messages and flush the result on a
specific interval
• Used to monitor logstash performance by counting messages and store their count value in Zabbix as Delta (speed per second)
• which gives us the messages this logstash server handles per second
Xifeo ICT | Dennis Kanbier
logstash.confinput { …}
filter { metrics { meter => "events" add_tag => "metric" add_field => { "[@metadata][zabbix_key]" => "logstash_events" } add_field => { "[@metadata][zabbix_host]" => “logstash-indexer-1“ } flush_interval => 20 }}
output { if "metric" in [tags] { zabbix { zabbix_server_host => “zabbix-server.example.com” zabbix_host => “[@metadata][zabbix_host]” zabbix_key => “[@metadata][zabbix_key]" zabbix_value => “[events][count]” } }}
Xifeo ICT | Dennis Kanbier
More results!• We now have the item “logstash_events” which
shows how many messages we process per second:
Xifeo ICT | Dennis Kanbier
Now about that last slide….
• Presenting reports and graphs are not exactly Zabbix’ strongest points
• Works all right to view single values like on the previous slide
• But try to align multiple graphs nicely on one screen to put on a wall and impress people
Xifeo ICT | Dennis Kanbier
Grafana• Build to present metric data in a simple and nice
manner
• Supports multiple data sources like Graphite and InfluxdB
• Alexander Zobnin created grafana-zabbix, which enables us to use the Zabbix database as a Grafana backend through the Zabbix API
• https://github.com/alexanderzobnin/grafana-zabbix
Xifeo ICT | Dennis Kanbier
Example dashboard
Xifeo ICT | Dennis Kanbier
Things to consider• Grafana queries the Zabbix database through the
API
• If you have a lot of graphs with a lot of items these queries are quite expensive
• Especially if you have multiple dashboards refreshing automatically on a high interval
• Still learning on how to best counter these challenges
Xifeo ICT | Dennis Kanbier
Live demo time!Did you remember to tweet things with hashtag #nlzgg?
Xifeo ICT | Dennis Kanbier
Questions?
Xifeo ICT | Dennis Kanbier
Contact details
• dkanbier on Freenode IRC
• https://github.com/dkanbier
• http://denniskanbier.nl/blog for tutorials
Resources• http://www.zabbix.com • https://github.com/alexanderzobnin/grafana-zabbix • https://www.elastic.co • http://www.xifeo.nl