Date post: | 11-May-2015 |
Category: |
Technology |
Upload: | david-lutz |
View: | 2,137 times |
Download: | 5 times |
logstash
Infrastructure Coders MelbourneApril 2013David Lutz
@dlutzy
What does logstash do?
It does "stuff" with log files.
Typical day (or night) in the life of a sysadmin...
Something's wrong.
Check the log files.
How?
grep
catgrepsedawktailsort
and pipes
lots of pipes
Fine if you have one server. But what if you have 10 or 100 or 1000for i in `seq 1 10` ; do ssh server$i blah blah; done
cluster ssh
Splunk perhaps?
Problems with Splunk...
1.eats log files
2. digests data
3. spits it out into other apps
inputs● amqp● drupal_dblog● eventlog● exec● file● ganglia● gelf● gemfire● generator● heroku● irc● log4j
● lumberjack● pipe● redis● relp● sqs● stdin● stomp● syslog● tcp● twitter● udp● xmpp● zenoss● zeromq
filters● alter● anonymize● checksum● csv● date● dns● environment● gelfify● geoip● grep● grok● grokdiscovery
● grokdiscovery● json● kv● metrics● multiline● mutate● noop● split● syslog_pri● urldecode● xml● zeromq
outputs● amqp● boundary● circonus● cloudwatch● datadog● elasticsearch● elasticsearch_http● elasticsearch_river● email● exec● file● ganglia● gelf● gemfire● graphite● graphtastic● http● internal● irc● juggernaut● librato
● loggly● lumberjack● metriccatcher● mongodb● nagios● nagios_nsca● null● opentsdb● pagerduty● pipe● redis● riak● riemann● sns● sqs● statsd● stdout● stomp● syslog● tcp● websocket● xmpp● zabbix● zeromq
How to: install logstash
wget http://logstash.objects.dreamhost.com/release/logstash-1.1.9-monolithic.jar
easy!
How to: run logstash
java -jar logstash-1.1.9-monolithic.jar agent -f logstash.conf -- web
easy!
How to: get some apache logs in
input { tcp { type => "apache" port => 3333 } }
How to: get some apache logs in
tail -f /var/log/apache2/access.log | nc localhost 3333
How to: digest the logs
filter { grok { type => "apache" pattern => "%{COMBINEDAPACHELOG}" }
date { type => "apache" timestamp => "dd/MMM/yyyy:HH:mm:ss Z" }}
How to: output to elasticsearch
output { elasticsearch { embedded => false }}
How to: output to elasticsearch and graphite via statsd
output { elasticsearch { embedded => false } statsd { increment => "apache.response.%{response}" }}