ELKBigData for DevOps
Javantura v3 // February 20, 2016 Maarten Mulders // @mthmulders
logstashvery modular: various inputs, filters and outputs
input: various application log files, but also syslog, stdin, xmpp, log4jsocket, irc, ...
filter: extract semantics (geo info, grok), add information, removeinformation, match fields (cidr, dates, numbers, dns, user agent), ...
output: send events to another system such as graphite,elasticsearch, email, file, stdout, irc, jira, nagios, s3, redis, xmpp, ...
elasticsearch and analytics engine
very scalable
stores collected log events in an uniform way
events can be filtered and queried by clients (e.g. kibana)
kibanaanalytics and search dashboard for elastic
just html and javascript (dashboards can be saved to elastic, too)
filtering determines what data is used to populate the dashboard,queries categorise data inside the dashboard
logstash setupprocessess technical logging and audit logging
adds information (hostname, environment, application name)
removes information (sensitive details about customers,transactions)
transforms information to a more usable form
ship events to redis
elastic setuplarge cluster that contains data
one month of history
also hosts kibana files and stores its dashboards
kibana configurationfilters based on environment and timestamp (last 24h)
automatically refreshed
queries for 'error', 'orange cell', specific error codes
rows and panels for optimal screen usage
logstash inputinput { file { path => "/path/to/application.log" codec => multiline { pattern => "̂%{TIMESTAMP_ISO8601} " negate => true what => previous } type => "application" } file { path => "/path/to/audit.log" type => "audit" } }
logstash filtersregular application log file
filter { if [type] == "application" { grok { match => { "message" => "(?m)%{TIMESTAMP_ISO8601:timestamp} \[%DATA\] %{LOGLEVEL:level} %{JAVACLASS} %{GREEDYDATA:line}" } remove_field => "message" } } }
logstash filters (ctd)audit log file
2015-01-28 01:32:15,098 [thread-1] INFO nl.ing.application.Class eventId=1401751935098~|~inChannel=MINGZ~|~odBeneficiaryAccount=NL28INGB0000000001
filter { if [type] == "audit" { grok { match => { "message" => "(?m)%{TIMESTAMP_ISO8601:timestamp} \[%DATA\] %{LOGLEVEL} %{JAVACLASS} %{GREEDYDATA:audit_message}" } remove_field => "message" } mutate { gsub => ["audit_message", "\~\|\~", "̀"] } kv { source => "audit_message" field_split => "̀" remove_field => "audit_message" } prune { blacklist_names => "̂od.+$" } } }
{ timestamp: "2015-01-28 01:32:15,098", eventId: "1401751935098", inChannel: "MINGZ" }
logstash filters (ctd)just in case...
filter { if "_grokparsefailures" in [tags] { prune { blacklist_names => [ "message", "audit_message" ] } } }