Bastian Widmer / @dasrecht

Are you visualizing your logfiles?

Bastian Widmer / @dasrecht

Visualizing Logfiles with ELK Stack

Hola ¿Com estàs?

Bastian Widmer

@dasrecht / bastianwidmer.ch

DrupalCI: Modernizing Testbot Initiative

Chief YoloOps Evangelist

Agenda 1 Introduction2

3

4

ArchitectureELK Stack

Tools!AutomationP22N - Performance Optim…

5

6

Visualizing Logfiles, why?

„Can you check the errors from yesterday between 15.02 and 15.07“

Visualization > Plaintext

Patch deployed, instant feedback!

Visualization > Plaintext

VISUALIZATION > Plaintext

Do you log to database? dblog?

Okay for one site, but what if you have 70+ sites logging into your

database?

Use Cases• Audit Trail - Who changed what?

• Content

• Modules

• Errors - Fixing errors and getting instant feedback by easy readable graphs

• Billing

• Application Speed

• Deep Inspection (TOR Nodes)

ELK Stack!ELK Stack!

ELK Stack!ELK Stack!ElasticsearchLogstashKibana

Sidenote : Things move fast! Even with minor releases

Elasticsearch

Elasticsearch®• Java

• Search and Index

• Distributed — Copies & Shards

• Clustering (Zen Discovery - Multi/Unicast)

• API — JSON / RESTful

• Apache Lucene

• Disk-Based Shard Allocation

Elasticsearch• Index

like a Database

• Replica Copies for Fault Tolerance

• ShardLucene Instance which indexes the Data see : http://blog.liip.ch/archive/2013/07/19/on-elasticsearch-performance.html

Elasticsearch{ "status" : 200, "name" : "es-03", "cluster_name" : "cluster01", "version" : { "number" : "1.7.1", "build_hash" : "b88f43fc40b0bcd7f173a1f9ee2e97816de80b19", "build_timestamp" : "2015-07-29T09:54:16Z", "build_snapshot" : false, "lucene_version" : "4.10.4" }, "tagline" : "You Know, for Search" }

Elasticsearch

ElasticSearch Plugins

• New Integrated Plugin System

• ‚Bundles‘ Plugins with Elasticsearch

• „bin/plugin -install YOURPLUGIN"

ElasticSearch Security

• Speak with me:

• „I will hereby solemnly swear not to expose my Elasticsearch Server to public, never-ever!“

• Elastic Shield - Provides Security (Subscription Feature)

ElasticSearch Security - cheap

• Run Elasticsearch bound to localhost

• use an internal network • ssh elasticsearch@elasticsearch.amazee.io -N -L

9200:127.0.0.1:9200'

Thankmelater™

• Security can be an issue

• curl -XDELETE ‚http://localhost:9200/*/’

• curl -XDELETE ‚http://localhost:9200/_all/’

• action.destructive_requires_name: true

https://www.elastic.co/guide/en/elasticsearch/reference/1.7/_parameters.html

Marvel

• Shows Cluster Health and Real-Time Analysis

• Free during development product

• Deep insights into index creation across cluster, routing decisions and much more

Logstash

Did the Catalan Citizens invent Logstash?

Logstash

• Multiple Input / Multiple Output

• Centralize and Process Log Data

• Collect

• Parse

• Store / Forward

The life of an event

• Input

• Filters

• Codecs

• Output

Logstash• JRuby*

• >1.4.0 - FlatJAR Release is gone

• Instead of running „java -jar logstash.jar“ — „bin/logstash“

• Contrib Plugins

• Daily Indices !

* see https://gist.github.com/jordansissel/978956

Input

• File

• Syslog

• Redis

• logstash-forwarder (former Lumberjack)

Filters

• Grok

• Mutate

• Drop

• Clone

• GeoIP (!!!)

Outputs

• Elasticsearch

• File / S3

• Graphite

• StatsD

Logstash 1 input {! 2 stdin { }! 3 }! 4 ! 5 output {! 6 stdout {! 7 codec => rubydebug! 8 }! 9 }!!

Logstash

1 vagrant@precise64$ ./logstash agent -f 1_simpleconfig.cfg! 2 very important log message!! 3 {! 4 "message" => "very important log message!",! 5 "@version" => "1",! 6 "@timestamp" => "2014-04-21T16:18:02.952Z",! 7 "host" => "precise64"! 8 }

Logstash 1 input {! 2 stdin { }! 3 }! 4 output {! 5 elasticsearch{! 6 host => "127.0.0.1"! 7 }! 8 stdout {! 9 codec => rubydebug! 10 }! 11 }

Logstash 1 input {! 2 file {! 3 path => "/var/log/syslog"! 4 start_position => beginning! 5 }! 6 }! 7 ! 8 output {! 9 stdout {! 10 codec => rubydebug! 11 }! 12 elasticsearch{! 13 host => "127.0.0.1"! 14 }! 15 }

Kibana

Some history

• Ruby

• PHP

• Just Javascript (the crowd applauds)

• Node Webserver and Javascript (Kibana 4)

Kibana 4

• D3.js - more fancyness

• More complex backend

• Much better flexibility

• Analytics and Aggregations

Architecture

Architecture

Shipper

Shipper

Shipper Broker IndexerSearch

and Storage

Architecture

Shipper

Shipper

Shipper Broker IndexerSearch

and Storage

Syslog

Architecture

Shipper

Shipper

Shipper Broker IndexerSearch

and Storage

Syslog Logstash

Architecture

Shipper

Shipper

Shipper Broker IndexerSearch

and Storage

Syslog Logstash Elasticsearch

But, Bastian…

ArchitectureThe real deal!

Logstash-Forwarder

• Written in Go

• Lightweight utility to forward logs to logstash

• Low resource usage

• TLS/SSL Encrypted Transfer

Indexer

Architecture

Shipper

BrokerBroker IndexerSearch

and Storage

Logstash Redis Logstash Elasticsearch

nginx.log

drupal.log

auth.log

Shipper

Shipper Shipper

Logstash-Forwarder

Indexer

Architecture

Shipper

BrokerBroker IndexerSearch

and Storage

Logstash Redis Logstash Elasticsearch

nginx.log

drupal.log

auth.log

Shipper

Shipper Shipper

Logstash-Forwarder

And from here you can go crazy!

Indexer

Architecture High-Available

Shipper

Broker

Broker Indexer

Search and

Storage

Search and

Storage

Logstash Redis Logstash Elasticsearch

nginx.log

drupal.log

auth.log

Shipper

Shipper

Logstash-Forwarder

But, Bastian!!!

No!

Indexer

High Available Setup with Rocketfuel!

Shipper

Broker

Broker Indexer

Search and

Storage

Search and

Storage

Logstash Redis Logstash Elasticsearch

nginx.log

drupal.log

auth.log

Shipper

Forwarder

Logstash Forwarder

HAProxy

HAProxy

KeepaliveD

Tools!(because anyone needs a bit help)

Elasticsearch Head

http://mobz.github.io/elasticsearch-head/ ./plugin -install mobz/elasticsearch-head

Elasticsearch Kopf

./plugin -install lmenezes/elasticsearch-kopf

Curator

• Time Series Indices? THIS IS THE TOOL!

• Close Indexes

• Delete (by space or time)

• Disable Bloom Filter

• Optimize / ForceMerge

• https://github.com/elasticsearch/curator

Curator

• Time Series Indices? THIS IS THE TOOL!

• Close Indexes

• Delete (by space or time)

• Disable Bloom Filter

• Optimize / ForceMerge

• https://github.com/elasticsearch/curator

Curator Perfect for Time Series Indexes

Curator

• Close indices older than 14 days, delete indices older than 30 days curator --host my-elasticsearch -d 30 -c 14

• Disable bloom filter for indices older than 2 days, close indices older than 14 days, delete indices older than 30 days:curator --host my-elasticsearch -b 2 -c 14 -d 30

Curator 1 root@precise64:/home/vagrant# curator -c 7 -b 2 -d 10! 2 2014-04-21T17:57:19.419 INFO main:333 Job starting...! 3 2014-04-21T17:57:19.420 INFO _new_conn:180 Starting new HTTP connection (1): localhost! 4 2014-04-21T17:57:19.422 INFO log_request_success:49 GET http://localhost:9200/ [status:200 request:0.002s]! 5 2014-04-21T17:57:19.423 INFO main:359 Deleting indices older than 10 days...! 6 2014-04-21T17:57:19.430 INFO log_request_success:49 GET http://localhost:9200/logstash-*/_settings?expand_wildcards=closed [status:200 request:0.007s]! 7 2014-04-21T17:57:19.433 INFO find_expired_indices:209 logstash-2014.04.21 is 10 days, 0:00:00 above the cutoff.! 8 2014-04-21T17:57:19.433 INFO index_loop:309 DELETE index operations completed.! 9 2014-04-21T17:57:19.433 INFO main:364 Closing indices older than 7 days...! 10 2014-04-21T17:57:19.434 INFO log_request_success:49 GET http://localhost:9200/logstash-*/_settings?expand_wildcards=closed [status:200 request:0.001s]! 11 2014-04-21T17:57:19.435 INFO find_expired_indices:209 logstash-2014.04.21 is 7 days, 0:00:00 above the cutoff.! 12 2014-04-21T17:57:19.435 INFO index_loop:309 CLOSE index operations completed.! 13 2014-04-21T17:57:19.435 INFO main:369 Disabling bloom filter on indices older than 2 days...! 14 2014-04-21T17:57:19.437 INFO log_request_success:49 GET http://localhost:9200/logstash-*/_settings?expand_wildcards=closed [status:200 request:0.002s]! 15 2014-04-21T17:57:19.438 INFO find_expired_indices:209 logstash-2014.04.21 is 2 days, 0:00:00 above the cutoff.! 16 2014-04-21T17:57:19.438 INFO index_loop:309 DISABLE BLOOM FILTER FOR index operations completed.! 17 2014-04-21T17:57:19.438 INFO main:379 Done in 0:00:00.020348.!

BigDesk

bigdesk.org Elasticsearch Plugin

Grok Filters?!

1 grok {! 2 match => { "message" => "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }! 3 add_field => [ "received_at", "%{@timestamp}" ]! 4 }!

Grok Debugger

grokdebug.herokuapp.com

The Logstash Book

logstashbook.com

Elasticsearch : The Definitive Guide

http://www.elastic.co/guide /en/elasticsearch/guide/current/index.html

Performance Optimisationor short P22N

Performance

• Remember: It’s just Java

• File Descriptors >32k

• Give enough Memory (-Xms -Xmx Values)

• Leverage File System Cache

https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html#_give_half_your_memory_to_lucene

Automation!

Puppet Modules

• elasticsearch/elasticsearch (PuppetLabs Approved)

• elasticsearch/logstashforwarder

• elasticsearch/logstash

Puppetclass { 'elasticsearch': repo_version => '1.7', manage_repo => true, java_install => true, config => { 'cluster.name' => 'cluster01' }, datadir => '/var/lib/elasticsearch/' } !

elasticsearch::instance { 'es-01': config => { 'node.name' => 'es-01' } }

Take Home• Centralized Logging saves time

• Is fun with the ELK Stack

• Gives you Graphs to Interpret

• „can you check the errors from yesterday between 15.02 and 15.07“ get’s A LOT easier

• Start here tomorrow: http://logstash.net/docs/1.4.2/tutorials/getting-started-with-logstash

Thank you for having me here! Slides: http://s.nrdy.ch/drupalcon-bcn

Feedback: http://s.nrdy.ch/rateme

Friday Sprints - Join us! http://s.nrdy.ch/sprints

Legal (because Legal…)

• Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries.

• Kibana is a trademark of Elasticsearch BV, registered in the U.S. and in other countries.

• Elastic, Logstash and Marvel are trademarks of Elasticsesarch BV

Images Used

• Elk : https://www.flickr.com/photos/ucumari/353839518/

• Architecture : https://www.flickr.com/photos/dasrecht/6743411525/

• VideoWall : https://twitter.com/webtuesday/status/433296964055470080/photo/1

• Tió de Nadal http://en.wikipedia.org/wiki/Image:Cagatio.jpg (CC-BY-SA 3.0)