Graylog2Lennart Koopmann, OSDC 2014

@_lennart / www.graylog2.org

About me

• 25 years old

• Living in Hamburg, Germany

• @_lennart on Twitter

• Co-Founder of TORCH - The Graylog2 company.

Graylog2 history• Started as open source project by Lennart Koopmann in 2010 and

was developed entirely in free time.

• TORCH founded as company behind it in late 2012 after seeing massive growth and worldwide distribution in large scale setups.

• Now team of 6 working full-time on it, three more people joining this summer. (and still hiring)

• www.graylog2.org

Graylog2 history• Big rewrite of Graylog2 started in 2012 and finished with releasing a

final v0.20.0 in February 2014 that addresses what we learnt from our first customers and all users.

• Web Interface now (like the server) written in Java and easy to install. Prior versions used Ruby On Rails and were hard to install.

• New web Interface focussing on powerful analytics.

• Unified REST API communication for easy extending and integrating with other products, tools and scripts.

Free and open source analysis of any machine data written in your datacenter.

!

Running on the JVM in your own environment. Not limited by licenses.

Basic architecturemessage sources

graylog2-server graylog2-server

ElasticSearch Cluster

MongoDB

graylog2-web-interface

Your own reporting scripts

Your own subscribers

RESTREST

Inputs

Architecture considerations

• Use graylog2-radio for HA and high level buffering

• Put load balancers in front and scale out horizontally

Architecture considerations

• graylog2-server / graylog2-radio: Focus on CPU

• ElasticSearch: Focus on RAM and IO

• MongoDB: Replication set for failover, not much load

• graylog2-web-interface: Not much load at all

Architecture considerations

• http://support.torch.sh/help/kb/general/graylog2-architecture-high-level-overview

No message left behind2014-04-04 14:05:43,147 INFO : org.graylog2.Core - SIGNAL received. Shutting down. 2014-04-04 14:05:43,150 INFO : org.graylog2.system.shutdown.GracefulShutdown - Graceful shutdown initiated. 2014-04-04 14:05:43,150 INFO : org.graylog2.system.shutdown.GracefulShutdown - Node status: [Halting [LB:DEAD]]. Waiting <5sec> for possible load balancers to recognize state change. !… !2014-04-04 14:05:49,156 INFO : org.graylog2.system.shutdown.GracefulShutdown - Attempting to close input <org.graylog2.inputs.raw.udp.RawUDPInput.531f89283004f7b66a87e163> [Raw/Plaintext UDP]. 2014-04-04 14:05:49,157 INFO : org.graylog2.system.shutdown.GracefulShutdown - Input [org.graylog2.inputs.raw.udp.RawUDPInput.531f89283004f7b66a87e163] closed. Took [1ms] … !2014-04-04 14:05:49,158 INFO : org.graylog2.caches.Caches - Waiting until all caches are empty. 2014-04-04 14:05:49,158 INFO : org.graylog2.caches.Caches - All caches are empty. Continuing. 2014-04-04 14:05:49,159 INFO : org.graylog2.buffers.Buffers - Waiting until all buffers are empty. 2014-04-04 14:05:49,159 INFO : org.graylog2.buffers.Buffers - All buffers are empty. Continuing. !… !2014-04-04 14:05:49,176 INFO : org.graylog2.system.shutdown.GracefulShutdown - Goodbye.

No message left behind

GELF http://graylog2.org/gelf

• The “Graylog2 Extended Log Format”. Structured and compressed, based on JSON.

• Optional UDP chunking allows sending a lot of data without having to care about connection management in your application (timeouts, …) if you don’t need transport security.

• Already over 30 libraries from the community and integrated into the first products.

Streams• Performant realtime routing of messages based on rules. Matching applied

when the message is received and processed.

• Create streams like “SSH logins” or “Exceptions in application X” for quick access in the web interface (like saved searches) or alerts.

• Be alerted based on message count thresholds or results of statistical computation of given relative time windows. “Send me an alert when the standard deviation of the response time in application X was higher than 100 in the last 10 minutes.”

• Forward to other systems based on matched streams. “Forward all business intelligence related logs to another system. (to save license costs)”