Graylog for OpenStack : 3 steps to know WHY

MediTech JSChttps://meditech.vn

Private Cloud

Storage

Monitor

Logging

Managed Services

About me

Dinh Van Manh

● System Integration Department in MediTechJSC● Member of Hocchudong● Interested in OpenStack, Linux, Monitoring, Logging

and new technology● Habbit : “tra da + thuoc lao” with friends

Agenda

1. Log Overview1.1. Logs : What & Where?1.2. Why look at Logs1.3. How to use Logs effectively

2. Log in OpenStack2.1. OpenStack log statistics2.2. OpenStack Log Management : in imagionation & in fact

3. Graylog for OpenStack3.1. Introduce about Graylog3.2. Key features3.3. Architecture/Mechanism/Model of Graylog3.4. Graylog for OpenStack: 3 steps to know WHY?

4. Demo + Q.A

Log OverviewWhat? Where? Why? How?

1.1. Logs : What & Where

What logs? (from the view of system administrator)

● System event diary

● System status records

● User activities

● Incident notify

Log format

1.1. Logs : What & Where

Log come from WHERE?

● Storage devices

● Application in Linux/Windows

● Cloud Services : OpenStack

● Servers

● Firewalls

● Routers, switches

1.2. Why look at Logs?

Basically :Incident response

higherTracking system event

higherMeasuring security : metrics, trends…

higher and higher Situational awarenessNew threat discoveryEstimating about user habit, trends...

1.3. How to use Logs effectively

Level 1 : Just SSH and view !● Understanding log location● Command to view log : tail, more, grep● Filtering by keyword

Level 2 : Use Syslog ● Collect syslog from client● Store in log server

Level 3 : Log management Software● Collect everything ● Retain most everything● Analyze enough● Summarize and report● Advance features : visualize, alert, share...

1.3. How to use Logs effectively

● Facility

○ Application Logs

○ Event Logs

○ Service Logs

○ System Logs

Log Keywords

● Severity○ 0 - emerg○ 1 - alert○ 2 - crit○ 3 - error○ 4 - warn○ 5 - notice○ 6 - info○ 7 - debug

● Rotention

○ Time to rotate log

● Retention

○ Delete, archive...log

● Syslog

○ protocol to transfer log

Log in OpenStackWhich level is appropriate?

2.1. OpenStack log statistics

OpenStack System : 3 Controller + 30 Compute node ● Controller Node

○ 6 log folder per OpenStack service○ system log : auth, dmesg, kernel…○ application log : apache, haproxy,

pacemaker…

● Compute Node○ 2 log folder per OpenStack service○ system log : auth, dmes, kernel…○ application log : libvirt○ log of instances

=> Total : ● ~ 220 log file● 10 GB log = 30 million messages / day

2.2. OpenStack log management : in imagionation & in fact

Communication think Colleagues think In fact

When i said : My job is OpenStack log management !

So Waste !!! What should we do?

Graylog for OpenStack: To infinity & beyond !

3.1. Graylog Introduce

● Log centralized management software

● Released in 2010 by Lenart Koopman with name is Graylog2

● In 1/2015 release Graylog v1., Graylog Inc was established

● Big change from Graylog version 2.0

● Newest version is Graylog 2.3.1, stable version is Graylog 2.3.0

3.2. Key features

Various Input & Output Analyze & Search

Visualize metricAlert & Trigger User management

3.3. Architecture/Mechanism/Model of Graylog

Overall architecture● Server

○ Graylog● Client

○ Client host○ Graylog sidecar○ Nxlog/Filebeat

Filebeat

Graylog Sidecar : Break the old path

● Configuration management system

● Config in client host only ONCE !

● All in Web● Secure with SSL/TLS

3.3. Architecture/Mechanism/Model of Graylog

Sidecar Work-flow : Easy config in 3 steps

Step 1 : Config in client● install sidecar● declare : graylog ip, client hostname, tags● start service

Step 2 : Config in Graylog Web● add tags● chose what logs you want to collect

Step 3 : Checking● Check colleted log

3.3. Architecture/Mechanism/Model of Graylog

Deep dive in architectureGraylog Server

● receive log message● execute log● communicate with other components

Elasticsearch ● store log message● search engine

MongoDB ● store meta infomation● store config data

3.3. Architecture/Mechanism/Model of Graylog

Log execute processingStep 1 :

● Spooling & store in disk temporarily● Prepare for buffer process

Step 2 : ● Messages from disk go in to Input Buffer● Mission : Filter, classify messages

Step 3 : ● Messages go in to Output Buffer● Onward to Elasticsearch or user defined

output

3.3. Architecture/Mechanism/Model of Graylog

Elasticsearch & Graylog

● Clustering

● Use API to communicate

● Use unicast-discovery to recogize other nodes

● Graylog as a Master Node

MongoDB & Graylog

● Client - Server mechanism

● Graylog use driver to communicate with MongoDB

Internal Graylog components mechanisms

3.3. Architecture/Mechanism/Model of Graylog

None HA - Small production HA - Bigger Production

Code show you HOW !Log show you WHY !

3.4. Graylog for OpenStack : 3 steps to know WHY?

Just 3 steps to exploiting log in OpenStack

3.4. Graylog for OpenStack : 3 steps to know WHY?

What should i do when instance spawning fail

A. Try to spawn again B. Blame for customer

D. Bug again! I’m quit ! C. Take a search in Graylog

Incident Response

Problem appear ! What should we do?

3.4. Graylog for OpenStack : 3 steps to know WHY?

Step 1 : Collect logsTake log from :

● nova log● neutron log● cinder log● glance log

Step 2 : AnalyzeMake a search in Graylog :Syntax : instance id + ERROR

Step 3 : Now you know WHYJust solve the problem & Go to sleep !

3.4. Graylog for OpenStack : 3 steps to know WHY?

Tracking a event

My instances was rebooted last night ??? When?

3.4. Graylog for OpenStack : 3 steps to know WHY?

Measuring metric

DEMO & Q.A

Bonus : Graylog vs ELK

Graylog is coming the closest to the Splunk architecture !

VS

Thank you !ManhDV

manh.dinhvan@meditech.vn

https://meditech.vn/https://github.com/hocchudong