LHCb Logging System

Nikolaidis Fotis ( fotis.nikolaidis @ cern.ch )University Of Crete, Greece

A computer log is a diary or archive of events, in this case generated by a computer system or systems. In the late- 23rd century, Federation starships were equipped with a "black box" that stored computer logs. The logs could be used in criminal investigations or to determine the cause of a lost ship. Computer logs were for official purposes only and were available to authorities only under specific legal circumstances or court-order.

Sources Web Servers Gateways Network Components Farm Nodes PVSS FMC

Storage SchemaFARM

HOSTS

PARTITIONS

PVSS

SERVICES

hlt[a-e][1-11] Messages, crond, maild, dnsd,secure, secureNagios

hostName Messages, crond, maild, dnsd,secure, secureNagios{Other files either from FMC or web sites}

hostName Project Name PVSS_II.logPVSS00ctrl50.logAnd other ...

LHCbTFCFESTECAL........

DAQ

TELL1 Messages, crond, maild, dnsd,secure, secureNagios

$partition.log

Dataremove, Dimrpc, Writerd, Xmlrpc

Needs

Forensic / Troubleshooting Splunk ( http://admin01/splunk )

Real Time Alert Ossec

Splunk

Is a High performance, scalable software server written in C/C++ and Python.

Index and Normalize logs (disk fail , disk error are the same)

Can be combined with with Ossec, Snort and other IDS via plugins

Does not need an external Database.

Splunk - Features

Advanced search Regular Expressions / Time Windows

Runtime statistical analysis Extensible

Modules, Patterns Dashboards

Splunk - More Features

Can correlate events of different hosts/formats Supports many log formats out of the box

(For non standard logs such as FMC configuration is needed)

If run on CLI , can be integrated to scripts

Have a closer look here ...

The first line is excludedThe second line is now the first

Who is keeping ssh busy ? ;p

New Patterns can be generated almost automatically

Internal Information

OSSEC

OSSEC

Open Source Host-based Intrusion Detection System.

Log analysis, file integrity checking, policy monitoring, rootkit detection, real-time alerting and active response.

OSSEC

Analyzes incoming logs runtime and reacts if needed

Every event can be ranked with a value [1-14] If event > mailRank , send a mail If event > scriptRank , execute a script Rules are defined in XML files

Message, frequency, priority, etc

Fault Tolerance

Normal

Logsrv01 failure

Log analysis failure

Logsrv02 failure