Large enterprise SIEM: get ready for oversizeSvetlana/Mona ArkhipovaQiwi

OWASP Meetup, Moscow, 28 Feb 2015

What are we talking about?• Log collecting != Security Information and

Event Management• Systems monitoring is not enough• Logs as a ‘Big Data’•

WTF is qRadar?

Hello IBM!

• Log management• Network activity/anomaly detection• SIEM• Nice API

WTF is qRadar?Administrator’s nightmare:

• Frontend: Java+Tomcat• Backend: Java daemons• DB: Ariel for collected+ indexed data, PostgreSQL for ‘static’ data• Painful performance metrics and load

balancing

Architecture

To log or not to logGuides/best practices• https://www.owasp.org/index.php/Logging_Cheat_Sheet • http

://www.syslog.org/logged/logging-and-syslog-best-practices/

• http://sniperforensicstoolkit.squarespace.com/storage/logging/Windows%20Logging%20Cheat%20Sheet%20v1.1.pdf

• https://zeltser.com/media/docs/security-incident-log-review-checklist.pdf

• …

To log or not to logHuston, we got a problem:• Standard syslog message size (RFC 5424)• Windows security logs permissions on

W7/2008+• Database audit – what to log?• Log files on FS (IIS and so on)• In-house developed apps

To log or not to logStandard sources: Windows

• Event collectors vs. agents• Extended system audit• Non-English logs:

To log or not to log

Standard sources: *nix, network devices

• Syslog as a standard• TCP syslog+network issues=pain

(google: “TCP is not reliable”)• UDP syslog message size• Auditd – what to log?

To log or not to log

Standard sources: Databases

• Is login history enough?• Syslog vs DB connection

To log or not to logNon-Standard sources:

• Exotic network devices• In-house developed apps• 1C (OMG…) and other specific apps• Integration with other security systems (NGFW,

DBFW, AV, Security scanners…)

To log or not to logWhen syslog is powerless: WAF CEF log file

Normalizing/indexingEvent at a glance• Standard properties: timestamp, src IP, dst IP, log

source identifier and so on• Custom event properties – KISS principle• No search – no property.

Indexing• Standard properties – index, index, index!• Custom event properties indexing: with great

power comes great responsibility…• BTW, watch your index size.

Over(sizing)Current Qiwi SIEM metrics:

• 1800 log sources• 10 000 - 24 000 RAW events per second (EPS)• ~11 600 network flows per second (FPS),

~700 000 flows per minute(FPM)

SIEM system: 39 virtual servers, 2 hardware servers with Napatech 2x10G cards, 1 archive server

Over(sizing)

Expectations (sizing) RealityvCPU 140 160vRAM 272 Gb 521 GbvHDD 15 TB 61 TB

Once upon a time in a far far galaxy we decided to build our own SIEM…

Online/offline storageDaily stats:• 67-145 Gb raw event logs per day• 37-53 Gb network communication events per

day

• Online storage – fast access (realtime + some previoius data)

• Offline – archive storage

What if...…EPS or FPM x2 ?

Internal security scanners“Normal paranormal” activity inside and outside.

• Butthurt :(• Log or drop events?• Custom rules set for nodes• Keep an eye on credentials!• Balancers NAT/SNAThttps

://f5.com/resources/white-papers/load-balancing-101-nuts-and-bolts

Autopilot: ON• Simple rules

• Chained rules:

Autopilot: ON

Questions?Svetlana/Mona Arkhipova

Lead information security expertQIWI infrastructure security team

mona@qiwi.com

mona.sax m0na_sax