Splunk Worldwide Users’ ConferenceThe Palace Hotel, San Francisco, CAAugust 9-11, 2010
Best Practices for Federal Compliance
Dan O’Donnell, CISSO
© Copyright Splunk 2010Splunk Worldwide Users’ Conference2
About this talkUsing Splunk to satisfy U.S. Gov’t NISPOM auditing reqs. (and maybe a little DCID/CNSS/ICD)
Audit multiple events, across multiple platformsGoal 1: Show how one organization uses Splunk for auditing.Goal 2: Start a dialog - maybe a community.Non-goal: to be an expert source
© Copyright Splunk 2010Splunk Worldwide Users’ Conference3
About the SpeakerDan O’Donnell, CISSO
- 8 yrs RAND; 5 yrs NBC- IANAP (Fortran 77)
Who/what is RAND.org?- FFRDC; non-partisan think tank on public policy: health, education, mil, etc.- many PhD scientists, engineers, social scientists, economists
Splunk users- Between 3-10, depending…
© Copyright Splunk 2010Splunk Worldwide Users’ Conference4
What Are NISPOM, DCID?NISPOM = National Industrial Security Program Ops Manual
- Chapter 8: computers and networks- Chapter 8-602: what we care about for auditing with Splunk- ISFO: Industrial Security Field Operations Manual
DCID 6/3, being replaced by ICD- Equivalent (sort of) for military and IC shops
© Copyright Splunk 2010Splunk Worldwide Users’ Conference5
Ch.8 Significant RequirementsCh.8-602 mandates several things, but we’ll only discuss…
‣ auditing of specific logs and trails‣ PL-1: 1 of 5, with 1 being lowest
‣ISLs are more specific
Auditing monitors computers for intrusive patterns and behavior.
© Copyright Splunk 2010Splunk Worldwide Users’ Conference6
DSS: Defense Security ServiceNISPOM: National Industrial Security Program Ops ManualISL: Industrial Security LettersSRO: Security Relevant ObjectsISSO/ISSM: Information Systems Security Officer/MgrISFO: Industrial Security Field OperationsDISA: Defense Information Systems AgencySTIG: Security Technical Implementation GuideICD (DCID): Intelligence Community Directive
Acronyms
© Copyright Splunk 2010Splunk Worldwide Users’ Conference7
Time efficiency!Audit frequency (mandated) = weeklyTime Requirements
1 machine ~ :10 minutes100 machines ~ 1,000 minutes ~ 2 days per week
Conclusion: Auditing does not scale.Also: log aggregator “remembers” the search strings.
Why Use a Log Aggregator
© Copyright Splunk 2010Splunk Worldwide Users’ Conference8
Labor efficiency = $ efficiencySystem Comparison:
We looked at 6 systems, including making our own.Splunk did all four OS platforms – no other commercial product did this.Splunk: superior to what we could do on our own, and less costly.Splunk: modifiable, and on our own hardware.
Nominally approved by DSS ODAA (v3.4), summer 2009YMMV – check with your ODAA or equivalent
Why Use Splunk as Log Aggregator?
© Copyright Splunk 2010Splunk Worldwide Users’ Conference9
RecommendationsRichard Bejtlich: network security, awareness, and APT
- “Federal security is the most frustrating…”- “Splunk is really awesome…” and- “Splunk is remarkably cheap for an enterprise app…”- YouTube: BSDconferences talk, April 21, 2009- http://taosecurity.blogspot.com/
© Copyright Splunk 2010Splunk Worldwide Users’ Conference10
Intro to NISPOM (1)NISPOM Chapter 8, Section 602 a, b, c, d
- ISL 2007-01,
Data, metadata to capture- 2 general categories
Identification and Authentication (I&A)Security Relevant Objects (SRO)
‣ Prohibited file or directory activity
© Copyright Splunk 2010Splunk Worldwide Users’ Conference11
Platforms
© Copyright Splunk 2010Splunk Worldwide Users’ Conference12
Intro to NISPOM (2): Info to CaptureDate/Time stampUser or agentResources involvedAction involved
© Copyright Splunk 2010Splunk Worldwide Users’ Conference13
Intro to NISPOM (3): I&AI&A: Identification and authentication
Login success
Logout success
Login attempts that fail – bad username or password
Login attempts to lockout – 5 attempts within :15min
Account lockouts
Password changes
User authentication changes: sudo, su, admin
© Copyright Splunk 2010Splunk Worldwide Users’ Conference14
Intro to NISPOM (4): SROSRO: Security Relevant Objects
- Windows types; *nix types
OS executablesOS config filesSystem management and maintenance executablesAudit system and dataSecurity-related software
© Copyright Splunk 2010Splunk Worldwide Users’ Conference15
ISL #44
© Copyright Splunk 2010Splunk Worldwide Users’ Conference16
ISL #45
© Copyright Splunk 2010Splunk Worldwide Users’ Conference17
Windows Event IDsUltimate Windows Security
- Windows: simplest and easiest- This list isn’t all, but most.- This is “XP family”.- Win7 is totally different.
© Copyright Splunk 2010Splunk Worldwide Users’ Conference18
In general:/bin /usr/bin /sbin /usr/sbin
Audit systems – BSM; Snare/var/audit /etc/security ; syslog
Avware (required)Disk utilities“Lemme es’plain… No, it is too difficult.”
“Let me sum up…”
SROs in *nix
© Copyright Splunk 2010Splunk Worldwide Users’ Conference19
Auditable Events & Objects: SummarySummary of: WinXP, Server 2003, *nix, OSX, *BSD
© Copyright Splunk 2010Splunk Worldwide Users’ Conference20
WindowsSystem EventsSecurity EventsApplication Events (coming)
LinuxSyslogSeclogSnare log (merged into syslog)
Streams
SolarisSyslogBSM (converted to text)
OSX, TrustedBSD, FreeBSD
SyslogSeclogBSM (converted to text)
© Copyright Splunk 2010Splunk Worldwide Users’ Conference21
Streams to Capture in Splunk
© Copyright Splunk 2010Splunk Worldwide Users’ Conference22
Windows: easy to config, easy to interpret.Configure file size for maximum.Configure persistence for “long”
Linux: moderately hard to config, fairly hard to interpretSyslogSnare – use the IA “one button config for NISPOM” or make your own.Check with your ISSP or DSS Rep., or DSS Academy.
Configuring Streams: Win, linux
© Copyright Splunk 2010Splunk Worldwide Users’ Conference23
Solaris: moderately hard to config; hard to interpretSyslogBSM (converted)
OS X: hard to config; hard to interpretSyslogSeclogOpenBSM (converted to text)
*BSD: moderately hard to config; hard to interpretBSM part is the same in OS X, sort of
Configuring Streams: Unix, OSX, BSD
© Copyright Splunk 2010Splunk Worldwide Users’ Conference24
Windows XP to Vista, including ServerAbout 14 of 114 events are audited: success and fail
Configure Events filesIncrease size (default 2MB -> 600MB), increase persistence
Snare can be usedActive Directory (AD) spews log entries.
Filtering with clever Splunk search strings can improve SNR.
Potential Problem: Active Directory and unix, linux, OSX
Case Analysis: Windows
© Copyright Splunk 2010Splunk Worldwide Users’ Conference25
Syslog: raw syslog is straightforward.Seclog: raw seclog is straightforward.Snare:
Freeware, from Intersect AllianceNISPOM config can be 1-touch, but “roll your own” may be better.Output is text, merges into syslog.Output is text strings, searchable with Splunk.Problem: interpretation of outputSplunk lookup tables as a solution?
Case Analysis: Linux
© Copyright Splunk 2010Splunk Worldwide Users’ Conference26
Snare necessary for linuxHooks into auditd.Recommended by DSS Academy.
Snare can be used with Windows.Increases detail and complexity.
Not yet ready for Win7.Snare can be used with Solaris.
Data and detail equivalent to BSM.Complexity slightly reduced since Snare outputs to text.Complexity increase as minimal Snare docs don’t include output interp.
Case analysis: Snare
© Copyright Splunk 2010Splunk Worldwide Users’ Conference27
Snare config
© Copyright Splunk 2010Splunk Worldwide Users’ Conference28
Syslog is easy; BSM is hard.About BSM:
Flags: lo,ad,-fr,-fw,-fc,-fd,-fm,-clLog file is binary: Splunk can’t handle it. Rotate regularly, export to text.
DocumentationSun docs, man pagesHal Pomeranz SysAdmin magazine article
Sun BSM similar to BSM on OSX, FreeBSD, TrustedBSD.Snare works on Solaris
Case analysis: Solaris
© Copyright Splunk 2010Splunk Worldwide Users’ Conference29
Syslog, seclog are easy.BSM (harder than Solaris):
Same flags: lo,ad,-fr,-fw,-fc,-fd,-fm,-clRequires script to rotate BSM binary, export to text.Rotation frequency; retention period (1 yr.)
ParseAuditLog (PAL) scriptDiffs between OSX 10.6 and earlier
10.6.x has OpenBSM v1.1, with more functionality than earlier 1.0.
Case Analysis: OS X
© Copyright Splunk 2010Splunk Worldwide Users’ Conference30
Small but powerfulGenerates big binary files, but very compressible
Very configurable, including “make your own masks”
~380 total events (x4); <50% are audited; (x 1/4)Diffs between Solaris, OSX, *BSDHistory of OpenBSMInterpretation of output (BSM output is Splunk input)
20100731123015.not_terminated; 20100731123015.20100731133015
Case Analysis: BSM
© Copyright Splunk 2010Splunk Worldwide Users’ Conference31
BSM Audit Classes
© Copyright Splunk 2010Splunk Worldwide Users’ Conference32
BSM Configs
OpenBSM v1.1etc/security/audit_control
© Copyright Splunk 2010Splunk Worldwide Users’ Conference33
BSM OutputLookup tables candidate???
© Copyright Splunk 2010Splunk Worldwide Users’ Conference34
Splunk and (most): routers, firewalls, switches, ips/idsMandated to audit these too.NISPOM and DSS don’t tell us what or how to audit.
Case Analysis: network appliances
© Copyright Splunk 2010Splunk Worldwide Users’ Conference35
WindowsSys.Evt, Sec.Evt
LinuxSyslog, SeclogSnare audit log
SolarisSyslogBSM audit log (converted)
Splunk Inputs (redux)
OSX, FreeBSDSyslog, SeclogBSM audit log (converted)
Network appliancesLogs from firewalls, ips/ids
Active Directory logs(lots of kruft to filter)
© Copyright Splunk 2010Splunk Worldwide Users’ Conference36
Auditable Objects | Splunk inputs
© Copyright Splunk 2010Splunk Worldwide Users’ Conference37
Overall Audit TablePL1 Login fail
Login success
Login fail to
lockout
Account lockout
Logout success
Password change attempt
SRO access
fail
Windows ✔ ✔ ✔ ✔ ✔ ✔ ✔
OS X ✔ ✔ ✔ ✔ ✔ ✔ ✔
Linux ✔ ✔ ✔ ✔ ✔ ✔ ✔
Unix ✔ ✔ ✔ ✔ ✔ ✔ ✔
© Copyright Splunk 2010Splunk Worldwide Users’ Conference38
Active Directory, *nix, and UID/GID conflictsUnique search string (saved search) for 100s of events???Lookup tables to convert Snare and BSMActive Directory and SNR (Signal to Noise Ratio)
AD spews a large volume of data – filtering requires knowledge and finesseAD and duplicate records
Issues or Potential Problems
© Copyright Splunk 2010Splunk Worldwide Users’ Conference39
RemainingOutputsInterpretation to actual intelligenceMetricsOther Splunk capabilitiesRe-architect?
© Copyright Splunk 2010Splunk Worldwide Users’ Conference40
Questions?