Splunk Enterprise SecurityFor Proactive Monitoring
AKA: Enterprise SecurityTips, Tricks, and Analytics
Purpose
● Describe Tips for a Clean Setup of ES
● Provide Tricks “From the Field” in Setup/Mgmt
● Demonstrate Analysis With and Without Training Wheels
Who Am I?
● Sean Wilkerson, Partner/Consultant, Aplura
● Speaker at SANS Log Mgmt Summits
● Splunk Pro Serv Partner Since 2008
Splunk/ES Experience
● 20+ ES Engagements
● Dozens of Different Federal Entities
● Many Commercial Customers
● My 4th .conf
● 5+ Years of Splunk Pro Serv.
● 6+ Years using Splunk
● 12+ Years Of Logs (Shell, Scripts, SIM, Splunk)
● 14+ Years of Network --> Systems --> InfoSec
Who Are You?
● You Know a Handful of Splunk Search Cmds
● You Have Worked With Splunk Conf Files
● You Know Generally What ES is demos/talks
● You May be a Splunk/ES User/Administrator
● You Are Analysts!! <--- Really Important
Content Available Now!
aplura.com/splunkconf2013
Splunk App for Enterprise Security
Scalability to manage multi-terabytes of real-time and historical data
Pre-built security correlation rules, reports, and dashboards
Statistical analysis for defining ‘normal’
Incident investigation and management framework
Solution with out-of-the-box content to manage known and unknown threats.
Security AnalystsSOC Staff Security Execs/Mgrs
Security Auditors
vocab(ES)
● ES = Enterprise Security
● TA = Technology Add-on (fields and tags)
● SA = Security Add-on (searches and corr logic)
● DA = Domain Add-on (dashboards)
● Macros = Shortcut to Splunk search string
● Correlation = Notable Event Searches
● Onboard = Inputs and TAs
● CIM = Common Information Model
Splunk Enterprise Security
Tips
ES From 10,001 Feet
● ES = Well-Organized Deployment
● Good Organization = Free Correlation
● CIM = Babelfish (One Language)
● Allows Fewer and Clearer Correlations
● Intelligently Doing More with Less...Overhead
● Improves the Speed to Root Cause Analysis
Deployment Steps
● First, Solidify Architecture
● Install: DS, SH->IDX; Validate Storage, etc.
● Ensure ES Storage Supports TSIDX (100GB is 340G/yr w/ ES-2.x)
● Onboard at least: Firewall, WEL (AD), IDS, AV
● Start ES and Validate TAs
● Enable and Schedule Desired Correlations
● Integrate Assets and Identities
● Onboard Other Supported Data-Sources
● Onboard Custom Data-Sources
● Tune and Optimize
General Housekeeping
● ASAP Start Defining Assets
● Time-Audit Before It's Too Late
● RT > index=* | eval timeDiff=_indextime-_time | timechart span=10m avg(timeDiff) by sourcetype
Leverage ES Strengths
● ES Reports on Security-related Decisions
● Information is Grouped into Three Domains:
● Access (Logins, Admin Activity)
● Endpoint (Malware, Systems, Time)
● Network (Firewall, IDS, VA, WebProxy)
● Some Data Doesn't Need ES
● ES Assumes a Framework, So Should You
**Hazards Ahead**
The journey is profitable; however,fumbled steps can land you in peril.
Hazard: Underpowered Hardware
● Splunk (like DBs) Can Run on An Old Laptop
● It doesn't mean that it should!!
● Meet or Exceed the “Reference“ “Architecture“
● Don't Skimp on Hardware!!
● Until [ $IOPS >= 1200 ]; do storage++; done;
Hazard: No EventGen in Production
● EventGen Creates Fake Data for DEMOS
● Do Not Enable This in Production!!
● Really? Do I Have to Say This?
● Yes, I do!!
Hazard: App Isolation● “App Isolation” allows apps to play nice
with each other with little to no regard for precedence.
● For testing: Edit any SA-$NAME/medatadata/local.meta to add your custom app
● For permanence: Edit app SplunkEnterpriseSecuritySuite default/inputs.conf
● Be Mindful of App Isolation – It Can Bite Hard
Hazard: Asset Formatting
● Assets provide the context between the data and correlations. VALUE++
● ES-2.2+ - “Asset Expander” - Validate/Format
● > index=_internal source=*lookup_expander.log
● Temporarily Adjust Input to Shorten Test Cycle
Hazard: RealTime Correlations
● Many of the Correlations are “RealTime”
● Switch these to scheduled (generally speaking)
Hazard: Customizations
● In Splunk – Custom = Immortalized
● This is the local vs default thing...
● Customizations Can Effect the Mechanics of ES
● Leverage the Customizations Encouraged in ES but Don't Make Your Own
● Customizations = Difficult Upgrades
● Customize Correlations With Care
● Do Not Customize Views, Assets-fields, or Scripts
Splunk Enterprise Security
Tricks
Tricks: Dynamic Lists
● This May Look Unexciting, but it is What I Get Asked for the Most
● Dynamic Assets/Identities (via SavedSearch) Whenever Possible
● Use SA-ldapsearch for Both, such as this:
| ldapsearch domain=$domain$ search="(&(objectClass=user)(!(objectClass=computer))(!(displayName=SystemMailbox*)))" attrs="cn,userPrincipalName,sAMAccountName,personalTitle,displayName,givenName,sn,suffix,mail,telephoneNumber,mobile,manager,priority,department,category,watchlist,whenCreated,accountExpires" | fields - _* | rename sAMAccountName as identity, personalTitle as prefix, displayName as nick, givenName as first, sn as last, mail as email, telephoneNumber as phone, mobile as phone2, manager as managedBy, department as bunit, whenCreated as startDate, accountExpires as endDate | table identity, prefix, nick, first, last, suffix, email, phone, phone2, managedBy, priority, bunit, category, watchlist, startDate, endDate | outputlookup simple_identity_lookup
Tricks: Nice Assets
● Create Asset Categories In SavedSearch
| `assets` | mvexpand category | dedup category | sort category | table category | outputlookup category_lookup
● Use CIDR Blocks in Assets
● This Allows for Inclusion/Exclusion of Network by category reference. This is big!
– E.G. All IDS alerts by category=oracle_cluster
Tricks: Nice Assets 2
● Plan Asset Categories and Benefit
● Use Built-ins When Available (e.g. email_servers)
● Don't Make More Granularity Than You Can Use
● Plan Supportive Naming Scheme For CIDR...
– foonet_nyc_dmz
– foonet_nyc_users
– foonet_chg_dmz
Note: Critical Point
Tricks: Upgrade !SNAFU
● Read ReleaseNotes
● Unpack ES, Extract TAs
● Sync Upgraded TAs to DS:
● Do them one at a time
● Watch for default changes and lookup overwrites
● Push TAs Out to Search/Parsing Tiers
● Use UI and Do ES Upgrade
● Remove Unnecessary TAs and Ensure Yours are Pushed
Tricks: Create a Custom TA
● It Looks Harder Than it is, but Don't Rush
● Have Your DataSource Manual Ready
● Prepare a regex parser too, as needed
● Work in Dev Environment Whenever Possible
● Copy a Similar TA
● Input the Data (Apply Necessary Parse-time Confs)
● Ensure Necessary Fields Are Present
● Ensure Necessary Tags/Eventtypes are There
● Validate Your TA (See Next Slide)
Tricks: Validate TAs
Use search or macros to verify TAs
Splunk Enterprise Security
Analytics (Unchained)
Drilldown Gets You Started (Demo)
ES Macros (Demo)
● `authentication`
● `ids_attack`
● `communicate`
● `malware`
● `proxy`
● `vulnerability`
ES Lookups (Demo)
● Assets
● | `assets` ; | inputlookup simple_asset_lookup
● | `categories` ; | inputlookup category_lookup
● Identities
● | `identities` ; | inputlookup simple_identity_lookup
● Trackers (on my)
● | `access_tracker`
● | `port_protocol_tracker`
● | `ids_attack_tracker`
Custom Analysis (Demo)
● `proxy` | search `get_subject(src, "10.11.36.23")`
● `proxy` | search NOT action="tcp_denied" [ search `proxy` | search action="tcp_denied" | dedup src | table src] | top dest by src
● `ids_attack` | search (severity="critical" OR severity=”high”) signature="dos*" `get_subject(src, "125.17.14.100")` category="dos"
Additional Resources
● docs.splunk.com - General Manuals
● docs.splunk.com/Documentation/ES - ES
● splunk-base.splunk.com - User forums
● Cheatsheet - duh!
Thank You!
ES: Trips, Tricks, Analytics (This Talk)
aplura.com/splunkconf2013
● Also:
● Best Practice PDF: aplura.com/splunkbp
● Talk: Security Analysis: aplura.com/splunklive2013
● Talk: Best Practice: aplura.com/splunklive2012
● Talk: SIEM Fails: aplura.com/lookbeforeyousim