Slide 2
Background – Scott Plichta
• Chief Information Security Officer • Development background • Over 20 years developing/operating SaaS solutions • Startups +
Slide 3
Corporation Service Company
• Established in 1899 • With 2,000+ employees worldwide, CSC: – Services more than 180,000 businesses,
including 90% of the Fortune 500® – Supports nearly 10,000 law firms and the largest
U.S. law firms – Provides Solutions to more than 3,000 financial institutions – Protects the top three and more than half of the 100 Best
Global Brands (Interbrand ®)
Slide 4
Our Products & Services
We manage more than 1.5 million corporate domain names, search the internet for brand abuses and provide online security assets including SSL digital certificates and anti-phishing services.
We help the general counsel efficiently handle and store sensitive data including service of process, contracts, and M&A documents.
Ranked #1 for Matter Management and Entity Management by The New York Law Journal
We enable banks and commercial lenders to conduct UCC searches and filings, prepare and record real estate documents, and manage titles and real property.
Slide 5
Information Technology at CSC
• Service business first and foremost • High availability Software as a Service component (SaaS) • Internal and external apps • Security is a top priority • Heterogeneous environment
– .Net / SQL Server – Java / Oracle / SOLR / Hadoop – Cold Fusion, PHP, Salesforce.com integration
Slide 6
Operational Environment
• Splunk environment - About 400 distinct sourcetypes. - 400GB App ingest, 1 TB security ingest - Production and pre-prod
• Security team structure – 2 Operational – 2 Senior engineers
• Part of IT Operations – ~ 100 people
Slide 7
Four Scenarios
1. Audit privileged access 2. Chasing ransomware 3. Phishing site targeting payments 4. APDEX customer experience
Slide 8
Auditing Privileged Access 1. Audit that only active employees
utilize privileged access (monthly) 2. Audit that appropriate individuals
access (quarterly) • Challenge:
– Time consuming à aggregate manual
– Tedious à manual correlation – Not consistently run à have to
remember
Privileged Access: • Allow user to modify client
permissions • Password reset • Perform functions on
behalf of clients • Only allowed by specific
active employees
Slide 9
Solution Design
• Evaluate all access – Do you have access? – Are you in the database? – Are you still employed? – Need to know admin functions
• Function tagged (MM_Admin) – Pre-assess threat – – And NOW we can alert on it
Slide 10
Tools
• Person DB – Database of all employees and contractors
• Splunk access logs – Tag assigned to all admin functions
(tag=MM_adminFunction)
Slide 11
DB Connect
• Active connection to production DB
• Allows lookup against active DB in real-time from Splunk
• Easy configuration and setup
• Lookup command syntax to access
Slide 12
The Results Output
• All relevant employee data: – Manager – Full Name – Active/Term – Term date – Last Access – Green – Yellow - Red
Green/Yellow/Red indicator
Slide 13
Breaking Down the Query sourcetype="prod-MM-perf" tag=MM_adminFunction | lookup local=1 Employees USERID as MM_UserName OUTPUT NAME, SUP_NAME, STATUS_CODE,TERMINATION_DATE | fillnull VALUE=null NAME,SUP_NAME,TERMINATION_DATE,STATUS_CODE | eval Date=strftime(_time,"%Y-%m-%d") |stats max(Date) as LastAccess, count by SUP_NAME, NAME, MM_UserName, MM_uri, STATUS_CODE, TERMINATION_DATE | eval flag=if(STATUS_CODE="null","Yellow", if(STATUS_CODE="A","Green",if(TERMINATION_DATE="null","Yellow",if(TERMINATION_DATE>LastAccess, "Green", "Red"))) )
Get values
Assign Alert level
Cleanup
Link to employee DB
Calc last access
Slide 14
Access Audits Simplified
• Alert – Any non-active employee access realtime
• Monthly audit of usage – Operations Manager reviews list of access
• Quarterly management audit – Email manager of employees using functions
Slide 15
Value Review
• Challenges: – Time consuming à
aggregate manual – Tedious à manual
correlation – Not consistently run à
have to remember
• Solution: – Splunk does all aggregation – Splunk highlights anomalies – Splunk sends finalized report
Slide 16
Three Scenarios
1. Validating privileged access 2. Chasing ransomware 3. Phishing site targeting payments 4. APDEX customer experience
Slide 17
Ransomware
• Blocks access on infected computer • Demands ransom to unlock • Payment *might* release access • Delivery mechanism vary
• CryptoLocker (and variants) – Encrypt all files on all drives – Infect other computers – Leaves ransom note
Slide 19
Malware Remediation
• Isolate “the patient(s)” • Fix the “the patient(s)” • How did it get in? • What’s the link? • Who else has the link? (vector) • Who else has clicked the link?
Slide 20
Isolate/Fix
• Disconnect shared drives • Find the program – but its gone by now…
• Assess the damage • Start the restore
Ø Let’s go find patient zero
Slide 21
Find the Source
• We know: – User IDs of encrypted files on shared drives
• We assume: – Payload/link was delivered by e-mail
• What does Splunk know: – Mail header logs – Proxy logs
Slide 22
Check Email… Email
Senders Each Infected
User
Every user received this email Same process for proxy logs
Slide 23
Find the Email
sourcetype="MSExchange:2010:MessageTracking” !recipients="*user1*” OR recipients="*user2*" OR recipients="*user3*" OR recipients="*user4*" ! !|eval user1=if(searchmatch(”user1"),1,0)!
|eval user2=if(searchmatch(”user2”),1,0)!|eval user3=if(searchmatch(”user3"),1,0) !|eval user4=if(searchmatch(”user4"),1,0) ! !| stats count,sum(user1),sum(user2), ! sum(user3), sum(user4) by !
sender_domain,sender!
Specify email
Show me a table of senders and receivers
Mark each email based on who received it
Select users
Slide 24
Who Else Got This email?
• Pull a sample email • Search by subject and time sourcetype="MSExchange:2010:MessageTracking" message_subject="*Statament*" OR message_subject="*August 2014*” OR sender=“[email protected]”
| stats count by message_subject, recipients
• Send copy to spam protection • Remove the emails from Exchange
Slide 25
Who Clicked the Link?
• So who has been to the link? • Search the proxy for all references to all URL’s in email
Ø Now we have users to contact
Slide 26
What if This Didn't Work
• Technique works – Splunk sources à more forensics
• Not always email – Could be links – follow the proxy
• Check proxy against threat lists
Slide 27
Chasing Ransomware
• Without Splunk – 4 day event, with re-infection – Multiple people, many hours
• With Splunk – Multiple hour – Small staff – Minimal WIP
Slide 28
Four Scenarios
1. Validating privileged access 2. Chasing ransomware 3. Phishing site targeting payments 4. APDEX customer experience
Slide 29
Phishing Sites
• Do you take payments online? • Do you worry about phishing? • Do hacker have good separation of duties? • Do hackers test in production?
Slide 31
lncorporate.com
• Marketing monitors referrers on webservers
sourcetype="prod_apache_logs" source="/opt/httpd/logs/www.incorporate.com-access.log" | rex field=http_referrer "http[s]://(?<ref_hostname>[^/]+)"| timechart count by ref_hostname
Slide 32
Lncorporate.com • Is something strange here?
0 2000 4000 6000 8000 10000 12000
incorporate.com
lncorporate.com
cscglobal.com
google.com
advendor1.com
advendor2.com
Incorporate.com Referrers
5/6/14
5/5/14
5/4/14
5/3/14
5/2/14
5/1/14
• lncorporate.com à Lncorporate.com
Slide 33
Fix the Problem & Monitor
• After a few whois/dns searches – This is not right
• Contacted CSC’s Take Down service – DNS, website offline in 30 minutes – Phone number disabled within 45 minutes
• New search to detect phishing attacks – Scan for new referrers and alert [lookup]
Slide 34
Fraudulent Site Process
• Phishers – Develop in production – Use your images
• Everyone should: – Baseline your referrers – Check your actual referrer
Slide 35
Lncorporate
• Conclusion: – Use Splunk to analyze web traffic – Anomaly detected and remediated
• EVERYONE is in security – Problem fixed while still in development
Slide 36
Four Scenarios
1. Validating privileged access 2. Chasing ransomware 3. Phishing site targeting payments 4. APDEX customer experience
Slide 37
User Experience
“Know before the customer does”
Monitoring is all green
Have them screen share Get their
credentials
Performance is awful
Report won’t finish 50 Employees
crawling Works on test acct
Slide 38
The Monitoring Dilemma What level of performance are
my users seeing today?
• Sitescope™ – How’s my server? • BAC™ – Synthetic read-only transactions • Gomez™ – Synthetic read-only transactions over the
Internet But what do our users really see (and why…) …
Slide 39
Apdex
• Application Performance Index • Sample Range – Satisfied = 0-1.5s – Tolerating = 1.5s – 7.0s – Frustrated = >7.0s
• Your ranges may vary by work type
Slide 40
Approach
• Gather render times • Bucket according to Apdex • Monthly reports/reviews by app/page • Trending reports • Baseline • Alerts for abnormal behavior
Slide 41
• | rangemap field=microseconds Sa3sfied=0-‐1500000 Tolera3ng=1500001-‐7500000 Frustrated=7500001-‐999999999999 default=Unknown …
• ….|chart values(pct) over Applica3on by range
41
Satisfied = 0-1.5s Tolerating = 1.5s – 7.0s Frustrated = >7.0s
Apdex
Slide 43
Splunk & Apdex
sourcetype="prod_apache_logs" NOT file=*.js NOT file=*.jpg httpstatus=200 | rangemap field=microseconds Satisfied=0-1500000 Tolerating=1500001-7500000 Frustrated=7500001-999999999999 default=Unknown | eval apdexScore=if(range=="Satisfied", 1, if(range=="Tolerating", 0.5, 0)) | streamstats count as Num, sum(apdexScore) as apScore window=500 global=f | eval apdex=apScore/Num | timechart avg(apdex)
Apdex =Satisfied+ Tolerating
2!
"#
$
%&
SampleSize
Slide 44
Splunk & Apdex
• Drill down to page level • Correlate to user – customer data – DB lookup to bring in customer stats – Proactive customer service
• Performance over time • Historic or real time metric
Slide 45
What Can I Do Tonight? • 1. Privileged access
– Employee database linked to Splunk – Compare access vs. HR records – Create and automate it
• 2. Malware Infections – Don’t stop with a re-image – Keep going – Find all possible infections
• 3. Internet sites – No reason not to look for invalid referrers
• 4. Apdex – Score your site against Apdex – Baseline and trend
Slide 48
What Can I Do Tonight?
1. Privileged access Employee database linked to Splunk Compare access vs. HR records Create and automate it
2. Malware Infections Don’t stop with a re-image – Keep going Find all possible infections
3. Internet sites No reason not to look for invalid referrers
4. Apdex Score your site against Apdex Baseline and trend
Slide 49
Put Totals in Your Searches?
Command: appendpipe Let’s count all our production sourcetypes sourcetype=prod_*
|stats count as Lines by sourcetype
|appendpipe [stats count as Types, sum(Lines) as Lines |eval sourcetype="Totals ...." ] | sort -Types
Slide 50
Recipes
• Can’t remember a sourcetype name: • |metadata type=sourcetypes | search
sourcetype=“prod*”
• Count by ip/user/host/dest • sourcetype=“prod_apache”| stats count by c_ip |
sort 20 –count
• Anonymizing Data • eval NAME=
substr(NAME,1,5). substr("**************",1,len(NAME)-5)
Slide 51 51
Security office hours: 11:00 AM – 2:00 PM @Room 103 Everyday
Geek out, share ideas with Enterprise Security developers Red Team / Blue Team - Challenge your skills and learn new tricks Mon-Wed: 3:00 PM – 6:00 PM @Splunk Community Lounge Thurs: 11:00 AM – 2:00 PM
Learn, share and hack
Birds of a feather- Collaborate and brainstorm with security ninjas Thurs: 12:00 PM – 1:00 PM @Meal Room