BSides Augusta

September 2016

Hunting: Defense Against The Dark Arts

Who We Are

Hunting: Defense Against The Dark Arts 2

• Jackie Stokes ....................................... @find_evil

• Danny Akacki ....................................... @dakacki

• Stephen Hinck ...................................... @stephenhinck

Hunting: Defense Against The Dark Arts 3

Problem Set

• Finding Evil

• Ways for Evil to do Evil Things

• Leverage data we already have / can readily obtain

• Drive maturation of monitoring & detection capabilities

HUNTDrive continuous improvement

Identify opportunities for action

Use internal and external data to

of the Information Security program

Solution: Threat Hunting

Hunting: Defense Against The Dark Arts 4

Hunting: Defense Against The Dark Arts 5

Hunting is a collection of processes

Not

❌ Tools

❌ Alerts

❌ Automation

Building a Hunt Program

Hunting: Defense Against The Dark Arts 6

"Understanding is the first step to acceptance, and only with acceptance can there be recovery." —Albus Dumbledore

Hunting Program

Mature detection capabilities

Use Cases + Playbooks

Guiding processes for SOC / CIRT

Technology & Tools

Operationally-driven and requirements-based

SOC + CIRT

Security operations and incident response

Formalized Security Program

Chartered and backed by an executive sponsor

Hunting: Defense Against The Dark Arts 7

Hunting Capability Pyramid

Must be this tall to ride

Hunting: Defense Against The Dark Arts 8

http://blog.sqrrl.com/the-cyber-hunting-maturity-model

Hunting Maturity Model

Building a Hunt Program

Hunting: Defense Against The Dark Arts 9

1. Establish executive sponsorship and mission charter/objectives

2. Establish and implement enterprise logging strategy

3. Aggregate, centralize, and process data

4. Make data available within searchable (fast) interface

5. Drive maturity

• Develop use cases

• Are we getting the right data?

• Review tooling and associated requirements

• Reintegrate hunt mission data to security operations

Hunting + IR Detection Maturation

Hunting: Defense Against The Dark Arts 10

HUNT SOC DETECT

IR USE CASE

Ongoing hunt missions

Feed Incident Response activities

IR outcomes affect

SecOps

Lessons Learned

incorporated to SecOps

Detection capability

improvement

Evil

Non-Evil Risk

Hunt Mission Outcomes

Hunting: Defense Against The Dark Arts 11

•Benefit: Activity shown not to be present

•Next Step: Evaluate hunt mission effectivenessNo Detection

•Benefits: Activity shown to be present

Hunt mission effectiveness validated

Identify best practice / compliance issues

•Next Step: Escalate as appropriate, monitor to closure

Detection: Non-Malicious

•Benefits: Activity shown to be present

Hunt mission effectiveness validated

Identify security incidents

•Next Step: Escalate as appropriate, monitor to closure

Detection: Malicious

Sorting Out Your Data

Hunting: Defense Against The Dark Arts 12

"Not Slytherin, eh? Are you sure? You could be great, you know."

Data Sources

- Remote Access- Web Proxy- IDS / IPS- Email- WAF

- DNS- DHCP- NetFlow- Firewall- Router / Switch- Wireless Infrastructure

- Agents- Antivirus- Operating Systems- Active Directory- File, Print, Database- Other Services

External Feeds- Paid, Free, OSINT

Internal Feeds- Recon data- IR Lessons Learned

- Critical Asset Inventory

- Privilege Management

- Approved Service Interruptions

- Terminated Users- Acceptable Use Policy- Employee Work Hours- Physical Access Data

Security

Network

Endpoint

IT

Threat Intel

HR

Hunting: Defense Against The Dark Arts 13

Two Types of Events

Hunting: Defense Against The Dark Arts 14

1. Observed Originated from a device which handled the event in some way

2. Synthetic Generated through automated analysis of event data

What is the Right Data?

Hunting: Defense Against The Dark Arts 15

• Original source data where-ever possible

• Ensure the presence of important fields

• Generally, observed events > synthetic events

• Synthetic events can provide useful context in the form of analytics

• Logs must enable pivoting

• Minimum one extractable / consistent data point to correlate log sources

Ready the Spells!

Hunting: Defense Against The Dark Arts 16

• Understand the network

• Learn critical assets

• Develop enterprise logging strategy

• Ensure data sources use consistent time settings; implement NTP, use GMT

• Plug in to asset, change, and configuration management processes

• Account for other organizational use cases

• IT Operations

• Forensics / Incident Response

• Compliance / Audit

• Clean up the dataset

• Normalization

• De-duplication

• Parsing

• Enrich and contextualize the dataset...!

Event Enrichment

Hunting: Defense Against The Dark Arts 17

• Internally-sourced Intelligence

• Attack trees

• Red Team / Penetration test output

• TTPs from previous incidents

• Deviances from baselines / Expected behavior

• Organizational risk profile / Threat context

• Externally-sourced Intelligence

• Paid subscriptions

• OSINT

• Free feeds

• Passive DNS, WHOIS, etc.

• Geographical data

• ISAC, Infragard, etc.

• Context

• Environmental

• Refer to "Data Source" slide

• Previous hunt and IR output

• Malware analysis

• Analytics, Ex.

• Geo-infeasibility

• Beacon detection

• DNS entropy

• Data exfiltration

Tools of the Trade

Hunting: Defense Against The Dark Arts 18

"It is important to fight, and fight again, and keep fighting, for only then could evil be kept at bay,

though never quite eradicated" —Albus Dumbledore

Criteria for a Working Hunt Platform

Hunting: Defense Against The Dark Arts 19

• Rapid search with high quality UI and / or API

• Stacking

• Group and reduce the dataset to more easily identify outliers

• Make manual analysis of an entire environment feasible

• Pivoting

• Move laterally through the dataset

• See the whole picture

Is It Worth It? Let Me Work It

• Tagging and Enrichments

• Intelligence Integration Support

• Automation: Rules & Alerting

• Evaluation Success Criteria

• Totally sweet dance moves

All About The Galleons

Hunting: Defense Against The Dark Arts 20

• Budget

• Driven by Operational Requirements

• Tool/Vendor Selection Process

• Multiple Tools: Diverse Perspectives

• Free and Open Source Software!

• NXLog

• Sysmon

• Moloch

• Wireshark

• Bro Network Security Monitor

• ELK (ElasticSearch, Logstash, Kibana)

• Security Onion Linux Distribution– Da Real MVP

+ A bunch of other stuff we didn't list here...

Analysis

Hunting: Defense Against The Dark Arts 21

"We teachers are rather good at magic, you know."

—Minerva McGonagall

Threat Hunting Loop

Hunting: Defense Against The Dark Arts 22

https://sqrrl.com/solutions/cyber-threat-hunting

Sample Hypotheses to Drive Hunt Missions

Hunting: Defense Against The Dark Arts 23

1. Sensitive corporate data stored only in approved locations

2. Large or extended outbound data transfers meet business needs

3. Reconnaissance activities against DMZ hosts provide advance warning of pending malicious activity

4. VPN logins by users are geographically feasible

5. Domain controller baselines are simple and deviations rarely occur

6. Service credentials are used only in expected ways and for their appropriate services

7. Web proxies are appropriately configured to block suspicious traffic

8. Our services communicate using secure, encrypted protocols

9. Tunneling HTTP traffic and other proxy avoidance techniques are not allowed in or out of our network

10. The use of management tools (such as PSExec) occurs only within approved change windows

11. Endpoints are not added to the network without infosec visibility

More Data, More Problems

Hunting: Defense Against The Dark Arts 24

"Dobby is... free." —Dobby the House Elf

Hunting: Defense Against The Dark Arts 25

Evil vs. Ways for Evil to do Evil Things

1. Remote Access

Hunting: Defense Against The Dark Arts 26

Hypothesis: Remote access to our environment is conducted using approved means

Discovery:

• Remote access is occurring over multiple protocols to / from unapproved hosts

• VNC to / from production network

• RDP to domain controllers from DMZ

• Evidence of unapproved remote access utilities such as LogMeIn, GoToMyPC, etc

Recommendation:

• Evaluate unapproved connections for mitigation or for risk acceptance

• Ensure that risk accepted software is fully patched and up to date

• Implement strong encryption, jump boxes / VPN ACLs, and two-factor authentication where possible

2. Data Storage

Hunting: Defense Against The Dark Arts 27

Hypothesis: Corporate data is only stored in approved locations

Discovery:

• Sensitive corporate data stored on unencrypted and infected external media

• Unrestricted use of common cloud data storage providers

• Unmanaged source code repositories (intellectual property)

Recommendation:

• Evaluate DLP implementation and allowed web proxy categories

• Consider establishing formalized agreement with a cloud storage provider

• Bring unmanaged data stores under management in support of development teams

3. Proxy Infrastructure

Hunting: Defense Against The Dark Arts 28

Hypothesis: Our proxy infrastructure is properly configured

Discovery:

• Not blocking known malicious categories

• Not blocking executable downloads

• Proxies not logging all necessary protocol metadata

• Ex. User Agent, Status Code, Byte Counts, X-Forward-For, etc.

Recommendation:

• Validate security operations' requirements of proxy infrastructure

• Re-evaluate proxy configurations for appropriate changes

• Ensure security operations are looped in to the change management process

4. Approved Protocols

Hunting: Defense Against The Dark Arts 29

Hypothesis: Protocols transiting our network are secure and approved for use

Discovery:

• Various insecure protocols identified in use across the network

• Unencrypted: Telnet, FTP

• Deprecated: SNMP v2, cleartext SMTP

• Risky: IRC, TOR / i2p

Recommendation:

• Identify opportunities to deploy secured versions of protocols

• FTP SFTP

• Telnet SSH

• SNMP v2 SNMP v3, etc.

• Evaluate implementation of risk detection and mitigation strategies

5. Approved Clients

Hunting: Defense Against The Dark Arts 30

Hypothesis: Internet access is achieved using known and approved client software

Discovery:

• Suspicious user-agents identified indicating potential latent infections

• Extremely out of date software, including client browsers, Flash, and Java

Recommendation:

• Begin incident response procedures to evaluate and triage endpoints

• Evaluate consistency of patch and vulnerability management processes

6. Privilege Management

Hunting: Defense Against The Dark Arts 31

Hypothesis: Account management is rooted in best practice

Discovery:

• Service accounts used for unrelated purposes or shared by users

• Regular and privileged users with non-specific accounts

• Direct privileged logins without approved privilege escalation process (e.g. sudo)

• Suspicious usernames that do not conform to the organizational standard

• User account belonging to terminated user active on the network

Recommendation:

• Evaluate suspicious or ambiguous accounts for mitigation or for risk acceptance

• Ensure security operations are tied into the HR termination workflow

• Update organizational username standard and privilege management processes

7. Security Architecture

Hunting: Defense Against The Dark Arts 32

Hypothesis: Event logs provide information needed to validate control effectiveness

Discovery:

• Non-security specific appliances with disabled security functionality

• Ex. Cisco ASA scan detection disabled

• Security specific appliances improperly placed

• Bro NSM placed post-proxy, post-NAT

Recommendation:

• Evaluate IT systems for security value (non-traditional security appliances)

• Ex. Network devices

• Modify configuration and placement of systems to meet requirements

8. Process Execution

Hunting: Defense Against The Dark Arts 33

Hypothesis: Endpoints only execute processes required for business functions

Discovery:

• Obfuscated PowerShell execution

• Mimikatz and other persistence toolkit execution

• Suspicious filenames/paths/registry entries, etc.

• Users installing browser toolbars and miscellaneous adware/spyware

Recommendation:

• Call the IR Team

• Adjust detections / controls to rapidly detect and prevent future occurrences

9. DNS

Hunting: Defense Against The Dark Arts 34

Hypothesis: DNS resolutions occur within the bounds of best practices

Discovery:

• "Weird" protocol deviations/padded packets suggesting exfil or C&C

• Uncontrolled resolutions that are not forced through corporate infrastructure

• Resolutions for unusual or risky domains

• Ex. Dynamic DNS domains, domains appearing to be algorithmically generated

• Initial resolutions for suspicious domains + subsequent unusual communication

Recommendation:

• Harden organizational DNS infrastructure

• Ex. Implement DNSSEC, prevent zone transfers, etc.

• Configure perimeter devices to only accept DNS requests from corporate DNS

• Implement protocol anomaly detection to identify protocol misuse

Thinking Ahead

Hunting: Defense Against The Dark Arts 35

"The one with the power to vanquish

the Dark Lord approaches..."

—Sybill Trelawney

Ensuring Successful Outcomes

Hunting: Defense Against The Dark Arts 36

• Goals

• Reduce attack surface

• Harden the environment

• Improve detection and monitoring

• Don't bother hunting without using the outputs!

• Lessons Learned / AAR

• Feedback loop on IR processes

• Create new or improve existing detections

• Metrics

• Cannot improve what is not measured

• The absence of something is still something

• Most metrics will trend upwards before they come down

• 'Time to Detect' and other metrics will trend downward over time

Hunt Methodology: From Art to Science

Hunting: Defense Against The Dark Arts 37

Begin evolution from an intuitive art form to a structured science

Happy Hunting!

Questions

Resources

Hunting: Defense Against The Dark Arts 39

FireEye Threat Analytics Platform: Hunting at Scalehttps://www.fireeye.com/products/threat-analytics-platform.html

Sqrrl: Thought leadership in the hunting spacehttp://blog.sqrrl.com

The Threat Hunting Project: Compendium of useful resourceshttp://www.threathunting.net

Loggly: Helpful logging guidelineshttps://www.loggly.com/intro-to-log-management

Security Onion: Peel back the layers of your networkhttps://securityonion.net