Seeing Red: Improving blue
teams through red teamingDave Hull
Tanium EDR Engineering
What is this?
Copyright 2015 Tanium Inc. All rights reserved.2
Intro
Copyright 2015 Tanium Inc. All rights reserved.3
Intro
Copyright 2015 Tanium Inc. All rights reserved.6
Intro
Copyright 2015 Tanium Inc. All rights reserved.7
Intro
Copyright 2015 Tanium Inc. All rights reserved.8
Intro
Copyright 2015 Tanium Inc. All rights reserved.9
Intro
Copyright 2015 Tanium Inc. All rights reserved.10
Intro
Copyright 2015 Tanium Inc. All rights reserved.11
Agenda
• Teaser
• Why red teaming
• What is red teaming
• Highlights and lessons learned
• Who should be red teaming
• When
• Practicalities of red teaming
• Conclusion
Copyright 2015 Tanium Inc. All rights reserved.12
Why red team?
Copyright 2015 Tanium Inc. All rights reserved.13
Because it delivers a security incident.
Pen testing delivers… a nice report.
Copyright 2015 Tanium Inc. All rights reserved.14
Why red team?
Because you will play like you practice.
Copyright 2015 Tanium Inc. All rights reserved.15
Why red team?
Copyright 2015 Tanium Inc. All rights reserved.16
“We run that play every day — end of every
practice,” [Phil] Booth said.
http://www.nytimes.com/2016/04/06/sports/ncaabasketball/villanova-national-championship.html?_r=0
Why red team?
Copyright 2015 Tanium Inc. All rights reserved.18
Why red team?
Because red teaming is quantifiable.
Copyright 2015 Tanium Inc. All rights reserved.19
Why red team?
Mean-time-to-compromise.
Copyright 2015 Tanium Inc. All rights reserved.20
Why red team?
Mean-time-to-detection.
Copyright 2015 Tanium Inc. All rights reserved.21
Why red team?
Mean-time-to-recovery.
Copyright 2015 Tanium Inc. All rights reserved.22
Agenda
• Teaser
• Why red teaming
• What is red teaming
• Highlights and lessons learned
• Who should be red teaming
• When
• Practicalities of red teaming
• Conclusion
Copyright 2015 Tanium Inc. All rights reserved.23
What is red teaming?
It is not threat modeling.
Copyright 2015 Tanium Inc. All rights reserved.24
What is red teaming?
It is not vulnerability assessment.
Copyright 2015 Tanium Inc. All rights reserved.25
What is red teaming?
It is not penetration testing.
Copyright 2015 Tanium Inc. All rights reserved.26
What is red teaming?
Red teaming is different.
Copyright 2015 Tanium Inc. All rights reserved.27
What is red teaming?
Some call it “adversary emulation.”
Copyright 2015 Tanium Inc. All rights reserved.28
What is red teaming?
Some call it “a force-on-force engagement.”
Copyright 2015 Tanium Inc. All rights reserved.29
Red teams:
Have mission objectives.
Copyright 2015 Tanium Inc. All rights reserved.30
Red teams:
Have mission objectives.
Enterprise or domain admin?
Copyright 2015 Tanium Inc. All rights reserved.31
Red teams:
Have mission objectives.
Customer pivot.
Copyright 2015 Tanium Inc. All rights reserved.32
Red teams:
Have mission objectives.
IP theft.
Copyright 2015 Tanium Inc. All rights reserved.33
Red teams:
Have mission objectives.
Burn it all down.
Copyright 2015 Tanium Inc. All rights reserved.34
Red teams:
Have mission objectives.
Test incident response capabilities and procedures.
Copyright 2015 Tanium Inc. All rights reserved.35
Red teams:
Have mission objectives.
Test incident response capabilities and procedures
of the organization... not just the blue team.
Copyright 2015 Tanium Inc. All rights reserved.36
Who responds, if...
Copyright 2015 Tanium Inc. All rights reserved.37
Who responds, if Brian Krebs is your IDS?
Not just the IR team.
Not just the security team.
Copyright 2015 Tanium Inc. All rights reserved.38
Agenda
• Teaser
• Why red teaming
• What is red teaming
• Highlights and lessons learned
• Who should be red teaming
• When
• Practicalities of red teaming
• Conclusion
Copyright 2015 Tanium Inc. All rights reserved.39
Lesson learned
Outliers may be leads.
Copyright 2015 Tanium Inc. All rights reserved.40
Outliers may be leads.
Copyright 2015 Tanium Inc. All rights reserved.41
Outliers may be leads.
Copyright 2015 Tanium Inc. All rights reserved.42
Do you even monoculture?
Copyright 2015 Tanium Inc. All rights reserved.43
Dan Geer:
Copyright 2015 Tanium Inc. All rights reserved.44
• "Internet security is quite possibly the most
intellectually challenging profession on the planet... for
two reasons... complexity... and rate of change [are] your
enemy.
Loathsome long tails...
Copyright 2015 Tanium Inc. All rights reserved.45
“... ever present everywhere...”
Copyright 2015 Tanium Inc. All rights reserved.46
Build systems that automate
data collection, analysis and remediation.
Copyright 2015 Tanium Inc. All rights reserved.47
Blue’s Prime Directive: Remediation
Copyright 2015 Tanium Inc. All rights reserved.48
Remediation, like security, is a process not a product.
Copyright 2015 Tanium Inc. All rights reserved.49
Investigate. Remediate. Repeat.
Copyright 2015 Tanium Inc. All rights reserved.50
Agenda
• Teaser
• Why red teaming
• What is red teaming
• Highlights and lessons learned
• Who should be red teaming
• When
• Practicalities of red teaming
• Conclusion
Copyright 2015 Tanium Inc. All rights reserved.51
Who should be red teaming?
Any organization that may have a security incident.
Copyright 2015 Tanium Inc. All rights reserved.52
Who should be red teaming?
Any organization with something worth protecting.
Copyright 2015 Tanium Inc. All rights reserved.53
Who should be red teaming, practically speaking?
Organizations meeting the previous criteria and having:
Some monitoring.
Some defenses.
Some IR capabilities.
Copyright 2015 Tanium Inc. All rights reserved.54
Who should be red teaming?
Probably an internal team, but not just the security team.
Copyright 2015 Tanium Inc. All rights reserved.55
Lesson learned
Documentation is wrong.
Code comments are wrong.
Assumptions are wrong.
Copyright 2015 Tanium Inc. All rights reserved.56
Agenda
• Teaser
• Why red teaming
• What is red teaming
• Highlights and lessons learned
• Who should be red teaming
• When
• Practicalities of red teaming
• Conclusion
Copyright 2015 Tanium Inc. All rights reserved.57
When should you red team?
Two, maybe three times a year.
Copyright 2015 Tanium Inc. All rights reserved.58
Agenda
• Teaser
• Why red teaming
• What is red teaming
• Highlights and lessons learned
• Who should be red teaming
• When
• Practicalities of red teaming
• Conclusion
Copyright 2015 Tanium Inc. All rights reserved.60
Practicalities
Have Rules of Engagement.
Copyright 2015 Tanium Inc. All rights reserved.61
Rules of engagement
Get approval from management and legal.
Copyright 2015 Tanium Inc. All rights reserved.62
Rules of engagement
Copyright 2015 Tanium Inc. All rights reserved.63
Rules of engagement
No accessing or tampering with customer data.
Copyright 2015 Tanium Inc. All rights reserved.64
Rules of engagement
No accessing or tampering with real customer data.
Copyright 2015 Tanium Inc. All rights reserved.65
Rules of engagement
No outages.
Copyright 2015 Tanium Inc. All rights reserved.66
Rules of engagement
No weakening of existing
controls.
Copyright 2015 Tanium Inc. All rights reserved.67
Rules of engagement
Give the red team access.
Copyright 2015 Tanium Inc. All rights reserved.68
Rules of engagement
Give the red team source code.
Copyright 2015 Tanium Inc. All rights reserved.69
Rules of engagement
Give the red team architecture diagrams.
Copyright 2015 Tanium Inc. All rights reserved.70
Rules of engagement
Keep the blue team in the dark.
Copyright 2015 Tanium Inc. All rights reserved.71
Rules of engagement – Don’t let blue do this
Copyright 2015 Tanium Inc. All rights reserved.72
Rules of engagement
Real incidents trump red team incidents.
Copyright 2015 Tanium Inc. All rights reserved.73
Rules of engagement
Red incidents are core hours only.
Copyright 2015 Tanium Inc. All rights reserved.74
Rules of engagement
Red incidents are core hours only,
plus a little.
Copyright 2015 Tanium Inc. All rights reserved.75
Rules of engagement
Cross team collaboration.
Copyright 2015 Tanium Inc. All rights reserved.76
Rules of engagement
Establish a situation room.
Copyright 2015 Tanium Inc. All rights reserved.77
Rules of engagement
Designate incident and investigative leads.
Copyright 2015 Tanium Inc. All rights reserved.78
Rules of engagement
Delegate and PM.
Copyright 2015 Tanium Inc. All rights reserved.79
Situation normal...
Investigate.
Copyright 2015 Tanium Inc. All rights reserved.80
Situation normal, practice how you want to play
Document.
Copyright 2015 Tanium Inc. All rights reserved.81
Situation normal, practice how you want to play
Report.
Copyright 2015 Tanium Inc. All rights reserved.82
Situation normal, practice how you want to play
Copyright 2015 Tanium Inc. All rights reserved.83
Situation normal, practice how you want to play
Plan for remediation.
Copyright 2015 Tanium Inc. All rights reserved.84
Situation normal, practice how you want to play
Execute remediation.
Copyright 2015 Tanium Inc. All rights reserved.85
Situation normal, practice how you want to play
Post remediation monitoring.
Copyright 2015 Tanium Inc. All rights reserved.86
Take aways
Postmortems.
Copyright 2015 Tanium Inc. All rights reserved.87
Postmortem: Who?
Stakeholders, blue team, red team.
Copyright 2015 Tanium Inc. All rights reserved.88
Postmortem: What?
No blame games.
Copyright 2015 Tanium Inc. All rights reserved.89
Postmortem: What?
But hold yourself accountable.
Copyright 2015 Tanium Inc. All rights reserved.90
Postmortem: Story time.
Blue team goes first.
Copyright 2015 Tanium Inc. All rights reserved.91
Postmortem: Tell all.
Copyright 2015 Tanium Inc. All rights reserved.92
Postmortem: The facts.
Red team goes second.
Copyright 2015 Tanium Inc. All rights reserved.93
Postmortem: Mind the gap.
Blue Red
Copyright 2015 Tanium Inc. All rights reserved.94
Goal: close gap over time
Postmortem: Takeaways.
All teams get bugs, feature requests.
Copyright 2015 Tanium Inc. All rights reserved.95
Agenda
• Teaser
• Why red teaming
• What is red teaming
• Highlights and lessons learned
• Who should be red teaming
• When
• Practicalities of red teaming
• Conclusion
Copyright 2015 Tanium Inc. All rights reserved.96
Lesson learned
Just-In-Time admin (JIT).
Copyright 2015 Tanium Inc. All rights reserved.98
Lesson learned
Dedicated admin workstations.
Copyright 2015 Tanium Inc. All rights reserved.101
Lesson learned
Zero human generated passwords.
Copyright 2015 Tanium Inc. All rights reserved.102
Lesson learned
2FA everywhere.
Copyright 2015 Tanium Inc. All rights reserved.103
Lesson learned
Don’t trust. Verify.
Copyright 2015 Tanium Inc. All rights reserved.104
Agenda
• Teaser
• Why red teaming
• What is red teaming
• Highlights and lessons learned
• Who should be red teaming
• When
• Practicalities of red teaming
• Conclusion
Copyright 2015 Tanium Inc. All rights reserved.105
Conclusion
Red teaming is hard.
Copyright 2015 Tanium Inc. All rights reserved.106
Conclusion
Real incidents may be harder.
Copyright 2015 Tanium Inc. All rights reserved.107
Conclusion
Practice how you want to play.
Copyright 2015 Tanium Inc. All rights reserved.108