10/19/15
1
Cyber&Metrics&in&the&DoD
or
How&Do&We&Know&What&We&Don’t&Know?
John S. Bay, Ph.D.Executive Director
Things& People&Have&Asked&MeThings& People&Have&Asked&Me
• How&much&money&should&I&spend&this&year&on&cyber&
defense&technologies?
• How&many&attacks&has&your&firewall&repelled&this&
month?
• If&I&only&had&a&dollar&to&spend&on&cyber,&where&
should&I&spend&it?
• Why&is&cyber&research&such&a&slog?
211/12/14
Answers
(which& did¬& go&over&well)
Answers
(which& did¬& go&over&well)
• How&much&money&have&you&got?
• We&repelled&all&of&them&…&except&that&one&you&read&
about&in&the&paper
• Spend&your&dollar&on&upgrades
• Cyber&research&is&a&slog&because&there&is&no&physics&
theory&underlying&it&all,& liker&Maxwells’&Equations&
or&Newton’s&Laws
311/12/14
10/19/15
2
But&really&…&it&DEPENDSBut&really&…&it&DEPENDS
• The&“threat”&factor&is&common&in&cybersecurity,&
but&mostly¬&elsewhere
• …&&and&it&IS& true&that&there&is&no&useful&PHYSICS&
for&the&problem
411/12/14
DoD&Taxonomy&of& ThreatsDoD&Taxonomy&of& Threats
5
From:.Defense.Science.Board,.Resilient(Military(Systems(and(the(Advanced(Cyber(Threat,.January.2013
Tier Description
I Practitioners . who. rely. on. others . to. develop. the. malicious . code,. delivery. mechanisms,. and. executionstrategy. (use.known. exploits ).
II Practitioners . with. a.greater. depth. of. experience,. with. the. ability. to.develop. their. own. tools . (from.publically. known. vulnerabilities ).
III Practitioners . who. focus . on.the. discovery. and.use. of. unknown. malicious . code,. are. adept. at. installing.user. and. kernel. mode. root. kits ,. frequently. use.data. mining. tools ,. target. corporate. executives . akey.users . (government. and. industry). for. the. purpose. of.s tealing. personal. and. corporate. data. with. the.expressed. purpose. of. selling. the. information. to. other. criminal. elements .
IV Criminal. or. s tate. actors . who. are. organized,. highly. technical,. proficient,. well.funded. profess ionals .working. in.teams. to. discover. new. vulnerabilities . and.develop. exploits .
V State. actors . who. create. vulnerabilities . through. an.active. program. to. “influence”. commercial. products .and. services .during. des ign,. development. or. manufacturing,. or. with. the. ability. to. impactproducts . while. in.the. supply. chain. to. enable. exploitation. of. networks . and. systems.of. interest
VI States . with. the. ability. to. success fully. execute. full. spectrum. (cyber. capabilities . in.combination. withall.of. their. military. and.intelligence. capabilities ). operations . to. achieve. a.specific.outcome. in. political,military,. economic,. etc.. domains . and. apply.at. scale.
11/12/14
And&The& Corresponding& CriticalityAnd&The& Corresponding& Criticality
611/12/14
10/19/15
3
What&Might&the&COSTS& Be?What&Might&the&COSTS& Be?
711/12/14
So&Then,&What&to&Measure?So&Then,&What&to&Measure?
• Qualitative
– Capabilities
– Missions&lost
• Quantitative
– Performance
– Cost
• To&achieve
• Not&achieving
811/12/14
Capabilities&and&MaturityCapabilities&and&Maturity
911/12/14
10/19/15
4
Dashboard&ApproachDashboard&Approach
1011/12/14
“Stoplight&Chart”&Assessments“Stoplight&Chart”&Assessments
1111/12/14
See:. SPIDERS.JCTD
Costs& to&UsCosts& to&Us
• All&vulnerabilities&are&bugs&
• All&code&has&bugs
• Bugs&are&expensive
• Exploits&are&cheap&! the&“asymmetry”&problem
1211/12/14
10/19/15
5
MissionUAssurance& ApproachMissionUAssurance& Approach
• Helps& focus& attention
• Requires& a&“map”& o& the&
mission
• Implies& a&prioritization&
on&missions& (something&
loses)
• Requires& reconfigurable&
systems& and& networks
• Is¬& cheap
13
From:. .DUSD(I&E). Office,. HANDBOOKFor( SELFAASSESSING ( SECURITY( VULNERABILITIES( &(RISKS( of(INDUSTRIAL( CONTROL(SYSTEMSOn(DOD( INSTALLATIONS,.December. 201211/12/14
Just& Good& Enough& (Incremental)
Approach
Just& Good& Enough& (Incremental)
Approach
• How& long&would& our& red&team& take&to&penetrate&the&
system?
– An&empirical&measure,&at&best.
– Implies&a&canonical&red&team
14
prob(first&
vulnerability& is&
discovered)
time
Bad&code
Better&code
Gamma(distribution?
11/12/14
The&Accountability&ApproachThe&Accountability&Approach
• NIST&800U53&guidelines
• The&“did&we&do&everything&we&know&how&to&do”&approach
15
From:.NIST.Special.Publication.800]53,.rev..4,.Security(and(Privacy(Controls(forFederal(Information(Systems(and(Organizations,.April.201311/12/14
10/19/15
6
Conclusions:&Which&is&Best?Conclusions:&Which&is&Best?
• None& of&them.& They& service&somewhat& orthogonal&
purposes.
– But&they&can&provide&applesUtoUapples&comparisons
• Can& they&answer& the&Generals’& questions?
– No
– …&except&maybe&the&one&about&the&firewall
– There& is&CERTAINLY&no&satisfactory&“physics”&to&guide&anybody
• Cyber&Metrics& is&still&an& extremely&important& and& highU
priority&problem& for&OSD!
1611/12/14
