Adversarial Machine Learning
from an
Adversarial Risk Analysis
Perspective
David Ríos Insua
AXA-ICMAT Chair and Royal Academy
ICANN, Alghero, September 2017
with D. Banks, J. Rios, F. Ruggeri, R. Soyer, J. Ortega, R. Naveiro, A. Redondo and CYBECO
Outline
• (Almost) All things adversarial
• Adversarial risk analysis
• Adversarial statistical decision theory
• Adversarial point estimation
• Adversarial hypothesis testing
• Adversarial classification
• Discussion and challlenges
Adversarial problems
• Stats/ML: Standard problems – Point Estimation
– Prediction
– Learning
– Hypothesis testing
– Classification
– ..
• Many applications (security, mktg,…) entail adversaries – Spam detection
– Fraud detection
– Network monitoring
– ….
• Intelligent attackers adapting their behaviour to remain undetected and obtain a benefit
• Comparatively few attempts to deal with the problem….
• ...Mostly modelled through noncooperative game theory
Example: Adversarial
classification as a game • C, classifier. A, adversary
• Two classes: + malicious; - innocent.
• C and A maximise expected utility under common knowledge conditions
• Finding Nash equilibria extremely complex
• Dalvi et al (2005) propose a scheme
Utility sensitive Naive Bayes
Forward myopic approach under strong common knowledge
Adversarial problems
• Adversarial classification (Dalvi et al,…)
• Adversarial signal processing (Barni et
al,..)
• Adversarial learning (Lowd and Meek,..)
• Adversarial machine learning (Tygar,..)
• Adversarial SVMs (Zhou et al,…)
• …
• Current adversarial competition in Kaggle
Outline
• (Almost) All things adversarial
• Adversarial risk analysis
• Adversarial statisticsl decision theory
• Adversarial point estimation
• Adversaril hypothesis testing
• Adversarial classification
• Discussion and challlenges
From RA to ARA…
Motivation
• RA extended to include adversaries ready to increase our risks
• S-11, M-11,.. lead to large security investments globally, some of them criticised
• Many modelling efforts to efficiently allocate such resources
• Parnell et al (2008) NAS review
– Standard reliability/risk approaches not take into account intentionality
– Game theoretic approaches. Common knowledge assumptions…
– Decision analytic approaches. Forecasting the adversary action…
• Merrick, Parnell (2011) review approaches commenting favourably on ARA
9
ARA • A framework to manage risks from actions of intelligent adversaries (DRI, Rios, Banks,
JASA 2009)
• One-sided prescriptive support – Use a SEU model – Treat the adversary’s decision as uncertainties
– Bayesian games Kadane, Larkey (1982), Raiffa (1982, 2002) made operational
• Method(s) to predict adversary’s actions
– We assume the adversary is a expected utility maximizer • Model his decision problem • Assess his probabilities and utilities • Find his action of maximum expected utility
(But other descriptive models are possible) • Uncertainty in the Attacker’s decision stems from
– our uncertainty about his probabilities and utilities – but this leads to a hierarchy of nested decision problems
(random, noninformative, level-k, heuristic, mirroring argument,…) vs (common knowledge)
• Lippman, McCardle (2012) • Stahl and Wilson (1995) D. Wolpert (2012) • Rothkopf (2007) • MacLay, Rothschild, Guikema (2013,2014)
• Banks, Rios, DRI (2015)
10
Sequential Def-Att game
– Two intelligent players
• Defender and Attacker. D knows A’s judgements
– Sequential moves
• Def, then Attacker
( | , )Ap S d a
( , )Du d S ( , )Au a S
( | , )Dp S d a
11
Standard GT Analysis
Solution:
Nasheq. Subgame
perfect equilibrium
Expected utilities at node S
Best Attacker’s decision at node A
Assuming Defender knows Attacker’s analysis
Defender’s best decision at node D
12
Supporting the Defender
Defender problem Defender’s view of Attacker problem
13
Supporting the Defender
Defender problem Defender’s solution
Modeling input: ??
14
Supporting the Defender:
The assessment problem
Defender’s view of
Attacker problem
Elicitation of
A is a EU maximizer
D’s beliefs about
MC simulation
where
Sequential D-A
16
Simultaneous DeffAtt game
• Decisions are made without knowing each other’s
decisions
17
Game Theory Analysis
• Common knowledge
– Each knows expected utility of every pair (d, a) for both of them
– Nash equilibrium: (d*, a*) satisfying
• When some information is not common knowledge
– Private information
• Type of Defender and Attacker
– Common prior over private information
– Model the game as one of incomplete information
18
Bayes Nash Equilibrium
– Strategy functions
• Defender
• Attacker
– Expected utility of (d,a)
• for Defender, given her type
• Similarly for Attacker, given his type
– Bayes-Nash Equlibrium (d*, a*) satisfying
19
Supporting the Defender
• Defender’s decision analysis
How to
assess it ??
20
Assessing • Attacker's decision analysis as seen by the Defender
21
The assessment problem
• To predict Attacker’s decision
The Defender needs to solve Attacker’s decision problem
She needs to assess
• Her beliefs about are modeled through a probability distribution
• The assessment of requires deeper analysis – D’s analysis of A’s analysis of D’s problem
• It leads to an infinite regress thinking-about-what-the-other-is-thinking-about…
22
Hierarchy of nested models
Stop when the Defender has no more information about utilities and probabilities
at some level of the recursive analysis. level-k thinking
ARA templates
More general interactions
ARA: Examples/Cases Problem Defender Attacker Specificities Template
ATC protection Airport authority Terrorist Single site D-> A
Piracy Ship owner Pirates Single site D- >A - > D
Metro Operator Pickpock
Fare evasion
Multisite
Multiattack,
Cascade
D->A
Urban security Police Mob Multisite spatial D->A->D
Train DoT, DoD Terrorist Multisite network D->A->D
Reliability Manufacturer Customer -- D->A
SME IS.
CYBECO
Company Competitor Cyber, Integrated
with RA
D->A
Oil rig
cybercontrolled
Oil company Sponsored
hackers
Cyber, Multiattack D->A->D
CI Owner Terrorist Multistage General
Cybersec res
allocation+cybins
IT Owner Hacker(s) Several decisions
Random and
targeted attacks
D-A, D-A-D
Social robots Robot User Sequential D->A
26
Other themes
• Different opponent models, beyond SEU
• Concept uncertainty, Mixtures
• Robustness and ARA (GT, ARA, Robust ARA)
• Multiple attackers, Multiple defenders
• Differential games
• Competition and cooperation
• Efficient computational schemes
• Computational environment
• …
Outline
• (Almost) All things adversarial
• Adversarial risk analysis
• Adversarial statistical decision theory
• Adversarial point estimation
• Adversaril hypothesis testing
• Adversarial classification
• Discussion and challlenges
Statistical Decision Theory
• Point estimation under quadratic loss
Adversarial Statistical Decision
Theory
Outline
• (Almost) All things adversarial
• Adversarial risk analysis
• Adversarial statistical decision theory
• Adversarial point estimation
• Adversarial hypothesis testing
• Adversarial classification
• Discussion and challlenges
Adversarial point estimation
• Quadratic loss
Adversarial point estimation
Concept uncertainty
Adversarial point estimation
• A Bayesian adversary
• Mixture, e.g.
Adversarial point estimation
• Normal-normal model, for certain
parameter choices
Outline
• (Almost) All things adversarial
• Adversarial risk analysis
• Adversarial statistical decision theory
• Adversarial point estimation
• Adversarial hypothesis testing
• Adversarial classification
• Discussion and challlenges
AHT
AHT
AHT
AHT
AHT
Numerical example, Spam detection example
Outline
• (Almost) All things adversarial
• Adversarial risk analysis
• Adversarial statistical decision theory
• Adversarial point estimation
• Adversarial hypothesis testing
• Adversarial classification
• Discussion and challlenges
Adversarial classification through
ARA. ACRA
Dalvi et al’s pioneer AC model from ARA perspective
ACRA. Classifier problem
?
?
ACRA. Adversary problem
random version
of
ACRA. Spam detection. Approach
• Preprocessing 1.
For a given
training set,
we estimate
e.g. utility
sensitive
Naïve
Bayes
Probability of
malicious
and innocent
class
Probability for
each email to be
malicious or
innocent
ACRA. Spam detection. Approach
• Preprocessing 2.
For each email,
we compute
The probabilities of the
relevant attacks, given the
email and if is malicious or
innocent
ACRA. Spam detection. Approach
• Operation.
Read a (possibly) modified email by
attacker
Compute all relevant
attacks
The Classifier maximize her expected utility to classify the email as spam
or not
Outline
• (Almost) All things adversarial
• Adversarial risk analysis
• Adversarial statistical decision theory
• Adversarial point estimation
• Adversarial hypothesis testing
• Adversarial classification
• Discussion and challlenges
Discussion
• Traditional statistical/ML problems perturbated by presence of
adversaries
• Traditionally treated from a game theoretic perspective (common
knowledge)
• An ARA approach to mitigate common knowledge
• Many challenges
– Multiple attackers vs Multiple defenders
– Efficient computation
– Generic approach: point estimation, interval estimation,…
• Classification: NB, NNs, SVMs,…
– Generative adversarial networks?
– Cybersecurity
Thanks!!!
Collabs welcome
SPOR DataLab https://www.icmat.es/spor/
Aisoy Robotics https://www.aisoy.com
It’s a risky life @YouTube
CYBECO https://www.cybeco.eu/
