Accountability in Cyber-Physical Systems

Anupam Datta

Associate Professor Computer Science &

Electrical and Computer Engineering

Cyber-Physical Systems (CPS)

Computing + Communication + Control

2

Cyber-Physical Systems (CPS) under Attack

Computing + Communication + Control

3

Thesis

Accountability is key to securing Cyber-Physical Systems

Causal information flow analysis will enable a unified foundation for accountability in Cyber-Physical Systems

4

Accountability

• Detection of attacks

• Explanation and responsibility assignment

• Correction

• Much work in CPS security already on pieces of the accountability puzzle

• Goal: A unified foundation

5

System

System

Dist

InputA

InputB

Causal Information Flow Measure

Prior work on causal information flow

• Non-interference in deterministic systems [Denning-Denning1977]

• Probabilistic Non-interference [Volpano et al. 1999]

• Measuring Probabilistic Interference [Tschantz et al. 2015, Datta et al. 2015]

• Measuring Quantitative Input Influence [Datta et al. 2016]

Confidentiality = Absence of information flow

Attack Detection= Presence of causal information flow

(e.g., gender causes difference in job ads)

Explanation= Quantifying causal information flow

(e.g., zipcode has higher influence than

income in credit decisions)

6

My Work: Privacy through Accountability

Work in my research group:

Accountability via audit log analysis: CCS 2011, Oakland 2012, CAV 2014, CCS 2015

Accountability in big data systems: Oakland 2014, CSF 2015, PETS 2015, Oakland 2016

7

Web

services Credit

Law

Enforcement Healthcare Education …

Accountability in Machine Learning Systems

Associative Information Flow Measures

• Popular in research on Quantitative Information Flow • Mutual information, correlation coefficients, Jacard Index etc. to

measure association between inputs and outputs

• Appropriate for measuring confidentiality leaks • Example: sexual orientation revealed by association with social network

friend links

• Not appropriate for accountability • Difficult to trace back to exact cause

• Example: Gender and Weightlifting Ability both associated with job decision classifier but Gender not a cause

8

Today’s Thesis…with Evidence

Accountability is key to securing control systems • Focus on detecting attacks

• Preliminary ideas on responsibility-assignment, corrective measures

Causal information flow analysis will enable a unified foundation for accountability in control systems

9

Joint work with Kar, Sinopoli, Weerakkody at CMU

Technical paper on arXiv

Control

System

Control

System

Dist

Normal Input

Normal Input

Attack Input Attack Output

Normal Output

Idea: Causal Information Flow Measure

Using KL-divergence to measure difference between output distributions

10

×

Physical System Plant

S

S

A

A

Controller

Communication

Network

S

Detector

Passive Detection

11

Physical System Plant

S

S

A

A

Controller

Communication

Network

S

Detector

Stealthy Attacks

✓

12

×

S

S

A

A

Communication

Network

S

Detector ✓

Controller

Physical System Plant

Active Detection

13

Topic Summary of Result Previous

Work

Passive

Detection

Information flow as a measure of optimal false

alarm rate.

[7]

Information flow, relation to Neyman-Pearson

detector

Neyman

Pearson Lem.

Stealthy

Attacks

0 – information flow related to left invertibility

of system (zero dynamics without initial state)

[8],[9]

False Data Injection Attacks: Information flow

equivalent to norm of residue bias

[10],[11],[12]

Bias on residues characterize attack detectability [10],[11],[12]

Active

Detection

Replay attacks for certain systems/controllers

are stealthy

[13],[14],[15]

Physical Watermarking can be used to detect

replay attacks

[13],[14],[15]

Unified Treatment

Causal information flow analysis useful for recovering ~10

attack detection results

14

Why useful?

•Systematization of work on CPS Security • Simple natural definition

• A tool kit of analysis techniques

• Recover existing results + prove new results

•Bridge CPS Security and mainstream Security & Privacy • Shared vocabulary of information flow concepts

• New results combining information flow analysis for computing + communication + control systems

15

Workshop Goal

Talk Outline

• System and attack model

• KL divergence measure for causal information flows

• Information flow and detection • Passive detection

• Stealthy attack scenarios

• Active detection

• Toward Responsibility Assignment and Correction

16

×

S

S

A

A

Communication

Network

Detector ✓

Controller

Physical System Plant

System Model

17

A discrete linear time-invariant control system

State: Velocity and Position

Inputs: Throttle

Outputs: Velocity and Position

Physical System/Plant

18

Definitions generalize to general nonlinear time varying systems

Physical System/Plant

19

×

S

S

A

A

Communication

Network

Detector ✓

Physical System Plant

Controller

Control Strategy

20

Control Strategy

• Control strategy leverages the defender’s information

which includes previous outputs, inputs and the system model in order to construct an input which meets system objectives.

• An admissible control strategy is a sequence of deterministic measureable functions

21

×

Passive Detection

S

S

A

A

Communication

Network

✓

Physical System Plant

Controller

Detector

22

Passive Detection

Detector leverages information in the system to determine if the system is operating normally or under attack .

Implements forms of hypothesis testing

An admissible detector is a sequence of deterministic measureable functions:

23

×

Physical System Plant

S

S

A

A

Controller

Communication

Network

S

✓ Detector

Attack Model

24

Attack Model

• Attacker can modify a subset of control inputs and sensor outputs.

• Vehicle Example: if the adversary modified the entire input and the position sensor (our second state):

25

Attack Model

• General Setup: Can model nonlinear adversaries

26

Attack Strategy

• An attack strategy leverages an attacker’s information to construct attack sequences.

27

Attack Strategy

• An admissible attack strategy is a sequence of deterministic measureable functions

• Can model adaptive adversaries

28

Attack Model Covers Known Attack Classes

• Replay attack

• False data injection attack

• Zero dynamics attack

• Covert attack

All these attacks on linear systems involve

• Additive attack input

• Varying degrees of knowledge about system

29

Output Distribution

The distribution of the system output given defender and attacker strategies is

30

Control

System

Talk Outline

• System and attack model

• KL divergence measure for causal information flows

• Information flow and detection • Passive detection

• Stealthy attack scenarios

• Active detection

• Toward Responsibility Assignment and Correction

31

Causal Information Flow Measures

•What measures are appropriate? • Open: Exploration of space of measures

•Today • Pick a specific measure

• Define it

• Justify choice

32

KL-divergence

• KL divergence measures difference between two distributions p and q defined over a set X

• Properties

33

Control

System

Control

System

Proposed Measure: Attack vs Normal AN-KL-Divergence

Measures information flow from

attacker’s inputs to defender’s outputs

34

AN-KL-Divergence and Probabilistic Non-interference

• Input x is probabilistically non-interfering with output y if changing x doesn’t alter probability distribution of y

• Here is probabilistically noninterfering with iff the AN-KL-Divergence = 0.

35

Agrees with existing qualitative

information flow property

Control

System

Definition: Weak Information Flow

Useful for characterizing stealthy

attacks against active detection

(e.g., zero dynamics attack)

36

Control

System

Control

System

Definition: (M,U)- Weak Information Flow

Useful for characterizing stealthy

attacks that might be revealed through

active detection

(e.g., replay attack)

37

Control

System

Control

System

Definition: Strong Information Flow

Useful for characterizing successful

active detection

(e.g., of replay attacks)

38

Control

System

Talk Outline

• System and attack model

• KL divergence measure for causal information flows

• Information flow and detection • Passive detection

• Stealthy attack scenarios

• Active detection

• Toward Responsibility Assignment and Correction

39

Topic Summary of Result Previous

Work

Passive

Detection

Information flow as a measure of optimal false

alarm rate.

[7]

Information flow, relation to Neyman-Pearson

detector

Neyman

Pearson Lem.

Stealthy

Attacks

0 – information flow related to left invertibility

of system (zero dynamics without initial state)

[8],[9]

False Data Injection Attacks: Information flow

equivalent to norm of residue bias

[10],[11],[12]

Bias on residues characterize attack detectability [10],[11],[12]

Active

Detection

Replay attacks for certain systems/controllers

are stealthy

[13],[14],[15]

Physical Watermarking can be used to detect

replay attacks

[13],[14],[15]

Unified Treatment

40

10,000 Foot View of Some Results

41

1. Passive Detection

• Result: AN-KL-Divergence measures optimal false alarm rate

• - strong information flow Detectability with false alarm rate

• - weak information flow Not detectable with false alarm rate

• Technically

• Cast results of Bai et al. 2015 in terms of information flow

• Conceptually

• We recognize that information flow is the fundamental concept; detectability is one consequence (there are others)

• Bai et al. have detectability as the fundamental goal; KL-divergence is an analysis tool

42

2. Stealthy Attacks

• Result: Complete characterization of conditions under which linear systems exhibit 0- weak information flow (i.e. probabilistic non-interference) for all time k • 0- weak information flow (probabilistic non-interference)

zero dynamics attack when defender does not have knowledge of initial state

• 0- weak information flow (probabilistic non-interference) attacks on systems that are not left invertible when defender has knowledge of initial state

• Technically

• Cast results of Pasqualetti et al. [8], Teixeira et al. [9] in terms of information flow

43

3. Active Detection

• Result: Characterization of conditions under which

1. Replay attacks can be stealthy based on (M,U)- weak information flow

2. A randomized watermarking active detector generates strong information flow and hence can detect replay attacks.

• Technically

1. More general result than Mo et al. [13,14,15], which assumed certain properties of the detector

2. Quantifying information flow allows us to directly characterize optimal detectability unlike Mo et al. [13], [14], [15]

44

1. Passive Detection

• Result: AN-KL-Divergence measures optimal false alarm rate

• - strong information flow Detectability with false alarm rate

• - weak information flow Not detectable with false alarm rate

• Technically

• Cast results of Bai et al. 2015 in terms of information flow

• Conceptually

• We recognize that information flow is the fundamental concept; detectability is one consequence (there are others)

• Bai et al. have detectability as the fundamental goal; KL-divergence is an analysis tool

45

Passive Detection

• Passive detector

• Probability of false alarm

• Probability of detection

46

Passive Detection: Optimality (1)

Theorem: Strong Information Flow => Detectability

Let

and

Then there exists a detector such that

1)

2) converges to 0 with rate at least

47

Casting result of (Bai et al 2015) [7] in terms of IF

Passive Detection: Optimality (2)

Theorem: Weak Information Flow => Stealthy

Let

and and is ergodic

Then there is no detector such that

1)

2) converges to 0 with rate at least

48

Casting result of (Bai et al 2015) [7] in terms of IF

2. Stealthy Attacks

• Result: Complete characterization of conditions under which linear systems exhibit 0- weak information flow (i.e. probabilistic non-interference) for all time k • 0- weak information flow (probabilistic non-interference)

zero dynamics attack when defender does not have knowledge of initial state

• 0- weak information flow (probabilistic non-interference) attacks on systems that are not left invertible when defender has knowledge of initial state

• Technically

• Cast results of Pasqualetti et al. [8], Teixeira et al. [9] in terms of information flow

49

0-Dynamics Attacks: Background

• 0-Dynamics of a system • Control input u produces non-zero state x but output y = 0

• Consider normal operating condition with (u*, x*, y*)

• 0- Dynamics Attack on linear system • Adversary adds u to u*

• State changes to x + x*

• Output = y + y* = y* (no change!)

50

S

S

Same!

Example

A vehicle with initial state operating normally will produce the same output as a system with initial state under attack! Cannot be detected without knowledge of initial state

51

0-Weak Information Flow

• Assume defender has no knowledge of initial state

• Theorem: There exists an attack which generates a 0- weak information flow for all k (there exists probabilistic noninterference for all time k) if and only if with probability 1 for some .

52

Leverages Teixeira et al. [9]

0-Dynamics Attacks: Result

• 0-weak information flow attacks are equivalent to 0-dynamics attacks Pasqualetti et al. [8], Teixeira et al. [9]

• Result: If the defender has no knowledge of the initial state, a zero information flow attack exists for all time k if there exists

and which satisfy

• One stealthy attack:

53

3. Active Detection

• Result: Characterization of conditions under which

1. Replay attacks can be stealthy based on (M,U)- weak information flow

2. A randomized watermarking active detector generates strong information flow and hence can detect replay attacks.

• Technically

1. More general result than Mo et al. [13,14,15], which assumed certain properties of the detector

2. Quantifying information flow allows us to directly characterize optimal detectability unlike Mo et al. [13], [14], [15]

54

2) Inject input 𝑢𝑘 = 𝑢𝑘∗ + Δ𝑢𝑘 1) with optimal inputs

3) with sub-optimal input 4) Binary Detector

Watermarking: Background

55

Watermarking: Strong Information Flow

Randomized watermarking generates

strong information flow for replay

attack

56

Watermarking: Detectability

Randomized watermarking can be used

to detect replay attacks

57

Talk Outline

• System and attack model

• KL divergence measure for causal information flows

• Information flow and detection • Passive detection

• Stealthy attack scenarios

• Active detection

• Toward Responsibility Assignment and Correction

58

×

S

S

A

A

Communication

Network

S

✓

Physical System Plant

Controller

Detector

Responsibility Assignment

59

Responsibility Assignment: Idea

60

×

S

S

A

A

Communication

Network

S

✓

Physical System Plant

Detector

Controller

×

×

Correction: Resilient Control

61

Resilient Control: Idea

• Leverage results on detection and responsibility assignment

• For sensor attacks:

1. Detect an information flow from attack inputs.

2. Identify malicious nodes which generate information flow.

3. Construct robust estimate with trusted sensors.

4. Perform resilient control using robust estimate.

62

Today’s Thesis…with Evidence

Accountability is key to securing control systems • Focus on detecting attacks

• Preliminary thoughts on responsibility-assignment, corrective measures

Causal information flow analysis will enable a unified foundation for accountability in control systems

63

Joint work with Kar, Sinopoli, Weerakkody at CMU

Technical paper on arXiv

Toward Accountability in CPS

• Cryptography + Control Systems

Example: Randomized watermarking • Watermark generated using a pseudorandom number generator

(PRNG) • What is an appropriate information flow measure? • Have to restrict to polynomial time adversaries

• Computing System Security + Control Systems

Example: Defending against 0-dynamics attacks • How to reliably communicate initial state to defender?

• A trusted path primitive for CPS?

64

Information flow analysis spanning

cryptography, computing systems,

control systems (models + code)

Thesis

Accountability is key to securing Cyber-Physical Systems

Causal information flow analysis will enable a unified foundation for accountability in CPS

65

Thanks!

66

[1] D. E. Denning and P. J. Denning, “Certification of programs for secure information flow,”

Communications of the ACM, vol. 20, no. 7, pp. 504–513, 1977. [Online]. Available:

http://doi.acm.org/10.1145/359636.359712

[2] J. A. Goguen and J. Meseguer, “Security policies and security models,” in IEEE Symposium on

Security and Privacy, 1982, pp. 11–20.

[3] D. M. Volpano and G. Smith, “Probabilistic noninterference in a concurrent language,” Journal of

Computer Security, vol. 7, no. 1, 1999.

[4] M. C. Tschantz, A. Datta, A. Datta, J. M. Wing, “A methodology for information flow experiments,” in

Proceedings of the 28th IEEE Computer Security Foundations Symposium, July 2015.

[5] G. Smith, "On the foundations of quantitative information flow.” in Foundations of software science

and computational structures, Springer Berlin Heidelberg, 2009, pp. 288-302.

[6] A. Datta, S. Sen, Y. Zick, “Algorithmic Transparency via Quantitative Input Influence”, in Proceedings

of 37th IEEE Symposium on Security and Privacy, May 2016.

[7] C. Z. Bai, F. Pasqualetti, and V. Gupta, “Security in stochastic control systems: Fundamental

limitations and performance bounds,” in American Control Conference (ACC), June 2015.

[8] F. Pasqualetti, F. Dorfler, and F. Bullo, “Attack detection and identification in cyber-physical systems,”

IEEE Transactions on Automatic Control, vol. 58, no. 11, pp. 2715–2729, Nov 2013.

[9] A. Teixeira, I. Shames, H. Sandberg, and K. H. Johansson, "A secure control framework for resource-

limited adversaries.” Automatica, vol. 51, pp.135-148, 2015.

[10] Y. Mo and B. Sinopoli, “False data injection attacks in control systems,” in First Workshop on Secure

Control Systems, Stockholm, Sweden, April 2010.

Related Work

[11] Y. Mo, E. Garone, A. Casavola, and B. Sinopoli, “False data injection attacks against state

estimation in wireless sensor networks,” in 49th IEEE Conference on Decision and Control, Atlanta,

Georgia, 2010, pp. 5967–5972.

[12]Y. Mo and B. Sinopoli. “Integrity attacks on cyber-physical systems.” In Proceedings of the 1st

international conference on High Confidence Networked Systems, pp. 47-54. ACM, 2012.

[13] Y. Mo and B. Sinopoli, “Secure control against replay attacks,” in 47th Annual Allerton Conference

on Communication, Control, and Computing, Sept 2009, pp. 911–918.

[14] Y. Mo, R. Chabukswar, and B. Sinopoli, “Detecting integrity attacks on SCADA systems,” IEEE

Transactions on Control System Technology, vol. 22, no. 4, pp. 1396–1407, July 2014.

[15] Y. Mo, S. Weerakkody, and B. Sinopoli, “Physical authentication of control systems: Designing

watermarked control inputs to detect counterfeit sensor outputs,” IEEE Control Systems Magazine, vol.

35, no. 1, pp. 93–109, Feb 2015.

[16] S. Sundaram, M. Pajic, C. Hadjicostis, R. Mangharam, and G. J. Pappas, “The wireless control

network: monitoring for malicious behavior,” in 49th IEEE Conference on Decision and Control, Atlanta,

GA, Dec 2010, pp. 5979-5984.

[17] H. Fawzi, P. Tabuada, and S. Diggavi, “Security for control systems under sensor and actuator

attacks,” in 51st IEEE Conference on Decision and Control, Maui, HI, Dec. 2012, pp. 3412–3417.

[18] ——, “Secure estimation and control for cyber-physical systems under adversarial attacks,” IEEE

Transactions on Automatic Control, vol. 59, no. 6, pp. 1454–1467, June 2014.

[19] S. Mishra, N. Karamchandani, P. Tabuada, and S. Diggavi, “Secure state estimation and control

using multiple (insecure) observers,” in 53rd IEEE Conference on Decision and Control, Los Angeles, CA,

Dec. 2014, pp. 1620–1625.

[20] Q. Zhu and T. Basar, “Robust and resilient control design for cyber-physical systems with an

application to power systems,” in 50th Decision and Control and European Control Conference (CDC-

ECC), Dec. 2011, pp. 4066-4071.

Related Work