Computer Science
Systematic Testing and Verification of Security Policies
Tao XieDepartment of Computer Science
North Carolina State Universityhttps://sites.google.com/site/asergrp/projects/policy
•Joint Work with Vincent Hu, Rick Khun, and ACTS group (NIST)•JeeHyun Hwang, Evan Martin (NCSU), Alex Liu (MSU)
Computer Science
Motivation
• Digital information is– Easy to access– Easy to search
• Sensitive information requires access control mechanisms
• Security policies are popularly in access control– Access control policies for applications– Firewall policies for networks
Computer Science
Motivation - cont.
• How to ensure the correct specification of security policies?– What you specify is what you get, but not
necessarily what you want
• Solution: systematic testing and verification of security policies
Computer Science
Example Access Control Policy
• Subjects: Student, Faculty• Actions: Assign, Receive• Resources: Grades
Rule 1: IF (faculty AND assign AND grades)Permit
Rule 2: IF (student AND receive AND grades)Permit
Rule 3: OTHERWISEDeny
Computer Science
Policy Verification
• Verify policy against specified property
What properties can you come up for this policy?
Rule 1: IF (faculty AND assign AND grades)Permit
Rule 2: IF (student AND receive AND grades)Permit
Rule 3: OTHERWISEDeny
Computer Science
Policy Verification
Property: student can never assign grades
Rule 1: IF (faculty AND assign AND grades)Permit
Rule 2: IF (student AND receive AND grades)Permit
Rule 3: OTHERWISEDeny
Violated with a counterexample request: faculty|student assign grades
Computer Science
Policy Verification
“when the specification language is sufficiently declarative, users have great difficulty providing a duplicate statement of behavior.” --- Shriram Krishnamurthi [RiseandRise 08]
Rule 1: IF (faculty AND assign AND grades)Permit
Rule 2: IF (student AND receive AND grades)Permit
Rule 3: OTHERWISEDeny
Computer Science
Our Approaches
• Systematic policy verification– Property inference [POLICY 06, SSIRI 09, DBSec 10]
– Property-quality assessment [ACASC 08]– Properties derived from access control models [POLICY 10DE]
• Systematic policy testing– Structural coverage criteria [ICICS 06]
– Fault models/mutation testing [WWW 07]
– Test generation [SESS 07]
– Policy engine performance [SIGMETRICS 08, TC]
– Policy engine correctness [TAV-WEB 08]
– Firewall policy testing/fixing [SRDS 08/09, LISA 10]
•XACML policies
•XACML engines
•Firewall policies
Computer Science
XACML
• A standard access control policy language used to express access control policies– who can do what when
• A request/response language used to express– queries about whether access should be allowed
(requests) and – answers to those queries (responses)
•http://www.oasis-open.org/committees/tc_home.php?wg_abbrev=xacml
Computer Science
XACML Policy Structure
•http://www.oasis-open.org/committees/tc_home.php?wg_abbrev=xacml
policy
target
target
rule2
target
rule1
cond1 cond2
• A Policy Set holds other policies or policy sets.
• A policy is expressed as a set of rules.
• Rules have targets and a set of conditions that determine if the rule applies to a given request.
• Both rule and policy combining algorithms exist to reconcile conflicts.
Computer Science
A Simple Scenario
• A Subject who wishes to perform an Action on a Resource must do so through a PEP.
• The PEP forms the XACML request and sends it to the PDP.• The PDP checks the request against the Policy and returns an
XACML response.• The PEP either Permits or Denies access to the resource.
Computer Science
Software Testing
Expected Outputs
Test Outputs
Test Inputs
Program
Expected ResponsesResponsesRequests Policy
Software Testing
Policy Testing
Policy Testing
Computer Science
Research Problems and Solutions
• Test generation - Request generation– Policy Coverage Criteria– Random request generation– Request generation based on change-impact
analysis– Mutation testing to assess fault-detection
capability
• Test-result inspection - Response inspection– Request selection and minimization based on
structural coverage
Computer Science
Structural Policy Coverage Criteria
policy
target
target
rule2
target
rule1
cond1 cond2
policy covered if target matches
rule2 covered if target matches
Condition must evaluate to True and False to be covered entirely
Computer Science
Random Request Generation
• The example policy:– Subjects: Student, Faculty– Actions: Assign, Receive– Resources: Grades
• Model the set of attribute values as a vector of bits and randomize the bits
Student Faculty Assign Receive Grades
1 0 0 0 00 1 1 0 1
Computer Science
Cirg: Change-Impact Request Generation
policy
version synthesis
policyversions
change-impact analysis
3. requestgeneration
counterexamples
requests
Computer Science
IF (faculty AND assign AND grades)
ELSE IF (student AND receive AND grades)Permit
ELSEDeny
• Counter-example– faculty, assign, grades : Permit Deny
Cirg Example
PermitDeny
Computer Science
Rationale: synthesize two versions whose differences are coverage targets
• All-to-Empty• One-to-Empty• One-Increment• All-to-Minus-One• All-to-Change-One-Effect
Synthesized Versions
Computer Science
Margrave – Change-Impact Analysis Tool
• Faculty (f) can assign (a) grades (g)• Students (s) can receive (r) grades (g)
[Fisler et al. ICSE 05]
Multi-Terminal Decision Diagrams
Computer Science
Margrave Sample Output
1:/Subject, role, Faculty/ 2:/Subject, role, Student/ 3:/Resource, resource-class, ExternalGrades/ 4:/Resource, resource-class, InternalGrades/ 5:/Action, command, Assign/ 6:/Action, command, View/7:/Action, command, Receive/ 8:/Subject, role, TA/12345678{00010101 N->P00011001 N->P00100101 N->P00101001 N->P01010101 N->P01011001 N->P01100101 N->P01101001 N->P}
Computer Science
Software Mutation Testing
Test Outputs
Test Inputs
Program
Mutator
MutantProgram
Mutant Outputs
Differ?Mutant Killed!
Mutation Operators
Computer Science
Policy Mutation Testing
ResponsesRequests Policy
Mutator
MutantPolicy
Mutant Responses
Differ?Mutant Killed!
Mutation Operators
Computer Science
Components of Mutation Testing Framework
ResponsesRequests Policy
Mutator
MutantPolicy
Mutant Responses
Differ?Mutant Killed!
Mutation Operators
Computer Science
Research Questions
• Does test selection based on structural coverage criteria produce request sets with high fault-detection capability?
• What are the individual characteristics of each mutation operator? – Are some more difficult to kill than others? – Are some easily killed by request sets selected
based on structural coverage criteria?
Computer Science
Sample Policies
•continue: 51 policies, 56 rules
Computer Science
# of Requests Generated and Selected
0
5
10
15
20
25
30
35
40
45
50
codeA
codeB
codeC
codeD
conference
default-2
demo-11
demo-26
demo-5
mod-fedora
simple-policy
Nu
mb
er o
f R
equ
ests
Cirg NumReq NumRedReq
•continue: 373 (cirg), 500 (random), 32 (reduction)
Computer Science
Coverage Results
Random Request Set Cirg
Subject PolCov RuleCov CondCov PolCov RuleCov CondCov
codeA 100 100 N/A 100 100 N/A
codeB 100 100 N/A 100 100 N/A
codeC 100 100 N/A 100 100 N/A
codeD 100 100 N/A 100 100 N/A
conference 0 0 N/A 100 100 N/A
default-2 100 92.31 75 100 100 50
demo-11 100 100 75 100 100 50
demo-26 100 100 50 100 100 50
demo-5 100 100 75 100 100 50
mod-fedora 100 84.62 58.33 84.62 84.62 33.33
simple-policy 100 100 N/A 100 100 N/A
•continue: 32% RuleCov (random) vs. 98% RuleCov(cirg)
Computer Science
Mutation Operators, Mutation, and Equivalent Mutant Detection
ResponsesRequests Policy
Mutator
MutantPolicy
Mutant Response
s
Differ?Mutant Killed!
Mutation Operators
Computer Science
Mutation Operators
• Each operator mutates a different policy element: policy set, policy, rule, condition, and/or their associated targets and effects.
Op Description
CPC Change Policy Combining algorithm
CRC Change Rule Combining algorithm
CRE Change Rule Effect
PSTF Policy Set Target False
PSTT Policy Set Target True
PTF Policy Target False
PTT Policy Target True
RCF Rule Condition False
RCT Rule Condition True
RMP Remove Policy
RMPS Remove Policy Set
RMR Remove Rule
RTF Rule Target False
RTT Rule Target True
Computer Science
IF (faculty AND assign AND grades)
ELSE IF (student AND receive AND grades)Permit
ELSEDeny
• The CRE mutation operator is performed on each rule and changes the decision effect (Permit Deny)
Change Rule Effect (CRE) Example
PermitDeny
Computer Science
Equivalent Mutant Detection
• An equivalent mutant is semantically equivalent although syntactically different than the original policy.
• They provide no value and waste resources.
• We use change-impact analysis to detect equivalent mutants and remove them.
Op # Mutants# Equivalent
CPC 1280 64
CRC 2316 32
CRE 968 0
PSTF 256 0
PSTT 256 0
PTF 579 0
PTT 579 0
RCF 670 0
RCT 18 0
RMP 579 0
RMPS 256 0
RMR 670 0
RTF 670 0
RTT 639 0
Computer Science
Request Evaluation and Mutant Detection
ResponsesRequests Policy
Mutator
MutantPolicy
Mutant Response
s
Differ?Mutant Killed!
Mutation Operators
Computer Science
Sun’s XACML implementation
• An open source implementation of the XACML standard in Java
• Developed by Sun as part of an ongoing project on Internet Authorization in the Internet Security Research Group
•http://sunxacml.sourceforge.net/
Computer Science
Mutant-Killing Ratios by Subject
0
10
20
30
40
50
60
70
80
90
100
codeA
codeB
codeC
codeD
confe
rence
defau
lt-2
demo-1
1
demo-2
6
demo-5
mod-fe
dora
sim
ple-p
olicy
Mu
tan
t-K
illin
g R
atio
s
Cirg Random Selected Random
Computer Science
Mutant-Killing Ratio by Operator
0
10
20
30
40
50
60
70
80
90
100
PSTTPSTF
PTTPTF
RTTRTF
RCTRCF
CPCCRC
CRE
RMPS
RMP
RMR
Mu
tan
t-K
illin
g R
atio
s
Cirg Random Selected Random
Computer Science
Our Approaches
• Systematic policy verification– Property inference [POLICY 06, SSIRI 09, DBSec 10]
– Property-quality assessment [ACASC 08]– Properties derived from access control models [POLICY 10DE]
• Systematic policy testing– Structural coverage criteria [ICICS 06]– Fault models/mutation testing [WWW 07]– Test generation [SESS 07]– Policy engine performance [SIGMETRICS 08, TC]
– Policy engine correctness [TAV-WEB 08]
– Firewall policy testing/fixing [SRDS 08/09, LISA 10]
•XACML policies
•XACML engines
•Firewall policies
Computer Science
Firewall Policy Structure• A Policy is expressed as a set of rules.• A Rule is represented as <predicate> → <decision>• <predicate> is a set of <clauses>
• An example firewall policy
•Rule r1’s <predicate>
•Rule r1’s <decision>•A range in each field refers <clause>
Computer Science
Structural Coverage Definition• Rationale: when the policy part with a fault is not
evaluated (i.e., “covered”), the fault is often not exposed.
• Rule coverage of a policy P by packets T = #rules evaluated by at least one packet in T
#rules in P• Predicate coverage of a policy P by packets T
#predicates evaluated to true or false by T at least once2 ×#predicates in P
• Clause coverage of a policy P by packets T #clauses evaluated to true or false by T at least once
2 ×#clauses in P
Computer Science
Test Packet Generation
Our objective: generating packets for achieving high structural coverage
1. Random Packet Generation – Randomly selects values for a packet
2. Packet Generation based on Local Constraint Solving– Considering individual rules in a policy
3. Packet Generation based on Global Constraint Solving– Considering multiple rules in a policy
Computer Science
Experiments (measuring coverage)• Test 14 firewall policies• Generate packets by our proposed three techniques• Measure structural coverage.
Computer Science
Experiments (measuring coverage)
Computer Science
Experiments (measuring fault detection capability)
• We also used reduced packet sets (maintaining the same level of structural coverage with the corresponding original packet set)
Computer Science
NCSU/NIST ACPT Architecture
•GUI
AC Model Templates
DataAcquisition
PolicyGeneratorStatic Verification
DynamicVerification
•Test inputs based•on structural or
•combinatorial coverage
•User,•attribute,•resource,
•role,•etc. data
•GUI allows specification of users, groups, attributes, roles, rules, policies, and resources
Generate enforceable
policies
.xml
Generate and evaluate test inputs
•Verify access control policies
•API/mechanism to consume/acquire external data related to policies
•Generate •test inputs
Test inputs with
their evaluated decisions
•Administrator
XACML
•http://www.nist.gov/itl/csd/set/acpt.cfm
Computer Science44
• Property specification in ACPT
ACPT
Computer Science45
• Verify the property against Policy A, the result return false with counterexample.
Static Verification
Computer Science46
• Verify the property against Policy B, the result return true.
Static Verification (cont.)
Computer Science47
Test Input Generation and Evaluation
Computer Science48
XACML Generation
Computer Science
Conclusion
• Systematic policy verification– Property inference [POLICY 06, SSIRI 09, DBSec 10]
– Property-quality assessment [ACASC 08]– Properties derived from access control models [POLICY 10DE]
• Systematic policy testing– Structural coverage criteria [ICICS 06]
– Fault models/mutation testing [WWW 07]
– Test generation [SESS 07]
– Policy engine performance [SIGMETRICS 08, TC]
– Policy engine correctness [TAV-WEB 08]
– Firewall policy testing/fixing [SRDS 08/09, LISA 10]
•XACML policies
•XACML engines
•Firewall policies
Computer Science
Questions?
https://sites.google.com/site/asergrp/projects/policy