1
Security Architecture and Analysis
• Software Inspections and Verification
• Software Testing and Certification
2
The Software System Development Process
Customerrequirements
Usagespecification
Architecture Defn(COTS, legacy), Increment Plng
Design andVerification
Testing and Certification
Functionspecification
Deployment and operation
Incremental Development
Highly Incremental Project team
3
Software Inspections
• Organization– Team-based activity– Specific roles and steps– Training required
• Management view– Proven path to quality improvement– A clear “exit criteria”– Inspect all work products: specifications, architecture,
designs, implementations, ...– Measure and manage results
4
Correctness Verification
• The next step beyond ordinary inspections
– Based on theory, but has practical application
– Can do verbal proofs in “verification inspections”
– Can do written proofs for more rigor
5
Programming Fundamentals
• All sequential programs, no matter what language or subject matter, can be represented in just three basic structures:– Sequence: do-end– Alternation: if-then-else-end – Iteration: while-do-end
• We will explore verification “look and feel” for sequence structures only, using the Trace Table method
6
An Example Sequence Structure
• In text form:
do
x := x + 1
y := y - x
x := x + y
end
• In flowchart form:
x := x - 1 y := y - x x := x + y
7
Verifying Long Division
• Division:
16
232 3712
232
1392
1392
0• Verification:
16 x 232 = 3712
Verification is by a different process than the one that produced the original answer
8
Software Correctness Verification Basics
• Verification is accomplished by comparing:
– What a program is supposed to do to variables -- this is its specification (called its intended function)
– What a program actually does to variables -- this is its net effect from beginning to end (called its program function)
• If the intended function and the program function are identical, the program is correct
9
The Verification Process
Write the IntendedFunction
Write the Program
ProgramFunction
Step 2:Compare forequivalence
Development Verification
(What the programactually does)
(What the programis supposed to do;its specification)
Step 1:Derive
10
Defining Intended Functions
• Concurrent assignment statements can be used to define intended functions
Values of expressions on the right are simultaneously assigned to corresponding variables on the left (old values on the right, new values on the left)
• Example:
[a, b, c := a + 1, a + b, b - c] means:
a := a + 1
b := a + b Assignments are simultaneous
c := b - c
12
Correctness Verification Steps
• Given a program and its intended function:
• Step 1: Derive the program function
– It is the net effect of the program on the variables to which it assigns new values
– “Net effect” mean the change from initial values at the start of the program to final values at the end of the program
• Step 2: Compare the program function to the intended function for equivalence (or not)
13
A Miniature Program -- Is it Correct?
• Intended function (all variables are integers):
[x, y, t := y, x, x]
• Program:
do
t := x
x := y
y := t
end
14
Trace Table Derivation of Program Functions
• Trace Table construction
– Define a row for each assignment, and a column for each variable whose value is changed
– Mechanically fill in the variable changes in each row
• Append initial values in the first row with a “0” suffix
• Increment the suffix with each successive row
• Trace Table Derivation
– Start with last row and mechanically substitute values until you get to the “0” suffixes from first row
– Final expression is the program function
15
Verifying the Miniature Program
Program t x y
1 t := x t1 := x0 x1 = x0 y1 := y0 2 x := y t2 := t1 x2 = y1 y2 := y1 3 y := t t3 := t2 x3 = x2 y3 := t2
Derivations:
t3 = t2 x3 = x2 y3 = t2 = t1 = y1 = t1 = x0 = y0 = x0 Derived program function:
[x3, y3, t3 := y0, x0, x0], or simply [x, y, t := y, x, x]
Therefore, the program computes the intended function’s exchange of the values of x and y, and the setting of t to the value of x.
Trace Table to derive the program function:
16
Is This Program Correct?
• Intended function:
[x, y := y, x]
• Program:
do
x := x + y
y := x - y
x := x - y
end
17
The Trace Table Proof
Program x y
1 x := x + y 2 y := x - y 3 x := x - y
Trace Table:
Derivation:
x3 = y3 =
Derived program function:
18
Is This Program Correct?
• Intended function:
[x, y := y, y + x - 1]
• Program:
do
x := x + 1
y := y - x
x := x + y
end
20
Verifying Branching Logic
[(x > y --> x, y := y, x) | (x <= y --> x, y := x + 1, y - 1)]
if
x > y
then
do [x, y := y, x]
x := x + y
y := x - y
x := x - y
end
else
do [x, y := x + 1, y - 1]
x := x + 1
y := x - 1
end
endif
21
Verification Perspective
• Other sources of errors– Human fallibility in verification – Compiler errors– Support software errors– Hardware errors
• Verification, combined with other rigorous software engineering practices, can produce software of extremely high quality
22
Testing and Certification
• You are the director of quality control for a light bulb factory
• 100,000 light bulbs come off the assembly line every hour:
– 50,000 60w bulbs
– 30,000 75w bulbs
– 15,000 100w bulbs
– 5,000 250w bulbs
• (This is also the distribution of customer purchases)
23
Define a Testing Process for Quality Control
• How will you predict customer quality experience with the light bulbs?
24
A Light Bulb Testing Process
• Sample the population of bulbs according to customer usage, then the quality of the sample will predict customer quality experience with the bulbs in the population that were not tested
• The sample composition should be:– 60W: 50%– 75w: 30%– 100w: 15%– 250w: 5%
• Test the sample to determine MTTF• Compare results to quality objectives
25
Software Testing Realities
• A software system is capable of a virtually infinite number of possible executions (each is a potential test case)
• No testing process, no matter how thorough, can exercise more than a minute fraction of these possible executions
• All testing is sampling; the only question is how to define the sample
26
Software Testing Methods -- 1
• Test to debug the code (40 years of widespread use)
– Objective of most testing today, but can result in buggy systems
• Testing can show the presence of bugs, but never their absence
– Often called coverage testing
– Test cases are often invented based on knowledge of code internals (this is the sample)
– Produces only anecdotal evidence of quality (“The code ran these 100 tests ok”)
– Not useful for predictions about all the executions not tested
27
Software Testing Methods -- 2
• Test to certify the code (10 years of minimal use)– Objective is not to debug, but to certify quality– Called statistical, usage-based testing – Test cases are based on the external usage that
actual users engage in– If sample the population of executions according to
usage, the results can predict user experience with all the possible executions that were not tested.
– Produces scientific measures of code quality
28
Statistical Usage-Based Testing
Develop test cases for thesample
Define a statistical sample of the population based on how the systemwill be used
Execute thetest cases, compute MTTF
Population ofpossibleexecutions
Compare toquality goals
Improve the production process as necessary
29
Testing According to Usage
• Example: Usage distribution for a banking system defines the test case distribution
– Account updates: 60% of usage and tests
– Fund transfers: 25%
– Statement generation: 10%
– Audit checks: 5%