+ All Categories
Home > Documents > Static Analysis versus Software Model Checking for bug...

Static Analysis versus Software Model Checking for bug...

Date post: 22-May-2020
Category:
Upload: others
View: 24 times
Download: 0 times
Share this document with a friend
29
Static Analysis versus Software Model Checking for bug finding Dawson Englers, Madanlal Musuvathi Stanford University, CA, USA Presented By: Prateek Agarwal ETH, Zürich
Transcript
Page 1: Static Analysis versus Software Model Checking for bug findingse.inf.ethz.ch/old/teaching/2009-S/0276/slides/agarwal.pdf · Static Analysis versus Software Model Checking for bug

Static Analysis versus Software Model Checking for bug finding

Dawson Englers, Madanlal MusuvathiStanford University, CA, USA

Presented By:Prateek Agarwal

ETH, Zürich

Page 2: Static Analysis versus Software Model Checking for bug findingse.inf.ethz.ch/old/teaching/2009-S/0276/slides/agarwal.pdf · Static Analysis versus Software Model Checking for bug

Agenda

● Goal● Aim● Scope● Methodologies used

– Meta Compilation

– C Model Checker● Case Studies● Conclusions

Page 3: Static Analysis versus Software Model Checking for bug findingse.inf.ethz.ch/old/teaching/2009-S/0276/slides/agarwal.pdf · Static Analysis versus Software Model Checking for bug

Goal

Checking validity of general perception

Static Analysis is easy to apply, finds shallow bugs

Model Checking is harder but strictly better

Page 4: Static Analysis versus Software Model Checking for bug findingse.inf.ethz.ch/old/teaching/2009-S/0276/slides/agarwal.pdf · Static Analysis versus Software Model Checking for bug

Aim

● Documentation of experience● Set of case studies● Comparison of two approaches, based on

– Bugs/Lines of Code

– False positives vs Actual Bugs

– Time & Effort required

– Types of Bugs: generic vs protocol specific● Find as many bugs as possible

Page 5: Static Analysis versus Software Model Checking for bug findingse.inf.ethz.ch/old/teaching/2009-S/0276/slides/agarwal.pdf · Static Analysis versus Software Model Checking for bug

Scope

● Verification of system/protocol software– Critical

– Hard to test/inspect/reason about manually

– Highly optimized● Protocol codes follow event driven model

Page 6: Static Analysis versus Software Model Checking for bug findingse.inf.ethz.ch/old/teaching/2009-S/0276/slides/agarwal.pdf · Static Analysis versus Software Model Checking for bug

Meta Compilation (MC)

● Extension of compiler● Used for Static Analysis & Model Checking● Metal: Language for

– Slice specification by pattern matching

– Code translation

– State machines● xg++ over g++ compiler

Page 7: Static Analysis versus Software Model Checking for bug findingse.inf.ethz.ch/old/teaching/2009-S/0276/slides/agarwal.pdf · Static Analysis versus Software Model Checking for bug

Meta Compilation: Key Features

● Code should be highly structured– e.g. special MACRO/functions for read/write

● No source code annotation needed● Emphasis on reducing false positives● No Simulation of code

– No model of heap, tracking variable values

– Simulation tends to become model checking

Page 8: Static Analysis versus Software Model Checking for bug findingse.inf.ethz.ch/old/teaching/2009-S/0276/slides/agarwal.pdf · Static Analysis versus Software Model Checking for bug

Static Analysis using MC● Patterns & Actions● Small State machines for checking

Example Rule: Sync before read...start: {WAIT_FOR_DB_FULL(addr);} ==> stop|{MISCBUS_READ_DB(addr);} ==>

{err(“Buffer read not sync");};....

Start

ErrorStop

misc_bus_read_db

wait_for_db_full

Engler & Masuvathi [1]

Page 9: Static Analysis versus Software Model Checking for bug findingse.inf.ethz.ch/old/teaching/2009-S/0276/slides/agarwal.pdf · Static Analysis versus Software Model Checking for bug

Model Checking using MC

● Extraction & Translation– protocol code Mur→ Φ Model

● Eliminates handcrafted system models● MurΦ

– Explicit state enumeration model checker

– Each reachable state visited once

Page 10: Static Analysis versus Software Model Checking for bug findingse.inf.ethz.ch/old/teaching/2009-S/0276/slides/agarwal.pdf · Static Analysis versus Software Model Checking for bug

Model Checking using MCState Variable(Metal Slicer)

Translation Pattern(Metal Printer)

CorrectnessProperties

Protocol Model

Protocol Code(Implementation)

Start StateHardware Model

Error List

xg++

MurØ

Automatically Generated

Hand Crafted by User

Program

Lie, Chou, Engler, Dill [2]

Legend:

Unedited source code

Page 11: Static Analysis versus Software Model Checking for bug findingse.inf.ethz.ch/old/teaching/2009-S/0276/slides/agarwal.pdf · Static Analysis versus Software Model Checking for bug

Example: Model Checking using MCMetal Slicer:.../* Patterns for n/w and proc mesgs, which use length field */pat sends =

{ NI_SEND (type, data, keep,swp, wait,nl);} | { P1_SEND(type, data, keep,swp, wait,nl);} ...Metal Printer...

/* Automatically insert length assertion before send */

{N1_SEND(type, data, keep, swap, wait, null);} ==>{

if(mgk_int_cst(data) != 0)amgk_e(“assert(nh.len = len_data);”);

elsemgk_e(“assert (nh.len = len_nodata);”);

mgk_e(“ni_send(%t, %t, procNum,nh”,type,swap);...

Lie, Chou, Engler, Dill [2]

Page 12: Static Analysis versus Software Model Checking for bug findingse.inf.ethz.ch/old/teaching/2009-S/0276/slides/agarwal.pdf · Static Analysis versus Software Model Checking for bug

Model Checking : C Model Checker

● Motivation– xg++ user needs to know system intricacies

– MurΦ lacks many C constructs

– Eliminate need of higher level system model● Use the implementation code itself● Process scheduling & execution by CMC

Page 13: Static Analysis versus Software Model Checking for bug findingse.inf.ethz.ch/old/teaching/2009-S/0276/slides/agarwal.pdf · Static Analysis versus Software Model Checking for bug

CMC Key Features● Searches for all transitions● Complete system state saved and restored● Heuristics & Optimizations for state space

explosion● Correctness properties

– Assertions from implementation code

– Memory leaks etc. in built

– User specified invariants● Environment Model still handcrafted

Page 14: Static Analysis versus Software Model Checking for bug findingse.inf.ethz.ch/old/teaching/2009-S/0276/slides/agarwal.pdf · Static Analysis versus Software Model Checking for bug

Case Study 1: FLASH

● Cache coherence protocol for FLASH– Multiprocessor architecture

– Code runs on each cache miss● 5 FLASH protocols● Code Size: 10K-18K

John Hennessy [3]

Page 15: Static Analysis versus Software Model Checking for bug findingse.inf.ethz.ch/old/teaching/2009-S/0276/slides/agarwal.pdf · Static Analysis versus Software Model Checking for bug

Results

● Static Analysis– Bugs Found: 33, False Positives: 28

– Most rules on buffer management

– e.g. allocation before use, deallocation after use, not used after deallocation

● Model Checking– 8 bugs

– e.g. queue overflows, no invalidation of an exclusive line

– Handwritten model lead to delays

Page 16: Static Analysis versus Software Model Checking for bug findingse.inf.ethz.ch/old/teaching/2009-S/0276/slides/agarwal.pdf · Static Analysis versus Software Model Checking for bug

Summary● Static Analysis Advantage:

– Better for same set of properties

– Works best for code visible rules● Model Checking Advantage:

– Can cover complex properties, invariants

– e.g. counter overflow missed by SA● Model Checking Disadvantage:

– Simple models Less Coverage→– Detailed model Increased Time & Effort→– Erroneous model False Positives→

Page 17: Static Analysis versus Software Model Checking for bug findingse.inf.ethz.ch/old/teaching/2009-S/0276/slides/agarwal.pdf · Static Analysis versus Software Model Checking for bug

Case Study 2: AODV Protocol● Ad-hoc On-demand Distance Vector Protocol● Routing protocol● Simplified environment

– Input: route request, timer interrupt, packet receipt● 3 publicly available implementations used● Average 6 KLOC● Model Checking using CMC

Page 18: Static Analysis versus Software Model Checking for bug findingse.inf.ethz.ch/old/teaching/2009-S/0276/slides/agarwal.pdf · Static Analysis versus Software Model Checking for bug

Results

● Model Checking– 42 unique bugs, 1 bug in AODV specification

● Static Analysis: 34 bugs

Both Model Checking Static Analysis

Generic Properties 21 1 13

Protocol Specific 0 20 0

Total 21 21 13

comparison of bug count classified by properties

Page 19: Static Analysis versus Software Model Checking for bug findingse.inf.ethz.ch/old/teaching/2009-S/0276/slides/agarwal.pdf · Static Analysis versus Software Model Checking for bug

Comparison

● Generic Properties:– CMC found only one bug not found by SA

– SA found 13 bugs not found by CMC● Protocol Specific Properties:

– Difficult to check statically 0 bugs→● SA hit more code but CMC hit more properties!

Page 20: Static Analysis versus Software Model Checking for bug findingse.inf.ethz.ch/old/teaching/2009-S/0276/slides/agarwal.pdf · Static Analysis versus Software Model Checking for bug

Summary

● Static Analysis Advantage:– Wins when checking same set of properties (again)

● Model Checking Advantage: – Simulates execution of 'entire system'

– Looks for actual errors (not 'causes' of error)

– Gives actual execution trace● More checks more bugs→

– CMC executed the code

Page 21: Static Analysis versus Software Model Checking for bug findingse.inf.ethz.ch/old/teaching/2009-S/0276/slides/agarwal.pdf · Static Analysis versus Software Model Checking for bug

Case Study 3: TCP

● TCP Code size ~ 10 * (AODV code size)● Frequently audited, Heavily Tested code● Only Model Checking done (using CMC)● System tightly coupled with Environment

– System: TCP stack

– Environment: Linux Kernel, Kernel Modules● Incremental refinement of model

– Start simple, keep adding complexity

Page 22: Static Analysis versus Software Model Checking for bug findingse.inf.ethz.ch/old/teaching/2009-S/0276/slides/agarwal.pdf · Static Analysis versus Software Model Checking for bug

Case Study 3: TCP● External Functions can be included in System● Advantage

– Model Checker will find errors in them

– No effort to create stubs● Disadvantage

– Increase in state space

– Dependencies on other functions● Essential Question

– Where to draw the line between system and environment?

Page 23: Static Analysis versus Software Model Checking for bug findingse.inf.ethz.ch/old/teaching/2009-S/0276/slides/agarwal.pdf · Static Analysis versus Software Model Checking for bug

System Separation● Cut on narrowest interface● TCP – Kernel module

– Poorly documented

– Effectively simulating kernel

– Months of effort!

– Difficult to determine false positives● New approach: Use well defined interface

– System calls interface

– Hardware abstraction layer

– Effectively running entire kernel in CMC!

Page 24: Static Analysis versus Software Model Checking for bug findingse.inf.ethz.ch/old/teaching/2009-S/0276/slides/agarwal.pdf · Static Analysis versus Software Model Checking for bug

Results

● 4 bugs found● Metric to measure coverage:

– Line Coverage: Lines of code covered

– Protocol Coverage: %age of abstract protocol behavior covered

Description Line Coverage Protocol Coverage Bugs

Standard server, client 47.4 % 64.7 % 2

+ Simultaneous connect 51.0 % 66.7% 0

+ Partial Close 52.7 % 79.5% 2

+ Message corruption 50.6% 84.3% 0

Combined Coverage 55.4% 92.1%

Page 25: Static Analysis versus Software Model Checking for bug findingse.inf.ethz.ch/old/teaching/2009-S/0276/slides/agarwal.pdf · Static Analysis versus Software Model Checking for bug

Conclusions

● No model is as good as implementation itself– Simplification , omission missing critical errors→

● Manual work scale badly for complex systems● Use well defined interfaces● More code coverage More bugs found→

Page 26: Static Analysis versus Software Model Checking for bug findingse.inf.ethz.ch/old/teaching/2009-S/0276/slides/agarwal.pdf · Static Analysis versus Software Model Checking for bug

My Opinion

● Usage of live software project is a big plus● Meta-Compilation & CMC give huge advantage● Better comparison if same tools used across

case studies● Time & Effort not measured accurately● No comparison with other existing tools

Page 27: Static Analysis versus Software Model Checking for bug findingse.inf.ethz.ch/old/teaching/2009-S/0276/slides/agarwal.pdf · Static Analysis versus Software Model Checking for bug

Future Scope

● How to increase code coverage in meta compilation and CMC?

● Analysis of class of properties checked

Page 28: Static Analysis versus Software Model Checking for bug findingse.inf.ethz.ch/old/teaching/2009-S/0276/slides/agarwal.pdf · Static Analysis versus Software Model Checking for bug

References

1)Experiences using static analysis & model checking for bug finding – Dawson Engler and Madanlal Masuvathi

2)A simple method for extracting models from protocol code – David Lie, Andy Chou, Dawson Engler, David L. Dill

3)Stanford FLASH Multiprocessor: Status, Some Lessons and Plans – John Hennessy

Page 29: Static Analysis versus Software Model Checking for bug findingse.inf.ethz.ch/old/teaching/2009-S/0276/slides/agarwal.pdf · Static Analysis versus Software Model Checking for bug

Q & A


Recommended