Static Analysis versus Software Model Checking for bug finding

Dawson Englers, Madanlal MusuvathiStanford University, CA, USA

Presented By:Prateek Agarwal

ETH, Zürich

Agenda

● Goal● Aim● Scope● Methodologies used

– Meta Compilation

– C Model Checker● Case Studies● Conclusions

Goal

Checking validity of general perception

Static Analysis is easy to apply, finds shallow bugs

Model Checking is harder but strictly better

Aim

● Documentation of experience● Set of case studies● Comparison of two approaches, based on

– Bugs/Lines of Code

– False positives vs Actual Bugs

– Time & Effort required

– Types of Bugs: generic vs protocol specific● Find as many bugs as possible

Scope

● Verification of system/protocol software– Critical

– Hard to test/inspect/reason about manually

– Highly optimized● Protocol codes follow event driven model

Meta Compilation (MC)

● Extension of compiler● Used for Static Analysis & Model Checking● Metal: Language for

– Slice specification by pattern matching

– Code translation

– State machines● xg++ over g++ compiler

Meta Compilation: Key Features

● Code should be highly structured– e.g. special MACRO/functions for read/write

● No source code annotation needed● Emphasis on reducing false positives● No Simulation of code

– No model of heap, tracking variable values

– Simulation tends to become model checking

Static Analysis using MC● Patterns & Actions● Small State machines for checking

Example Rule: Sync before read...start: {WAIT_FOR_DB_FULL(addr);} ==> stop|{MISCBUS_READ_DB(addr);} ==>

{err(“Buffer read not sync");};....

Start

ErrorStop

misc_bus_read_db

wait_for_db_full

Engler & Masuvathi [1]

Model Checking using MC

● Extraction & Translation– protocol code Mur→ Φ Model

● Eliminates handcrafted system models● MurΦ

– Explicit state enumeration model checker

– Each reachable state visited once

Model Checking using MCState Variable(Metal Slicer)

Translation Pattern(Metal Printer)

CorrectnessProperties

Protocol Model

Protocol Code(Implementation)

Start StateHardware Model

Error List

xg++

MurØ

Automatically Generated

Hand Crafted by User

Program

Lie, Chou, Engler, Dill [2]

Legend:

Unedited source code

Example: Model Checking using MCMetal Slicer:.../* Patterns for n/w and proc mesgs, which use length field */pat sends =

{ NI_SEND (type, data, keep,swp, wait,nl);} | { P1_SEND(type, data, keep,swp, wait,nl);} ...Metal Printer...

/* Automatically insert length assertion before send */

{N1_SEND(type, data, keep, swap, wait, null);} ==>{

if(mgk_int_cst(data) != 0)amgk_e(“assert(nh.len = len_data);”);

elsemgk_e(“assert (nh.len = len_nodata);”);

mgk_e(“ni_send(%t, %t, procNum,nh”,type,swap);...

Lie, Chou, Engler, Dill [2]

Model Checking : C Model Checker

● Motivation– xg++ user needs to know system intricacies

– MurΦ lacks many C constructs

– Eliminate need of higher level system model● Use the implementation code itself● Process scheduling & execution by CMC

CMC Key Features● Searches for all transitions● Complete system state saved and restored● Heuristics & Optimizations for state space

explosion● Correctness properties

– Assertions from implementation code

– Memory leaks etc. in built

– User specified invariants● Environment Model still handcrafted

Case Study 1: FLASH

● Cache coherence protocol for FLASH– Multiprocessor architecture

– Code runs on each cache miss● 5 FLASH protocols● Code Size: 10K-18K

John Hennessy [3]

Results

● Static Analysis– Bugs Found: 33, False Positives: 28

– Most rules on buffer management

– e.g. allocation before use, deallocation after use, not used after deallocation

● Model Checking– 8 bugs

– e.g. queue overflows, no invalidation of an exclusive line

– Handwritten model lead to delays

Summary● Static Analysis Advantage:

– Better for same set of properties

– Works best for code visible rules● Model Checking Advantage:

– Can cover complex properties, invariants

– e.g. counter overflow missed by SA● Model Checking Disadvantage:

– Simple models Less Coverage→– Detailed model Increased Time & Effort→– Erroneous model False Positives→

Case Study 2: AODV Protocol● Ad-hoc On-demand Distance Vector Protocol● Routing protocol● Simplified environment

– Input: route request, timer interrupt, packet receipt● 3 publicly available implementations used● Average 6 KLOC● Model Checking using CMC

Results

● Model Checking– 42 unique bugs, 1 bug in AODV specification

● Static Analysis: 34 bugs

Both Model Checking Static Analysis

Generic Properties 21 1 13

Protocol Specific 0 20 0

Total 21 21 13

comparison of bug count classified by properties

Comparison

● Generic Properties:– CMC found only one bug not found by SA

– SA found 13 bugs not found by CMC● Protocol Specific Properties:

– Difficult to check statically 0 bugs→● SA hit more code but CMC hit more properties!

Summary

● Static Analysis Advantage:– Wins when checking same set of properties (again)

● Model Checking Advantage: – Simulates execution of 'entire system'

– Looks for actual errors (not 'causes' of error)

– Gives actual execution trace● More checks more bugs→

– CMC executed the code

Case Study 3: TCP

● TCP Code size ~ 10 * (AODV code size)● Frequently audited, Heavily Tested code● Only Model Checking done (using CMC)● System tightly coupled with Environment

– System: TCP stack

– Environment: Linux Kernel, Kernel Modules● Incremental refinement of model

– Start simple, keep adding complexity

Case Study 3: TCP● External Functions can be included in System● Advantage

– Model Checker will find errors in them

– No effort to create stubs● Disadvantage

– Increase in state space

– Dependencies on other functions● Essential Question

– Where to draw the line between system and environment?

System Separation● Cut on narrowest interface● TCP – Kernel module

– Poorly documented

– Effectively simulating kernel

– Months of effort!

– Difficult to determine false positives● New approach: Use well defined interface

– System calls interface

– Hardware abstraction layer

– Effectively running entire kernel in CMC!

Results

● 4 bugs found● Metric to measure coverage:

– Line Coverage: Lines of code covered

– Protocol Coverage: %age of abstract protocol behavior covered

Description Line Coverage Protocol Coverage Bugs

Standard server, client 47.4 % 64.7 % 2

+ Simultaneous connect 51.0 % 66.7% 0

+ Partial Close 52.7 % 79.5% 2

+ Message corruption 50.6% 84.3% 0

Combined Coverage 55.4% 92.1%

Conclusions

● No model is as good as implementation itself– Simplification , omission missing critical errors→

● Manual work scale badly for complex systems● Use well defined interfaces● More code coverage More bugs found→

My Opinion

● Usage of live software project is a big plus● Meta-Compilation & CMC give huge advantage● Better comparison if same tools used across

case studies● Time & Effort not measured accurately● No comparison with other existing tools

Future Scope

● How to increase code coverage in meta compilation and CMC?

● Analysis of class of properties checked

References

1)Experiences using static analysis & model checking for bug finding – Dawson Engler and Madanlal Masuvathi

2)A simple method for extracting models from protocol code – David Lie, Andy Chou, Dawson Engler, David L. Dill

3)Stanford FLASH Multiprocessor: Status, Some Lessons and Plans – John Hennessy

Q & A