University of Maryland

Bug Driven Bug Finding

Chadd Williams

University of Maryland2

Motivation

Finding bugs in software is important Statically checking code has been effective

– finds complex errors– no need to run the code

Many static checkers available– some with specific bug patterns to find– some allow the user to define the patterns– what kinds of bugs are really out there?

Lots of false positive error reports– can we rank the errors better?– can previous bug history help?

Where to start?– Bug reporting databases– CVS commit messages

University of Maryland3

Bug Database

Inspect fixed bugs– review bug discussions– tie fixed bug to source code change– classify the type of the bug– look for bugs that can be found statically

Users Developers

Bug Database

University of Maryland4

Bug Database: Practical Experience

We inspected the Apache httpd bug database– inspected 200 bug reports marked as fixed– not as helpful as we expected

Only 24% tied directly back to a source code change– bug reports include a discussion of the

problem– rarely is a diff or a CVS revision noted

Most are logic errors/feature requests– not the type found by static checkers

University of Maryland5

Bug Database Bug Types

Most classified bugs are logic errors

NULL pointer check

Return Value check

Logic Errors/Feature Request

Uninitialized Variable Errors

Error Branch Confusion

External Bugs (OS or othersoftware failed)

System specific pattern

University of Maryland6

Bug Database: Practical Experience

Most bug reports originate from users– 197 out of 200– does not capture bugs found by developers

Most bug reports came against a release of the software, not a CVS-HEAD– 198 out of 200– does not capture bugs between releases

What about the bugs that don’t make it into the release?– they may be in the CVS repository…

University of Maryland7

CVS Repository

Commits may contain useful data– any bug fix must show up in a commit– will commit messages lead us to bug fixes?

Shows bugs fixed between releases Bugs caught by developers

– bugs that could be found by static checking

1.1

1.2

1.3CVSRepository

University of Maryland8

CVS Repository: Practical Experience

Inspected commit messages– looked for ‘fix’, ‘bug’ or ‘crash’– ignored those with bug number listed– looked at mature source files

Commit messages are useful– trivially tied to source code change– less logic errors

Common errors found– NULL pointer check– failing to check the return value of a

function before use

University of Maryland9

CVS Repository Bug Types

NULL pointer bugs and return value bugs can be found by static analysis

NULL pointer check

Return Value Check

Feature Request

Uninitialized Variable Errors

Failure to set value of pointerparameter

Error caused by if conditionalshort circuiting

Loop iterator increment error

System specific pattern

University of Maryland10

Return Value Check Bug

Returning error code and valid data from a function is a common C idiom

int foo(){ … if( error ){

return error_code; } …. return data;}

…value = foo();newPosition + = value; // ???

– the return value should be checked before being used

– lint checks for this error

Error types– completely ignored

• foo();– return value used directly

as an argument• bar(foo());

– others …

University of Maryland11

Return Value Checker

Some functions don’t need their return value checked– no error value returned– could lead to many false positives

Naively flagging all unchecked return values leads to many false positives– over 7,000 errors reported for the Apache

httpd-2.0 source Need to determine which are most

likely true errors– use historical data – present this data to the user

University of Maryland12

Which return values need checked?

Infer from historical data– look for an add of a check of a return value in

a CVS commit– implies the programmer thinks it’s important

Infer from current usage– does the return value of a function get

checked in the current version of the software– how often?

…value = foo();newPosition + = value; // ??? …

…value = foo();if( value != Error) { // Check newPosition + = value;}…

Commit Bug Fix

University of Maryland13

Our Tool

Static checker that looks for return value check bugs– built on ROSE by Dan Quinlan, et al.

Classify each error by category– ignored return value– return value used as argument, etc.

Produce a ranking of the errors– group errors by called function– rank most promising errors higher

• rank functions that most likely need their return value checked higher

University of Maryland14

Return Value Checker: Ranking

Rank errors in two ways Split functions into two groups

– functions flagged with a CVS bug fix commit• at least one CVS commit adds a check of

the function’s return value– functions not flagged with CVS bug fix

commit

Within each group:– rank by how often the function’s return

value is checked in the current software distribution

– checked more often means rank higher

University of Maryland15

Case Study

Apache httpd-2.0 on Linux– core system– modules– Apache Runtime Library

Checked all the CVS commits for a return value check bug fix– 6100 commits checked– 2600 commits failed to go through our tool

• wrong (too new) version of autoconf• parser problems• compile bugs in the CVS commits

University of Maryland16

Case Study: Results

Our checker marked over 7,000 errors– individual call site for non-void function

where the return value is not checked

Too many too look at!– expect many are false positives

Rank errors– inspect CVS bug fix commit flagged

functions– inspect functions with return value checked

more than 50% of the time in the current source tree

value = foo(); // ERRORnewPosition + = value; …result = foo(); // ERRORzoo(result);

University of Maryland17

Case Study: Error Breakdown

Inspected 453 errors (of 7,000)– found 98 that may be bugs!

231 errors associated with a CVS bug fix flagged function– 61 of the 98 bugs found here– false positive rate of 74%

222 errors associated with a function that has its return value checked > 50% of the time– 37 of the 98 bugs found here– false positive rate of 83%

University of Maryland18

Case Study: A Bug

We investigated an error and found it did crash httpd– error reported near the top of the ranking

The called function builds a filename– arguments represent file and pathname– a char array is returned and directly used

as an argument to strcmp() – strcmp(foo())– NULL return value will cause a seg fault– return value is NULL if the path is too long!

University of Maryland19

Analysis

False positive rate too high!– overall false positive rate: 78% (1-(98/453))

A false positive rate closer to 50% would be acceptable– the user is likely as not to find a true error– cluster them near the top of the ranking

We did cull 7,000 errors down to 453– lint would have flagged only the ‘ignored’

errors and not ranked them

University of Maryland20

Conclusion

Bug databases are not useful in understanding much about low-level bugs– good for logic errors– good for misunderstood specifications

CVS commit messages give a better picture of low-level bugs– especially bugs that don’t enter a release

CVS commits can give useful data to help classify error reports

University of Maryland21

Future Work

What other types of bugs are common?

What other checkers can benefit from CVS data?

How can we cut the false positive rate?

Can we dynamically gather data on functions called via function pointers?– many of the error messages involved calls

through function pointers– Dyninst will allow us to instrument function

pointer call sites and gather data