Configuration Coverage in the Analysis ofLarge-Scale System Software

Reinhard Tartler, Daniel Lohmann,Christian Dietrich, Christoph Egger, Julio Sincero

System Software Group

Friedrich-Alexander UniversityErlangen-Nuremberg

October 23, 2011

supported by

Configuration Complexity

Linux has become incredibly configurable

Complexity increases considerably

; Source of bugs!

R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)2 – 10

Configuration Complexity

Linux has become incredibly configurable

Complexity increases considerably

; Source of bugs!

R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)2 – 10

Configuration Complexity

Linux has become incredibly configurable

Complexity increases considerably

; Source of bugs!

R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)2 – 10

Configuration Complexity

Linux has become incredibly configurable

Complexity increases considerably

; Source of bugs!

Linux v3.0 contains:

7.702 Features

893 Kconfig files

31.281 Source files

88.897 #ifdef blocks

R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)2 – 10

Finding Bugs with Tools for Static Analysis

Bugs in declaration and implementation

Excellent tool support for static analysis:

Coccinelle: Faults in Linux: Ten Years Later (ASPLOS’11)

Dingo: Taming Device Drivers (EuroSys’09)

KLEE: Automatic generation of high-coverage tests (EuroSys’08)

RWset: Attacking path explosion (TACAS’08)

EXE: Automatically generating inputs of death (CCS’06)

...

Each of them checks a single configuration:

What does allyesconfig actually cover?

R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)3 – 10

Finding Bugs with Tools for Static Analysis

Bugs in declaration and implementation

Excellent tool support for static analysis:

Coccinelle: Faults in Linux: Ten Years Later (ASPLOS’11)

Dingo: Taming Device Drivers (EuroSys’09)

KLEE: Automatic generation of high-coverage tests (EuroSys’08)

RWset: Attacking path explosion (TACAS’08)

EXE: Automatically generating inputs of death (CCS’06)

...

Each of them checks a single configuration:

What does allyesconfig actually cover?

R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)3 – 10

Finding Bugs with Tools for Static Analysis

Bugs in declaration and implementation

Excellent tool support for static analysis:

Coccinelle: Faults in Linux: Ten Years Later (ASPLOS’11)

Dingo: Taming Device Drivers (EuroSys’09)

KLEE: Automatic generation of high-coverage tests (EuroSys’08)

RWset: Attacking path explosion (TACAS’08)

EXE: Automatically generating inputs of death (CCS’06)

...

Each of them checks a single configuration:

What does allyesconfig actually cover?

R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)3 – 10

Configuration Coverage

We define Configuration Coverage as:

fraction of selected configuration-conditional blocksdivided by the number of available configuration-conditional blocks.

How to catch bugs in obscure configurations?

Use a static analyzer that is able to detect the bug

Test each possible configuration individually

⇒ Not feasible

Static analyzers scan only a particular kernel configuration

⇒ How to effeciently extend their coverage?

R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)4 – 10

Approach

MEMORY MODEL

FLATMEM

DISCONTIGMEM

SPARSEMEM NUMA

depends on

#ifdef CONFIG DISCONTIGMEM

// Block1

static . . . int pfn_to_mid(. . .)

# ifdef CONFIG NUMA

// Block2

# else

// Block3

# endif

#endif

Analysis of both Model and Implementation

Extraction of logical constraints

Find set of configuration that maximizes coverage

R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)5 – 10

Approach

MEMORY MODEL

FLATMEM

DISCONTIGMEM

SPARSEMEM NUMA

depends on

#ifdef CONFIG DISCONTIGMEM

// Block1

static . . . int pfn_to_mid(. . .)

# ifdef CONFIG NUMA

// Block2

# else

// Block3

# endif

#endif

Analysis of both Model and Implementation

Extraction of logical constraints

Find set of configuration that maximizes coverage

R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)5 – 10

Solution Outline

Build Variability Model

Reuse of the Implementation Variability Model [GPCE’10]

Additional constraints from Kconfig [Eurosys’11]

Use SAT solvers to find a “good” inital solution

Repeat until finished:

find a solution that adds additional blocksstore this selection of features as partial configuration

Caveat: Terminate even with dead blocks

Scan the source code with the resulting set of valid configurations

R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)6 – 10

Implementation for Linux

KConfigFiles

config HOTPLUG_CPU bool "Support for ..." depends on SMP && ...

undertaker

Dead BlockDetection

KConfigParser

#ifdef CONFIG_HOTPLUG_CPU...#endif

Linuxsource file

CalculatePartialConfigs

ExpandPartialConfigs

Buildand

Testing

R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)7 – 10

Implementation Challenges

Proper extraction of Configurations constraints

Kconfig (implemented in undertaker)Kbuild constraints (largely unhandled)

Expansion of Partial Configurations (Kconfig Fragments)

Naıve approach has some surprising effects (i.e., fails sometimes)Kconfig-sat seems promising, but unfortunately discontinued

R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)8 – 10

Evaluation with Linux 2.6.35

Analyzed files 10,365Files with variability 3,163Rate of files with variability 30.52%Sum of all (partial) configurations 4,435

Sum of configuration controlled conditional blocks 16,444Sum of blocks selected by allyesconfig 11,511Sum of all blocks selected by undertaker-coverage 13,844Coverage allyesconfig (non-dead-corrected) 70.00%Coverage undertaker (non-dead-corrected) 84.19%

Dead blocks 1,778Selectable blocks (excluding dead blocks) 14,666Selected by allyesconfig 11,511Covered by undertaker 13,844allyesconfig coverage (dead-corrected) 78.49%undertaker coverage (dead-corrected) 94.40%undertaker coverage / allyesconfig coverage 1.20

R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)9 – 10

Evaluation with Linux 2.6.35

Analyzed files 10,365Files with variability 3,163Rate of files with variability 30.52%Sum of all (partial) configurations 4,435Sum of configuration controlled conditional blocks 16,444Sum of blocks selected by allyesconfig 11,511Sum of all blocks selected by undertaker-coverage 13,844Coverage allyesconfig (non-dead-corrected) 70.00%Coverage undertaker (non-dead-corrected) 84.19%

Dead blocks 1,778Selectable blocks (excluding dead blocks) 14,666Selected by allyesconfig 11,511Covered by undertaker 13,844allyesconfig coverage (dead-corrected) 78.49%undertaker coverage (dead-corrected) 94.40%undertaker coverage / allyesconfig coverage 1.20

R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)9 – 10

Evaluation with Linux 2.6.35

Analyzed files 10,365Files with variability 3,163Rate of files with variability 30.52%Sum of all (partial) configurations 4,435Sum of configuration controlled conditional blocks 16,444Sum of blocks selected by allyesconfig 11,511Sum of all blocks selected by undertaker-coverage 13,844Coverage allyesconfig (non-dead-corrected) 70.00%Coverage undertaker (non-dead-corrected) 84.19%Dead blocks 1,778Selectable blocks (excluding dead blocks) 14,666Selected by allyesconfig 11,511Covered by undertaker 13,844allyesconfig coverage (dead-corrected) 78.49%undertaker coverage (dead-corrected) 94.40%undertaker coverage / allyesconfig coverage 1.20

R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)9 – 10

Evaluation with Linux 2.6.35

Analyzed files 10,365Files with variability 3,163Rate of files with variability 30.52%Sum of all (partial) configurations 4,435Sum of configuration controlled conditional blocks 16,444Sum of blocks selected by allyesconfig 11,511Sum of all blocks selected by undertaker-coverage 13,844Coverage allyesconfig (non-dead-corrected) 70.00%Coverage undertaker (non-dead-corrected) 84.19%Dead blocks 1,778Selectable blocks (excluding dead blocks) 14,666Selected by allyesconfig 11,511Covered by undertaker 13,844allyesconfig coverage (dead-corrected) 78.49%undertaker coverage (dead-corrected) 94.40%undertaker coverage / allyesconfig coverage 1.20

R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)9 – 10

Evaluation with Linux 2.6.35

Analyzed files 10,365Files with variability 3,163Rate of files with variability 30.52%Sum of all (partial) configurations 4,435Sum of configuration controlled conditional blocks 16,444Sum of blocks selected by allyesconfig 11,511Sum of all blocks selected by undertaker-coverage 13,844Coverage allyesconfig (non-dead-corrected) 70.00%Coverage undertaker (non-dead-corrected) 84.19%Dead blocks 1,778Selectable blocks (excluding dead blocks) 14,666Selected by allyesconfig 11,511Covered by undertaker 13,844allyesconfig coverage (dead-corrected) 78.49%undertaker coverage (dead-corrected) 94.40%undertaker coverage / allyesconfig coverage 1.20

With 30 percent more static analysis runs(compiler calls)

→ 15 percent more Configuration Coverage

R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)9 – 10

Conclusions

Configurability has to be seen as a significant cause ofsoftware defects in its own respect

The C Preprocessor imposes challenges for existing static analyzers

Configuration Coverage makes existing tools more effective

Further Discussions Points:

How accurate and useful is the current metric?Suggestions for other static analyzers?Analyze non-Linux projects

http://vamos.informatik.uni-erlangen.de/trac/undertaker

R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)10 – 10

Conclusions

Configurability has to be seen as a significant cause ofsoftware defects in its own respect

The C Preprocessor imposes challenges for existing static analyzers

Configuration Coverage makes existing tools more effective

Further Discussions Points:

How accurate and useful is the current metric?Suggestions for other static analyzers?Analyze non-Linux projects

http://vamos.informatik.uni-erlangen.de/trac/undertaker

R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)10 – 10

Conclusions

Configurability has to be seen as a significant cause ofsoftware defects in its own respect

The C Preprocessor imposes challenges for existing static analyzers

Configuration Coverage makes existing tools more effective

Further Discussions Points:

How accurate and useful is the current metric?Suggestions for other static analyzers?Analyze non-Linux projects

http://vamos.informatik.uni-erlangen.de/trac/undertaker

R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)10 – 10