Scientific Debugging

Errors in SoftwareErrors are unexpected behaviors or outputs in programs

As long as software is developed by humans, it will contain potential errors

Typical methods of debugging

Typically our approach to debugging is very scattered, and we learn primarily from experience. The more bugs we encounter, the more quickly we can debug.

For most programmers, sourcing and systematically solving problems is challenging

Debugging intuition is not easy for many developers, and can’t be relied on

Our approach to errors

We can choose to see errors as independent frustrations

We can assume everyone has the same intuition regarding

debugging, but this proves unrealistic

A smarter technique is approaching all errors from a scientific

perspective

Scientists approach problems with the scientific method

Programmers are capable of approaching errors with a similar

approach

To do this we have to consider errors a natural phenomena

Scientific Debugging method - simplified approach

1.Observe a failure

2.Create a hypothesis for cause of failure based on observations

3.Use hypothesis to make predictions

4.If the experiment satisfies the predictions, refine or limit the hypothesis

5.If the experiment does not satisfy predictions, create a new hypothesis

6.Repeat 3,4,5 as needed until further refining is not possible

Eventually we will arrive at a cause for the software failure

What is a theory?

In informal speech, the word theory is sometimes confused with a “good guess”

In a scientific context, a theory

is a rigorously tested hypothesis

attempts to explain past observations

makes testable claims about future observations

In scientific debugging, theories are proven causes of software failure

Shell Sort revisited

Shell Sort debugging example - 1

For shell sort, we will note the following:

Hypothesis - The Sample program works

Prediction - Attempting to sort the collection “11 14” will result in “11 14”

Experiment - Run sample on given input

Observation - the rejected output is “0 11”

Conclusion - rejected hypothesis

Shell Sort debugging example - 2For our new hypothesis, let’s try restricting to a simpler case.

Hypothesis - our array start index, a[0] will stay 0

Prediction - If we insert a breakpoint in our debugger at 37, a[0] is 0.

Experiment - Run sample with a[0] as 0

Observation - the correct output is 0

Conclusion - supported hypothesis

Continuing down this trend, we can observe more stringent hypotheses and reach our final theory

Maintaining this approach of systematic checking will ensure consistent debugging

Explicit DebuggingScientific Debugging tests hypotheses which are explicitly stated

Explicit in this case means problems are well defined and documented

When debugging, developers often keep error notes in their mind of the problems

This is an enormous mistake because

we are human, and humans are forgetful

much harder to see patterns emerge

coworkers can’t help you nearly well enough

Organizing thoughts in and of itself is useful

It becomes vital once we have multi-layered issues

“Writing bugs” only useful when organized

Explicit Debugging One solution: keep a logbook

Useful because:

we avoid relying on own memory

there is a permanent record of bug solutions, in case similar bugs occur in other software

it’s easier finding related errors (multiple errors with a single fix)

Algorithmic DebuggingTry to automate the process of debugging

Walk through execution of a program

Observe current state and values

Follow valid input until it “breaks”

Stop when we transition from valid -> invalid

Can be thought of as a “tree” of execution states

Insertion Sort Example - Algorithmic

Example: Algorithmic Debugging

Sort([2,1,3]) == [3,1,2]? no → first check left execution path

Example: Algorithmic Debugging

Sort([3]) == [3] ? yes → mark OK, check other path

Example: Algorithmic Debugging

Insert(1,[3]) != [3,1] → algorithmic debugger marks invalid

Scientific Debugging - Pros and ConsScientific Debugging is very helpful when:

the software is designed as small modular parts for easier separate testing

our program performs a linear procedure of instructions

it is easy to tell the difference between “unfinished” and “bad” answers

our output is easy to parse and test

mission critical software needs testing

Not as helpful when:

“unfinished” and “bad” output are very similar, and confuse our debugging method

we have very convoluted data structures

we perform several datatype conversions which break many types of testing

we combine multiple languages in a single piece of software

Obtaining a Hypothesis - Understanding the Problem

In order to use scientific debugging, we have to arrive at meaningful hypotheses

There are a handful of tools to do this to help narrow our effective results

There are several key ingredients of debugging:

Hypothesis Formation - Problem DescriptionA problem you cannot clearly describe is a problem you

cannot clearly solve

“Shell sort doesn’t work on input [2,1,3]” is not a useful description

Observing that shell sort fails once we call insert() is useful

Problem reports and problem tracking are vital to narrowing problem scope

Hypothesis Formation - Program Code

It is intuitive that programmers should know their code in order to debug it

However this becomes very difficult for large projects reliant on others’ work

Important to keep in mind when relying on libraries and external projects as well

For our shell sort example, would have been impossible to see insert() problem

Hypothesis Formation - Earlier HypothesesOne other crucial rule to keep in mind is

hypothesis refinement

When we update a hypothesis:

Include earlier hypotheses that passed

Exclude earlier hypotheses that failed

Future hypotheses must also keep track of prior observations

Reasoning about Programs - using DeductionDeduction is reasoning that moves from the general to the particular

These take the form of mathematical proofs

If these abstractions of our program are true, so are properties we can deduce

This technique is useful because it doesn’t require running the program

If we don’t run the program, it is static analysis

If we use deduction but also test against execution, it is a dynamic analysis

In general

static attempts to predict the future

dynamic holds onto values from the past

It’s also difficult because abstracting our program can be challenging

Reasoning about Programs - using ObservationObservation is a dynamic method

dynamic: requires execution of program for testing

take in facts from execution

majority of these methods also look at program code so they usually include deduction

There are multiple different kinds of observational methods including:

Inductive methods

Experimental methods

Deductive methods (if using dynamic deduction)

All use information about the past to use for debugging solutions

Reasoning about Programs: using Induction Reasoning from particular to general (unlike deduction)

Attempt to summarize multiple runs through formal inductive proof

1. Start with base case for program input (for shell sort, sort a single element)

2. Attempt to show that if it works for n inputs,

3. It will also work for n+1 inputs

This requires multiple executions to test, so induction is also dynamic

All inductive methods are also observational

because they are dependent on observing output of multiple runs

Reasoning about Programs - using Experimentation

Searching for a failure through experiments

constantly refining and updating a hypothesis

this reduces our hypotheses to a theory

Based on multiple program runs, so also dynamic

each run is guided by a specific experiment to narrow the hypotheses

In general, experimentation techniques are so named if:

they generate findings from multiple runs

these runs are controlled by some hypothesis or test