Testing

Kurt Schmidt

Intro

TestingAssertions

Debugging

Testing

Kurt Schmidt

Dept. of Computer Science, Drexel University

December 5, 2018

Testing

Kurt Schmidt

Intro

TestingAssertions

Debugging

A QA engineer walks into a bar:A QA engineer runs into a bar:A QA engineer skips into a bar:

She orders 1 beerShe orders -1 beersHe orders 99,999 beersIt orders sfojbeerkiwHe orders ’0 ; select id, password from users ; dropbeers ; commit’

Opening day a customer walks in, asks for the bathroom.The bar catches fire, everybody dies.

Testing

Kurt Schmidt

Intro

TestingAssertions

Debugging

Intro

Testing

Kurt Schmidt

Intro

TestingAssertions

Debugging

Testing and Debugging

Testing “...a determined, systematic attempt to break aprogram that you think is working.”

Running a program with the intent offinding bugs“...testing can demonstrate the presenceof bugs, but not their absence.”

Debugging Finding the cause of an error, and fixing it.

Testing

Kurt Schmidt

Intro

TestingAssertions

Debugging

Goals of Testing

Discover bugs, not to show that program worksReduce the risk of failure to an acceptable levelDesigning a test before writing code is a great way toreduce bugs

Testing

Kurt Schmidt

Intro

TestingAssertions

Debugging

Complete Testing

Complete testing coverage is generally not possibleOr, not practical

E.g., consider a program that takes an input of 10characters

280 distinct inputsAt 1µs/test, would take more than twice the age of theuniverse

Testing

Kurt Schmidt

Intro

TestingAssertions

Debugging

Testing Caveats

People naturally assume what they do is correctPeople by nature overlook minor deficiencies in theirown workEasy to overlook or ignore bad resultsEasy to choose only test cases that show the program“works”It’s useful to get another’s help

Testing

Kurt Schmidt

Intro

TestingAssertions

Debugging

Software Testing Myths

Really good programmers don’t have bugsTesting implies an admission of failureTesting is a punishment for our errorsTesting can be avoided if we

ConcentrateUse OO methodsUse a good programming language

Testing

Kurt Schmidt

Intro

TestingAssertions

Debugging

Software Testing Reality

Humans make mistakesEspecially when creating complex artifacts

Even good programs have 1-3 bugs per 100 lines ofcodePeople who claim they write bug-free code likelyhaven’t coded much

Testing

Kurt Schmidt

Intro

TestingAssertions

Debugging

Defensive Programming

Anticipate potential problemsDesign, code the system so problems are accountedfor, or detected as early as possible

Defensive design – Minimise confusion due to complexityDefensive coding – Take steps to localise problems

Testing

Kurt Schmidt

Intro

TestingAssertions

Debugging

Defensive Design

Simplicity of designEncapsulationDesign with error in mindPrototype, walk-throughMake all assumptions and conditions explicitly

Testing

Kurt Schmidt

Intro

TestingAssertions

Debugging

Encapsulation

Minimise coupling (dependencies) between objects1

Provide a sufficient interfaceHide all data behind the interfaceHave enough functionality so that the client needn’taccess the data directly

1Loosely; i.e., a collection of related data, not necessarily an instanceof a class

Testing

Kurt Schmidt

Intro

TestingAssertions

Debugging

Designing With Error in Mind

Error handling is often added as an afterthoughtShould be part of the interface

Decided before/as you code

Ask “What if?” often

Testing

Kurt Schmidt

Intro

TestingAssertions

Debugging

Design Reviews

Show the design to another programmer, or leadDiscussions and critiques are an excellent way to learnIt is much easier to see another’s mistakes, andassumptions

Another pair of eyeballs is always helpful

Testing

Kurt Schmidt

Intro

TestingAssertions

Debugging

Pre- and Post-Conditions

Pre-conditions Assumptions that are made by a function ormethod upon entry, of necessity or efficiency

Units or state of argumentsState of the objectState of globals

Post-condition Anything a function guarantees upon exit (ifpre-conditions were met

Return valueAny side-effects

Testing

Kurt Schmidt

Intro

TestingAssertions

Debugging

Class Invariants

Conditions on an object which are always trueFor a Vector V of length n, elements V0 through Vn−1are validA Fraction might always be in lowest terms

Conditions public (interface) functions can assumeupon entryMust guarantee are true upon exit

Note, this does not necessarily apply to helper functions

Testing

Kurt Schmidt

Intro

TestingAssertions

Debugging

Evolutionary Programming

Compile and test your code oftenImplementation is bottom-up

Write small piecesTest themGather into bigger pieces

Evolve (and test) your program by writing modules,using stubs

Testing

Kurt Schmidt

Intro

TestingAssertions

Debugging

Testing

Testing

Kurt Schmidt

Intro

TestingAssertions

Debugging

General Testing Guidelines

A necessary part of a test case is the expectedoutput/behaviorTest border conditionsTest error conditionsCheck that the program doesn’t do what it shouldn’t(harder to impossible)Never be the last to test your own codeTesting is an art; your skills will develop

Testing

Kurt Schmidt

Intro

TestingAssertions

Debugging

Functional vs. Structural Testing

Functional (black box) testing:Implementation details are invisibleCode is tested against its specs – look at advertisedbehavior

Inputs, outputsSide effectsError codes and exceptions

Cases can be written before the code is writtenStructural (clear box) testing:

Tester must be familiar with the codeExercise all control paths, loop boundariesTools such as gcov might be helpful

Testing

Kurt Schmidt

Intro

TestingAssertions

Debugging

Integration, Unit, System, and AcceptanceTesting

These labels often get in the way of talking abouttesting1

I don’t want to confuse this discussion this term“Black-box” does not imply “unit” testing“Clear-box” does not imply “integration” testing

1Mike Kelly, searchsoftwarequality.techtarget.com/

Testing

Kurt Schmidt

Intro

TestingAssertions

Debugging

The assert statement

Used during development, testing and debugging totest programmer’s assumptions

Not to be used in released code

#include <assert.h>...assert( x!=0 ) ;

If condition failsPrint conditionPrint a line numberDump the program

To turn it off:

$ gcc -DNDEBUG ...

Testing

Kurt Schmidt

Intro

TestingAssertions

Debugging

Debugging

Testing

Kurt Schmidt

Intro

TestingAssertions

Debugging

Types of Bugs

SyntacticDesignLogic

InterfaceMemory

Testing

Kurt Schmidt

Intro

TestingAssertions

Debugging

How to Find a Problem

ThinkI’ll spend a few minutes with print/debug statementsI’ll move to a debugger before too longIf you reach an impasse, sleep on itIf you reach an impasse, show it to someone else

Testing

Kurt Schmidt

Intro

TestingAssertions

Debugging

Why Use a Debugger?

Debuggers are very powerfulDesk checking code can be tedious and error-pronePrint/Debug statements might require re-compilation

Error-proneLeaves a mess to clean up

Testing

Kurt Schmidt

Intro

TestingAssertions

Debugging

Basic Common Debugger Functions

BreakpointsSet at line or functionCan be conditional

Run programExecute line at a timeRun to next breakpointMove execution pointer aroundSet command-line argumentsSee stack trace

DataEvaluate any variable, expressionModify values in memory

Testing

Kurt Schmidt

Intro

TestingAssertions

Debugging

Think Before Repairing Errors

Be careful you fix the problemDon’t simply address the symptom

Remember, don’t fix bad code, rewrite it!