cs498dmSoftware Testing

Darko Marinov

January 22, 2008

Course Overview

• Introduction to software testing– Systematic, organized approaches to testing– Based on models and coverage criteria– Testing is not (only) about finding “bugs”– Improve your testing (and development) skills

• Teaching staff– Insructor: Darko <marinov@cs.uiuc.edu>– TA: Vilas <vbangal2@cs.uiuc.edu>

Administrative Info

• FAQ: Deliverables?– No exams: no final, no midterm– Five problem sets (5*15%) and a project (25%)

• Project: proposal, two reports, hopefully bug reports• Project is on testing a piece of refactoring engine

• Undergrads must be registered for 3 hours– For more, you can do an independent study– If interested in research on testing, contact me

• Juniors: ask for permission if you didn’t

“Assignment”

• Did you sign up on Wikihttps://agora.cs.uiuc.edu/display/cs498dmsp08/Home

• Did you receive emails on the listcs498dm@cs.uiuc.edu

• Did you try out Eclipse/NetBeans?– Which refactorings did you try?

“Prizes”

• Potential categories– The “buggiest” bug found in the course

• Hardest to find, most important, realistic case study

– Most bugs found or tests generated• Not the best measures of testing effort

• Occasional candies to encourage your participation and discussion in lectures– Used to be a part of grade but not any more

Several Warm-Up Topics

• Your opinion about the guest lecture?– Do you want to have more of them?

• My report from the Workshop on Teaching Software Testing (WTST 7)– No technique is perfect– Abstraction/modeling is important

• Think of various ways that software could break

– Testing for testers vs. testing for developers• Do you look at source code while testing?

This Lecture: Introduction (Cont’d)

• Why look for bugs?

• What are bugs?

• Where they come from?

• How to detect them?

• Topics that will be covered in the course

• Related topics that will not be covered

“Bugs” in IEEE 610.12-1990

• Fault– Incorrect lines of code

• Error– Faults cause incorrect (unobserved) state

• Failure– Errors cause incorrect (observed) behavior

• Not used consistently in literature!

Correctness and Quality

• Common (partial) properties– Segfaults, uncaught exceptions– Resource leaks– Data races, deadlocks– Statistics based

• Specific properties– Requirements– Specification

Traditional Waterfall ModelRequirements

Analysis

DesignChecking

ImplementationUnit Testing

IntegrationSystem Testing

MaintenanceRegression Testing

We will look at general techniques, applicable in several phases of testing

Phases (1)

• Requirements– Specify what the software should do– Analysis: eliminate/reduce ambiguities,

inconsistencies, and incompleteness

• Design– Specify how the software should work– Split software into modules, write specifications– Checking: check conformance to requirements,

using for example conformance testing

Phases (2)

• Implementation– Specify how the modules work– Unit testing: test each module in isolation

• Integration– Specify how the modules interact– Integration testing: test module interactions– System testing: test the entire system

• Maintenance– Evolve software as requirements change– Regression testing: test changes

Testing Effort

• Reported to be >50% of development cost [e.g., Beizer 1990]

• Microsoft: 75% time spent testing– 50% testers who spend all time testing– 50% developers who spend half time testing

When to Test

• The later a bug is found, the higher the cost– Orders of magnitude increase in later phases– Also the smaller chance of a proper fix

• Old saying: test often, test early

• New methodology: test-driven development(write tests even before writing code)

Software is Complex

• Malleable

• Intangible

• Abstract

• Solves complex problems

• Interacts with other software and hardware

• Not continuous

Software Still Buggy

• Folklore: 1-10 (residual) faults per 1000 nbnc lines of code (after testing)

• Consensus: total correctness impossibleto achieve for complex software– Risk-driven finding/elimination of faults– Focus on specific correctness properties

Approaches to Detecting Bugs

• Software testing

• Model checking

• (Static) program analysis

• …

Software Testing

• Dynamic approach

• Run code for some inputs, check outputs

• Checks correctness for some executions

• Main questions– Test-suite adequacy (coverage criteria)– Test-input generation– Test oracles

Other Testing Questions

• Selection

• Minimization

• Prioritization

• Augmentation

• Evaluation

• Fault Characterization

• …

• Testing is not (only) about finding faults!

Current Status

• Testing remains the most widely used approach to finding bugs– Validation: are we building the right system?– Verification: are we building the system right?

• Testing is gaining importance with test-first development and increased reliability needs

• A lot of research on testing (part of mine too)– We’ll look mostly at techniques used in practice

“Schools” of Software Testing

• Bret Pettichord described four schools– Analytic (a branch of CS/Mathematics)– Factory (a managed process)– Quality (a branch of quality assurance)– Context-Driven (a branch of development)

• This course focuses on artifacts, not process

• Do you want a guest speaker from industry?

Topics Related to “Finding Bugs”

• How to “eliminate bugs” (localize faults)?– Debugging

• How to “prevent bugs”?– Programming language design– Software development processes

• How to “show absence of bugs”?– Theorem proving– Model checking, program analysis

Testing Topics to Cover

• Test coverage and adequacy criteria– Graph, logic, input domains, syntax-based

• Test-input generation

• Test oracles

• Model-based testing

• Testing software with structural inputs

• Test automation

• Testing in your domain of interest?

Your Interests

• Testing methods and methodology (*2)• Writing good/correct test cases (*3)• Better/more effective testing (*5) • Working with testers/developers (*2)• Relationship of testing and development cycle• Improve development (*4)• Software verification and (test) automation (*2)• Job skills (*3) (short-term or long-term?)• Please update me as we go through the course

Summary of the Introduction

• Eliminate bugs to save lives and money

• “Bugs” may mean faults, errors, failures

• Several approaches for detection: software testing, model checking, static analysis…

• Software testing is the most widely used approach for validation and verification– We will cover systematic approaches to testing,

based on coverage criteria for various models– Testing is not (only) about revealing faults

Next Lecture

• Thursday, January 24, 3:30pm, 1304 SC– Example Interactive Testing Session

• Test some refactoring• It should be fun• We can learn from mistakes as we go• We will learn some testing terminology

• Assignments– Really sign up on Wiki– Really try out Eclipse and/or NetBeans