ICST Tutorial

8/6/2019 ICST Tutorial

1/83

Software Testing:A Newcomers Guide

Paul Ammann

Software Engineering Group

George Mason University

Presented at ICST

April 2, 2009

Denver, Colorado

Slides Co-Produced with Jeff Offutt


2/83

Software Testing: A Newcomers Guide 2009 22

Here! Test This!

Testing two decades ago

A stack of computer printoutsand no documentation

Testing a decade agoTesting Now?

A disk and no documentationA complex conglomeration only low level documentation


3/83


A Talk in 4 Parts

1. Why do we test ? Costs and Attitudes

2. What should we do during testing ? Test Activities

Software Testing Terms

Evolving Notion of Coverage Criteria

3. How do we get to this future of testing ? Beizers Maturity Levels

4. Background for ICSTsessions

We are in the middle of aWe are in the middle of a revolutionrevolution in how software is testedin how software is tested

ResearchResearch isis finallyfinally meeting practicemeeting practice


4/83


Part 1 : Why Test?

Written test objectives are rare or useless

The software shall be easily maintainable What fact is each test trying to verify?

Requirements definition teams should include testers!

Ensures testing can satisfy test objectives

Provides rationale for tests

How much testing is enough?

If you dont knowIf you dont know whywhy youre conductingyoure conductinga test, it wont be very helpfula test, it wont be very helpful


5/83


Cost of Testing

In the real-world, testing is the principle post-design activity

Restricting early testing usually increases cost

Extensive hardware-software integration requires more testing

Youre going to spend at least half ofYoure going to spend at least half ofyour development budget on testing,your development budget on testing,whether you want to or notwhether you want to or not


6/83


Cost of Not Testing

Not testing is even more expensive

Planning for testing after development is prohibitivelyexpensive

Hardware developers spend huge amounts on testing

Software test tools arent that expensive. Many are free!!!

Program Managers often say:Program Managers often say:Testing is too expensive.Testing is too expensive.


7/83


Caveat: Impact of New Tools andTechniques

Theyre teaching a new wayof plowing over at the Grangetonight - you going?

Naw - I alreadydont plow as goodas I know how...

Good Techniques and Tools are Not Enough:Good Techniques and Tools are Not Enough:

Practitioners Need to Adopt Them Too!Practitioners Need to Adopt Them Too!


8/83


Part 2 : What ?

But But WWhathat should we do ?should we do ?

1. Types of test activities

2. Software testing terms

3. Changing notions of testing

test coverage criteria

criteria based on structures


9/83


Testing in the 21st Century

We are going through a time of change Software Defines Behavior

network routers

financial networks

telephone switching networks

other infrastructure

Embedded Control Applications

airplanes, air traffic control

spaceships

watches

ovens remote controllers

Safety critical, real-time software

Web apps must be highly reliable

And of course security is now all about software faults !

PDAs

memory seats

DVD players

garage door openers

cell phones

Testing ideas haveTesting ideas have

matured enough tomatured enough to

be used in practicebe used in practice


10/83


Test Activities Testing can be broken up into four general types of activities

1. Test Design2. Test Automation

3. Test Execution

4. Test Evaluation

Each type of activity requires different skills, backgroundknowledge, education and training

No reasonable software development organization uses the samepeople for requirements, design, implementation, integration

and configuration control

10

Why do test organizations still use the same peopleWhy do test organizations still use the same people

for all four test activities??for all four test activities??

This is clearly aThis is clearly a wastewaste of resourcesof resources

1.a) Criteria-based1.b) Human-based


11/83


1. Test Design (a) Criteria-Based

This is the most technicaljob in software testing

Requires knowledge of :

Discrete math Programming

Testing

Requires much of a traditional CS degree

This is intellectually stimulating, rewarding, and challenging

Test design is analogous to software architecture on thedevelopment side

Using people who are not qualified to design tests is a sure way toget ineffective tests

11

Design test values to satisfy coverage criteriaDesign test values to satisfy coverage criteria

or other engineering goalor other engineering goal


12/83


1. Test Design (b) Human-Based

This is much harder than it may seem to developers

Criteria-based approaches can be blind to special situations

Requires knowledge of : Domain, testing, and user interfaces

Requires almost no traditional CS

A background in the domain of the software is essential

An empirical background is very helpful (biology, psychology, )

A logic background is very helpful (law, philosophy, math, )


But not to typical CS majors they want to solve problems and buildthings

12

Design test values based on domain knowledge ofDesign test values based on domain knowledge of

the program and human knowledge of testingthe program and human knowledge of testing


13/83


14/83


3. Test Execution

This is easy and trivial if the tests are well automated

Requires basic computer skills

Interns

Employees with no technical background

Asking qualified test designers to execute tests is a sure way toconvince them to look for a development job

If, for example, GUI tests are not well automated, this requires alot ofmanual labor

Test executors have to be very careful and meticulous withbookkeeping

14

Run tests on the software and record the resultsRun tests on the software and record the results


15/83


4. Test Evaluation

This is much harder than it may seem

Requires knowledge of :

Domain

Testing

User interfaces and psychology

Usually requires almost no traditional CS

A background in the domain of the software is essential

An empirical background is very helpful (biology, psychology, )

A logic background is very helpful (law, philosophy, math, )


But not to typical CS majors they want to solve problems and buildthings

15

Evaluate results of testing, report to developersEvaluate results of testing, report to developers


16/83


Other Activities

Test management : Sets policy, organizes team, interfaces withdevelopment, chooses criteria, decides how much automation isneeded,

Test maintenance : Tests must be saved for reuse as softwareevolves

Requires cooperation of test designers and automators Deciding when to trim the test suite is partly policy and partly technical

and in general, very hard !

Tests should be put in configuration control

Test documentation : All parties participate

Each test must document why criterion and test requirement satisfiedor a rationale for human-designed tests

Traceability throughout the process must be ensured

Documentation must be kept in the automated tests

16


17/83


Approximate Number of Personnel

A mature test organization may need only one test designer towork with several test automaters, executors and evaluators

Improved automation will reduce the number of test executors

Theoretically to zero but not in practice

Putting the wrong people on the wrong tasks leads to

inefficiency, lowjob satisfaction and lowjob performance A qualified test designer will be bored with other tasks and look for a job

in development

A qualified test evaluator will not understand the benefits of test criteria

Test evaluators have the domain knowledge, so they must be freeto add tests that blind engineering processes will not think of

17


18/83


19/83


Applying Test Activities

19

To use our people effectivelyTo use our people effectively

and to test efficientlyand to test efficiently

we need a process thatwe need a process that

lets test designerslets test designersraise their level of abstractionraise their level of abstraction


20/83


Model-Driven Test Design

20

softwareartifact

model /structure testrequirements

refined

requirements /test specs

inputvalues

testcases

testscripts

testresults

pass /fail

IMPLEMENTATION

ABSTRACTION

LEVEL

DESIGN

ABSTRACTION

LEVEL


21/83


22/83


Model-Driven Test Design Activities

22

softwareartifact

model /structure testrequirements

refined

requirements /test specs

inputvalues

testcases

testscripts

testresults

pass /fail

IMPLEMENTATION

ABSTRACTION

LEVEL

DESIGN

ABSTRACTION

LEVEL

Test DesignTest Design

TestTest

ExecutionExecutionTestTest

EvaluationEvaluation

Raising our abstraction level makestest design MUCH easier


23/83


Software Testing Terms

Like any field, software testing comes with a large number ofspecialized terms that have particular meanings in this context

Some of the following terms are standardized, some are used

consistently throughout the literature and the industry, but somevary by author, topic, or test organization

The definitions here are intended to be the most commonly used

23


24/83


Important TermsValidation & Verification (IEEE)

Validation : The process of evaluating software at the end ofsoftware development to ensure compliance with intendedusage

Verification : The process of determining whether the productsof a given phase of the software development process fulfill therequirements established during the previous phase

IV&V stands for independent verification and validation


25/83


Test Engineer & Test Managers

Test Engineer : An IT professional who is in charge of one ormore technical test activities

designing test inputs

producing test values

running test scripts

analyzing results

reporting results to developers and managers

Test Manager : In charge of one or more test engineers

sets test policies and processes

interacts with other managers on the project otherwise helps the engineers do their work


26/83


Test Engineer Activities

Test

Designs

Output

Executable

Tests

Computer EvaluateP

Test

Manager

Test

Engineer

Test

Engineer

design instantiate

execute


27/83


Static and Dynamic Testing

Static Testing : Testing without executing the program This include software inspections and many forms of analysis

Very effective at finding certain kinds of problems especially potential faults,that is, mistakes that may only be visible after the program evolves

Example faults identifiable with static analysis:

Overriding equals(), but not hashCode()

Implementing clone() by calling a constructor

Using foreign data prior to scrubbing

Hypertext inputs

Parts of SQL queries

Array indices

Dynamic Testing : Testing by executing the program with realinputs

Mostly what this talk is about


28/83


29/83


Testing & Debugging

Testing : Finding inputs that cause the software to fail

The principle topic of this lecture

Debugging : The process of finding a fault given a failure

Fault localization is a huge issue

Some faults have failures that are hard to reproduce

Often desirable to recast a problem report as a short test case that also

reveals the failure General principle: Faults usually have short revealing test cases

These test cases are excellent candidates for regression tests


30/83


Fault & Failure Model

Three conditions necessary for a failure to be observedThree conditions necessary for a failure to be observed

1. Reachability : The location or locations in the program thatcontain the fault must be reached

2. Infection : The state of the program must be incorrect

3. Propagation : The infected state must propagate to cause someoutput of the program to be incorrect


31/83


Test Case

Test Case Values : The values that directly satisfy one testrequirement

Expected Results : The result that will be produced whenexecuting the test if the program satisfies it intended behavior


32/83


33/83


Inputs to Affect Controllability andObservability

Prefix Values : Any inputs necessary to put the software intothe appropriate state to receive the test case values

Postfix Values : Any inputs that need to be sent to the softwareafter the test case values

Two types of postfix values

1. Verification Values : Values necessary to see the results of the test case values

2. Exit Commands : Values needed to terminate the program or otherwise return itto a stable state

Executable Test Script : A test case that is prepared in a formto be executed automatically on the test software and producea report

Example: Junit tests


34/83


Top-Down and Bottom-Up Testing

Top-Down Testing : Test the main procedure, then go downthrough procedures it calls, and so on

Bottom-Up Testing : Test the leaves in the tree (procedures thatmake no calls), and move up to the root.

Each procedure is not tested until all of its children have been tested


35/83


White-box and Black-box Testing

Black-box testing : Deriving tests from external descriptions ofthe software, including specifications, requirements, and design

White-box testing : Deriving tests from the source code internalsof the software, specifically including branches, individualconditions, and statements

This view is really out of date.This view is really out of date.

The more general question is:The more general question is:from what level of abstractionfrom what level of abstractionto we derive teststo we derive tests??


36/83


Evolving Notion of Coverage Criteria

Old view of testing is of testing at specificsoftware development phases

Unit, module, integration, system

New view is in terms ofstructures and criteria

Graphs, logical expressions, syntax, input space


37/83


38/83


Beizers Insight : Find a Graph and Cover It

Tailored to:

a particular software artifact

code, design, specifications

a particular phase of the lifecycle

requirements, specification, design, implementation

But graphs do not characterize all testing techniques well

Four abstract models seem to suffice


39/83


40/83


New : Criteria Based on Structures

1. Graphs

2. Logical Expressions

3. Input DomainCharacterization

4. Syntactic Structures

(not X or not Y) and A and B

if (x > y)

z = x - y;

else

z = 2 * x;

Structures : Four ways to model software

A: {0, 1, >1}

B: {600, 700, 800}

C: {swe, cs, isa, infs}


41/83


42/83


43/83


44/83


2. Logical Expressions

((a> b) orG) and(x


45/83


46/83


2. Logic Active Clause Coverage orMultiple Condition/Decision Coverage (MCDC)

((a>b)orG)and(x


47/83


48/83


49/83


50/83


51/83


52/83


53/83


Two Ways to Use Test Criteria

1. Directly generate test values to satisfy the criterion oftenassumed by the research community most obvious way to usecriteria very hard without automated tools

2. Generate test values externally and measure against thecriterion usually favored by industry

sometimes misleading

if tests do not reach 100% coverage, what does that mean?

Test criteria are sometimes called metrics


54/83


55/83


56/83


57/83


58/83


Beizers Maturity Levels

Level 0 : Theres no difference between testing and debugging

Level 1 : The purpose of testing is to show correctness

Level 2 : The purpose of testing is to show that the software

doesnt work Level 3 : The purpose of testing is not to prove anything specific,

but to reduce the riskof using the software

Level 4 : Testing is a mental discipline that helps all ITprofessionals develop higher quality software


59/83


60/83


Level 1 Thinking

Purpose is to show correctness

Correctness is impossible to achieve

What do we know ifno failures?

Good software or bad tests?

Test engineers have no: Strict goal

Real stopping rule

Formal test technique

Test managers are powerless

This is what hardware engineers often expectThis is what hardware engineers often expect


61/83


62/83


63/83


64/83


Summary

More testing saves money Planning for testing saves lots of money

Testing is no longer an art form Engineers have a tool box of test criteria

When testers become engineers, the product gets better The developers get better


65/83


66/83


67/83


Part 4: Testing Topics by ICST Section

ICST has 16 Sessions with Papers: 13 Research and 3 Industry R1: GUI Testing

R2: Model Checking

R3: Embedded and Real-Time Testing

R4: QA and Test Management

R5: Model-Based Testing

R6: Static Analysis

R7: Security Testing

R8: Empirical Studies

R9: Test Case Generation

R10: Web Testing

R11: Aspects and Faults

R12: Mutation and Non Functional Testing

R13: Assertions and Failure States

I3: Real World Testing I2: Test Management

I3: Automation

Following slides briefly overview foundations for each Session Tall Order!

Ill be brief

67


68/83


Section R1: GUI Testing

Basic Issues

Many systems have only a GUI interface

Problems may exist in GUI, or in underlying application

Structural Tests

Legal event sequences / hiding of uninteresting events

Model Based Tests Operational / Behavioral / Graph models

Research Foci

Case studies on industrial software

Reuse and scalability are huge issues

Particularly in the presence of maintenance

Crash testing

68


69/83


Section R2: Model Checking

Basic Issues

Define a labeled state space with transitions: A state machine

Check (temporal) properties over paths in that state space

Bonus: Counterexamples

Standard Problem: Scalability

Research Foci Show two descriptions consistent (or identical)

Test case generation

Interpreting counterexamples as test cases

Using model checker as test generation engine

Model checking real code

Direct application to programs (eg JavaPathFinder)

69


70/83


Section R3: Embedded and Real-Time Testing

Basic Issues

Embedded software has a huge observability problem

How do you know the exact state of the SUT?

Testing timeliness constraints as well as functional correctness

Research Foci

Abstract models for SUT Notions such as Partial Observability

Passive Testing

Non interference of test harness with observations

Test specification frameworks for specific domains

TTCN-3

70


71/83


72/83


73/83


74/83


75/83

S ti R9 T t C G ti


76/83


Section R9: Test Case Generation

Basic Issues

Old, but very important problem

We need tools that can automatically generate good test data!

Research Foci

Generating feasible paths in finite state machines

Hard problem whether a given transition is, in fact, feasible Generating test inputs to follow paths

Key factors

Constraint solving

Path explosion

Relation of paths to other coverage goals

Test inputs to satisfy complex predicates

Efficient algorithms

76


77/83


78/83


79/83

S ti R13 A ti d F il St t


80/83


Section R13: Assertions and Failure States

Basic Issues

Assertions are statements that must be true at given points in execution

Extremely powerful approach to fault detection

Failure states are program states after something bad happens.

Research Foci

Extracting tests from runtime failures

Automatically turning any failure (in operation) into a test

Huge potential practical impact

Clearly, overhead needs to be low

Assertion-based test validation

When software is modified, which assertions need to be checked?

Metamorphic testing

Solving the oracle problem by relating multiple program runs

Experience with implementing this approach via assertions

80

S ti I1 R l W ld T ti


81/83


Section I1: Real World Testing

Basic Issue

Testability:

The degree to which components are easy to test

Components can be designed to be easier to test

The efficiency of the test process

Research Foci Why testability is hard inpractice

Organizational factors

Technical Factors

81


82/83

S ti I3 A t ti


83/83

Section I3: Automation

Basic Issues

Test generation

Code for multi-core processors

Specialized languages

Research Foci

Test generation to satisfy coverage criteria

Application to safety critical system

How to test code optimized for multi-core?

Testing multi-threaded is much harder than single-threaded!

Developing high level languages for specific domains

High level test approaches are a necessary complement

Date post:	07-Apr-2018
Category:	Documents
Upload:	dao-anh-hien
View:	231 times
Download:	0 times

ICST Tutorial

Documents