1

Software Testing – Present Problems and Future Solutions

Vipul Kocher

2

Jules Verne’s “Twenty Thousand leagues under the sea” (1869) – Simon Lake built “Argonaut” first submarine to operate in the open seas (1897)

Arthur C. Clark Geosynchronous satellites (1945) – Actual satellites (1963)

Cloning - Dolly

Science Fiction – Fiction today, Reality tomorrow

3

What About These?

Androids Time Teleportation

How about self testing code?

May that never become a reality as

long as I live…

What about ? travel? ?

4

assumptions

5

Assumptions

• All future solutions are based on today’s technologies

• Disruptive innovations like new class of operating systems, development paradigms, languages may significantly alter the landscapes

6

Assumptions

I have a few ideas on

• new class of Operating Systems

• new class of programming languages and paradigms to support that operating system

• ultimately supported by new class of hardware

I have no clue about it !

7

problems in testing

8

Problems in Testing

Are there problems in testing? We all seem to be doing fine!

Probably the biggest problem in testing is- “we do not have a problem”

9

Problems in Testing

Why is this a problem? Is there a solution in future?

10

challenges in testing

11

Challenges in Testing

• What does testing mean for different stakeholders?• Have we tested enough?• How much time do I need to test it?• How many test cases should I write?• Oracle problem• How do I choose right set of regression tests?• How do I get developers to do better testing?• Do I need coding skills? What will I do with those?• How do I teaching testing to my testers?

12

What does testing mean for different stakeholders

13

What does testing mean for different stakeholders?

Let us face itNobody wants testing…

14

What does testing mean for different stakeholders?

If one assumes that “testing is necessary” one is likely to try to improve the way testing gets done

If one believes that testing is unnecessary then one is more likely to find ways to minimize or

eliminate testing

Mindset Matters !

15

testing world’s biggest problem

16

Testing World’s biggest problem

• Where is the innovation in testing? • Innovation as in Breakthrough Innovation?

17

Testing World’s biggest problem

Can you name three breakthrough innovations in testing? At least one?

18

HITE

H I T EHave I Tested Enough

19

HITE

First define “ENOUGH” – • Today

- No catastrophic or critical or major failure for users

• Tomorrow - ?

20

HITE

• Integrated software status monitor - Explained later (Patent Pending)

• New software technology - Self monitoring (Patent Pending), Self healing systems

• Software testing as a distributed, automated service - you pay software companies to install software on your system and test it (Patent Pending)

21

How much time do I need to test it

22

How much time do I need to test it

What are the problems you face in answering this

question?What are the solutions today?

• Estimation techniques? Work

Breakdown Structure? Function Points? Test Points? Use Case Points?

• Historical Data? Rule of

thumb? Gut feel?Are these accurate? What about

• Different levels of quality, workmanship (developers and testers both), project constraints such as budget, time and Subsequent Regression Cycles?

Future solution

• A mechanism to define quality level and a mechanism to measure that level (Patent Pending)

• Collection and analysis of data in automated manner and predictions based on current state of the project (Patent Pending)

23

Bug – The Only Perfect Being

24

Bug – The Only Perfect Being

Most bugs are because imperfect requirements are imperfectly translated into imperfect design which is imperfectly translated into imperfect code

25

Whole Is More Than The Sum Of Parts

26

Whole Is More Than The Sum Of Parts

Requirements are captured in

a granular mode, risk

prioritized and tracked and

checked automatically

for consistency, ambiguity,

incompleteness

Automated/assisted impact analysis is done

on other requirements/d

esign/code

Requirements are mapped to individual elements of design/code

Design is also assisted along

with code generation and

reverse engineering

Unit and integration tests

are automatically generated for

code

System tests of various

granularity are automatically generated and assisted based on some new and old test

design techniques

An integrated environment where…

27

Whole Is More Than The Sum Of Parts

The tests are executed in

an automated

manner and results

compared

The whole stack trace is maintained

and the entire

application can be

replayed line by source

line

All log/user messages are bubbled up by the tool

and test written to

execute those

conditions and all un-displayed

error messages are

flagged as holes

Performance and coverage

data is maintained

The map of code

executed has statistics on

requirements related to

that piece of code, tests,

testers, who wrote that code, bugs, modules,

risks, performance

etc

Each code fix is bubbled up to impact on other related

modules, features, GUI

elements, performance

etc

Regression tests are

suggested and run and

coverage, performance, security etc.

are all checked

again

An integrated environment where…

28

Integrated Environment

Requirements Visual Prioritized Self-checking Testable Constrained

Design – Mapped to Requirement

Code – Mapped to Requirements/DesignTests – Mapped to Requirements/Design/Code

29

Regression Test Selection Tool

30

Test Design Tools for automated / assisted creation of test cases – Q-Patterns, Unified

test design method, Noun and

Verb

Requirem ents

Requirements

Requirem ent based m odels

Requirement based models

D esign

Design

D esign based m odels: U M L

and future

Design based models: UML

and future

Code

Code

Code based m odels: U M L

and future

Code based models: UML

and future

D efect database

Defect database

Executi on tools – CA SE,

CA ST, A utom ati on

Execution tools – CASE,

CAST, Automation

Tool-Chain

31

User Messages and API based test cases

Code and Repository of “user” messages Log messages generated

Where is this message called?

How? Has it been called? Has this

log been generated?

API tests. Have all APIs been

covered? With what data?

32

Coverage & Data – Integral Part of Tool-Chain

Executable

External Interface coverage

Requirement Coverage

Risk Coverage

Test Coverage

Non-functional Coverage

Code Coverage

Code profiling

Code Security check

Customer care –

database, emails, blogs

Defect database,

data-mining for bug-

taxonomy

Data from application

monitoring in production

Internet security

advisories

StandardConformance

33

New format for writing test cases: visual format

34

An example of Visual format (http://www.exampler.com/software.html)

35

Ability to replay code

Crash

36

Testing as a Framework

37

38

Far Future – Where Testing?

• Tool driven

• Integral part of development

• Better ways to tie all development artifacts together

• Integration with various sources of information • Better analysis and action

39

Q & A

Feel free to contact – vipul@puretesting.com