Test Design and Techniques Continues….

Negative Testing

To think outside the box to determine just how many ways he/she can cause the software product to get wrong answers or abort execution.

By finding out what has to be in place before executing the program, a good tester can identify a series of test cases that determine how the target software product would react if one of these prerequisite conditions were not satisfied.

Tester will execute the program

• On platforms it wasn’t intended to execute on

• With missing communication lines, or bad incoming, out going data

• With missing data files, missing data records in databases or scrambled data in data files

• With misspelled links, or undefined, wrong, or missing configuration parameters

• Power-off peripherals, like printers, scanners, external CD drive, external hard drives, and so on

Negative testing also encompasses inputting bad data. This bad data can come in the forms of illegal user inputs, illegal index values seeded into indexed files, etc.

Robustness of software is the quality of being able to withstand negative testing without failure.

Software developer must be challenged to code and test for the bad data as well as good data.

However, the strategy for negative testing should never be a shotgun approach. Instead, the test plan should be well thought out, consistent, complete and productive in the finding of bugs.

Risk-Based Testing

Conventional approach to “Risk-Based” testing:Define a risk-ranking metric and establish appropriate policies and procedures for its use.

Risk Level Standard for selection of this level

1 Bug results in termination of all applications/communications; a restart is required and session data may be lost or incomplete

2 Bug results in wrong answers that may impact future results, should the user continue using application

3 Bug results in correct results, but user requirements not fully met

4 User-facing information or performance does not meet user usability requirements

5 A feature that could be improved, but users are not negatively impacted

Test the highest-risk things most

Risk-Based testing checklist

•Test based on quality priorities•Test features where cost of failure is likely to be high

–Customer impact–Business operational risk

•Test features likely to be used most•Test where defects are most likely

–Prior defect history–Areas that are newly added or modified–Relative complexity–Developers’ confidence levels–Testers’ observations

Ross Collard Systems Testing & Quality Assurance Techniques

Proactive Risk-Based Testing

Conventional testing is often reactive and often poorly received:

• Tests are defined late in the development process

• Tests are based mainly on what developers have done

Key Proactive testing concepts

• Intertwine testing with each development deliverable

•Plan before acting at any point/level• Define acceptance criteria at start• Prepare test plans/designs, promote reuse• Prioritized by level, avoid wasted effort

• Let testing drive development to• Enable developers to find more problems themselves• Selectively use resources to do important testing early

Use-Case Testing

For object-oriented programming, a design technique call use-case design is often in use today. Use cases depict what entities interact with one another, based on a scenario. The strong mapping of scenarios to use cases allow us to conclude that there is a set of popular use cases within a design.

By the percentage of usage frequency data, testers can budget more than 50% of their test case development and test case implementation time on less than 50% of the code that has the highest usage frequencies – the popular use cases.

The use-case design, along with the estimated usage frequency of each, provides a rank order of the priorities for the test cases. Test case development and the order in which the tests are executed should be based on these priorities in order to optimize test resources and testing productivity.

Before applying the use-case approach to testing, the testers and developers must work together to ensure that the use cases match the requirements and the code. Letting requirements documentation get unsynchronized with the code has been a frequent audit finding for many software development projects. If the use cases that were developed from original requirements are given to the test staff without validating them against current requirements, volatility in the original requirements may have occurred.In this case, if testers use the use-case information, they might get false detections of bugs and upon further investigation find out the that code satisfies the requirements, but not the use-case information.

Bebugging/Mutation

Bebugging is a form of program mutation, it is a way of determining the effectiveness of the testing strategies that are being taken on a project.

A primary test progress metric is: number of bugs found divided by the number of latent bugs to be found, express as a percentage.Bebugging is the act of changing the software to create bugs, creating a build that is tested by the test team, and then comparing the bugs found against the created bugs to determine test effectiveness. The ratio between the created bugs found divided by the number of created bugs should be the same as the ratio between the bugs found divided by the number of latent bugs in the product.

Buy-in of both the testers and developers must be won before launching a bebugging effort.

Define Testing Criteria and Quality Checkpoints

This is to set rules in place to govern the flow of testing. For example, if you are counting on the software passing a set of unit and integration tests before it come to system test, you need to say so up front in your test plan.Setting and adhering to testing criteria cab help a project stay on schedule by avoiding the waste of time associated with testing software that is not ready to be tested, or by continuing to test software that needs to be fixed.

Five types of criteria that can be defined before the start of system test:

• Entry criteria: explain what needs to be done before you begin testing

• Exit criteria: describe what you consider to be necessary in order to conclude testing

• Suspension/resumption criteria: describe what will happen if testing is blocked by bugs

• Test pass/fail criteria: each test that is run should have a defined expected result. If the expected result is obtained, the test passes, otherwise it fails

• Other criteria determined by process or standards: if your product must comply with a standard, or if your company places requirements on your process, you may have additional test criteria to consider.

Testing Object-Oriented software

Testing application built using object-oriented design employs many of the same techniques presented earlier. However, unique characteristics of object-oriented systems present some new challenges and require different strategies for testing.

Object-oriented programming provides features not available in procedural software. Object-oriented software centers on a class (the basic unit) and the inheritance and encapsulation that affect a class. Procedural programming controls the flow between software functions. Testing object-oriented features call for new strategies.

Source: Introducing Software Testing by Louise Tamres

Some Object-oriented terminology:

Class is a blueprint that defines the data and behavior (methods) common to all objects of a certain kind. An object is an instance of a particular class.

Inheritance provides the ability to derive new classes from existing classes. The child class (derived class) inherits all the data and behavior of the parent (base class), and it may also add new features or redefine existing features.

Encapsulation provides the programmers with a well defined interface to access an object’s features in a way that prevents the programmers from directly accessing the object’s internal code.

Object-oriented programming also provides the capability to limit the access to class members. An object can always access members defined within its own class. There are public class members and private class members, where objects of different class can access a public class member, and private members are only available to objects of the same class.

Let’s look at testing object-oriented applications from both system test and unit test points of view.

System testing is independent of the process used to create any application. The testers evaluates the application from a user perspective. Internal design details are irrelevant and do not affect how tests are defined. The application’s behavior, whether presented as use cases or other forms of requirements, drives the development of test cases. Therefore, knowledge about object-oriented software is not necessary for defining system test level test cases.

Unit testing of an software application requires programming knowledge and access to implementation details. When dealing with object-oriented code, the meaning of unit is different. A class is the smallest unit that can be complied. The key difference in unit testing the two design methodologies is that unit testing of procedural software accesses the units (procedures or functions) directly, whereas access to the class’s data and methods may be more restricted.

A thorough unit test of object-oriented software consists of ensuring that all the methods within a class and their interfaces function as required. Designing test cases for class inheritance requires additional considerations.

It is important to note that some object-oriented languages can be implemented in a procedural manner, thus requiring a procedural testing strategy.

Most of the techniques for defining test cases apply equally for both procedurally developed applications as well as those developed using object-oriented design. A major difference is in executing the test cases. Executing test cases for object-oriented software undergoing unit testing requires an execution environment that deals with encapsulation.

System test example:

A Mall Guide system in a local shopping mall has a few select stores that specialize in the following categories:

1. Women’s clothes (3 stores)2. Men’s clothes (2 stores)3. Children’s clothes (4 stores)4. Women’s shoes (2 stores)5. Men’s shoes (2 stores)6. Flowers (1 store)7. Gifts (2 stores)8. Restaurants (5 stores)

2 types of actors interact with the Mall Guide. The actors are:

1. Shoppers who seek to find an appropriate store in the mall2. Administrators who updates store names and locations

The current requirements, up to this point uses the following user input information, and there is no details on the administrator’s needs.

A partial list of “User Inputs” for the Mall Guide:Shopper commands: main menu1. shopper touches a store category from a list and it gives a list of

stores from which to select2. shopper touches map key and a map of the mall is provided3. shopper touches help key and is provided with a help screenShopper commands: list of stores1. shopper touches a particular store name and a map highlighting

the store’s location is provided on the screen2. shopper touches main menu key and returns to the main menu3. shopper touches help key and is provided with a help screenShopper commands: maps1. shopper touches print key and a map is printed out2. shopper touches “return to previous screen” key and the prior

screen is displayed3. shopper touches main menu key and returns to the main menu4. shopper touches help key and is provided with a help screen

Different strategies exist for developing system test cases. We’ll look at the approach employs use cases, which focus on actor scenarios.

Use cases describe the application’s behavior from the actor’s perspective. Well-constructed use cases translate easily into test cases. However, use cases rarely presents the entire system requirements, as they often omit information pertinent to performance and load tests.

Let’s look at the following use case:

Use case #1: shopper wants to find a specific store in the mall

Success guarantees

Shopper receives a printed map to the desired store

Main success scenario:

1. Shopper selects a store category2. Shopper selects a store name3. Shopper selects print function

Extensions

1a. Shopper wants help selecting a store category1b. Appropriate store category does not exist2a. Shopper wants help selecting a store name2b. Desired store does not exist

Use case #1 generates many test cases. At lease one test case must exist for each extension to ensure that the system properly handles that event. The following is a partial list of test cases that came from use case#1

Test case ID

Test source Initial system state

input Expected results Actual results

U1-1 Use case #1 Start test from main menu

1.Select “men’s clothes”

2.Select the first store name

3.Hit print key

1.The list of stores screen appears and lists 2 men’s clothes store

2. Get a map to this store posted on the screen

3. Get a printed copy of the map

U1-2 Use case 1

Extension 1a

Start test from main menu

1.Hit help key

2.Hit return to previous screen key

1. Get the help screen providing info about the main menu

2. Return to main menu

U1-3 Use case 1

Extension 1b

Start test from main menu

No input: shopper wants to find a book store

Main menu does not lists books as a store category

More test cases….

…..

Unit testing of classes

Although classes are unique to object-oriented programming, the approaches to defining test cases are often the same as for procedural applications. One instance involves classes that implement state machines. State diagrams and state tables can easily be translated into a set test cases as we have discussed earlier.

Another test design technique – robust testing using orthogonal arrays is also effective in testing classes. Using the orthogonal testing technique we have discussed earlier, one can select the best possible pair-wise combinations from each class to create test cases.

Testing Inheritance

Inheritance creates new classes that reuse and extend previously created classes. This allows for expanding the features of existing code without actually modifying the original code.

Let’s consider a generic example:

•Class A•Class B, child of A•Class C, child of B and grandchild of A

Class A

Public method m1Private method m2Public method m3Public method m4

Class B

Public method m3*Private method m4*Public method m5Public method m6Public method m7

Class C

Public method m4*Private method m6*Public method m8

* Denotes a new method that overrides the previous method

Even though prior testing confirms that the parent class works correctly by itself, the question is whether the methods defined by the parent work when called by the child. Inheritance allows a child to redefine (override) some of the methods. In addition, specifying whether the method is public or private affects whether it is accessible to the child.

Because of inheritance, one cannot assume that methods of class A will work when called from an object of class B. Thus we need to test the interactions of A in the context of B, likewise, we need to test the interactions of A in the context of C and the interactions of B in the context of C.i.e. we need to test the method m1 (defined by class A) in the context of class A, class B, class C.

Test execution

After defining the test cases, the next step is to formulate an environment in which to execute the tests. This requires special test software known as a driver to invoke the software-under-test. A typical driver consists of the following features:

•Packaging input and data

•Invoking software-under-test

•Capturing the resulting outcome

However, a driver to test a class calls for additional consideration. The very nature of object-oriented design uses encapsulation to hide complexity, thereby limiting which data and methods are visible outside of the class.

However, a driver to test a class calls for additional consideration. The very nature of object-oriented design uses encapsulation to hide complexity, thereby limiting which data and methods are visible outside of the class.

The driver must get around encapsulation to control and observe a class’s internal data and methods. There are several ways to achieve this:

•Providing a test case method for each class•Creating a parallel class, which appears identical to the original except for the addition of code needed for testing

•Creating a child class that inherits the methods to be tested

All of these approaches alter the class and thus modify the actual application. The tested implementation may not be identical to the release code. It is important that the testing environment be as close as possible to the deployment environment, otherwise, you are not fully testing the end product.

Homework: week 4 (02/10/05)

• Read chapters 9,10

• Review EZBay Product Descriptions and Requriements document for discussion in class next week