Software Testing - NC State Computer Science · PDF fileSoftware Testing Reading: Software...

8/22/2016

1

CSC216: Programming Concepts – Java © NC State CSC216 Faculty

Software Testing

Reading: Software Testing

1

CSC216: Programming Concepts – Java © NC State CSC216 Faculty 2

8/22/2016

2

CSC216: Programming Concepts – Java © NC State CSC216 Faculty 3


Testing Introduction

• Testing is the process of finding software faults

– Fault: “an incorrect step, process, or data definition in a program”

• Testing: “the dynamic verification of the behavior of a program on a finite set of test cases, suitably selected from the usually infinite executions domain, against the expected behavior”

• Test cases uncover failure by finding where the actual behavior of a program deviates from the expected behavior.

• Investigation of failures uncovering faults.

4

8/22/2016

3


Why Test?

• Testing increases confidence that your program works correctly – meets the customer requirements

– Better scores on assignments

– Save companies millions, if not billions, of dollars

5


Verification vs. Validation

• Verification: process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase– Are we building the product right?

• Validation: process of evaluating a system or component during or at the end of the development process to determine whether is satisfies specified requirements– Are we building right product?

6

8/22/2016

4


Verification vs. Validation (2)

7


Black Box Testing

• Ignores the internals of the program – program treated as a black box

• Finds– Incorrect or missing function,

– Interface errors,

– Errors in data structures or external data base access,

– Behavior or performance errors, and

– Initialization and termination errors.

8

Executable ProgramInput Output

Black Box Test

8/22/2016

5


White Box Testing

• Code under test is known!

• Use code to guide test

• Your tests should exercise

– Independent paths within the source code (“important” paths)

– Logical decisions as both true and false

– Loops at their boundaries

– Internal data structures

9


Types of Testing

• Unit Testing

• Integration Testing

• Functional/System Testing

• Acceptance Testing

• Regression Testing

• Beta Testing

10

Our focus is here

8/22/2016

6


Unit Testing

• Testing of individual hardware or software units or groups of related units

– Focus on the method, interleaved methods, and class level

• Done by programmer(s)

• Generally all white box

• Automation desirable for repeatability

• Tools: JUnit

11


Integration Testing

• Testing in which software components, hardware components, or both are combined and tested to evaluate the interaction between them

– Focus on the interaction between classes due to composition or inheritance

• Done by programmer as they integrate their code into code base

• Generally white box, maybe some black box (e.g., closed source libraries)

• Automation desirable for repeatability

• Tools: JUnit

12

8/22/2016

7


Functional / System Testing

• Testing conducted on a complete, integrated system to evaluate the system compliance with its specified requirements

– Tests are generated from the requirements

– Includes stress testing, performance testing, and usability testing

• It is recommend that this be done by external test group

• Mostly black box so that testing is not “corrupted” by too much knowledge

• Test automation desirable, but non-trivial

– Beyond the scope of 21613


Acceptance Testing

• Formal testing conducted to determine whether or not a system satisfies its acceptance criteria (the criteria the system must satisfy to be accepted by a customer) and to enable the customer to determine whether or not to accept the system

• Generally done by customer/customer representative in their environment through the typical user interface.

• Black Box

14

8/22/2016

8


Regression Testing

• Selective retesting of a system or component to verify that modifications have not caused unintended effects and that the system or component still complies with its specified requirements.

• Smoke test group of test cases that establish that the system is stable and all major functionality is present and works under “normal” conditions.

• Combination of white and black box

• Automated for repeatability

15


Beta Testing

• 3rd party testing by a subset of customers

• Unstructured/unscripted testing

• Black box

16

8/22/2016

9


Development and Verification Activities

17

Level of Detail

Project Time

Low

High

RequirementsElicitation

Analysis

Design

SystemTesting

Object Design Unit Testing

Integration Testing

AcceptanceTesting


Testing Process

18

Write test casesWrite program

Run test on program

Actual == Expected?

Deadline?

Deliver

NoFailure

YesPass

Yes

NoWrite more test cases

8/22/2016

10


Test Case Information

• Unique Identifier– Black box: name of test in document

– White box: name of test method or specific assert

• Input into the program or program unit– Black box: how the user runs and interacts with the program

• Could be redirection input and output on the console

– White box: inputs to methods that set up test

• Expected output from the program or program unit– What you expect to get based on input and requirements

• Actual results of running the test case– Black box: what the user gets from the program

• Could be redirection of std out

– White box: return values from functions or check on state via other methods

19


Seasons ProgramThere are four seasons that correspond to the given general dates for the northern hemisphere:

• Winter: December 21 to March 20

• Spring: March 21 to June 20

• Summer: June 21 to September 20

• Fall: September 21 to December 20

A customer wants a program that prompts the user a month and a day. If the month and day are valid, the program will print the season for the month and day. If the month isn’t a match to full name of one of the calendar months, then the error message “Invalid month <month>” is. If the day is not valid for the given month (e.g., 30 in February), the error message “Invalid day <day> for month <month>.” is printed. For this program, we will assume that we are not in a leap year, so February 29th would be considered invalid.

8/22/2016

11


Black Box Test Plan

Test ID Description Expected Results Actual Results

TestName

(Test Author)

Test Type:

Preconditions:

Test Inputs

Test outputs Actual outputs

• Formal document outlining the black box test cases for a project

• Description must be repeatable – have specific values!

• Expected results require specific values too!

• Write black box tests before writing your program

21


Test Requirements

• Ensure that all of the customer requirements are tested!

22


invalidMonth Preconditions: Seasons program

started

Enter a month: Jan

Enter a day: 3

Stop program

Invalid month Jan.

Program stops execution

8/22/2016

12


Equivalence Classes

• Input/output space is broken into different classes

• Each equivalence class is tested

• Tests are written to include “middle” input values from each of the possible classes

• One test may consider multiple equivalence classes

• One for each type of input/output

• A test focuses on one equivalence class, but other values are needed for a full test. Those other values should be “middle” values.

• Helps further test requirements by considering groups of inputs/outputs


Equivalence Classes – Month Month (Input) Example Value

Valid January

Invalid Jan


winter Preconditions: Seasons program

started

Enter a month: January

Enter a day: 15 Winter


invalidMonth Preconditions: Seasons program

started

Enter a month: Jan

Enter a day: 3

Stop program

Invalid month Jan.


8/22/2016

13


Equivalence Classes – DayDay (Input) Example Value

< 1 ‐3

1 <= day <= max 15

> Max 50



started




negativeDay Preconditions: Seasons program

started


Enter a day: ‐3

Stop program

Invalid day ‐3 for month January.



Equivalence Classes – SeasonDay (Input) Example Value

Winter February 15

Spring May 15

Summer August 15

Fall November 15



started

Enter a month: February



spring Preconditions: Seasons program

started

Enter a month:May

Enter a day: 15 Spring


8/22/2016

14


Boundary Value Analysis

Test ID Description Expected Results Actual

Results

zeroDays Preconditions: Seasons program

started


Enter a day: 0

Stop program

Invalid day 0 for month January.


day < 1 1 <= day <= max day for month day > max day for month

• Programmers tend to make mistakes at boundaries

• Want to test program boundaries and values to either side of the boundary


Diabolical or Dirty Test Cases

• Divide by zero?

• Wrong input type? String instead of a double

• Illogical path through functionality

• Don’t enter mandatory fields?

• Program aborted or device unplugged, dropped, turned off?

28

8/22/2016

15


White Box Testing Techniques

• Basis set testing

– Cyclomatic complexity V(G) – estimate of number of paths

– Test all program paths in a method

– Test all conditionals on true AND false side

– Control flow diagram

• Equivalence Classes

• Boundary Value Analysis

• Think diabolically

30


Control Flow Diagram

• Pictorial description of the flow of program control

• Diamonds: decisions

• Rectangles: program statements

• Break apart compound conditionals

• Loops have one decision (the continuation test)• Unless the loop has compound conditional tests – those should be broken up

31

8/22/2016

16


Control Flow Templates

T F

do-while Loops

T

F

for and while Loops

T

F

if Statement

TF

if/else Statement

T

T

T

F

F

F

Sequential ifStatements

Nested if/else/ifStatements

T

T

T

F

F

F

Nested if/elseStatements

T

T

F

F

32


Compound Predicates Control Flow Templates

F

T T

F T

F F

T

predicate1 && predicate2 predicate1 || predicate2

33

8/22/2016

17


Cyclomatic Complexity

• Measure of a method’s complexity

• Number of potential paths through the source code

• Use to estimate number of tests to write– Some paths may not be possible

• Cyclomatic Complexity = # decisions (diamonds) + 1

34


CalendarDay.validDayForMonth()

4a

12345678

private boolean validDayForMonth(int day) {for (int i = 0; i < MONTHS.length; i++) {

if (MONTHS[i].equalsIgnoreCase(month)) {return day >= 1 && day <= DAYS_IN_MONTHS[i];

}}return false;

}

32 4b

true

7 false

TF

T

F

T

T

F

F

8/22/2016

18


Writing White Box Test Cases

• Focus on methods – how can we test?

– Automation!

• Create a separate test class for each class that you need to test

• Exercise all paths of your class’ public or protected methods

– Create control flow diagram

– Inputs for basis set of paths

– Inputs for equivalence classes and boundary values

– Inputs for diabolical tests

37


JUnit for White Box Testing

• JUnit is an API for automating white box testing– Need to include JUnit libraries on classpath

– Focus on testing paths through methods of the class under test

• Different versions of JUnit (we’re using JUnit 4)

– Import• static org.junit.Assert.*;

• org.junit.Before;

• org.junit.Test;

– Test case methods have @Test tag before each test method

– setUp() automatically called before EACH test method• @Before tag required

38

8/22/2016

19


Assert Methods

• Assert methods provide information about the expected and actual values of a test case– assertEquals(expected, actual);

• For doubles, you will have a third argument, delta

– assertTrue(actual);

– assertFalse(actual);

– assertNull(actual);

– assertNotNull(actual);

• Could also include a leading parameter containing a message (id)

39


Example JUnit Programimport static org.junit.Assert.*;

import org.junit.Before;

import org.junit.Test;

public class CalendarDayTest {

@Test

public void testCalendarDay() {

//Test invalid month

CalendarDay cDay = null;

try {

cDay = new CalendarDay("Jan", 14);

fail();

} catch (IllegalArgumentException e) {

assertNull(cDay);

assertEquals("Invalid month Jan", e.getMessage());

}

}

}40

8/22/2016

20


White Box Testing Techniques

• Basis set testing

– Cyclomatic complexity V(G) – estimate of number of paths

– Test all program paths in a method

– Test all conditionals on true AND false side

– Control flow diagram

• Equivalence Classes

• Boundary Value Analysis

• Think diabolically

41


Failing Tests

• If a test fails, use the debugger to find and fix the problem.

42

8/22/2016

21


References

• L. Williams, “A (Partial) Introduction to Software Engineering Practices and Methods,” 2009.

• L. Williams, CSC326 Slides

• http://stuffthathappens.com/blog/2008/04/11/what-its-like-to-be-a-programmer/

43

Date post:	30-Jan-2018
Category:	Documents
Upload:	dotruc
View:	221 times
Download:	3 times

Software Testing - NC State Computer Science · PDF fileSoftware Testing Reading: Software...

Documents