Test Design and Management in context of IFDK reference product

About this course material - This material if for general training for Test

Design and management - Material is more supportive in class room - Material will be updated during courses - FreeNest Portable Project Platform is used to

demonstrate things only in practice. This is not limiting usage for material for other training environments (I hope )

About material

Example of IFDK product concept

IFDK = Internal Flame Drum Kit

Requirement Management

Yläotsikko

Seller

Customer

Developer

Different aspects to product

QA Tester

Different requirement levels

Customer/Business/Stake

Holder Requirements

System Requirements

Design Requirements

Component Requirements

Implementation

Scalability

Stability

Performance

Security

Performance

Stress

Usabilty

V-Model in testing

Customer Requirements

Sub System Requirements

Component Requirements Component/Unit Testing

Integration Testing

System Testing

Acceptance Testing

System Requirements

Verification and Validation

Verification = Are we building the product right? Validation = Are we building the right product?

Yläotsikko

System Testing

Integration Testing

Unit Testing

Customer/Business/Stake

Holder Requirements

System Requirements

Design Requirements

Component

Requirements

Acceptance Testing

Why we need requirements from testing point of view?

Implementation

„Developer's Area“

„Test Engineers Area“

”Traditional Testing Levels”

Customer/Business Requirements

Sub System Requirements

Component Requirements

Component / Unit Testing

Integration Testing

System Testing

Acceptance Testing

System Requirements

IFDK System Verification and Validation

Use Cases

User Storys

Features

Requirements

Validation = Are we building the right product? Verification = Are we building the product right?

Architecture& Design&

Implementation

Product VALIDATION

VERIFICATION

IFDK Product Ideas

Yläotsikko

VALIDATION vs VERIFICATION?

You have vision of product What means testing in brief?

Features Test Case

Use Cases

User Storys

Product Design & Implementation

Requirements

Test Case

Test Case

Test Case

Ready to test

Testing & Quality Assurance

Can we deliver Product

Ready to Deliver

? Customer Not ready to deliver

Yläotsikko

Verification vs Validation?

Features Test Case

Use Cases

User Storys

Product Design & Implementation

Requirements

Test Case

Test Case

Test Case

Ready to test

Testing & Quality Assurance

Can we deliver Product

Ready to Deliver

? Customer Not ready

to deliver

Verification Validation

Validation = Are we building the right product? Verification = Are we building the product right?

SW Development Process (Waterfall)

Requirement Gathering/Evaluatio

n

Design

Implementation

Verification & Validation

Maintenance Error Managment Process

Task1 Task1 Task1 Task1

Mile Stone 1 Mile Stone 2 Mile Stone 3

Task1 Task1

Testing Orientation

http://en.wikipedia.org/wiki/Software_testing

White Box Testing

Grey Box Testing

Black Box Testing “SYSTEM TESTING”

Perspective

“CODE LEVEL TESTING” Perspective

Black Box vs White Box Testing

Unit Testing is White Box testing Black Box Testing

for selected

component

? ?

Yläotsikko

TESTING LEVELS

Unit/Module/Component Testing

Customer/Business Requirements

Sub System Requirements

Component Requirements

Component / Unit Testing

Integration Testing

System Testing

Acceptance Testing

System Requirements

Architecture&

Design&

Implementatio

n

Product

VALIDATION

VERIFICATION

Yläotsikko

How to Test?

What should be tested? How ?

How to test?

What should be tested? How ?

How to verify component implementation

-Unit Testing

-Code Coverage

-Branch Coverage

-Complexity

Analyse

Yläotsikko

Component /Unit Testing

Class

Attributes

Class

Methods

Class

Attributes

Class

Methods

Attributes

TestClass

TestMethodCall

Unit Test Frame Work

Test Method Call

Method Result

A=1 B=2 C=Class.TestMethodCountValues(A+B) C<>3 FAIL C=3 PASS

Implemented Class

MethodCountValues( int x, int y) z=x+y+1 Return z

Developer

Code Coverage

An analysis method that determines which parts of

the software have been executed (covered) by the

test suite and which parts have not been executed,

e.g. statement coverage, decision coverage or

condition coverage.

http://en.wikipedia.org/wiki/Code_coverage

http://www.atlassian.com/software/clover/

Yläotsikko

Branch coverage

The percentage of branches that have been

exercised by a test suite. 100% branch coverage

implies both 100% decision coverage and 100%

statement coverage.

Yläotsikko

Code Complexity

Example tool CCCC

https://wiki.jenkins-ci.org/display/JENKINS/CCCC+Plugin

http://sourceforge.net/projects/codeanalyze-gpl/?source=recommended

Yläotsikko

Integration Testing

Customer/Business Requirements

Sub System Requirements

Component Requirements

Component / Unit Testing

Integration Testing

System Testing

Acceptance Testing

System Requirements

Architecture&

Design&

Implementatio

n

Product

VALIDATION

VERIFICATION

Yläotsikko

Integration Testing

How To Test ?

http://prosentti.vero.fi/veropros_tietojen_syotto2011.asp

What should be tested? How ?

Why Integrate first? Avoid Big Bang!

HW Component

Data Base

Component/Application

10% tested

Web Service

Tested Component/Application

Yläotsikko

System Testing

Customer/Business Requirements

Sub System Requirements

Component Requirements

Component / Unit Testing

Integration Testing

System Testing

Acceptance Testing

System Requirements

Architecture&

Design&

Implementatio

n

Product

VALIDATION

VERIFICATION

Yläotsikko

How to test?

What should be tested? How ?

System Testing in Large

What should be tested? How ?

Sales DB

Room Reservation DB

DB ?

CRM DB

Application & Gateways

Application

Application & Gateways

System Acceptance Testing

Customer/Business Requirements

Sub System Requirements

Component Requirements

Component / Unit Testing

Integration Testing

System Testing

Acceptance Testing

System Requirements

Architecture&

Design&

Implementatio

n

Product

VALIDATION

VERIFICATION

Yläotsikko

What to test?

What should be tested before so customer could be so happy ? How ?

http://www.123rf.com http://www.123rf.com

Feature Pack

Project Manager

Designer/Coder

Integration Test Engineer

Test Manager

System Testing

IFDK System Testing

Feature Unit/Integration Testing

IFDK System Acceptance Testing

System Test Engineer

Error Manager

Acceptance Test Engineer

Validation

Verification

IFDK Verification/Validation (Testing Organization)

Test Automation

Product Release

Error

Database

Test

Management

Database

Report

Staus

What should be tested?

Input ?

Output?

What is Test Design?

IDEAL Implementation

REQ-X

REQ-Y

REQ-Z

REQ-O

Yläotsikko

What I should check ?

Why we need test design?

• Discuss about reasons for test design?

• Why we need to do design?

• Stupid work ! I wan’t to progress!??

Test Case Design

Feature Pack

Project Manager

Designer/Coder

Integration Test Engineer

Test Manager

System Testing

IFDK System Testing

Feature Unit/Integration Testing

IFDK System Acceptance Testing

System Test Engineer

Error Manager

Acceptance Test Engineer

Validation

Verification

UNDERSTAND YOUR TEST LEVEL

Test Automation

Product Release

Error

Database

Test

Management

Database

Report

Staus

Verify what? Using configuration?

With tools?

What is verdict?

• Verify drum track player pause mode functionality. • Do this with IFDK software release X and playing song

”Show must go on by Freddy Mercury” • Test should be done using android emulator

environment and using your hands, ears and eyes”

Add Information about case

• Pre State: • Android emulator is running • Release X is installed on emulator

• Test Case Steps: • 1. Open drum kit player application • 2. Select song ”Show must go on” • 3. Start to play • 4. Press Pause and check song is paused • 5. Check memory usage from system application • 6. Press Play • 7. jump to 4 several time (<10) • 8. Listen song to the end • 9. Exit player using ”exit button”

• End State: • IFDK Kit in main screen mode

• Test Case Id • Test Case owner/writer • Date • comments

• If Pause is working result is PASS. If Pause mode failed result is FAIL

Define pre-state Define Steps

Define end-state

What Information Test Case should contain?

Why we need test design again!

• Stupid work! This takes ages! This Test Case documentation is old as soon I have changed some implementation? Why you need to do so hard documentation? Give me a one good reason!

Checklist?

Check UI is working Working? Check color change Working? Check Counter value after 50 logins Working? Check disable mode for counter Working?

Checklist can be working great in small team!

What happens if team is disbanded to other projects? And you are new maintainer for this project?

Agile Thinking?

• We have to automate all tests!! No sense to create documentation ?

• Who does automation without a design?

Where I find sources for test design?

Test

Des

ign

Met

ho

d

Negative Testing

Requirement based test design

Defect based test design

Design based test desing

Functional test design

NonFunctional test design

Specification based Testing design

Test Case

Step Step Step

• Customer's Idea

• Brainstorm

• Intitution

• Exploratory

Test Engineer’s Daily Job?

Mixed

Write

What is Testing level

Check

different

sources &

strategy

for Test

Case

design

Functional Test Case

Non-Functional Test Cases

How to create Test Case???

Acceptance Test Case

Field Test Case

Interoperability Test Case

Conformance Test Case

Regression Test Case

Choose Case Type

WRITE A

Test Case!

Test Case Design in agile framework

Definition for test cases

Definition for Acceptance Criteria

Check list 1. Verify Test 2. Verify Test 3. Verify Test 4. Verify Test

User Story:

Tested using test

automation?

As a user I would like to use my google account for login Implementation

Done

Test Driven Development TDD in all levels!

Design Tests Implement Code

Design draft Tests Case Define Architecture &

Design?

Unit Testing

Test Driven Development and Unit Testing

Class

Attributes

Class

Methods

Class

Attributes

Class

Methods

Attributes

TestClass

TestMethodCall

Unit Test Frame Work

Test Method Call

Method Result

A=1 B=2 C=Class.TestMethodCountValues(A+B) C<>3 FAIL C=3 PASS

Implemented Class

MethodCountValues( int x, int y) z=x+y+1 Return z

Developer

DEFINE TEST

CASES FIRST!!!

IMPLEMENT CODE

AGAINST TESTS

Ideal project team and unit testing

Implemented Software

Component #1

Implemented Software Component

#3

Implemented Software Component

#4

Implemented Software Component

#2

Software Product Integration

Developer 1 Developer 2 Developer 3 Developer 4

Test Sand Box Test Sand Box Test Sand Box Test Sand Box

TESTS TESTS TESTS TESTS

Integration test engineer #2 Integration test engineer #1

Integration Test with stubs/mocs

Tested Component/Application

Log

STUB/MOCK Component

Scripted STUB Interface

Control

Configure

Simulated

Interface

Messages/Events

STUB/MOCK Component

Control Interface

Integration Testing with simulated Interfaces

Fake Spotify Server

Fake Facebook

Server

Simulated Bluetooth Interface

Simulated

MIDI Interface

Integration Test with stubs

Tested Component/Application

Log

STUB/MOCK Component

Scripted STUB Interface

Control

Configure

Simulated

Interface

Messages/Events

STUB/MOCK Component

Control Interface

Yläotsikko

in practice #1 IFDK android setup

Tested Component

Application

Activate/Control

STUB/MOCK Component

Scripted STUB Interface

Control

Configure

Simulated

Interface

Messages/Events

WEB SERVER simulating

Service interface

Control Interface

Trace/Log

Yläotsikko

in practice #2 server component testing

Tested Component

Application

Trace/Log

Activate/Control

Mock Server/Daemon

Scripted STUB Interface

Automated

Test Interface

Simulated

Interface

Messages/Events

WEB SERVER

Control Interface

Operating System

Needed

Fake Application

Junit Scripted Interface

Yläotsikko

Integration Testing

Implementa

tion

Test

Framework

Error

Database

Test

Plan

Implementa

tion

Test

Framework

Component

2

Regression Test

Testing of a previously tested program following modification to ensure that defects have not been introduced or uncovered in unchanged areas of the software, as a result of the changes made. It is performed when the software or its environment is Changed. (ISTQB Syllabus)

Regression testing reason for automation?

Test Case

#1

Test Case

#3

Test Case

#7

Test Case

#4

Test Case

#8

REGRESSION

TEST PLAN

Test Automation Solution

Execute Regression

Test when needed?

?

Export Results as

Report

Create a Test Case!

Functional?

Your

Sources

For Test

Case

Functional Test Case: ●Verify functionality of

XXXX

Non-Functional Test Cases ●Verify Stability of XXXX

●Verify Performance of

XXX

●Verify Security of XXXX

●Verify Usability of XXXX

●Verify Scalability of

XXXX

●etc...

Non-Functional?

Which Type?

●Requirement

●Use Case

●Feature

●User Story

●Customer's Idea

●Brainstorm

●Intitution

●Exploratory

Check also..... ●Correct functionality

path

●Miss-usage of

functionality

●Boundary Check

Check also..... ●Check Possiblity to

automated testing?

How to create Test Case???

Regression Test Case??

Write a Case

Write a Case

Yläotsikko

Feature Pack

Project Manager

Designer/Coder

Integration Test Engineer

Test Manager

System Testing

IFDK System Testing

Feature Unit/Integration Testing

IFDK System Acceptance Testing

System Test Engineer

Error Manager

Acceptance Test Engineer

Validation

Verification

IFDK Verification/Validation (Testing Organization)

Test Automation

Product Release

Error

Database

Test

Management

Database

Report

Staus

System Testing in Small

What should be tested? How ?

DB

Appcliation

Application Functional testing

What should be tested? How ?

Application

Application code level testing

Test Case Management Basics

• Using Testlink

• How to write test case?

• Creating a test plan

About Testlink

TestLink is a web based Test Management tool. The application provides Test

specification, Test plans and execution, Reporting, Requirements specification and

collaborate with well-known bug trackers.

Test Management – Testlink

Test

Management

Database

Feature

Implementation

TestCase

Bug Report

Error

Database

PASS

FAIL

TestCase

TestCase

TestCase

TestCase

Test Plan/Suite

SourceCode

Version

Control

TESTLINK

Test Plan Life Cycle

Test

Plan 1.0

Week 0 Week 26 Week 52

REL 0.1 REL 0.2 REL 0.5 REL 1.0

Test

Plan 2.x

Routine

Test

Needs

Still

Work

Effort

Selected Test Cases = Test Plan/Suite

Test Case 1

Test Case 2 V2.0

Test Case 3 V2.0

Test Case 1

Test Case 2 V2.0

Test Case 3 V2.0

Test Plan V 0.1

Test Case Pool and version control

Test Case 1 Test Case 2

V1.0

Test Case 3 V1.0

Test Case 2 V2.0

Test Case 2 V2.0

Test Case 2 V3.0

Test Case 4 Draft

Test Management

What we can do with Testlink Tool

Who executes test cases and when?

Test Case 1

Test Case 2 V2.0

Test Case 2 V2.0

Test Plan V 0.1

What should be tested? Priority setting

Test Case 1 Test Case 2

V2.0 Test Case 3

V2.0

P1 P3 P4

What is a target?

Test Case 1

Test Case 2 V2.0

Test Case 3 V2.0

Test Plan V 0.1

PASS / FAIL

PASS / FAIL

Executes

Executes

Executes

Product

RELEASE/BUILD

VERSION 1.1

Product

RELEASE/BUILD

VERSION 1.3

Product

RELEASE/BUILD

VERSION 1.2

Setting Configuration

Version 0.4

Version

0.2.2.2

Version

0.2.2

Test

Plan

Test

Cases

For

Features

Tested

Release/configuration

Error/De

fect

Report

Error/De

fect

Report

Error/De

fect

Report

Testlink use….

Needed Tools

TESTLINK: http://sourceforge.net/projects/testlink/

BUGZILLA: http://www.bugzilla.org/

Execution of Test Case

Test Engineer daily job ?

Yläotsikko

Test Case

SUT/DUT ”IFDK-unit”

What is generated as results from test case execution

LOG FILE

Indications Events

NOTIFICATIONS

SUT = System Under Test DUT = Device Under Test

ENVIRONMENT/ EMULATOR

TOOLS

TEST CASE

Yläotsikko

Regression Test

Testing of a previously tested program following modification to ensure that defects have not been introduced or uncovered in unchanged areas of the software, as a result of the changes made. It is performed when the software or its environment is Changed. (ISTQB Syllabus)

Test Plan & Regression

Test Case

#1

Test Case

#3

Test Case

#5

Test Case

#7

Test Case

#9

Test Case

#2

Test Case

#4

Test Case

#6

Test Case

#8

Test Case

#10

TEST PLAN 1

Test Case

#1

Test Case

#3

Test Case

#7

Test Case

#4

Test Case

#8

REGRESSION

TEST PLAN

Test Case

#5

Test Case

#9

Test Case

#2

Test Case

#6

Test Case

#10

Test Case

#9

Test Case

#10

Test Case

#11

TEST PLAN 1.1

Test Case

#12

Test Case

#13

New

TCases

Added!

Regression testing reason for automation?

Test Case

#1

Test Case

#3

Test Case

#7

Test Case

#4

Test Case

#8

REGRESSION

TEST PLAN

Test Automation Solution

Execute Regression

Test when needed?

?

Export Results as

Report

Feature Component

Project Manager

Designer/Coder

Integration Test Engineer

Test Manager

System Testing

Feature Unit/Integration Testing

System Acceptance Testing

System Test Engineer

Test Automation Engineer

Acceptance Test Engineer

Validation

Verification

Product verification/validation and test automation

Regression Testing

Integration Testing

Functional System Testing

Acceptance Testing

Unit Testing

Product Release

Component Integration Testing

A

A

A

A

A Load, Stress, performance

Heading

- Sed posuere interdum sem. - Quisque ligula eros ullamcorper quis, lacinia

quis facilisis sed sapien. - Mauris varius diam vitae arcu. Sed arcu

lectus auctor vitae, consectetuer et venenatis eget velit.

- Sed augue orci, lacinia eu tincidunt et eleifend nec lacus.

Yläotsikko

Component Design

Class

Attributes

Class

Methods

Class

Attributes

Class

Methods

Yläotsikko

Feature X * n

Feature example 1 (Invented on course 2009-2010)

Calory Counter:

Player can measure calories during training session. This can be

seen as exercise result in web service eg. Facebook application

Energy usage

Yläotsikko

SW Development Process (Agile)

User Story Y

Sprint

Task1

Task2

Task3

Sprint Sprint Sprint

Task4

Task5

Task6

User Story X

Design Implementation

Verification

Product Backlog

Design Implementation

Verification

Design Implementation

Verification

Design Implementation

Verification

User Story Z

Task7

Task8

Task9

User Story Z

Yläotsikko

Yläotsikko