An Automated Test Strategy Based An Automated Test Strategy Based on UML Diagramson UML Diagrams

F. Basanieri *, A. Bertolino *, E. Marchetti *, A. Ribolini *, F. Basanieri *, A. Bertolino *, E. Marchetti *, A. Ribolini *, G. Lombardi **, G. Nucera **G. Lombardi **, G. Nucera **

  *PISATEL LAB, IEI-CNR, Pisa, Italy*PISATEL LAB, IEI-CNR, Pisa, Italy

  ** ERICSSON LAB ITALY** ERICSSON LAB ITALY

PISATEL

Page 2

Agenda

• Cow_Suite tool • New features • Traceability

• Cow_Suite vs RUP• Guidelines

• Discussion and future work

Page 3

Objectives• Use for test planning the same UML diagrams developed

(e.g., by Rational Rose) for design and analysis• Tool-supported generation and maintenance of an

updated and strategic test plan, as the design evolves.

PISATEL

Sequence Diagra...

Sequence Diagram: Desi... Sequence

Diagram: To ...

Sequence Diagram: CheckList...Sequence

Diagram: Kernel / Kernel_Sequ... User Model Design Crit ics To Do List

CheckListKernel

GEF UML Meta-Model Argo

LibrariesArgoUML

ArgoUML vers.0.7

Designer

(from Logical View)

DecisionModel

_decision : String

DecisionModel()GetDecisions()

(from Logical View)GoalModel

_goals : String

GoalModel()getGoals()

(from Logical View)

Decision

_name : String_priority : Integer = 0

Decision()getName()getPriority()

(from Logical View)

+contains

Designer

(from Logical Vi...

Goal

_name : String = "a"_priority : Integer = 3

Goal()getName()getPriority()

(from Logical View)

+contains

: Designer

: Decision...

: Decision : GoalModel

: Goal

1. DecisionModel( ) 1.1. Decision(name, priority)

2. GoalModel( ) 2.1. Goal(name, priority)

3. GetDecisions( ) 3.1. getName( )

3.2. getPriority( )

4. getGoals( ) 4.1. getName( )

4.2. getPriority( )

Test Case getDecisions()getName()/getPriority()

Opening a saved file_decisions

NamingTest Case getDecisions()getName()/getPriority()

Opening a saved file_decisions

Naming Test Case getDecisions()getName()/getPriority()

Opening a saved file_decisions

NamingTest Case

getDecisions()getName()/getPriority()

Opening a saved file_decisions

Naming

Test Case getDecisions()getName()/getPriority()

Opening a saved file_decisions

Naming

Test Case getDecisions()getName()/getPriority()

Opening a saved file_decisions

Naming

Test Case getDecisions()getName()/getPriority()

Opening a saved file_decisions

Naming

Page 4

Cow_Suite approach

• The processing of the Cow_Suite approach starts by analysing the project description*, and proceeds in parallel with the design and analysis refinement steps.

• The reference documentation is derived by collecting together and automatically organizing the diagrams developed during the design phase without additional manual effort of formalization

[*: the internal description of the project model, given by Rational Rose in a .mdl file]

PISATEL

Cow_Suite consists of two main components: UIT (Use Interaction Test) and CoWTeSt (Cost Weighted Test Strategy)

Cow_Suite =Cowtest pluS UIT Environment

Page 5

Cow_Suite Scheme

Test caseTest case

Test case

Test case

Test caseTest case

Test case

Test case

Test case

Test case

Test case

Test case

Test case

Test case

Test case

Test caseTest Test

Procedures Procedures

listlist

Strategy

StrategySele

ction

Selecti

on

Obj 1 Obj 2 Obj 3

Method1()

Method3()

Method2()

U I TU I T

SDs selection according to the strategy chosen

Automatic UIT application for Test

cases derivation

Test specific

ation

Interaction with the user for deriving executable test procedures

Sequence Diagra...

Sequence Diagram: Desi... Sequence

Diagram: To ...

Sequence Diagram: CheckList...Sequence

Diagram: Kernel / Kernel_Sequ... User Model Design Crit ics To Do List

CheckListKernel

GEF UML Meta-Model Argo

LibrariesArgoUML

ArgoUML vers.0.7

Designer

(from Logical View)

Sequence Diagra...

Sequence Diagram: Desi... Sequence

Diagram: To ...

Sequence Diagram: CheckList...Sequence

Diagram: Kernel / Kernel_Sequ... User Model Design Critics To Do List

CheckListKernel

GEF UML Meta-Model Argo

LibrariesArgoUML

ArgoUML vers.0.7

Designer

(from Logical View)

0.250.25 0.50

0.500.50

0.7 0.3

0.30.35 0.15

Cow_SuiteCow_Suite

PISATEL

Page 6

Tree Derivation

Use Cases (UCs) and Sequence Diagrams (SDs) of the .mdl project design realized with Rational Rose are automatically organized by Cow_Suite tool in a sort of hierarchical (pseudo-)tree

Hierarchical TreeHierarchical Tree

The case study is Course Registration System from RUP documentation

PISATEL

Page 7

Tree Derivation

The users can get a complete view of the status of the functionalities specification and a detailed documentation:

•UCs and their realization•SDs associated to each specification level

•The links between Use Case and Design Model

•Reused nodes•Not linked model elements

Hierarchical TreeHierarchical Tree

PISATEL

Page 8

Use Case View vs Logical View

The Main Tree can be analyzed from two different points of view which correspond to two different typologies of test.

•Using the information of the Use Case View System test Subsystem test

•Using the information of the Logical View Subsystem test Component test

PISATEL

Page 9

Use Case View

describes the set of UCs (and scenarios) that represents the significant functionalities and the interaction of the system with the external world.

PISATEL

The Main Tree derived represents a detailed documentation of how features/system requirement are realized in the Use Case View.

•The path from a high level Use Case to a leaf (UCs or SDs) describes the refinement of the associated system feature

The Test Cases derived from the SDs of the Use Case View are specifically designed for

•System integration Test•Subsystem integration Test

Page 10

Logical ViewDescribes how the system functionalities are provided in term of static structure and dynamic collaborations of the objects.

PISATEL

•Cow_Suite tool analyses the Logical View information about:–Use Cases realization –Sequence diagrams, classes and their organization in packages and subsystems

•The tool automatically derives:The UC traceability by analyzing the package Use Case Realizations.The Design Tree: derived by analyzing the Design Model packages

Page 11

Use Case Diagram “Traceabilities”

• The “Use Case Realizations” package links the information of the Use Case View and the Logical View .

• The Use Case Diagram “Traceabilities” traces the evolution of the Use Cases in the Design Model

PISATEL

•Each Traceability Use CaseRepresents the evolution of a UC of the Use Case View into the Design ModelDetails the sub-flow of a UC with Sequence (Collaboration) DiagramsSpecifies the Classes and the relationship that participate in the realization of the UC (Class Diagrams)Optionally holds the package Design Link

Page 12

Use Case Realization analysis The Main Tree derived from the Use Case View is automatically

extended with the information of the Use Case Realizations Package

The path from a high level Use Case, through its Use Case Realization, to a leaf documents and traces the implementation of a system feature

Considering the subtree rooted in a Traceability Use Case, its SDs: contain low level information with respect to those of the Use Case

View describe in a detailed manner the collaboration of the involved

objects The Test Cases derived from these SDs are specifically designed

for• Subsystem integration Test

• Component integration Test

PISATEL

Page 13

Design Link Package• Design Link package: for every use case realization it collects

the package/s that represent the implementation of the relative UC.

PISATEL

It is an explicit link between the Main Tree and the Design tree

Page 14

Design Tree

Analyzing this tree it is possible to:

Observe the dependence relationship between packages

Know which are the reused packages and where the reuse happens

Connect to the main tree the low level SDs not explicitly linked in the .mdl file

PISATEL

Design Tree: is derived directly from the Design Model and represents the organization of the packages in a hierarchy.

By selecting the SDs of the Design Tree and applying to them the Cow_Suite test derivation strategies, a list of low level Test Cases can be derived

Page 15

Deduce the Critical Profile The effort devoted to the test of different functionalities must be

planned and scheduled considering their relative “importance” (risk-based testing, reliability guided testing, etc…).

The importance, represented by number (weight) belonging to [0,1] interval, can be assigned in two different manner:

• Interacting with Cow_Suite Tool: Following level by level the tree structure of the UCs and SDs, the user annotates the corresponding node in the tree with its weights

• Directly in the .mdl file (UCs only): During the development of the Use Case Model architects/designers assign a value, belonging to [1,9] interval, to each Use Case.

The value is memorized in the use case documentation tab with the following structure: WGH = value (For SDs or when no value is expressed for UCs the default value is 5)

PISATEL

Page 16

Weighted tree derivationThe tool : reads the assigned

values the .mdl file (if they exist)

sums the values of the node at the same level in the tree

normalizes the weights among the nodes at the same level so that their sum is 1.

PISATEL

The user can always modify the assigned weights

Page 17

Integration Stage selectionThe Cow_Suite user selects the integration stage at which the test is conducted.

The nodes that are considered by Cow_Suite tool for the subsequent evaluations are all and only those belonging to the selected integration stage plus all the tree leaves existing at higher levels.

PISATEL

Page 18

Cow_Test StrategyTwo different situations are considered in test cases planning:

1: Cowtest_ing with fixed number of tests

• A certain budget is available, which is translated in terms of number of tests, NT, to execute.

• In this case the tool select the most adequate selection of NT tests from among the (many) various test cases that could be potentially conceived.

PISATEL

Test Case

Test Case Test Case

Tes

t Cas

e

Test Case

Tes

t Cas

e

Test Case

Test C

ase

Test Case Test Case

Test C

ase

Test Case Test Case

Test Case Test Case

Test Case Test Case

Test Case

Test Case

Test Case

Test Case Test Case

Test Case Test Case

Test Case Test Case

Test Case

Test Case Test Case

Test Case Test Case

Test Case Test Case

Test Case Test Case

Test Case Test Case

Test Case Test Case

Test Case Test Case

Test Case Test Case

Test Case Test Case

Test Case Test Case

Test Case Test Case

Test Case Test Case

NTNT

Page 19

Cow_Test Strategy

2: Cowtest_ing with fixed functional coverage• A certain percentage of functionalities must be covered.

• In this case by the tool is possible to define in advance which are the functionalities to be covered and the minimum number of tests to

execute.

PISATEL

Functional Coverage = 90%

Sequence Diagra...

Sequence Diagram: Desi... Sequence

Diagram: To ...

Sequence Diagram: CheckList...Sequence

Diagram: Kernel / Kernel_Sequ... User Model Design Critics To Do List

CheckListKernel

GEF UML Meta-Model Argo

LibrariesArgoUML

ArgoUML vers.0.7

Designer

(from Logical View)

0.250.25 0.50

0.500.50

0.7 0.3

0.30.35 0.15

10

2 2

11

Minimum Number of test to execute = 16

Page 20

Select the Test Strategy

PISATEL

Cowtest_ing with fixed functional coverage.

The most critical system functionalities are

selected for reaching the fixed coverage.

Test Procedures are distributed accordingly.

Cowtest_ing with fixed number of tests.

The selection of the most suitable distribution of test procedures is developed on the basis of leaves weights.

Page 21

For each fixed integration stage, a weighted subtree is derived according to the chosen test criterion. Only the relative SDs are considered for Test Case generation.

SDs selectionSDs selection

Sequence Diagrams Selection

PISATEL

N. Test Procedures

Final weight

Page 22

Test Case Derivation

The tool uses the UIT method, to derive the Test Cases.

The method is mainly based on the SDs (particularly objects, messages and parameters)

SDs SelectionSDs Selection

PISATEL

Page 23

Sequence Diagram analysis

: Designer

: Decision...

: Decision : GoalModel

: Goal

1. DecisionModel( ) 1.1. Decision(name, priority)

2. GoalModel( ) 2.1. Goal(name, priority)

3. GetDecisions( ) 3.1. getName( )

3.2. getPriority( )

4. getGoals( ) 4.1. getName( )

4.2. getPriority( )

•Test Units: horizontal axis analysis.

•Interactions Categories and Settings Categories: vertical axis analysis.

• MessageSequences: set of Messages (in temporal order), involved in a SD, used by objects to define and elaborate specific functionalities.

PISATEL

Page 24

Test Case DerivationFor each traceable Message Sequence a Test Case is generated. It contains the list of all its Settings and Interactions Categories and their values.

The final weight of every SDs is used to automatically derive the number of Test Procedures associated to each Test Case.

Detailed Test Case Description

PISATEL

Number of associated Test Procedures

Page 25

Test Procedures Specification and derivation

Test Specification

For each identified category, we consider all its possible values and constraints (“choices”).

PISATEL

The user interacts with the tool for inserting Choices values.

A Test Procedure is automatically generated from a Test Specification, for each possible combination of choices of every category involved in a Messages Sequence.

Page 26

Cow_Suite vs RUP

• RUP is based on an iterative process characterized by four phases divided in workflows.

• A workflow is a set of activities that produces a observable result (artifacts). The involved workflows are:

– Project Management

– Requirement

– Analysis and Design

– Test

PISATEL

• Guidelines for using Cow_Suite inside RUP

• Best Practices for Cow_Suite application

25

60

15

0

0

10

20

30

40

50

60

70

80

Inception Elaboration Construction Transition

RUP Phases

Cow_Suite Integration

Page 27

Inception PhaseDuring the inception phase:• The scope of the project and the external entities with which the system

will interact (actors) are defined.

• The use cases describing the most significant interaction are also

identified

PISATEL

Workflow Requirement

1. Analyse the Problem

2. Understand Stakeholder Needs

3. Define the System

4. Manage the scope of the System

5. Refine the System Definition

6. Manage Changing Requirements

7. Classify Requirements

Workflow Analysis and Design

• Perform Architectural Synthesis

• Evaluate Use Case

• Derive Cow_tree

Workflow Test• Test Plan• Testing Cost Estimation

Page 28

Elaboration PhaseDuring the inception phase:• The analysis of the problem domain and the basic architecture are established• The project plan is developed and the possible risk elements are evaluated• A prototype is built, and the use case model is almost complete

PISATEL

Workflow Project Management• Plan for Next Iteration• Manage Iteration• Monitor ad Control Project• Analyse Cow_tree • Reevaluate Project Scope and Risk

Workflow Requirement1. Analyse the Problem2. Understand Stakeholder Needs3. Define the System4. Manage the scope of the System5. Refine the System Definition6. Manage Changing Requirements7. Classify Requirements

Workflow Analysis and Design1. Define a Candidate Architecture

2. Process UCs and SDs

3. Derive Cow_tree3. Analyse Behaviour4. Implement Traceability 5. Evaluate Traceability6. Design component/design for Real-Time7. Cow design tree8. Design databaseWorkflow Test• Testing Cost Estimation• Plan test• Derive UIT Test • Test Traceability• Design test• Implement test• Execute Test in Integration Test Stage• Execute Test in System Test Stage• Evaluate Test

Page 29

Construction Phase• Includes the development of the system remaining components and features and

the test of the entire system.

PISATEL

Workflow Project Management• Plan for Next Iteration• Manage Iteration• Monitor ad Control Project• Analyse Cow_tree • Reevaluate Project Scope and Risk

Workflow Requirement1. Manage Changing Requirements2. Classify Requirements

Workflow Analysis and Design1. Define a Candidate Architecture

2. Process UCs and SDs

3. Derive Cow_tree

4. Analyse Behaviour5. Implement Traceability 6. Evaluate Traceability7. Design component/design for Real-Time8. Cow design tree9. Design database

Workflow Test• Testing Cost Estimation• Plan test• Derive UIT Test • Test Traceability• Design test• Implement test• Execute Test in Integration Test Stage• Execute Test in System Test Stage• Evaluate Test

Page 30

Inception Phase – Workflow Requirement Classify Requirements

The requirements must be classified and grouped depending on:

criticality system importance difficulty in the development

Optional: a number in [1,9] interval representing this importance value should be assigned to each requirement/requirement set.

PISATEL

Page 31

Inception Phase – Workflow Analysis and Design Evaluate Use Case

Basing on the requirement classification an importance value ( [1,9] interval) must be assigned to each developed UC.

The importance is an absolute value characterizing the UC The value is memorized in the use case documentation

tab with the following structure:

WGH = value

PISATEL

Page 32

Inception Phase – Workflow Analysis and Design Derive Cow_Tree

Apply the Cow_Suite and derive the main tree.

Possible design weaknesses and incompleteness are

displayed.

The user profile is derived

The most important UCs, i.e. system functionalities, are

highlighted

PISATEL

Page 33

Inception Phase – Workflow Test Testing Cost Estimation

Applying Cow_Suite test strategies it is possible to : Distribute the Test Cases among the system

functionalities Know how many Test Cases are necessary to cover

the developed system functionalities Derive the completion time or “cost” (in terms of man

hours/required budget) of the Test Phase by assigning a value to each typology of Test Case

PISATEL

Page 34

Elaboration Phase – Workflow Project Management Analyse Cow_tree

Apply the Cow_Suite and derive the main tree that: Evidences the degree of detail reached in the requirement

development. Highlights the design weaknesses in the Use Case

Model/Design Model developed. The main tree integrates the information derived from the Use Case

View with those of the Logical view and can be used for the completion of the Sw Architecture Document.

Marks the most important functionalities of the developed system The final weight associated to each node represent the importance of

the implemented functionalities for the system. The correctness of the values assigned must be checked and

eventually modified

PISATEL

Page 35

Elaboration Phase – Workflow Requirement Classifying Requirements

Software architects and Program managers:

highlight the most critical UCs

select the UCs that will be refined

They assign or modify the importance value associated to

the UCs during the Inception Phase.

PISATEL

Page 36

Elaboration Phase – Workflow Analysis and Design Process UCs and SDs

• Basing on the requirement classification an importance value ( [1,9] interval) must be assigned to each developed UC. – The value is memorized in the use case documentation tab

with the following structure:

WGH = value • Optional: Insert the default values in the SDs. They

represent the values of the Interaction Categories of the UIT method. – The values are memorized in the message/class method

documentation tab with the following structure:

DEF (value1, value2…)

PISATEL

Page 37

Elaboration Phase – Workflow Analysis and Design Derive Cow_tree

Apply the Cow_Suite and derive the Main Tree.

An extension/modification of the main tree corresponds to every refinement of the Use Case/Design Model.

The evolution of the system requirement (Use case View), into diagrams of the Logical View is highlighted

The missing links between the two views and the diagram

not properly connected are shown and evidenced.

PISATEL

Page 38

Elaboration Phase – Workflow Analysis and Design Implement Traceability

o Associate to every Traceability Use Case a value representing the criticality in the realization of the associate UC, and annotate in the UC documentation tab this number.

o Link to every Traceability Use Case the SDs representing the most important scenarios and the interaction between the involved classes. In this manner every requirement will be automatically associated to the proper set of Test Cases

o Optional: create a class diagram, called the Design Link, with the list of the packages in the Design Model that represent the implementation of the UC

PISATEL

Page 39

Elaboration Phase – Workflow Analysis and Design Evaluate Traceability

Apply the Cow_Suite and derive the Main tree and the Design Tree

The Design Link evidences the evolution of the UCs in the Design model:

The Design Tree represents the degree of detail reached in the requirements (UCs) implementation

The objects or packages involved in the requirement implementation are organized in a hierarchical tree using packages dependencies.

The weights associated to the Traceability Use Cases are analyzed and possibly modified

PISATEL

Page 40

Elaboration Phase – Workflow Analysis and Design Cow Design Tree

The Design Tree: shows the hierarchy between the different system

components, the subcomponents, the reused components and the associated SDs

Represents a detailed documentation which associates to each requirement the relative system components.

following the path in the main Tree from an high UC, throw Use

Case realization, to the Design Link node. Then automatically

jump to the proper position in the Design tree and cover it down till

the leaves.

PISATEL

Page 41

Elaboration Phase– Workflow Test Testing Cost Estimation

o Considering the main tree:• Evaluate the correctness of the assigned weights

• Possibly modify the values

o Applying Cow_Suite test strategies it is possible to : • Distribute the Test Cases among the system functionalities

• Know how many test are necessary to cover the developed system functionalities

• Derive the completion time or “cost” (in terms of man hours/required budget) of the Test Phase by assigning a value to each typology of Test Case

PISATEL

Page 42

Elaboration Phase– Workflow Test Derive UIT Test

Apply the UIT method. The Test Cases derived from the main tree can be used

for system or subsystem or component test depending on the integration level chosen.

The Test Cases derived from the Design tree can be used for component/unit test.

The Test Procedures, as derived by the tool, are not directly executable ( Test Scripts). The derived Test Procedures have to be executed and logged by a test driver.

The degree of detail of the Test Set depends directly on, the design detail level and granularity of the inserted values (choices of the UIT categories).

PISATEL

Page 43

Elaboration Phase– Workflow Test Test Traceability

Considering the main tree and the Design tree:

the SDs on the path from an high level UC to a leaf (of the main tree or Design tree) produces Test Cases specialized for the testing and the verification of the requirement (system functionalities) associated to the UC.

The degree of detail of the derived Test Cases depend on hierarchical position of the SDs in the tree (the higher is the level, the lower is the degree of detail)

PISATEL

Page 44

Discussion and Future Work

PISATEL

Tool integrated with the design tool (e.g. Rational Rose)

Systematic and automated support for test planning and test case generation, based on the same UML diagrams developed for design and analysis

Test case derivation (UIT) to be validated on ERI case studies

Page 45

PISATEL