99 CHAPTER 5 GENERATING TEST SCENARIOS AND TEST CASES FROM AN EVENT-FLOW MODEL 5.1 INTRODUCTION The survey presented in Chapter 1 has shown that Model based testing approach for automatic generation of test cases is currently drawing lot of attention from the software testing community. Existing Model-based testing approaches discussed in Chapter1 have extensively used UML models. As far as GUI testing is concerned, the existing approaches have considered GUI events only and have not dealt with other types of events like business events, state events or control events that happen in a system. Not much of an effort has been devoted to test functional requirements of a system more specifically for event-based systems. Existing event-based test case generation approaches have used events either from source code, Petri net model, or from GUI applications, so they require the applications or code to be made available prior to generating test cases. These applications or codes are run on existing GUI automation framework to reverse engineer an event-flow of an application. Not much of an effort has been devoted to use events captured directly from the requirements specification. The proposed approach has used events in place of UML models to generate a model of SUT. Events are used to generate event sequence-based test scenarios from an Event-flow model. A Fault Model for the SUT and test coverage criteria is defined for testing. An algorithm is also proposed to generate test scenarios from an Event-Flow model and help to detect faults described in the proposed Fault model. Regression testing is performed on modified software to provide confidence that the software behaves correctly and that modifications have not adversely affected the software’s quality. Events are also used to specify and model changes in software requirements either in terms of addition and/or modification of events or event interdependencies. The proposed approach has also applied a safe and efficient regression
Transcript
99
CHAPTER 5
GENERATING TEST SCENARIOS AND TEST CASES
FROM AN EVENT-FLOW MODEL 5.1 INTRODUCTION The survey presented in Chapter 1 has shown that Model based testing approach for
automatic generation of test cases is currently drawing lot of attention from the software
testing community. Existing Model-based testing approaches discussed in Chapter1 have
extensively used UML models. As far as GUI testing is concerned, the existing approaches
have considered GUI events only and have not dealt with other types of events like
business events, state events or control events that happen in a system. Not much of an
effort has been devoted to test functional requirements of a system more specifically for
event-based systems. Existing event-based test case generation approaches have used
events either from source code, Petri net model, or from GUI applications, so they require
the applications or code to be made available prior to generating test cases. These
applications or codes are run on existing GUI automation framework to reverse engineer an
event-flow of an application. Not much of an effort has been devoted to use events captured
directly from the requirements specification.
The proposed approach has used events in place of UML models to generate a model of
SUT. Events are used to generate event sequence-based test scenarios from an Event-flow
model. A Fault Model for the SUT and test coverage criteria is defined for testing. An
algorithm is also proposed to generate test scenarios from an Event-Flow model and help to
detect faults described in the proposed Fault model.
Regression testing is performed on modified software to provide confidence that the
software behaves correctly and that modifications have not adversely affected the
software’s quality. Events are also used to specify and model changes in software
requirements either in terms of addition and/or modification of events or event
interdependencies. The proposed approach has also applied a safe and efficient regression
100
test selection technique on event-flow model, so as to select tests from test scenario that are
deemed necessary to validate a modified software.
5.2 PROPOSED APPROACH TO EVENT-BASED TESTING
The entire event-based approach for generating test scenarios, test cases and perform
regression testing of a system consists of following steps:
1. Document events identified from requirements using proposed Event Templates.
2. Generate Event-Flow interdependencies from Event Templates.
3. Generate an Event-Flow Graph and precedence relations.
4. Generate Event sequence-based Test Scenarios and Test Cases from an Event flow
Graph.
5. Specify and model changes in software requirements if any, either in terms of addition
and/or modification of events or event transitions.
6. Apply a safe and efficient regression test selection technique on an event-flow model to
select tests from test scenario that are deemed necessary to validate the modified software.
The detailed process of documenting events from requirements using proposed Event
Templates is described in Chapter 3 and the process of creating an Event-Flow model from
Event Templates based on various possible event-flow interdependencies is described in
Chapter 4. This chapter describes the remaining steps in detail. Each of these steps is also
illustrated with a running example of a case study named ‘Automatic Production
Environment (APE)’ on a real time system taken from Real-Time Lab of Embry-Riddle
Aeronautical University [197].
101
5.3 EVENT-FLOW GRAPH AND PRECEDENCE RELATIONS In this sub section, a formal definition of an Event-Flow graph is presented based on Event
interdependency. Precedence relations are also defined among events in an Event-Flow
Graph.
5.3.1 Formal Definition of Event-Flow Graph An Event-Flow Graph represents events, their interaction in terms of causative events and
trigger vector. It has one event designated as a start event and another designated as an exit
event. A start event indicates initialization of a system i.e. once a system is turned on all
other events can get executed. An exit event indicates the completion of functionality. In an
Event-Flow graph, events are represented using circles, and dependencies between events
are represented using arrows. The proposed Event-Flow Graph is different from the one
used in Graphical User Interface (GUI) testing [223, 224]. An Event-Flow Graph used in
GUI testing represents all possible interactions among the events in a GUI component
whereas the proposed Event-Flow Graph depicts all possible event interdependencies
among all events happening in a system. Formally, An Event-Flow Graph is defined as:
An EFG is a 3-tuple <V, T, C> where
• V is a nonempty finite set of vertices representing all events in a system with two events
designated as a Start (S) node and an Exit (E) node to indicate a system startup and exit
events.
• T is a nonempty finite set of transitions. T ⊆ V * V . An event ei causes an event ej iff ei is
a causative event of ej or similarly an event ei triggers an event ej iff event ei triggers
event ej. An edge (vx,vy) ∈ T iff events vx and vy are interdependent either due to causes or
triggers relationship.
•C is a finite set of vertices representing connector nodes {OR , XOR, FORK, JOIN, NOT}
A connector node with more than one outgoing transition is classified as an Event-and split
(fork), an Event-or split or an Event-xor split node. A connector node with more than one
incoming transitions is classified as an Event-and join (join), an Event-or join or an Event-
102
xor join. An OR and XOR splits and joins are non-deterministic while an AND splits and
joins are deterministic. In an Event-Flow Graph, an Event-and split operator is represented
with a ‘fork’ label, an Event-and join operator is represented with a ‘join’ label, an OR-
split/join operator is represented with an ‘or’ label and an XOR-split/join operator is
represented with a ‘xor’ label.
5.3.2 Event-Flow Precedence Relations Events in an event-flow graph satisfy identical set of partial order relations among them.
There are various precedence relations among events that are possible in an event-flow.
Precedence relations, denoted as ‘ p ’, over a set of events SE in an event-flow graph are
defined as follows.
1. If an event Ei ∈ SE precedes another event Ej ∈ SE in an event flow graph, then, there
exists a partial order relation between two events Ei and Ej , denoted as Ei p Ej. It signifies
that event Ei occurs before event Ej in a system.
2. If an event Ei ∈ SE precedes a fork, a xor-split or an or-split and another event Ej ∈ SE
is the first event that exists in any thread originated from a fork, an xor-split or an or-split,
then Ei p Ej.
3. If an event Ej ∈ SE follows next to a join, a xor-join or an or-join and another event Ek∈
SE is the last event in any thread joining a join , an xor-join or an or-join, then Ek p Ej.
4. If an event Ei ∈ SE and another event Ej ∈ SE are two consecutive concurrent events in a
thread originated from a fork, an xor-split or an or-split, where Ei exists before Ej, then Ei p
Ej.
5. If an event Ei ∈ SE and another event Ej ∈ SE in an event flow graph are involved in a
loop (cyclic relation), then, there exists two partial order relations between events Ei and Ej,
103
denoted as Ei p Ej and Ej p Ei . It signifies that both Ei and Ej are involved in a cyclic
loop.
5.4 GENERATING EVENT SEQUENCE-BASED TEST SCENARIOS
AND TEST CASES FROM AN EVENT FLOW GRAPH This sub section first defines a fault model and presents different event paths that are
possible in an Event-flow graph. Then some of the existing test coverage criteria are
discussed followed by the test coverage criterion that is able to cover faults defined in the
proposed fault model. Subsequently, an approach of generating event sequence-based test
scenarios and test cases from an Event-Flow graph is presented.
5.4.1 Fault Model A Fault model describes certain categories of faults that a test strategy must detect [273]. A
fault model defines a small set of possible fault types that can occur in a system. Given a
fault model, the goal is to generate a test set T from an Event-Flow graph such that any
fault in a system of a type described in a fault model is guaranteed to be revealed when
tested against T. The proposed test scenario generation scheme is based on the following
fault model that covers eight different types of faults.
1. Event-XOR-Split Fault: This fault occurs at an event-xor- split node. Consider the
following case given below in Figure 5.1.
Figure 5.1: Event-XOR-Split Fault
Here, E1 is causative event of E2, E3, E4 and E2, E3, E4 are Triggered events of E1 with
an xor–spilt dependency. In an ideal state, E1 should trigger exactly one of the events in a
Trigger Vector (i.e. E2, E3, E4). In case, when E1 has occurred but none of the events in a
Trigger Vector gets triggered or more than one event gets triggered, an Event-XOR-Split
Fault occurs.
E1 E2 xor
E3
E4
104
2. Event-OR-Split Fault: This fault occurs at an event or-split node. Consider the
following case given below in Figure 5.2.
Figure 5.2: Event-OR-Split Fault
Here E1 is causative event of E2, E3, E4 and E2, E3, E4 are Triggered events of E1 with an
or–spilt dependency. In an ideal state, E1 triggers one, more or all events in a Trigger
Vector (i.e. E2, E3, E4). In case, when E1 has occurred but none of the events in a Trigger
Vector gets triggered, an Event-OR-Split Fault occurs. An Event-or split is more common
in real time systems where lots of independent events can be triggered together.
3. Event-AND-Split Fault: This fault occurs at an event-and- split node. Consider the case
as shown in Figure 5.3.
Figure 5.3: Event-AND-Split Fault
Here E1 is causative event of E2, E3, E4 and E2, E3, E4 are Triggered events of E1 with
and –spilt (fork) dependency. In an ideal state, E1 has to trigger all events in a Trigger
Vector (i.e. E2, E3, E4). The fault occurs, if all events are not triggered.
4. Event-NOT Fault: This fault occurs at an event-not node. Consider the following case
given below in Figure 5.4.
Figure 5.4: Event-NOT Fault
E1 E2 or
E3
E4
E1 E2 fork
E3
E4
E1 E2 not
105
Here E1 is causative event of E2 and E2 is Triggered event of E1 with an event-not
dependency. An Event-not connector node indicate non-occurrence of an event. The fault in
this case can occur, if E1 occurs in the presence of Event-not operator. This is a special
case of Non-event event as described by Ward and Mellor in [57].
5. Event-AND-Join Fault: This fault occurs at an event-and –join (merge) node. Consider
the following case given below in Figure 5.5.
Figure 5.5: Event-AND-Join Fault
Here, E1 is Triggered event of E2, E3, E4 and E2, E3, E4 are Causative events of E1 with
join (and-join) dependency. Causative events E2, E3, E4 all of them, when happen trigger
E1. Fault occurs if either (a) E1 is not triggered even though E2, E3, E4 have occurred or
(b) E1 occurs even though all the causative events have not taken place.
6. Event-XOR-Join Fault: This fault occurs at an event-xor- join node. Consider the
following case given below in Figure 5.6.
Figure 5.6: Event-XOR-Join Fault
Here, E1 is Triggered event of E2, E3, E4 and E2, E3, E4 are Causative events of E1 with
an event-xor join dependency. In an ideal state, exactly one of the causative events E2, E3,
E4 should happen to trigger E1. Fault occurs if more than one causative event occurs to
trigger E1.
7. Event-OR-Join Fault: This fault occurs at an event-or- join node. Consider the
following case given below in Figure 5.7.
E1 E2
join
E3
E4
E1 E2
xor
E3
E4
106
Figure 5.7: Event-OR-Join Fault
Here, E1 is Trigger event of E2, E3, E4 and E2, E3, E4 are Causative events of E1 with an
event-or join dependency. In an ideal state, one, more or all the causative events E2, E3, E4
should happen to trigger E1. Fault occurs, if none of the causative event occurs but E1 is
triggered.
8. Synchronization Fault: This fault occurs when some event takes place before
completion of all preceding events. The reason for this fault is that events are not executed
in timely manner as per the precedence relations between them. Such synchronization fault
can either be seen with sequential flow of one event following another without involving
any event operator. It can also be seen in concurrent events in which all parallel events are
not triggered together at the same time.
5.4.2 Event Paths There are various types of event paths that are possible in an event flow graph namely-:
non-concurrent path, simple basic path, cyclic basic path and concurrent path. Each of the
type is described in detail using Figure 5.8.
a) Non-concurrent event path is a sequence of non-concurrent events (that is, events
which are not executed in parallel) from the start event to an end event in an event-flow
graph, where each event in the sequence has at most one occurrence except those events
that exist within a loop. Each event in a loop may have at most two occurrences in a
sequence. Also all events satisfy the precedence relations among them. Simple Basic path
and Cyclic Basic path are two Non-concurrent event paths.
E1 E2
Or
E3
E4
107
Figure 5.8: Event-Flow Graph of APE Case Study
• Simple Basic event path- A simple basic path is a sequence of events where each
event in a path occurs exactly once and all events satisfy the precedence relation
among them. In an Event-Flow Graph shown in Figure 5.8, s 31 32 e is a
simple basic path.
• Cyclic Basic event path- A cyclic basic path is a like a simple basic path but it
involves a loop. All events in a cyclic basic path also satisfy the precedence relation
among them. In an Event-Flow Graph shown in Figure 5.8, s 29 30 29 30 e
is a cyclic basic path. In cyclic basic path some of the events may occur more than
once. In the above example, event 29 and event 30 occur more than once.
• Partial Basic event path- A partial basic path is a path that has sequence of events
starting from a start event but in terminates on another event instead of an end
event, each event in that path occurs exactly once and all events satisfy the
108
precedence relation among them. In an Event-Flow Graph shown in Figure 5.8,
s 3 11 xor 12 is a partial basic path since it starts from s and ends on event
with id 12.
b) Concurrent event path has concurrent events. It is a special case of non-concurrent
event path, which consists of both non-concurrent and concurrent events satisfying
precedence relation among them. Concurrency among events occurs due to presence of a
fork, or-split, xor-split connector nodes in an event path. For a complex and large system, it
is common to have explosion of concurrent event paths because there would be large
number of threads and every thread on an average would have large number of events. In
Figure 5.9, Event-flow graph with concurrent events E2, E3, E4, E5 is shown.
Figure 5.9: Event-Flow Graph with concurrent events
In an event-flow graph of Figure 5.9, there are six precedence relations among events: S p
E2, S pE3, E2 p E4, E3 pE5, E4 p E, E5 p E. There are six concurrent event paths
which satisfy all these relations specified above and consist of same set of events.
P1 = s E2 E3 E4 E5 E
P2 = s E2 E3 E5 E4 E
P3 = s E2 E4 E3 E5 E
P4 = s E3 E2 E4 E5 E
P5 = s E3 E2 E5 E4 E
P6 = s E3 E5 E2 E4 E
Depending on runtime environmental condition, execution thread of a system would follow
one of the concurrent event paths, but which concurrent event path would be followed, can
S fork
E2
E3 E
E5 join
E4
109
not be known at an analysis level, before execution of a system. For effective testing with
limited resource and time, the proposed approach aims to test only relative sequence of
concurrent and non concurrent events (i.e. set of precedence relations exist among these
events). For this, one representative concurrent event path from a set of concurrent event
paths is to be chosen that have same set of events and satisfy same set of precedence
relations.
The proposed test scenario generation approach selects a concurrent event path such that
sequence of all concurrent events encapsulated in that path, corresponds to breadth first
search traversal of them in an event-flow graph. This ensures that all precedence relations
among events in an event-flow graph are satisfied. One can avoid generation of entire set of
concurrent event paths by finding representative concurrent event path from an event-flow
graph. This will make the task of test case generation process easier and at the same time
avoid path explosion problem, hence, reduce testing efforts. E.g. path in Figure 5.8, start —
xor—16—20—34—E, is a concurrent path in which events 9, 12 and 15 to be executed
concurrently.
5.4.3 Test coverage criteria Test coverage criteria [274] is a set of rules that guide to decide appropriate elements to be
covered to make test scenarios and test cases adequate. In an Event-Flow Graph, there are
many, possibly infinite, paths between a start event and an end event. Structural test
strategies, also known as "path-oriented" strategies selects a subset of paths to be executed.
The selection of paths is aimed at covering a particular set of events of an Event-Flow
Graph along with the faults presented in the proposed Fault model. In this section, first
existing Test case coverage criteria are discussed in the context of the proposed Event-Flow
Graph, followed by the criterion used in generating Test Scenarios and Test Cases.
All-Paths The All-Paths criterion requires executing all possible event flow paths through
an Event-Flow Graph. This strategy is generally impossible to achieve, since an Event-
Flow Graph with cycles / loops have an infinite number of event flow paths.
110
All-L-Paths This criterion provides a restriction of All-Paths, by limiting the number of
iterations of cycles / loops in a path. The paths selected are those that do not iterate loops
more than k times, for a given integer k. The most commonly used k value is 1, where each
loop is to be executed at most one. It may be noted that presence of cycles / loops and
decision among concurrent events at connector nodes (fork, xor, or) results in path
explosion and hence, it is infeasible to consider all paths due to limited capability of
resource and time.
Event Coverage: Event coverage requires that each event in an Event-Flow Graph must be
executed at least once. Such a requirement is necessary to check whether each event
executes as expected. This kind of coverage criteria is able to detect synchronization faults
but fails to cover other faults due to the fact that an event may occur independently from
some other trigger point rather than through connector node. Hence execution of events
through connector nodes remains uncovered. E.g. in Figure 5.8, event 15 is triggered
independently by event 13, and also through a xor and a fork connector node via two other
paths. Thus, only event coverage will not be able to detect faults due to a xor-split and a
fork node (and-split). Due to these limitations, the proposed approach uses an Event Path
Coverage criterion, as an improved test coverage criterion.
Event Path Coverage Criterion- Event path coverage criterion is based on event paths
identified in an Event-flow Graph. This coverage criterion is used for testing both loop
(cycle) and concurrency among events in an Event-Flow Graph. In the proposed Event-
Flow Graph, different events may flow from special connector nodes {OR , XOR, FORK,
JOIN, NOT}. Each branch leads towards an event. Testing branches in an Event-flow
require that every branch leading to an event be exercised at least once under some test.
This ensures that Event-AND split, Event-OR split, Event-XOR split, Event-AND join,
Event-OR join, Event-XOR join faults are covered. Both the concurrent and non-concurrent
event paths are used to cover these faults.
111
5.4.4 Algorithm to generate Test Scenarios Test Scenario is a set of test cases or test scripts along with a sequence in which they are to
be executed. Test scenarios are test cases that ensure that all event flows are tested from
start to end. Every test scenarios in the proposed approach is an independent tests
consisting of a sequence of events that follow each other, where each event is dependent on
the occurrence of the previous event. Test scenarios are designed to represent both typical
and unusual situations that may occur in an application.
In our proposed approach, Test scenarios are generated using event path coverage criterion.
To do this, all event paths are obtained from a start event node marked S to an end event
node marked E in an event flow graph. An algorithm GenerateEventPaths is proposed to
generate event paths. In this algorithm, combination of both depth first search (DFS) and
breadth first search (BFS) techniques is used. An event flow graph is traversed by depth
first search technique except a portion of a graph (which contains a fork node that initiates a
set of concurrent events) with node type marked ‘fork’, following which sub tree is
traversed following a breadth breath first search traversal of a graph. A breadth first search
traversal helps to avoid path explosion problem as discussed before. It ensures that only one
representative path is chosen from a set of concurrent event paths. These event paths are not
necessarily linearly independent paths due to consideration of loop/ cycle more than once.
Algorithm: Generate_Event_Paths
Algorithm to generate event paths is divided in two parts. One part deals with generating
event paths and other deals with completion of generated paths that are partially completed.
Algorithm uses global data structure to record count of visits of every node, a stack for
DFS, a Queue for BFS and two 2-D arrays. One array called “Join Array” records different
types of join nodes traversed and other array called “Path Array” records duplet structure
<path as string, status as flag>. Path is kept as a string so that a path can be easily
manipulated and occurrence of any event node can be easily found. Current path traversed
is represented as a Path Array. Status flag can have value ‘C’ indicating a completed path
i.e. a path having both a start and an end node; ‘T’ indicating trailing path i.e. a path that
112
has an end node but a missing start node; ‘L’ indicating leading path i.e. a path that has a
start node but a missing end node; ‘B’ indicating both start and end node missing.
Completed paths are used in the second part of the algorithm to complete partially
completed paths. Global data structure to record count of visits of every node has a record
structure <event id, number of visits>
Each record whether pushed on a stack, a queue, a 2-D array that keeps join nodes, has a
triplet structure with <parent node, current node, valid flag>. Parent node keeps track of a
parent of the visited node whereas a current node keeps track of node currently visited. A
valid flag is used to indicate the status (active or deleted) of a record. When a record is first
pushed on a stack, its valid flag is set 1 and when it is popped from a stack; its valid flag is
set 0. This is to ensure that a record is not physically deleted from a stack as a stack history
is needed in the second part of algorithm that completes partially completed paths.
Construction of Event paths BEGIN
Traverse the entire event flow graph using DFS. For each node visited during
traversal, count its number of visits and update the record <event id , number of
visits>.
If <parent node, current node> is not already present on a stack, then
Push <parent node, current node, valid flag> and set the valid flag to 1.
EndIf
Update the record <event id , number of visits> by incrementing visit count by 1.
/* There is no specific order during DFS while traversing adjacent event nodes, however if an event
node has an adjacency with an end event node as well as other event nodes, then all other event
nodes are pushed on a stack first prior to end node. This is to ensure proper printing of event
paths.*/
While not empty Stack
{
113
Pop the record from a stack, set valid flag to 0 and add parent node and
current node to the current path.
If event marked by S && event marked by E are encountered then
Add the path to a path array.
Endif
If (find_on_stack(<parent node, current node>) ) then
Break the path and add the path to a path array.
Endif
If (a node has no other outgoing event node other than
connector nodes (xor, or , join, or-join, xor- join) ) then
Break the path and add the path to a path array.
Endif
If ( node is marked with E ) then
Break the path and add the path to a path array.
Endif
Update Status Flags in a path array with ‘C’, ‘L’,’T’ or ‘B’.
}
If (node is marked with join, xor-join, or-join) then
Insert node in a Join Array.
Endif
If (node is marked with fork) then
Start traversing the event flow graph in BFS using a Queue,
starting from a fork node.
/*During BFS traversal*/
Insert a triplet <parent node, current node, valid flag>in a Queue.
Add fork node to the current path.
While Queue is not empty
{
Delete the triplet <parent node, current node, valid
flag> from queue by making the valid flag 0.
114
Add node to a current path.
}
For (i=0 to length of a Join Array)
{
If (join type matches fork type) then
Delete the entry from a Join Array
EndIf
}
Start DFS traversal from the join node.
EndIf
END
Completion of Partially completed paths
This part of the algorithm aims to complete event paths that are not completed. Such paths
in a path array are either marked with ‘T’ ,‘B’, or ‘L’ in their status flag. In this process
stack and queue history is back traced to complete event paths. Paths that end in an event
node are completed first. Paths that ends in nodes like XOR/ OR event operators are not
completed. For (i=0 to length of a Path Array)
{
do
{
If the path has status flag ‘T’ or ‘B’ then
Current node=First node
Back track Stack history to search for a node that matches the
current node
If (found)
{
Fetch a parent node of the current node, add to the current
path
Current node=Parent node
}
}
While (Start node not found)
115
Set flag ‘B’ to ‘L’ and ‘T; to ‘C’
Fetch next event path.
}
For (i=0 to length of a Path Array)
{
do
{
If the path has status flag ‘L’ then
Current node=First node
Back track Path Array to search for a node that matches the
current node
If (found and status flag is ‘C’)
{
Complete the partial path by copying nodes from a complete path
starting from current node till end node marked by E
Change the status flag from ‘L’ to ‘C’.
}
If (found and status flag is not ‘C’)
{
Merge the partial path (other than those that end in a join node)
with complete path on the basis of common node
Change the status flag from ‘L’ to ‘C’.
}
}
While (Start node not found)
Fetch next event path.
}
Partial paths that end with nodes that lead to join nodes are left as such. These paths are not
completed in the algorithm as partially they are able to detect the faults due to an Event-OR
split and an Event-XOR split faults of the fault model.
5.4.5 Augmenting the Event-Flow Graph with necessary test information
In this sub section, a process is described to annotate an event-flow graph with necessary
test information. Whenever an event occurs in a system its occurrence has to be detected at
116
an analysis stage. The condition or detection criteria become annotation for an Event flow
graph. E.g. an event ‘Guest request for a room’ is detected by arrival of a ‘Booking request’
in a system. The construction of an Event-Flow Graph is based on the identification of
triggers or causes relationships among events. When an event ei triggers an event ej, an
edge between an event ei and an event ej is placed to indicate a transition. A transition is
labeled with a string that describes test criteria to detect the occurrence of an event ej. At
this level, details of each event such as actions encapsulated in an event and the inputs &
output parameters of each action are not considered in order to preserve simplicity of an
event-flow graph. This information is used at the level of testing individual events.
Information of each event node of an event-flow graph and its related event detection string
is stored separately in a data structure, called Event node Description Table (EnDT).
5.4.6 Generate Test Cases Test case in the proposed approach consists of following components- Event-sequence
based Test Scenario and Sequence of transitions to detect occurrence of events in a test
scenario. Event-sequence based Test Scenarios constitute the expected system behavior. On
the other hand, the sequence of transitions to detect occurrence of events in a test scenario
makes up the source of test input. The values for the sequence of transitions to detect events
may be identified with the help of system analyst. As a part of test case generation,
necessary values of all components of a test case are obtained from the corresponding
Event flow graph and its corresponding EnDT.
5.5 Application of Test Scenarios and Test Case Generation process This section describes application of the proposed approach to generate test scenarios and
Test Cases from an Event-Flow Graph on a case study named ‘Automatic Production
Environment (APE)’ on a real time system taken from Real-Time Lab of Embry-Riddle
Aeronautical University [197]. In this case study, after applying the proposed steps 1- 4 of
Event based OOA as described in Chapter 3, 34 events are extracted which are listed in the
Table 5.1.
117
Table 5.1: Some events from APE Case Study
5.5.1 Event Templates of Case Study APE All the events identified from Automatic Production Environment Case Study, are
documented in the proposed Event Templates. Event Templates corresponding to two
events listed in Table 5.1 are shown in Table 5.2 and 5.3.
Table 5.2: Event Template for event “Sensor 1 sense the package at start place”
1 Event ID EA05 2. Event Name
(verb phrase) Sense package at the start place
3. Description Sensor 1 sense the package at start place (State/Control Event) 4. Initiator Sensor 1 Count 5. Facilitator ALCS / Belt 1(Start place) Count 6. Affecter Package Count 7. Timestamp 8. Causative Events
(Preconditions) EA01 or Independent
9. Inputs 10. Trigger Vector Sensor 1 generate no-detect signal at start place
Sensor 1 generate detect signal at start place 11. Change-event Connection between Sensor 1 and Package
1. User places package at the start place of the Belt 1 (External Event).2. User places package at the scanner place of the Belt 1 (External Event). 3. User places package at the transition place of the Belt 2 (External Event). 4. User places package at the end place of the Belt 2 (External Event). 5. Sensor 1 sense the package at start place (State/Control Event) 6. Sensor 1 generate no-detect signal at start place 7. Sensor 1 generate detect signal at start place 8. Sensor 2 sense the package at scan place 9. Sensor 2 generate no-detect signal at scan place 10. Sensor 2 generate detect signal at scan place 11. Sensor 3 sense the package at transition place 12. Sensor 3 generate no-detect signal at transition place 13. Sensor 3 generate detect signal at transition place 14. Sensor 4 sense the package at end place 15. Sensor 4 generate no-detect signal at end place 16. Sensor 4 generate detect signal at end place 17. Motor 1 starts conveyor belt 1 18. Motor 1 stops conveyor belt 1 19. Motor 2 start conveyor belt 2 to move the package. 20. Motor 2 stops conveyor belt 2. 21. Motor 3 moves the scanner to down position and wait for 10 sec. 22. Motor 3 moves the scanner to home position and wait for 5 sec 23. Scanner down limit sensor detects down position. 24. Scanner up limit sensor detects home position. 25. Motor 4 moves pusher to extend position. 26. Motor 4 return pusher to home position. 27. Pusher forward limit sensor detects extended position. 28. Pusher back limit sensor detects home position. 29. System reports the status of system (sensors and motors) after every two seconds to web server. 30. User refreshes status earlier than 2 seconds. 31. Package is removed from sensor 4 at end place. 32. User toggle (manual / automatic) the control mode of ALCS. 33. ALCS toggle (manual / automatic) the control mode. 34. ALCS reports items track after every 5 sec.
118
Table 5.3: Event Template for event “Sensor 1 generate no-detect signal at start place”
1 Event ID EA06 2. Event Name
(verb phrase) Generate no-detect signal
3. Description Sensor 1 generate no-detect signal at start place (State/Control Event) 4. Initiator Sensor 1 Count 5. Facilitator ALCS / Belt 1(Start place) Count 6. Affecter Signal Count 7. Timestamp 8. Causative Events
(Preconditions) EA05
9. Inputs Signal type 10. Trigger Vector NULL 11. Change-event Connection between Sensor 1 and Signal
5.5.2 Generating Test Scenario The Event-Flow interdependencies are generated from Event Templates based on
information from causative events and trigger vector section of an Event Template. The
resultant Event-Flow Model is transformed into an Event-Flow Graph as shown in Figure
5.10.
Figure 5.10: Event-Flow Graph of APE Case Study
119
After applying the first part of algorithm Generate_Event_Path on the event flow graph
shown in Figure 5.10, the following event paths are obtained.
12 Sensor 3 generate no-detect signal at transition place
S3signal=nodetect
13 Sensor 3 generate detect signal at transition place
S3signal=detect
14 Sensor 4 sense the package at end place Generate S4signal=true 15 Sensor 4 generate no-detect signal at end place S4signal=nodetect 16 Sensor 4 generate detect signal at end place S4signal=detect 17 Motor 1 starts conveyor belt 1 Motar1cmd=forward 18 Motor 1 stops conveyor belt 1 Motar1cmd=stop 19 Motor 2 start conveyor belt 2 to move the
package. Motar2cmd=forward
20 Motor 2 stops conveyor belt 2. Motar2cmd=stop
21 Motor 3 moves the scanner to down position and wait for 10 sec.
Scannercmd=scan
22 Motor 3 moves the scanner to home position and wait for 5 sec
Scannercmd=home
23 Scanner down limit sensor detects down position.
Scannerdownsignal=true
24 Scanner up limit sensor detects home position. Scannerhomesignal=true
25 Motor 4 moves pusher to extend position. Pushercmd=extend
26 Motor 4 return pusher to home position. Pushercmd=home
Regression testing is performed on modified software to ensure that the software behaves
correctly and that modifications have not adversely affected the software’s quality.
124
Regression testing is an expensive activity as it can account for as much as one-half of the
cost of software maintenance. The primary goal of regression testing is to validate that
changes introduced in a program are correct and do not adversely affect the unchanged
portion of a program [278, 282 - 285].
An important difference between regression testing and development testing is that, during
regression testing, an established suite of tests may be available for reuse. During the
process of regression testing, these previously developed test cases are deployed for
revalidating a modified program. New test cases are also generated for the validation of a
program. Thus regression testing consumes large amount of time and computing resources.
In order to save cost and time, selective regression testing approaches are applied. One of
the regression-testing strategies is retest-all approach that reruns all previous tests, but this
may consume excessive time and resources. Selective retest techniques, in contrast,
attempts to reduce the time required to retest a modified program by selectively reusing
tests and selectively retesting a modified program. Next section presents application of
selective retest technique on testing using an event-flow graph.
5.6.1 Role of Regression Testing in Event-based testing
A regression test selection technique chooses, from an existing test set, tests that are
deemed necessary to validate modified software [287]. There has been significant amount
of research in designing effective regression testing techniques to reduce cost of regression
testing. These techniques can be broadly divided into two types: Code-based regression
testing and Specification based regression testing [277 – 281, 285, 286]. Since the proposed
work is on model based testing (i.e. Event-flow model is used as basis of generating test
scenarios), so it has used specification based regression testing approach.
Generally, in large and evolving software systems, specifications are frequently changed
for variety of reasons like: to correct errors in specifications, to enhance or change
functionality, to add a new functionality, to delete existing functionality. Specification
125
changes are typically made on a requirement level, i.e. system engineers make changes to a
specification by changing individual requirements. Based on the work done in [275, 276]
three types of elementary modifications are identified at the requirement level: (a) Addition
of a new requirement, (b) Deletion of existing requirement or (c) Modification of existing
requirement.
In the context of an event-based testing, addition and/or modification of events in a system
specify changes in software requirements. An Event-flow model is used in regression
testing. In an event-flow model, changes in requirements could be either due to addition
and/or modification of events, these changes are easily reflected in an Event-flow graph. In
such a case, event templates will change for those events in a trigger vector and causative
events which are affected due to modification. Subsequently, changes can also be reflected
in an event flow graph in terms of addition or deletion or changes in transitions in a graph.
In both cases, new modified set of test scenarios if any, can be easily identified from an
event-flow graph. It is possible to identify all potentially affected test scenarios from an
Event-flow graph. Next section presents various possibilities of changes that may occur in
an Event-flow graph.
5.6.2 Possibilities of changes in requirements that can generate need for Regression testing
Requirement for Regression Testing arises due to new events or event dependencies
(transitions) getting introduced in a system. It can also be in terms of changing or deleting
existing events or event dependencies. These changes are first reflected in event templates
by either introducing new templates or changing the existing event templates. Accordingly,
Trigger vector and Causative events will be changed in event templates of affected events.
Subsequently, changes are reflected in an event flow graph in terms of addition or deletion
or changes in nodes or transitions in a graph. These changes are categorized as follows:
126
a) Addition or Replacement of new event nodes as predecessor or successor of existing
nodes. The existing nodes affected can be a simple event node, an xor-node, an or-node or
a fork node. The changes in this category can be:
1. Addition of a new event node in a simple path of events leading to leaf event
node (event template with null trigger vector): In this case, depending on the
location (predecessor or successor) Trigger vector and Causative event field of the
existing and new event will change. If a new event is added as a last event in a
sequence, then a transition will be added towards an end node.
2. Addition of a new event to the Trigger vector of an xor-node, an or-node or a fork
node: In this case, a new event added will become part of trigger vector of the
parent event of an xor-node, an or-node or a fork node affected. The new event will
add a transition towards an end node, if a parent event is of an xor-node or an or-
node. In case the parent node is of a fork node, a transition will be added from the
new event towards join node.
3. Replacing entire set of Trigger vector of an xor-node, an or-node or a fork node
with a new event node. In this case, new event added will be trigger vector of the
parent event affected. Replaced set of trigger vector will become trigger of new
event.
4. Replacing some events from the set of Trigger vector of an xor-node, an or-node
or a fork node with a new event node. In this case, new event added will be trigger
vector of the parent event of an xor-node, an or-node or a fork node affected and the
replaced event will become trigger of new event.
b) Addition or Replacement of transitions among event nodes on a simple path or among
event nodes on paths at an xor-node, an or-node or a fork node. The existing nodes affected
can be simple event nodes, an xor-node, an or-node or a fork node. In case of events on the
127
path from an xor-node, all events are mutually exclusive so an edge can not be added
among events. In case of events on the path from an or-node, all events are independent so
addition of an edge will not affect the existing transitions on that path. In case of events on
the path from a fork node, addition of an edge from e1 to e2 will also involve
corresponding addition of an edge from fork node to e2 and from e2 node to join-node. In
case of two events e1 and e2 on two independent simple paths p1 and p2, addition of an
edge from e1 to e2 involves updation of trigger vector and causative events of e1 and e2
respectively.
a) Deletion of nodes or transitions among event nodes on a simple path or among event
nodes on paths at an xor-node, an or-node or a fork node. The changes in this category can
be:
1. An Event node deleted on a simple path.
2. An Xor-node is deleted when all events triggering from it are deleted, except one
event.
3. A Fork-node is deleted when all events triggering from it are deleted.
4. A Join-node is deleted when all transitions merging on it are deleted.
5. An Event node common on an xor-node, an or-node or a fork node path is deleted
when all its causative events are deleted.
6. A Transition from an xor-node, an or-node or a fork node can be deleted. In this
case, if a transition from a fork node is deleted its corresponding transition from
merging on the join-node is also deleted. If an event node becomes a leaf event
node, then a corresponding transition is added to an end node.
Modified Event-flow graph shown in Figure 5.11 shows changes with respect to possible
options described above:
1. Event node N1 added prior to 31, earlier transition is replaced by two new
transitions
2. Event node N2 added post to 33, earlier transition is replaced by two new
transitions
128
3. Event node N3 added after an xor-node, two new transitions added from an xor
node to a new event and from a new event to an end node
4. Event node N4 added after a fork-node, two transitions added from a fork node to a
new event and from a new event to a join node
5. Event node N5 replaced entire set of trigger vector of xor node at event-8, one
transitions added from an event 8 to an event node N5
6. Event node N6 replaced entire set of trigger vector of fork node at event-18, one
transitions added from an event 18 to an event node N6
Figure 5.11: Modified Event-Flow Graph of APE Case Study
5.6.3 Applying Regression Test Selection on Event-flow graph In this section, a regression test selection technique [287, 288] is applied that chooses from
an existing test set, tests that are deemed necessary to validate modified software. The
proposed approach [287, 288] is applied on an Event flow graph constructed from initial
129
requirements (Figure 5.10) and its modified version (Figure 5.11) constructed from
changed requirements. These two graphs are used to select test scenarios that cover
modified parts of an Event flow graph. The set of test scenarios that the approach selects
includes every test scenario that can expose faults in the modified Event flow graph.
Let R1 be initial requirements with an Event-flow graph G1 and T be the initial test suite
consisting of various test scenarios generated from algorithm presented in section 5.5.4.
When R1 is tested with a test scenario ti from test suite T, it records branch trace from an
Event-Flow Graph that consists of branches taken during this execution. An edge (N1, N2)
in G1 is traversed by ti if and only if , when R1 is executed with ti events corresponding to
nodes N1 and N2 are sequentially executed at least once during the execution. This
information is called edge trace for ti on R1.
A test history with respect to T is constructed by gathering edge trace information for each
test scenario in T and representing it such that for each edge (N1, N2) in G1, the test history
records all test scenarios that traversed (N1, N2). This representation is done in O (e*t) bits
where e is the number of edges and t is the number of test scenarios in T. Table 5.6 shows
Test History where for every edge (N1, N2) a bit vector is shown of size t in which kth bit in
bit vector is set, if and only if test scenario k in T traversed the edge (N1, N2) in G1. E.g, In
Figure 5.10 an edge (16—20) is traversed by test scenario 2, 3, and 6 so 2nd , 3rd and 6th bit
is set in the bit vector This selective retest technique reduces the cost of Regression Testing
by selectively reusing existing tests and identifying changed portion in the modified version
of an Event-Flow Graph that should be tested.
To find tests that are modification-traversing for original requirements R1 and modified
requirements R2, both Event-Flow Graphs G1 and G2 are synchronously traversed
beginning with a start node S, looking for pairs of event nodes N1 and N1’ whose labels are
not lexicographically equivalent. A pair wise comparison of an event node compares labels
on the first node in G1 to corresponding second node in G2 and so on. If a pair wise
comparison of event nodes reaches a first pair of nodes N1 and N1’ whose labels are not
130
lexicographically equivalent, then two event nodes N1 and N1’ are said to be non-
equivalent but the edge trace of the two Event-Flow Graphs are equivalent up to and not
including N1 and N1’. In such a case, bit vector is scanned in the Table 5.6 to search for an
edge consisting of parent of N1 and N1’ itself. All the test scenarios corresponding to bit
positions where value is 1 in the bit vector are selected for retest. For example, on
traversing Event flow graphs constructed from initial requirements (Figure 5.10) and its
modified version (Figure 5.11), it is determined that both start nodes have same labels so
successors of respective start nodes are compared i.e. (S-31) in G1 and (S-N1) in G2. Their
labels differ along pair of identifying labeled edges, i.e. 31 and N1 are not lexicographically
equivalent, so edge (S-31) is used to select test scenarios that have used this edge. Based on
the Test History, a Test Scenario with id 11 is selected. Similarly, successive comparisons
are done with pairs (2, 2), (8, 8) and (xor, N5). In these cases, all pairs have
lexicographically equivalent labels except pair (xor, N5) whose labels differ along pair of
identifying labeled edges. So edge (8-xor) is used to select test scenarios that have used this
edge. Based on the Test History, Test Scenarios with id 6 and 7 are selected. Thus test
scenario 6, 7 and 11 are selected as modification-traversing. This process is repeated for on
all test scenarios to get the final set of test scenarios to be retested for Regression Testing. Table 5.6: Test History
