Date post: | 18-Oct-2014 |
Category: |
Documents |
View: | 605 times |
Download: | 0 times |
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
1/41
SA-based Code SA-based Code TestingTesting
Henry MucciniComputer Science Department
Universita’ dell’Aquila – Italy
([email protected])[http://www.HenryMuccini.com]
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
2/41
My ResearchMy Research
● Ph.D. Thesis on “Software Architecture for Testing, Coordination Models and Views Model Checking”, year 2002.
● Software Architecture and Coordination● Software Architecture and Model Checking
● Software Architecture and Testing
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
3/41
● Brief introduction on Software Architecture and Testing
● SA-based Conformance Code TestingSA-based Conformance Code Testing ● SA-based Regression TestingSA-based Regression Testing● PLA-based TestingPLA-based Testing
● Ongoing and future work● Some references
Talk OverviewTalk Overview
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
4/41
Software Architecture (SA)Software Architecture (SA)
● 1992 and 1993: – SA is recognized as an independent area of research;
● 1993-1995: – SA described through box and line, informal, diagrams;
● 1995-2002: – Architectural Description Languages (ADLs) are introduced to
formally describe SA;● 1997-2003:
– more interest on dynamic aspects of SA;– SA used in academia and industry;– SA and analysis– SA recognized as a valid tool to describe complex, concurrent, real
systems;
The History: [PhDThesis, Ch2] [PhDThesis, Ch2]
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
5/41
In general terms…In general terms…● Describes (in a formal language) how a system is
structured into components and connectors…
– Components
● computation
– Connectors● interaction
– Ports, Interfaces, …
… how these
components interact
SA Structure (topology)
SA Dynamics (behavior)
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
6/41
SA dynamicsSA dynamics
● A model of Software Architecture behavior:– Labeled Transition System (LTS)– Finite State Machine – Petri Nets– …
● Given the SA formal description, the model can be automatically generated; the dynamics is expressed in terms of components interaction via connectors
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
7/41
Motivations on SA-based Analysis Motivations on SA-based Analysis ● For Analysis: [PhDThesis] [PhDThesis]
– SA management is expensive and time consuming ● maximize the benefits
– Analyze SA as much as possible
– SA and Testing – SA and Coordination models– Model checking SA
– SA and Performance Analysis– Deadlock detection on SA-based specification– SA and Security– Refining SA – SA-based development processes– ADL- based analysis of SA
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
8/41
Module Unit Integration SystemModule Module
M
M
M
M
Structural FunctionalCode LevelOriented to SyntaxUnit Tests
Functional propertiesFormal specifications
TestingTesting
● Testing is the process of executing a program with the purpose of – finding errors or defects– increasing the confidence in the proper functioning of the
software (dependability).● Not exaustive approach…
… based on the selection of an adeguate set of tests
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
9/41
Architecture-based Integration Architecture-based Integration and Conformance Testingand Conformance Testing
● Integration Testing: unit tested components are integrated to build complex systems, and – Integration Testing is aimed at exposing problems in the
interfaces and interactions between the system components
● Functional: focus on the functional requirements● Architectural: information for testing is extracted from
the Architectural specification... ● Down to the Code: … and propagated down to the
Code... Conformance Testing
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
10/41
The main ideaThe main idea
ClientA
ClientB
ServerMaster
ServerSlave
Recovery
ClientC
SA CodeXClient
ClientA ClientB
ClientC
Class Client--------------------------
● Goal:– Derive test cases– using software architectural description– run test cases on the Code
Identify SA-Tests Apply the Tests
Transform SA-Tests
Into Code-level
tests
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
11/41
Working DirectionsWorking Directions
1 - SA-based Conformance Code Testing(Joined work with Antonia Bertolino and Paola
Inverardi)
2 - SA-based Regression Testing– Testing C2-style architectures
(Joined work with Marcio Dias and Debra Richardson)
3 - PLA-based Testing (Joined work with Andre’ van der Hoek)
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
12/41
A
B
A
B
D
A
B C
D
Test ClassesTest Execution
Obj1
m1
Obj5Obj3Obj2
m6
m3
ObjN
Objk
Obj8
ObjL
m1m6
m3
m1
m2
m2
îe¤íiÛìoö
1act
ïî
ïíS
ì
3act
{2act{2act
step4
[ICSE01][ICSE01]
LTS Model
201964
1
144
1
12
9
1 1332
2019532
1172
4
12
1
96 1
113
2
75
2
6
4
1
5
32 1
0
8
7
6
5
321
0
4
2926
2
2822
20
12
10
13
11
3
9
36
35
33
32
34
15
18
17
16
3114
13
16
37
292830
24
34
27
33
22
Step2
Abstraction
step3
ALTS Model
A
B C
D
Components:User, Router, Server, TimerUser, Router, Server, Timer
Connectors:AlarmUR, AlarmRS, AckSR,AlarmUR, AlarmRS, AckSR,
AckRU, Check, Nofunc, Clock AckRU, Check, Nofunc, ClockComponents Behavior:
…, …, ……, …, …Connectors Behavior:
…, …, ...…, …, ...
SA Specification
Step1
modeling
[ICECCS’97][ICECCS’97]
[ICSE00][ICSE00]
1. SA-based Conformance Code Testing1. SA-based Conformance Code Testing [PhDThesis, ch. 4][PhDThesis, ch. 4]
[BookChapter03][BookChapter03]
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
13/41
Step1: Formal SA specificationStep1: Formal SA specification
Molecule syntax:Molecule: Process| Operation| M.MProcess: User, Router, Server, TimerChannel: check, alarmUR, alarmRS,ackRU, ackSR, nofunc, clockOperation: i(Channel), o(Channel), ...
InitialState (S0):Multiset of Molecules: m1, m2, ..., mn
Rules:m1, m2, ..., mk m1', m2', ..., ml'
User1
Router Server
Timer
AlarmUR AlarmRS (c)Check1
Nofunc
Clock
AckSR (c)
AckRU1
User2
AlarmUR1
AlarmUR2
Check2
Check
AckRU2
● Architectural Description Language (ADL) that produces a Labeled Transition System (LTS)
● Chemical Abstract Machine (CHAM) [ICSE00][ICSE00] ● Finite State Process (FSP) [ICSE01][ICSE01]
● Dynamics in terms of Component interactions
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
14/41
20
1911
10
6
3
6
1614126
3
63
4
5
3
97
5
11
7
6
11
96
5
4
3127
5
4
117
6
4
97
65
20
196
4
1
14
4
1
12
9
1 13
32
20
19
532
117
2
125
4
3
11
6
43
96
53
6
5
4
7
4
12
1
9
61
11
3
2
7
5
2
65
4
3
6
4
1
5
3
2
2 1
0
10
11
8
7
6
5
4
321
0
9
4
2926
2
28
2220
12
10
13
11
3
9
36
35
33
32
34
15
18
17
16
31
14
13
12
16
2051439051
37
22
21
20
2928
27
26
25
24
23
30
19
25
24
21
23
20
22
19
24
34
27
33
22
44
43
42
48
46
4140
39
38
43
40
41
42
39
38
45
43
38 49
72
71
159 107
67
107
117
77
69
68111
113
107
112
109
70
43
23
The The AArchitecture-level rchitecture-level BBehaviorehavior
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
15/41
● Architectural Views [Kruchten]
– Flow view– Component Based view– Concurrency view– ...
● Obs_Functions to view the system only from a – perspective that is relevant for testing– Obs : L D {}
● Abstract LTS (ALTS)● Minimization and Reduction (trace-equivalence)
Step2 : The Step2 : The AbstractionAbstraction
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
16/41
F R n o
F R n o F R n o
T R a ck 1T R a ck 2
F R n o
T R a ck 2T R a ck 1
F R a 1F R a 2
F R a 2F R a 1
D
A
BC
Flow view:
ComponentBased View
Alarm Flow
Check Flow
Clock Flow
User Component
Router Comp.
Server Comp.
R e c e iv e A c k 1
R e c e iv e A c k 2R e c e iv e A c k 1
S e n d A la r m 1S e n d A la r m 2
S e n d A la r m 2S e n d A la r m 1
A
B C
D
R e c e iv e A c k 2
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
17/41
Step3 - Architectural Test ClassStep3 - Architectural Test Class
● ALTS Path Coverage● Each complete path corresponds to an Architectural Test Class
– McCabe path coverage criteria
– FSP hiding operators
...
R e c e iv e A c k 1
R e c e iv e A c k 2R e c e iv e A c k 1
S e n d A la r m 1S e n d A la r m 2
S e n d A la r m 2S e n d A la r m 1
A
B C
D
R e c e iv e A c k 2
R e c e iv e A c k 1
R e c e iv e A c k 2R e c e iv e A c k 1
S e n d A la r m 1S e n d A la r m 2
S e n d A la r m 2S e n d A la r m 1
A
B C
D
R e c e iv e A c k 2
R e c e iv e A c k 1
R e c e iv e A c k 2R e c e iv e A c k 1
S e n d A la r m 1S e n d A la r m 2
S e n d A la r m 2S e n d A la r m 1
A
B C
D
R e c e iv e A c k 2
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
18/41
Step4 - Testing the SoftwareStep4 - Testing the Software ImplementatioImplementationn– How are the LTS paths implemented by the Code?
● We can test whether the system as-built implements this architectural behavior
Comp1 Comp4Comp3Comp2
act1
act2
act3
act1
Obj1
m1
Obj5Obj3Obj2
m6
m3
ObjN
Objk
Obj8
ObjL
m1
m6m3
SA test Code test
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
19/41
ApplicationsApplications
● Tele Remote Medical Care System (TRMCS) [ICSE2000,ICSE2001]
● The Whiteboard case study [BookChapter]
● The Elevator case study [Submitted]
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
20/41
● LTS can be automatically derived from an Architectural specification (LTS generator Tool, LTSA Tool);
● ALTS can be semi-automatically generated (FC2Tools + The Abstractor Tool)
ToolTool SupportSupport
LTS generatorLTSA tool
LTSAbstractor
SA formal specification LTS ALTS
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
21/41
● Step1:– SA specification and modelization– State explosion problem… completeness – Considerations:
● we used Cham and FSP
● Step2:– Observation and Abstraction– Considerations:
● It is an empirical task, based on the software architect experience● Classification of observations could help● Methods similar to SAAM or SCENT, in which architectural information
is empirically captured, could help in this task● The ALTS construction has been automated using the
CAESAR/ALDEBARAN tool
Topics of interest and Topics of interest and ConsiderationsConsiderations
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
22/41
● Step3:– Path coverage– Considerations:
● McCabe's test technique seems a reasonable coverage criterion ● it may be automated
● Step4:– Traceability and development process– Deterministic and non deterministic Testing– Considerations:
● it is the most difficult part● relating SA specification to code● key concepts: traceability, development process
TopiTopics of interestcs of interest and and ConsiderationsConsiderations
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
23/41
Step4: Step4: Some ConsiderationsSome ConsiderationsMapping problems:
● It is not so obvious which classes and methods implement an architectural functionality
● More sequences of method calls can implement the desired architectural behavior
● SA description is abstract and hides functionalities and objects defined at the code level
● The SA model usually describes only the expected behaviors, while the code also catches exceptional ones
● Test execution: nondeterministic or deterministic [CarverTai_TSE98] [CarverTai_TSE98] approach
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
24/41
User2User2User1User1 RouterRouterAlarm1Alarm1
Alarm2Alarm2
Ack2Ack2
Ack1Ack1
SA behavior Source CodeSource Code
class MasterRouter{ ServerConnection allarm; ServerConnection okFunction; // the services ReceiveUserAllarm serviceReceiveUserAllarm; ReceiveUserOkFunz serviceReceiveUserOkFunz; String name,serverName; static public PrintWriter user_router_ok_funz; static public PrintWriter user_router_alarm; ...
??????
drivesdrives
drivesdrives
SA (topology and model)
Design and Source Code Def.Design and Source Code Def.
Source Code Source Code (abstractions)(abstractions)
SA (model)
SA (model)
Source CodeSource Code
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
25/41
● Motivations and Goals:
– To reduce the testing effort during architecture or code maintenance
– To handle the “architectural drift”– To maximize benefits in using Software
Architecture
2. SA-based 2. SA-based RegressionRegression Testing Testing [Submitted][Submitted]
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
26/41
● Test modified software and provide a certain confidence that no new errors are introduced into previously tested code.
● Given program P and P’, T is a test suite for P, regression testing techniques:– provide a certain confidence that P’ is still correct
with respect to a set of test T’, which is a subset of T;
– Helps to identify new test cases for T’.
Regression TestingRegression Testing
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
27/41
● Select T’, subset of T and relevant for P’;● Test P’ with respect to T’;● If necessary, create T’’, to test new
functionality/structure in P’;● Test P’ with respect to T’’;● Create T’’’, a new test suite and test history;
Regression Test Selection Regression Test Selection techniquetechnique
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
28/41
Project GoalProject GoalSoftw are Architecture SA1
(version 1)
Code1(version 1)
Code2(version 2)
Thecode
evolves
Softw are Architecture SA2(version 2)
The SA evolves
Goal1: Test thenew code
conformance tothe initial SA
Architectural Test Cases(ATCs)
are extracted to test thesource code
Component AComponent C
Component B
Component AComponent C'
Component B
Component Y
Goal2: Test thenew SA and
code, reusingprevious results
Code
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
29/41
From Traditional to SA-based Regression From Traditional to SA-based Regression TestingTesting
Build GP',the graph
for P'
Run T over Pand Evaluate Results
Create aTest History
Select T' for P'
Compare GPw ith GP'
SA-based Testing
SA-basedRegression Testing
ExtractingSA-level Test Cases
Mapping SA-level testsinto Code-level tests asa w ay to select T for P
SA-spec for S
a: S tep 0
b: S tep 1
c: S tep 2
e:S tep 4
2) S tep B
3) S tep C
4) S tep D
TestingCriterion
d: S tep 3Build GP,the graph
for P
1) S tep A
All such stepsallow the
selection of atest suite T for P,driven by the SA
Build GP,the graph
for P
Build GP',the graph
for P'
Select T for P
Create aTest History
Select T' for P'
Compare GPw ith GP'
Testing
Regression Testing
Run T over Pand Evaluate
Results
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
30/41
Initial experimentInitial experiment● The approach has been applied to the Elevator case study
– Architecture in the C2 style– Implemented using the C2 framework
● Tool support:– The SA is formally specified using the C2 specification language
and FSP. – A behavioral model of the system is automatically produced from
the FSP specification, using the LTSA tool.– The abstraction over the LTS behavioral model is realized again in
FSP. – The mapping between architectural tests and code-level tests is
provided for by the C2 Framework. – Test cases are run over the code using the Argus-I environment.– The selective regression testing step is supported by the DejaVOO
tool.
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
31/41
3. PLA-based Testing3. PLA-based Testing [Tacos 2003][Tacos 2003]
● A product line architecture precisely captures, in a single specification, the overall architectureoverall architecture of a suite of closely-related productssuite of closely-related products [Bosch2000]
● A PLA explicitly specifies: – i) elements that are present in all productspresent in all products, – ii) elements that are optionaloptional, and – iii) elements which may be incorporated in one of
many forms (variantsvariants)
● Whereas a “regular” architecture defines the structure of a single product, a product line architecture (PLA) defines the common architecture for a set of common architecture for a set of related productsrelated products [Bosch2000]
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
32/41
Testing PLATesting PLA
● A new challenge: – how to deal with optional elements or with the
magnitude of productsmagnitude of products that may be present?
● The goal of this paper is to highlight the challenges and opportunities for software challenges and opportunities for software testing of PLAstesting of PLAs
● We believe that existing mechanismsexisting mechanisms with which SAsSAs are tested can be adaptedcan be adapted to PLAs
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
33/41
PLA examplePLA example
foo
barbar
goop
bar foobar
mandatory
optional
variant
variant
In total, twenty-four different product architectures can be formed.
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
34/41
Testing PLA- Unit TestingTesting PLA- Unit Testing
● SA:– Each SA component need to be unit tested
● PLA:– allall components should be unit tested as well
● Including each optionaleach optional component and each varianteach variant of a variant component
– However,the order in which they have to be tested can be adjusted based on “prioritypriority”.
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
35/41
Testing PLA – Integration Testing PLA – Integration TestingTesting
● SA:– components and connectors are combined together
according to the architecture configurationconfiguration
● PLA:– no single architectural configuration exists
– Possible solutions:● Iterative build-upIterative build-up integration approach: powerful but
expensive● big bangbig bang: limited but “effortless”
– Core-first + big bang approachCore-first + big bang approach● Integrated with “heuristics” to test only particular combinations
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
36/41
Testing PLA – Conformance Testing PLA – Conformance TestingTesting
● SA:– conformance testing has been used to detect detect
conformance errors between the SA and its conformance errors between the SA and its implementationimplementation.
● PLA:– an implementation an implementation II conforms to its PLA when conforms to its PLA when:
● I conforms to a single PA● I conforms to all the possible PAs out of the PLA● I conforms, at least, to all the constraintsconstraints and functionalitiesfunctionalities
associated to the mandatory, core elementscore elements of the PLA
– a product architecture PA conforms to its PLAa product architecture PA conforms to its PLA
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
37/41
Testing PLA – Regression Testing PLA – Regression TestingTesting
● SA:– If a new version P1v2 of an implementationimplementation P1v1 is
produced, regression testing techniques can be used to test the conformance of P1’ to the initial SAto the initial SA
– If a new version (SAv2) of an architecturearchitecture SAv1 is produced, SAv2 test cases may be selected reusing SAv1’s test cases.
● PLA:– PLA evolutionPLA evolution: PLA that becomes PLA’
– PA becomes PA’PA becomes PA’
– Code becomes Code’Code becomes Code’
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
38/41
Future WorkFuture Work
SA-based Code SA-based Code TestingTesting
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
39/41
● SA-based Conformance Code TestingSA-based Conformance Code Testing – Testing and Formal Methods
Integration with the TGV Integration with the TGV [TGV] [TGV] (Test Generation (Test Generation and and Validation) environment used to test Validation) environment used to test formal formal specificationsspecifications
– Testing C2 style architectures
C2 framework and Dradel C2 framework and Dradel [ongoing work]
– Integration with XADL, Menage, Behavioral modelIntegration with XADL, Menage, Behavioral model
● SA-based Regression TestingSA-based Regression Testing– Implement the second goalImplement the second goal
● PLA-based TestingPLA-based Testing– Introduce an approach for PLA-based testingIntroduce an approach for PLA-based testing– Integration with XADL, Menage, Behavioral modelIntegration with XADL, Menage, Behavioral model
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
40/41
Integration with XADL, Integration with XADL, Menage, Behavioral modelMenage, Behavioral model
● SA specified in XADL – with an additional behavioral specification
● The XADL architecture is implemented by using the new C2 Framework – May be something different
● Menage features are used to help the regression testing steps
● The testing capability is embedded into Menage
d
USC, October 2003 - Henry MucciniSoftware
Engineering and Architecture Group
i
41/41
Some ReferencesSome References● [ICSE2000][ICSE2000] A. Bertolino, F. Corradini, P. Inverardi and H. Muccini. "Deriving Test Plans from
Architectural Descriptions". In ACM Int. Conf. on Software Engineering (ICSE2000), pp. 220-229, Limerick (Ireland), June 2000.
● [ICSE2001][ICSE2001] A. Bertolino, P. Inverardi and H. Muccini. "An Explorative Journey from Architectural Tests Definition downto Code Tests Execution". In IEEE Int. Conf. on Software Engineering (ICSE2001), Toronto, May 2001.
● [PhDThesis][PhDThesis] H. Muccini. Software Architecture for Testing, Coordination Models and Views Model Checking, PhD thesis, year 2002.
● [Tacos 2003] [Tacos 2003] H. Muccini, A. van der Hoek. "Towards Testing Product Line Architectures“In Proc. of the ETAPS2003 workshop on "Test and Analysis of Component Based Systems" (Tacos), Warsaw, Poland, April 2003. In Electronic Notes of Theoretical Computer Science, Vol. 82, N. 6 (2003).
● [BookChapter][BookChapter] A. Bertolino, P. Inverardi and H. Muccini. “Formal Methods in Testing Software Architectures”. Chapter in Formal Methods for Software Architectures, SFM-03:SA Lectures, Eds. M. Bernardo, P. Inverardi, LNCS 2804, 2003, p. 124-149.
● [Submitted] [Submitted] H. Muccini, M. Dias and D. Richardson. Software Architecture-based Conformance and Regression Testing. Submitted for publication.
● [ongoing work] [ongoing work] H. Muccini, M. Dias. Systematic Testing of C2 style Software Architecture. Under Submission for publication.
● [CarverTai_TSE98][CarverTai_TSE98] Carver, R.H., Tai, K.-C.Use of Sequencing Constraints for Specification-Based Testing of Concurrent Programs. IEEE Trans. on Software Engineering 24, 6 (June 1998).
● [Kruchten][Kruchten] P. Kruchten. Architectural Blueprints - The “4+1” View Model of Software Architecture. IEEE Software 12 (6), November 1995, pp. 42-50.
● [TGV] [TGV] Fernandez, J.-C., Jard, C., Jeron, T., Nedelka, L., Viho, C.: An Experiment in Automatic Generation of Test Suites for Protocols with Verification Technology. Special Issue of Science of Computer Programming, Vol. 29, pp. 123-146, 1997.