ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing
Nikolay Pakulin
Contributors:Ina Schieferdecker
Jens Grabowski
TTCN-3 User Conference2012, Bangelore, India
Agenda
STF 430: TTCN3 Evolution TTCN3 Overview Advanced TRI Continuous Signals
STF 433: TTCN3 Tool Conformance Testing Motivation Methodology Challenges Requirements-driven Test Development
Slide 2ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing
TTCN-3 Evolution
TTCN-3 User Conference2012, Bangelore, India
Nikolay Pakulin
on behalf of STF430
TTCN-3 User Conference2012, Bangelore, India
STF administrative data STF 430: April 2011 – December 2011 (78 mDays)
Members: Gyorgy Rethy, Ericsson Jens Grabowski, University of Goettingen Ina Schieferdecker, Fraunhofer FOKUS Jacob Wieland, Testing Technologies
STF 430 sessions in 2011 Session 1: 24-27 May Session 2: 27 June - 1 July Session 3: 26 - 30 September Session 4: 28 November - 2 December
Slide 4ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing
TTCN-3 User Conference2012, Bangelore, India
IDL XSDLanguagemappings ASN.1
ExtensionsAdvanced
parameteri-zation
Behaviourtypes
Staticconfiguration
Documentationt3doc
TTCN-3 Structuring: Imports, Groups, Attributes
Corelanguage
TTCN-3 Technology Overview
TTCN-3 Behaviour
TTCN-3 Data
C C++TRI/TCImapping Java XML C#
Real-timesupport
Continuoussignalsupport
AdvancedTRI
Slide 5ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing
TTCN-3 User Conference2012, Bangelore, India
TTCN-3 language evolution Tasks
Resolution and implementation of CRs
Prepare the drafts for ETSI publication:Parts 1 and 4 to 10 of the ES 201 873 seriesThe 4 extension packages (ES 202 781, ~782, ~784,
~785)
Cooperation with STF 160 on language requirements of LTE testing
Two new language extension packages Advanced TRI Continuous signal support
Slide 6ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing
TTCN-3 User Conference2012, Bangelore, India
The version forest Approved new versions
TTCN-3 Part-1: Core Language V4.4.1 (44 CRs) TTCN-3 Part-4: Operational Semantics V4.4.1 (1 CR) TTCN-3 Part-5: TTCN-3 Runtime Interface V4.4.1 (2 CRs) TTCN-3 Part-6: TTCN-3 Control Interface V4.4.1 (8 CRs) TTCN-3 Part-7: Using ASN.1 with TTCN-3 V4.4.1 (1 CR) TTCN-3 Part-8: Using IDL with TTCN-3 V4.4.1 (2 CR) TTCN-3 Part-9: Using XML with TTCN-3 V4.4.1 (12 CRs) TTCN-3 Part-10: TTCN-3 docum. tags V4.4.1 (1 CRs) TTCN-3 Extension: Behaviour Types V1.2.1 (3 CRs) TTCN-3 Extension: Configuration and deployment V1.2.1 (1 CR) TTCN-3 Extension: Real-Time and Performance Testing V1.2.2 (2 CRs)
Not republished: TTCN-3 Extension: Advanced Parameterization V1.3.1
New: TTCN-3 Extension: Advanced TRI V1.1.1 TTCN-3 Extension: Continuous signal support V1.1.1
Slide 7ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing
TTCN-3 User Conference2012, Bangelore, India
New features in Part 1
5417: Support of parametrized map/unmap var MyConfigType cfg:={ option := 1, lock := false};...map(mtc:Port4, system:PCO2) param (cfg);
5262: Partially constrained structured types 5938: Type restriction by template list
type MyRecord MyRecordSub5 ( {f1 := *, f2 := "user", f3 := pattern "password|Password" }, { f1 := (1 .. 10), f2 := "User“, f3 := ? })
5937: Generalized annotation of attributes to declarations/members
Slide 8ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing
TTCN-3 User Conference2012, Bangelore, India
New: Advanced TRI 1(3)
Historically, TTCN has been used to test communication protocols which typically use encoded messages. This has been reflected in the TRI SA and TCI CD design of TTCN-3 by encoding and decoding messages to/from bitstrings.
However, TTCN-3 also supports signature-based communication for which the transformation of objects into bitstrings and vice versa is cumbersome.
Therefore, an alternative API is being proposed along which TTCN-3 values can be directly passed to/from the SUT.
Old and new TRI can be used in combination!Slide 9ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
TTCN-3 User Conference2012, Bangelore, India
New: Advanced TRI 2(3)
Redefinition of TRI Data Interface TriMessageType use Value TriAddressType use Value TriAddressListType use ValueList TriSignatureIdType unchanged TriParameterType unchanged (not being
used) TriParameterPassingModeType
unchanged (not being used)
TriParameterListType TciParameterListType (by that, the TciParameterPassingMode and Values for parameters will be used)
TriExceptionType use Value
Slide 10ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing
TTCN-3 User Conference2012, Bangelore, India
New: Advanced TRI 2(3) Changes to Message-based communication operations
triSend xtriSend triSendBC xtriSendBC triSendMC xtriSendMC triEnqueueMsg xtriEnqueueMsg
New Value xtriConvert(in any value, in Type typeHypothesis)
This operation shall be called whenever the TE has to convert a value. The TE might convert immediately after reception of the value, or might for performance considerations postpone the conversion until the actual access to the value.
Changes to Procedure-based communication operations triCall xtriCall etc.
Changes to Miscellaneous operations triExternalFunction xtriExternalFunction
Slide 11ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing
TTCN-3 User Conference2012, Bangelore, India
TTCN-3 Extension: Continuous signal support 1(4)
This new package defines concepts for testing systems using continuous signals.
The package introduces notions of time and sampling, notions of streams, stream ports and stream variables,
and definitions allowing an automaton alike control flow
structure supporting the specification of hybrid behavior (i.e., combined definition of discrete and continuous behaviors).
Slide 12ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing
TTCN-3 User Conference2012, Bangelore, India
TTCN-3 Extension: Continuous signal support 2(4)
The following items shall give an impression about the contents of the package. Details and further operations can be found in the package definition.
Concepts for time and sampling: access to global time: now operator Setting stepsize for sampling: stepsize operator
Streams New port type for streams, e.g.,:
type port StreamIn stream { in float } type port StreamOut stream { out float }
Operations for accessing data streams: retrieving or setting actual values: value operation retrieval of time related information: timestamp operation Retrieval of actual stepsize of a port: delta operation Slide 13ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
TTCN-3 User Conference2012, Bangelore, India
TTCN-3 Extension: Continuous signal support 3(4)
Streams (cont.) navigation on streams
previous event: prev operation event at a certain point in time: at operation
stream extraction Retrieval of (partial) history: history operation
Modes define states in hybrid automata. Modes may
be composed sequential or parallel have entry and exit conditions have invariant parts have a duration be parameterisable etc.
Slide 14ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing
TTCN-3 User Conference2012, Bangelore, India
TTCN-3 Extension: Continuous signal support 4(4)
15Slide 15ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
A Conformance Test Suite for TTCN-3 Tools
Bogdan Stanca-Kaposta, Testing TechnologiesAndras Kovacs, Broadbit
Nikolay Pakulin, ISPRAS
TTCN-3 User Conference2012, Bangelore, India
Motivation
Many TTCN-3 compilers (~7 commercial, 2 internal) High interest in the language Complex standard: hundreds of pages No way to check the compliance of TTCN-3 tools Quality of a test tool is critical
Relevance: For users:
Is my tool standard compliant? Can I switch tools easily / vendor lock-in?
For tool vendors: Where can I improve the standard compliance of my tool? Possibly later: compliance certification
For standard developers: Where is need for clarification?
Slide 17ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing
TTCN-3 User Conference2012, Bangelore, India
STF 433
Specialist Task Force initiated by ETSI MTS Manpower: 3 persons for 100 days total Dates: November 2011 – Early 2012 Work is based on the existing framework.
Task: Update the existing TTCN3 ATS to the newer TTCN-3
revision Extend the conformance test suite for TTCN-3 tools.
Three deliverables: ETSI TS 102 950-1 V1.2.1 (2012-05) – ICS ETSI TS 102 950-2 V1.2.1 (2012-05) – TSS & TP ETSI TS 102 950-3 V1.2.1 (2012-05) – ATS & IXIT Approved by MTS in May 2012.
Downloadable from the ETSI download area right now!
Slide 18ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing
TTCN-3 User Conference2012, Bangelore, India
Methodology (1/7)
Main steps:1. Validate existing tests against new revision of the
standard.2. Develop conformance tests.
Follow the priority list discussed with MTS and vendors.3. Develop adapters for selected tools. Validate and fix
tests.4. Automate test execution.
Slide 19ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing
TTCN-3 User Conference2012, Bangelore, India
Methodology (2/7)
Adjusting Conformance ATS from TTCN3 rev. 4.2.1 to rev. 4.3.1: Was: 733 tests, 130 pp. of test purposes
Validate that test purposes still hold Update modified test purposes Revoke outdated tests Update ATS folder structure
Approximately 30% of the STF 433 effort
New tests development Clause priority list:
Not all clauses can be covered. Selection was made: Vendor feedback + most basic functionality.
There is no perfect selection!
Slide 20ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing
TTCN-3 User Conference2012, Bangelore, India
Methodology (3/7)
Development of conformance tests: The ATS is not the test automation, but the test input! In most cases: 1 module = 1 conformance test. Test automation provided for certain vendors
Elvior and Testing Technologies Test suite structure: organized according to clauses.
Expected output for verdict determination: @verdict pass accept/reject [expectedoutput]
Examples: @verdict pass reject @verdict pass accept, ttcn3verdict:pass @verdict pass accept, noexecution
Purpose specification: @purpose documentreference, description Documentreference: part:clause
Examples: @purpose 1:5, Ensure that when the IUT loads a module containing some
definitions before the module declaration then the module is rejected.
Slide 21ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing
TTCN-3 User Conference2012, Bangelore, India
Methodology (4/7)
Development of conformance tests (cont.): Complete TTCN-3 test input example:
/*************************************************** ** @purpose 1:5.2.2, Ensure that the IUT correctly handles the uniqueness of variable names in its scope. ** @verdict pass accept, ttcn3verdict:pass ***************************************************/module Sem_050202_Uniqueness_001 {
import from Sem_050202_Uniqueness_001_import { const all;
}type component GeneralComp {}function f_funcScope() {
var boolean repeatedIdentifier := true;
if(repeatedIdentifier==true) { setverdict(pass); }
}testcase TC_Sem_050202_Uniqueness_001()
runs on GeneralComp { f_funcScope(); }
control { execute(TC_Sem_050202_Uniqueness_001()); }}
module Sem_050202_Uniqueness_001_import {// repeated indentifier // from imported module// is allowed
const integer repeatedIdentifier := 0;}
Slide 22ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing
TTCN-3 User Conference2012, Bangelore, India
Methodology (5/7)
Creation of documentation: Framework already provided by STF393 (Proforma). Little time for documentation. Working on a single source saves time.
Solution: Write test purposes and expected tool outputs directly
into the test inputs. Generate tables for the documentation from the ATS:
Test purpose tables. Test suite structure.
Much easier to maintain! Documentation generation tools developed as part of
STF409.
Slide 23ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing
TTCN-3 User Conference2012, Bangelore, India
Methodology (6/7)
Validation and fixing: Validation and fixing took more than half of the
development time!
STF internal validation with 2 tools: TTworkbench: compile-time and execution. TestCast: compile-time and execution.
Developed necessary adapters Loopback, proc. call
Lots of feedback from TTech and Elvior!
Slide 24ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing
TTCN-3 User Conference2012, Bangelore, India
Methodology (7/7)
Executing the ATS:ATS
Sem_060101_TopLevel_001.ttcnNegSem_060303_component_types_001.ttcn…
TTCN-3 Tool (SUT)
TTCN-3 FileTTCN-3 FileTTCN-3 FileSem_060101_TopLevel_001.ttcn
Tool output
Rejection as invalid Rejection to executeExecution with results(e.g., pass/inconc/fail)
Tool output evaluation
** @verdict pass accept, ttcn3verdict:pass
failpass
Test inputExpected
output
Slide 25ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing
TTCN-3 User Conference2012, Bangelore, India
Discussion
Compliance does not imply safety from a vendor lock-in! Testing is never complete! With v1.1.1 of the ATS, about 32% of the core standard clauses
are covered with at least one test. Ideally, we should have many tests for each clause.
Compliance means less likelihood for vendor lock-in. With better coverage and higher number of test cases, we
improve the quality of this statement. Compliance reduces the likelihood for major tool errors
Tests range from simple arithmetic operations to communication operations and verdict handling.
ATS checks syntax and semantics. ATS does not check the correctness of every single tool
message, but of the overall output.
TTCN-3 ToolTTCN-3 Conformance ATS Meaning?
pass
Slide 26ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing
TTCN-3 User Conference2012, Bangelore, India
Challenges (1/2)
Various standard clarifications were necessary: 13 CRs for TTCN-3 core languages have been written. Unclarified test cases have been excluded from the delivery
("on-hold"). Tools sometimes handle the standard differently.
Tests need to be expressed as tool independent as possible. Test Automation:
No standard command line interface for batch run Tool-specific adapters No common test log format
TTCN3 semantics TTCN3 standard part 1 defines both static and dynamic
semantics No definition of well- and ill- formed programs
What to check at compile time is tool-dependent
Slide 27ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing
TTCN-3 User Conference2012, Bangelore, India
Challenges (2/2)
Synchronize ATS with new TTCN3 revisions Time consuming manual activity As ATS grows the effort will grow as well To which core language standard version does the
conformance test suite relate to? How to maintain compatibility across different language
versions? Against which version should the conformance test suite be
developed in the future? Which TTCN3 revision to support
Should we certify newest TTCN3 revision only?
Slide 28ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing
TTCN-3 User Conference2012, Bangelore, India
Conclusion
The conformance test suite is available now! ATS Metrics:
Total number of conformance tests: 1077 (340 new) 20 tests fixed 9 removed since deprecated or duplicates
Number of positive syntactic conformance tests: 138 Number of negative syntactic conformance tests: 82 Number of positive semantic conformance tests: 517 Number of negative semantic conformance tests: 340 Number of clauses (at least partially) covered: 134 (24 new)
Slide 29ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing
TTCN-3 User Conference2012, Bangelore, India
30
TTCN-3 CONFORMANCE TEST SUITE STF 433
STATUS REPORT
Nikolay Pakulin
Requirements tracking case study
Slide 30ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing
TTCN-3 User Conference2012, Bangelore, India
Case study in a nutshell
4 chapters: total 7,5 pages 12 Declaring messages, 13 Declaring timers, 23 Timer operations, 24 Verdict operations;
56 requirements elicited, 7 requirements refined into 34 test purposes; covered by 174 test cases
Average 7,5 requirements per page, 3 test cases per requirement
Slide 31ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing
TTCN-3 User Conference2012, Bangelore, India
Requirements tracking case study
ISP RAS contributed to the project requirements management tool Requality Hierarchical grouping of requirements Requirements elicitation from DOC/HTML documents
Requirements are connected to certain locations in the documents
Requirements locations are highlighted Requirements refinement through test purposes
Test purposes define specific cases to be tested in order to treat corresponding requirement ‘covered’
Requirements coverage by test cases report Source document change management (beta)
Update requirements when the text of the standard is modified Requality is an open-source plugin to eclipse Integrated with version control software (SVN)
Slide 32ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing
TTCN-3 User Conference2012, Bangelore, India
Sample view: Requality in TTWorkbench
Slide 33ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing
TTCN-3 User Conference2012, Bangelore, India
/*************************************************** ** @author STF 433 ** @version 0.0.1 ** @purpose 1:12, Ensure timer can be declared in components ** @verdict pass accept, ttcn3verdict:pass ***************************************************//* * #reqname /Requirements/12 Declaring timers/Timers can be declared in component type **/module Sem_12_toplevel_timer_001 { type component TComp{ timer t_timer:=10.0; } testcase TC_Sem_12_toplevel_timer_001() runs on TComp{ t_timer.start; if (t_timer.running){ setverdict(pass) } else { setverdict(fail) } } control{ execute(TC_Sem_12_toplevel_timer_001()) }}
Requirements Tracking (1/2)
Fully qualified name of the requirement
Slide 34ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing
TTCN-3 User Conference2012, Bangelore, India
Requirements Tracking (1/2)
Requirements coverage report Integrated into
Eclipse Evaluates
coverage of requirements and test purposes
Slide 35ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing
TTCN-3 User Conference2012, Bangelore, India
Suggested use case
Add requirements markup to TTCN3 standard Manual
Introduce references between requirements and TTCN3 test cases Manual
Track requirements changes between TTCN3 standard revisions Automated (beta)
Track required changes in TTCN3 ATS Manual, to be automated
TTCN3 standa
rd update
d
Changes in reqs.
reviewed
New reqs. added
ATS update
d
Slide 36ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing
Thank You!
Questions?
Slide 37ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing