1
11th TTCN-3 User ConferenceAn Introduction to TTCN-3
Bangalore, June 2012
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 3
Speaker
Theofanis Vassiliou-Gioles Founder and CEO of Testing Technologies
Master in Electrical Engineering Started communication testing 1996 ATM test specification Standardization ETSI TTCN-3 Standardization Application of test automation
in new domains
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.©
Agenda
MotivationBasic Concepts of Communication TestingIntroduction to TTCN-3 –TTCN-3 by Example
The TTCN-3 LanguageIntroduction to TTCN-3Motivation
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 6
„ ... the national annual cost estimates of an inadequate
infrastructure for software testing are estimated to be
$59.5 billion.The potential cost reductionfrom feasible infrastructureimprovements is $22.2 billion.“
The Economic Impacts of Inadequate Infrastructure for Software Testing
Study by NIST, May 2002
How Much Does Testing Cost?
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 7
Testing Today
IsImportantExpensiveTime critical
ButOnly rarely practicedUnsystematicPerformed by handError-proneUncool („If you are a bad programmer you might be a tester.“)Unconstructive
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 8
Why Using TTCN-3 (2)
Speed to Market
HighIntegration
HighReliability
LawChanges
Optimal Cost
Complexity
WLAN3G/4G
IP
MATCH
Req
uire
men
ts
Test
Met
hods
High Quality
TTCN-3
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 9
Testing is
a technical processperformed by experimenting with a software productin a controlled environment following a specified procedurewith the intent of observing one or more characteristics of the productby demonstrating the deviation of the product’s actual status from the required status/specification
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 10
Testing Today‘s Systems
Component-basedTest-components contribute to SUT functionality and performance
DistributedNot only local, but also distributed test setups
Dynamic in terms of behavior and configurationTesting of static and dynamic aspects; dynamic creation of test components
Use various type systems to exchange dataOpen to all type systems
Service is essentialConcentration on service-oriented black-box testing
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 11
Design Principles of TTCN-3
One test technology for different kind of testingDistributed, platform-independent testingIntegrated graphical test development, -documentation and -analysisAdaptable, open test environment
One test technology for distributed IT and telco systems and beyond
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 12
History (1)
TTCN-2
TTCN-2++
1997
1984
1994
1992
TTCN
TTCN (1992)Published as an ISO standardTree and Tabular Combined NotationUsed for protocol testing only
GSM, N-ISDN, B-ISDN
TTCN-2/2++ (1997)Concurrent testsModularizationManipulate external dataRather for conformance testingDeveloped by ETSI MTS
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 13
History (2)
2000
1998
2012
u
TTCN-3 (2000)Testing and Test Control NotationDeveloped by ETSI MTSStandard language
Well defined syntax and semanticsEnhanced communication, configuration and controlStandard test specification
SIP, SCTP, M3UA, IPv6HiperLan, HiperAccess, Wimax3GPP LTE, OMATETRAMOST, AUTOSAR
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.©
History (3)
Since 2002 standard bodies using TTCN-3 to define test specifications
ETSI 3GPPWiMAX ForumOMATETRAAUTOSARMOST
14
19982000
ETSI: SIP
Start of developing TTCN-3
2002
2003
2004
ETSI IPv6
AUTOSAR: Pilot
20082011
TTCN-3 Standard /TT Foundation
ETSI: HiperACCESS
ETSI: HiperLAN2
ETSI: HiperMAN
WiMAX Forum: 802.1d2005
2006WiMAX Forum: 802.1e
WiMAX Forum: Interop
ETSI: VoIP/POTS Testing
OMA: Interop Pilot
AUTOSAR: Rollout
2007
OMA: Rollout
TETRA: Pilot / Rollout
2012 EUROCONTROL: Rollout
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 15
Maintenance of TTCN-3
Standard is constantly maintainedThrough Change Requests (CRs)Extension proposalsActive contributions in the TTCN-3 community
TTCN-3 mailing list, TTCN-3 users conferenceETSI STFs (Specialist Task Force)Change requests result in new editions of the standard
2000: Edition 12003: Edition 22005: Edition 32010: Edition 4.2.12011: Edition 4.3.1
Resources: http://portal.etsi.org http://www.ttcn-3.org
The TTCN-3 LanguageIntroduction to TTCN-3Basic Concepts
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.©
Generic Protocol Architecture(s)
17
L1 / PhyL2 / Data LinkL3 / NetworkL2 / TransportL5 / Session
L6 / Presentation
L7/ Application
Link Layer
Internet LayerTransport Layer
Application Layer
OSI View TCIP/IP View
Ethernet, Token Bus, Token Ring, FDDI, IPoAC
IPv4, IPv6TCP, UDP, SCTP
HTTP, FTP, SMTP, POP, Telnet, DNS
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.©
Generic Protocol Architecture(s)
18
L1 / PhyL2 / Data LinkL3 / NetworkL2 / TransportL5 / Session
L6 / Presentation
L7/ Application
Link Layer
Internet LayerTransport Layer
Application Layer
OSI View TCIP/IP View
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.©
When we test we ...
Select the protocol or application to test DNSSelect the test access UDP, IPv4, Ethernet
19
Ethernet
IPv4 UDP
DNS
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.©
When we test we would like to ...
Concentrate on the protocol (application) on an abstract levelDo not care for the concrete technical details like test access
20
Test Access
DNS
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 21
What is TTCN-3?
Testing and Test Control NotationInternationally standardized testing language for formally defining test scenarios. Designed purely for testingTaking the best bits of TTCN-2 and combining them with a new more powerful textual notation
testcase tc_Hello_Bob () { p.send("How do you do?"); alt { [] p.receive("Fine!"){ setverdict( pass );} [else]{ setverdict( inconc );} //Bob asleep! }}
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 22
testcase tc_Hello_Bob () { p.send("How do you do?"); alt { [] p.receive("Fine!"){ setverdict( pass );} [else]{ setverdict( inconc );} //Bob asleep! }}
TTCN-3 Execution
SUT
Tester
TTCN-3
API / Network
API / Communication
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 23
Application Areas
Multiple application areasSoftware testingCommunication protocols …
Two communication paradigmsMessage-based communicationProcedure-based communication
Different kinds of testingFunctional testingConformance testingScalability testing …
Addresses the complete development cycleFrom unit via integration testing up to system level tests
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 24
Main Aspects of TTCN-3
Triple CConfiguration: Dynamic concurrent test configurations with test componentsCommunication: Various communication mechanisms (message-based, procedure-based)Control: Test case execution and selection mechanisms
FeaturesWell-defined syntax, static and operational semanticsDifferent presentation formatsModule conceptExtendibility via attributes, external function, external dataHarmonization with ASN.1, integration of XML, IDL, …
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 25
TTCN-3 Standards
ETSI ES 201 873-1 TTCN-3 Core Language (CL)ETSI ES 201 873-2 TTCN-3 Tabular Presentation Format (TFT)ETSI ES 201 873-3 TTCN-3 Graphical Presentation Format (GFT)ETSI ES 201 873-4 TTCN-3 Operational Semantics ETSI ES 201 873-5 TTCN-3 Runtime Interface (TRI)ETSI ES 201 873-6 TTCN-3 Control Interfaces (TCI)ETSI ES 201 873-7 Integration of ASN.1ETSI ES 201 873-8 Integration of IDLETSI ES 201 873-9 Integration of XML ETSI ES 201 873-10 T3DocETSI ES 202 784 TTCN-3 Extension: Advanced ParametrizationETSI ES 202 785 TTCN-3 Extension: Behaviour TypesMaintenance on the basis of change requests by ETSIStandard available for download at http://www.etsi.orgTesting Tech tools support Edition 4.2.1Also standardized by the ITU-T as ITU-T Z.140 series
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 26
TTCN-3 Standards
ETSI ES 201 873-1 TTCN-3 Core Language (CL)ETSI ES 201 873-2 TTCN-3 Tabular Presentation Format (TFT)ETSI ES 201 873-3 TTCN-3 Graphical Presentation Format (GFT)ETSI ES 201 873-4 TTCN-3 Operational Semantics ETSI ES 201 873-5 TTCN-3 Runtime Interface (TRI)ETSI ES 201 873-6 TTCN-3 Control Interfaces (TCI)ETSI ES 201 873-7 Integration of ASN.1ETSI ES 201 873-8 Integration of IDLETSI ES 201 873-9 Integration of XML ETSI ES 201 873-10 T3DocETSI ES 202 784 TTCN-3 Extension: Advanced ParametrizationETSI ES 202 785 TTCN-3 Extension: Behaviour TypesMaintenance on the basis of change requests by ETSIStandard available for download at http://www.etsi.orgTesting Tech tools support Edition 4.3.1Also standardized by the ITU-T as ITU-T Z.140 series
The TTCN-3 LanguageIntroduction to TTCN-3TTCN-3 by Example
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 28
Tester
Main Test Component
TTCN-3 By Example
System Under Test
Local Domain Name Server
Local Network
Client
Send fully qualified hostname
Return IP-address
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 29
Tester
Main Test Component
System Under Test
TTCN-3 By Example
DNSClient
pass
(www.testingtech.com,A)
(www.testingtech.com,212.227.57.158,A)
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 30
TTCN-3 Modules
Main building block of TTCN-3 is a module
Unit of compilationContains definitionsPlus optional control part
module DNS {
}
// module definitions
// module control (optional)
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 31
Module Definitions
Contains descriptions forWhat type of data the System Under Test understandsHow the System Under Tests can be accessed and what environment a test component needsWhen to communicate what with the SUT and whyDependencies between test cases, if any
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 32
Module Definitions (1)
Module definitionsType definitionsPort definitionsComponent definitionsTemplatesTest case
Control partControls the execution of test cases
type record DNSQuery { charstring hostname, AnswerType answer optional, QueryType qtype}type union AnswerType { Byte ipAddress[4], charstring hostname}type integer Byte (0 .. 255);type enumerated QueryType { A, NS, CNAME, MX}
DNSQueryanswer qtypehostnamehostname
ipAddress
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 33
Module Definitions (2)
Module definitionsType definitionsPort definitionsComponent definitionsTemplatesTest case
Control partControls the execution of test cases
type port DNSPort message { inout DNSQuery; // a port may send/receive messages // of more than one type}
Component definitionstype component DNSTester { port DNSPort P; timer t := 3.0; // a component may have more than one port}
DNSTester P
Port definitions
DNSQuery
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 34
Module Definitions (3)
Module definitionsType definitionsPort definitionsComponent definitionsTemplatesTest case
Control partControls the execution of test cases
query"www.testingtech.com" A
answer"www.testingtech.com" A212, 227, 57, 158
template DNSQuery query := { hostname := "www.testingtech.com", answer := omit, qtype := A}template DNSQuery reply modifies query := { answer := { ipAddress := {212,227,57,158} }}
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 35
Module Definitions (4)
(www.testingtech.com,A)
(www.testingtech.com,212.227.57.158,A)
Client DNS
pass
Module definitions
Type definitionsPort definitionsComponent definitionsTemplatesTest case
Control partControls the execution of test cases
testcase tc_testcase1() runs on DNSTester { P.send(query); P.receive(reply); setverdict(pass);
}
// there may be more than one in a module
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 36
Module Definitions (5)
Module definitionsType definitionsPort definitionsComponent definitionsTemplatesTest case
Control partControls the execution of test cases
control {
execute(tc_testcase1(), 5.0); while( /* condition */) { };
// more testcases might follow // C-like control structures available
}
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 37
control {execute(Testcase1(), 5.0);while( /* condition */) { };
}
Execution of a Test Case
Tester
testcase tc_testcase1() runs on DNSTester { P.send(query); P.receive(reply); setverdict(pass);
}
MTC Pverdict =
pass
P SUTreply = (www.testingtech.com, 212.227.57.158,A)
query = (www.testingtech.com,A)
Is this test case definition adequate?
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 38
reply = (www.testingtech.com, 127.0.0.1,A)
Tester
Dealing with Erroneous Behavior (1)
query = (www.testingtech.com,A)
SUT???
P.receive (reply) blocks until it receives a message that matches the replyIf an unexpected message is received, any other correct message does not unblock the tester, which then blocks foreverIf no message is received, the tester will also block forever
MTCverdict = ???
P P
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 39
Client DNS
failinconc
(www.testingtech.com,A)
failpassfailfailfailpass
Dealing with Erroneous Behavior (2)
testcase tc_testcase2() runs on DNSTester {
P.send(query);
t.start;
alt {[] P.receive(reply) {
setverdict(pass);}[] P.receive { // any message
setverdict(fail);}[] t.timeout {
setverdict(inconc);}
}stop;
}
(www.testingtech.com, 127.0.0.1,A)
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 40
Code Reusability – Altsteps and Defaults
altstep a_RefactoredAltstep() runs on DNSTester {[] P.receive { // any message
setverdict(fail);}[] t.timeout {
setverdict(inconc); }}
refactoralt {[] P.receive(reply) {
setverdict(pass);}[] P.receive { // any
messagesetverdict(fail);
}[] t.timeout {
setverdict(inconc); }
}
var default d := activate(a_RefactoredAltstep());
P.send(query);t.start;P.receive(reply);setverdict(pass);
becomes
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.©
Tester
The internet‘s rootname service
(root NS)
Remote DNS (NS)
Local NetworkClient (Client) Parallel Test
Component 2
Parallel Test Component 1
Parallel Test Component 3
41
Non-Local DNS Query (1)
Main TestComponent
System Under Test
Local Domain Name Server
(DNS)
Ask for remote DNS
Get remote address
Ask for IP address
Return desired IP address
Send fully qualified hostname
Return IP address
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 42
From Simple To Complex Test Scenarios
Test system needs more interfacesTest System Interface has to be extended
Test behavior required at additional test interfacesBehavior of Local Network Client already covered in tc_testcase2Behavior of RootNS and NS required
Test case that combines all parts
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 43
DNSQuery
Parallel Test Components
Test system interface type component TestSystemInterface {
port DNSPort CLIENT; port DNSPort ROOT; port DNSPort NS;}
DNSQuery
DNSQuery
Tester
P
Tester
CLIENT
ROOT
NS
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 44
From Test Case to Behavior Function
function f_clientBehavior() runs on DNSTester { var default d := activate(a_refactoredAltstep()); P.send(query); t.start; P.receive(answer); setverdict(pass); stop;}
becomes
testcase tc_testcase2() runs on DNSTester { var default d := activate(a_refactoredAltstep()); P.send(query); t.start; P.receive(answer); setverdict(pass); stop;}
Functions can be used to define the behavior of the parallel test components
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 45
Additional Test Behavior
function f_rootBehavior() runs on DNSTester { alt { [] P.receive(rootquery) { P.send(rootanswer); setverdict(pass);} [] P.receive { setverdict(fail);} }}
Simple „react-on-request“ behavior
function f_nSBehavior() runs on DNSTester { alt { [] P.receive(nsquery) { P.send(nsanswer); setverdict(pass);} [] P.receive { setverdict(fail);} }}
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 46
Additional Test Behavior
function f_rootBehavior() runs on DNSTester { alt { [] P.receive(rootquery) { P.send(rootanswer); setverdict(pass);} [] P.receive { setverdict(fail);} }}
Simple „react-on-request“ behavior
function f_nSBehavior() runs on DNSTester { alt { [] P.receive(nsquery) { P.send(nsanswer); setverdict(pass);} [] P.receive { setverdict(fail);} }}
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.© 47
Dynamic Configuration
testcase tc_testcase3() runs on MTC system TestSystemInterface {
var DNSTester ClientComp, RootComp, NSComp;
ClientComp := DNSTester.create; RootComp := DNSTester.create; NSComp := DNSTester.create;
map(ClientComp:P, system:CLIENT); map(RootComp:P, system:ROOT); map(NSComp:P, system:NS);
RootComp.start (f_rootBehavior());NSComp.start (f_nSBehavior());ClientComp.start(f_clientBehavior());
ClientComp.done; // block until ClientComp is done stop; } Tester
Re-configuration during run time is possible
MTC
CLIENT
ROOT
NS
PTCClient
P
PTCNS P
PTCroot NS P
Copyright Testing Technologies 2012. Confidential Information. All Rights Reserved. More Information at www.testingtech.com.©
Some References
The languagewww.ttcn-3.orgwww.testingtech.com/ttcn3/introduction.php de.wikipedia.org/wiki/TTCN-3en.wikipedia.org/wiki/TCN-3t-ort.etsi.org
The TTCN-3 Certificatewww.german-testing-board.info/en/ttcn3_certificate.shtm
The Quick Reference Cardwww.blukaktus.com/card.html
Some toolswww.ttcn-3.org/commercialtools.htm
48