Model-Based Testing at ETSIModel-Based Testing at ETSI
Stephan SchulzETSI TC MTS chair
TC INT IMS Workshop Nov 2010 © ETSI 2010. All rights reservedTC INT IMS Workshop Nov 2010
SCOPE of the Technical Committee
Who is MTS?
Who is TC MTS?SCOPE of the Technical Committee
Methods for Testing and Specification (MTS) creates guidelines, frameworks, notations, and methodologies for specification and testing to help other ETSI committees to efficiently develop
their standards and test specifications.
Past Achievements
pTC MTS cooperates closely with ETSI’s Center for Testing & Interoperability (CTI).
Present Activities Possible Future AreasTTCN-3 Evolution
NotationsTTCN-3, TPLan, Descriptive SDL
TTCN 3 EvolutionEnhancements for LTE testing, real time
Performance testing
Further TTCN-3 Development
Update of best practice for specification
Test specifications for core IP technologies SIP, SIGTRAN, IPv6 (core, sec, mob, 4to6), H.225, H.248
Automated interoperability testingMethodology & framework
M d l B d T ti
p
ETSI security framework
Measurement framework formob, 4to6), H.225, H.248
Methodologies & guidelinesMaking Better Standards,IP testing framework
Model Based TestingConcepts for modelling
Securitye-Passport testing framework
Measurement framework for physical layers
Further work on Model Based Testing
22
e Passport testing framework and platform
Testing
More info: http://portal.etsi.org/mts
About Model-Based Testing (MBT)An umbrella of approaches that generate tests from models
Here: Test generation from system modelsSystem Model
Here: Test generation from system modelsIs about automating test design, i.e., produces test scripts and documentation
I li i
Generation
(TTCN-3) Test ScriptsIt compliments test execution
Considered today industrial best practice in software testing1 Test Execution Engine
(TTCN-3) Test Framework
gNumber of commercial tools available
ETSI TC MTS is creating standards on MBT since 2008 S t U d T t
Test Adaptation
MBT since 20082 TRs,1 ES, and more are to come
System Under Test
3
1 ”The Testing Tools Landscape: 2010. Functional Testing Tools Are Not Enough.” by M. Visitacion & M. Gualtieri. Forrester Research, Inc.
First ES: Concepts for Model Specification Identifies and collects all relevant concepts that model specifications developed for testing should include
It will essentially mandates concepts that MBT tools need to supportIt will essentially mandates concepts that MBT tools need to support to facilitate the generation of such testsConcepts are specified independent of a specific modelling language
Standard is currently in drafting stageStandard is currently in drafting stageFinal draft expected for TC MTS meeting in Dec 2010Key MBT tool vendors, industrial users, and research institutes who are active in this field are participating(Ericsson, Wipro, NSN, Siemens, SAP, MS, Conformiq, Smartesting, SeppMed, FOKUS, IRISA)
N t th t thi t d d i t d t b l th fi t i iNote that this standard is expected to be only the first in a series of ES’s for model-based testing tools
4
Classic vs. MBT Test Development
BaseSpec
TestPurposes
TTCN-3Tests
Reqstestcase t1() {
…}Specify
SystemModel
BaseSpec
GenerateReqs
5
testcase t1() {…
}
An Example MBT Tool Workflow
DevelopSystem Model
Direct & ReviewTest Design
Generate Scripts& DocumentationSystem Model Test Design & Documentation
Model Editor MBT Tool Test ExecutionTool(s)
6
Why is MBT interesting?
Model reuse Simplified test harness
Savings in test maintenance
Enabling new ways of working
Improved test Customizable
t t t
Higher test
coverage test sets
The business case for
MBT
Higher test design
productivityTest scripting
language independence
77
MBT
First Ideas on using MBT at ETSIShould not change ETSI’s existing way of standardizing tests!
Merely offer a new method for test development, e.g., in STFsModels specify externally observable behavior as defined by aModels specify externally observable behavior as defined by a base specification
But should be part of test specifications - not base specificationsIt is the generated tests that should be standardized
Models should be considered informative testing artifactsETSI TCs should set requirements for test selectionETSI TCs should set requirements for test selection
Perceived benefits of this approach includeImproved quality of ETSI test specifications More transparency in the ETSI test specification processAbility for ETSI members to reuse models for own test generation
8
The Road AheadCreation of further ETSI standards on model-based testing
Guide on the specification of models for testing in the context standardizationMinimum requirements on test generation/selectionConcretization of modelling concepts
O i ti f k h f h i i d t i l iOrganization of a workshop for sharing industrial experiences with model-based testingA MTS STF for developing a guide and performing case studies p g g p gwith other ETSI TCs/WGs in second half of 2011
So far TC INT and TC ITS have shown interest
9