Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
SMB2 Model-Based Testing
Nico Kicillof, Albert Lee
Microsoft
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
222 open protocol specifications tested
22,847 pages
~10,000 document bugs
66,962 person days (250+ years)
Tool dev center in Beijing
Vendor test teams
Hyderabad: 250
Beijing: 100
2
“BlueLine” Technical Document
Testing Project
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
Formalization
Technical DocumentTest Suite
Normative
statementsAssertions
Checked against Windows
(Accuracy)
• Validate
consistency
• Simulate
implementation
process
Test-Driven Document Analysis
3
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
Model-Based Testing
A Lightweight Formal Method
4
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
Model
Implementation
Test OracleTest Sequences
Control
Verdict
Informal Spec/Reqs
Generate
Observe
Feedback
Feedback
Author
Issue
Feedback
5
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
Requirements
Gathered from technical document alone Gatherers not protocol experts
Clean-room approach
Windows-specific behavior as separate requirements
Document testing Not implementation
Independent reviewers
Main input for model design
Technical Document
Requirements Spec
Model
Test Suite
Logs
Network Captures
6
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
The Value of Modeling
Model-Based Testing vs. traditional testing comparison
Empirical SE Research group assessed statistical significance
Uniform sample from one testing site
Modeling resulted in 42% productivity gain
Overall, modeling saved 50 person-years (12,547 days)
No formal methods background was required
Teams of junior engineers
Mostly vendors, fresh out of college
7
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
Test Representation
8
Getting data on the wire and back
Known problem for protocol testing (e.g. TTCN-3)
Our Solution: Protocol Test Framework (PTF)
Test adapters written in a managed language (C#)
Custom support for dealing with protocols
Automatic data packet (de)serialization
Based on declarative data definitions
Protocol-specific logging capacity
Beacon packets
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
Test Adapters
Adapter Interface
TCTest Cases(traditional or MBT)
TC TC
TC TC TC
Adapter Implementation
SUT(System under test)
•Abstracts SUT functionality
•Contract between teams
•Test case team
•Adapter team
•Pluggable
•Different server setups
•Different transports
9
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
Spec Explorer
A Model-Based Testing Tool from Microsoft
10
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
Features
Multiple modeling styles and languages
Programs, patterns, diagrams
Asynchronous & non-deterministic systems
State machine extraction from model program
Test code generation from state machine
Model composition
11
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
Spec Explorer Usage
Internally used to test several Microsoft products
External Early Adopter Program
Shipping plans
MSDN DevLabs
Visual Studio 2010 Beta 2 timeframe
12
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
SMB2 Interoperability Test Suite
13
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
Test Suite Scope
Validate the Technical Document against Windows for
interoperability
Accuracy
Consistency
Usability
Aimed at the Microsoft Communications Protocol
Program
Only server behavior is tested
14
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
Test Approach (First Pass)
Assumptions
Single connection
No compounded message
No nested shares
Out of model scope
GSS-API authentication
Race conditions
Exhaustive flag combinations
121 scenarios
15
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
Test Suite Architecture
Test Suite
Test generated by Model Traditional Test
MS-SMB2 Infrastructure SDK
SMB2 Server
SMB2 Adapter Interface
Request
Call
Response
Event
SMB2 Adapter Core
Traditional Adapter Interface
Traditional Adapter Core
Request
Call
Response
Event
Server
Config
Adapter
Interface
· Server
Config
Adapter
Core
16
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
Demo
The SMB2 Test Suite
17
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
Conclusions
Interoperability test suites developed for 200+
protocols
Thousands of document bugs filed and fixed
Test suite further usage
Regression testing
Third-party implementations (Plugfests)
SMB2 test suite (first pass)
Document improvement
80% requirement coverage
18
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
Questions
{nicok, albertle} @ microsoft . com
http://www.microsoft.com/protocols/
19