Date post: | 07-Jul-2015 |
Category: |
Technology |
Upload: | eurostar-software-testing-conference |
View: | 91 times |
Download: | 0 times |
Thomas Kauders
Agile Test Design and Automationof a Life-Critical Medical Device
A usable model for Agile Test Automation
1 © PrettyGoodTesting 2010
Agile Test Design and Automation of a Life-Critical Medical Device
Project Summary- about me, the project and our test team
Project Challenge- why we needed to automate test
Our Approach- how we designed the automation process
Automation Implementation- how did it take place in practice
Lessons Learned- our successes and concerns
Agenda
2 © PrettyGoodTesting 2010
Thomas KaudersM.Sc., Test and test manager since 2002
ISEB/ISTQB Certified
Currently:
Assigned at a large medical company
About
3 © PrettyGoodTesting 2010
More than
11.000 pages
of specifications
About the Project
“The Device”
Not the actual devicedepicted! Used for
illustration only!
700 Man-years
...as from Guthenberg
...to ”The Internet”
4 © PrettyGoodTesting 2010
The Test Team
1 Test Manager
1 Test Designer
2 Manual Testers
3 Test Automators
...1 Model-Based Test (MBT) Automator
...2 Functional Test Automators
5 © PrettyGoodTesting 2010
Large and complex system
Life critical system
Volatile requirements R&D
Agile technical requirements
Formal V&V(Verification & Validation)
U.S. Food and Drug Administration
The Project Challenge
Test
Development
Requirements
6 © PrettyGoodTesting 2010
Importance of formal test
Huge volume of manual test
Short development sprints
Limited ressources
Strict deadline
Possible solutions:• More ressources • Time extension• Test Automation
The Testing Challenge
7 © PrettyGoodTesting 2010
Considering test automationfrom the very beginning of test design
Cooperating Manual- and Automated Teams
Reusing Test Design across teams
Modularising the test
Gradually adding more ”Pay As You Go”
A New Approach
8 © PrettyGoodTesting 2010
SRS MMI
Consider Test Automation from the Start
Modular mindset,during analysis andTestCase design!
Aut. TestCases
Man. TestCases
TestDesign
Requirements
ActionWords
Traceability to Reqs?
Based on SRS and MMI?
TestCases OK for Test Automation?
9 © PrettyGoodTesting 2010
To keep it simple we developed a concept of:
PRIMITIVES (aka ActionWords, KeyWords etc.)
Primitives is a known term by key project members
Sticking to this term eased our ”selling the idea”
Naming the Principle
10 © PrettyGoodTesting 2010
Close cooperation btw. Man. & Aut. teams
Re-using Test Design across teams
11 © PrettyGoodTesting 2010
Use of ”mnemonics” resulted in ”readable” names, e.g.:
ST-PRM-FTUS :System Test, Primitive, First Time Use
ST-PRM-LGBK :System Test Primitive, Log Book
Using “Call to test” to use Primitives
Call to test
12 © PrettyGoodTesting 2010
PTC:
Set Year ()
PTC:
Set Month ()
PTC:
Set Day ()
Primitives (PTC)
Call to test PTC:
Set Month ()
Call to test PTC:
Set Day ()
Call to test PTC:
Set Time ()
Call to test PTC:
Confirm Setup
PTC:
Confirm Cancel Setup
PTC:
Set Time ()
PTC:
Cancel Setup
Cancel Cancel Setup
PTC:
Confirm Setup
Test Cases (STC)
Call to test PTC:
Set Year ()
TC:Navigate to Setup D&T
Call to test PTC:
Set Month ()
Call to test PTC:
Set Day ()
Call to test PTC:
Cancel Setup
Call to test PTC:
Confirm Cancel Setup
Call to test PTC:
Set Year ()
Call to test PTC:
Set Month ()
Call to test PTC:
Set Day ()
Call to test PTC:
Cancel Setup
Call to test PTC:
Cancel Cancel Setup
Call to test PTC:
Set Year ()
TC:Navigate to Setup D&T
Call to test PTC:
Set Time ()
Call to test PTC:
Confirm Setup
PTC:
Verify Setup Complete
PTC:
Navigate to Setup D&T
Call to test PTC:
Verify Setup Canceled
PTC:
Verify Setup Complete
PTC:
Verify Setup Complete
TC:Navigate to Setup D&T
Think of Lego Bricks
13 © PrettyGoodTesting 2010
Manual test not coping with the volume of tests
Automated tests running on unstable test system
The ”Death by test script maintenance” issue
Milestones not reachable if not:
- increasing the extent of the automated regression
- increasing the quality of the automated regression
- creating better manual regression testing
- developing smarter approach to test scripting
Not possible without effective test automation!
The Test Automation Challenge
14 © PrettyGoodTesting 2010
Navigation
Go from A to B (Start to adjust Time)
Action
Input to system (Set Time)
Verification
Expected result (Verify Time set)
Execution
Practical test implementation (Note Test Start Time)
Use parameters to keep numbers of mode
Primitive Classification
15 © PrettyGoodTesting 2010
Traditional SequentialTestCases
A pseudo test case broken down into Primitives,each Primitive is unique to the SUT (Software Under Test)
PRIM 6
PRIM 5
PRIM 4
PRIM 3
PRIM 2
PRIM 1
Products of Module-based Test Design
16 © PrettyGoodTesting 2010
PRIM 7
PRIM 6
PRIM 3
PRIM 2.1
PRIM 1
PRIM 7
PRIM 5.1
PRIM 4
PRIM 3
PRIM 2.2
PRIM 1
PRIM 7
PRIM 5.2
PRIM 4
PRIM 3
PRIM 2.3
PRIM 11
2.1 2.2 2.3
3
6 4
5.1
7Saving above 50% of work for just this example.
5.2
An Advantage
17 © PrettyGoodTesting 2010
During test analysis try to break down each Test Case in unique and identifiable user action sequences
A unique sequence must not exist elsewhere
(stick to this…)
The sequences are to be re-used and combined in anyorder representing complete user actions
Why does this work?
18 © PrettyGoodTesting 2010
New test delivered once a week
Tests fixed from day to day
Change identified (By Developer or Tester)
Impact assesed
Affected Primitive(s) identified
Change(s) implemented
Test Scripts tested
Delivered and verified that tests are running
Dealing with Change in Automation
19 © PrettyGoodTesting 2010
Deliver tests Verify tests
Test Test Scripts
ImplementChange
Identify affectedPrimitive(s)
Assess impact
Changeidentified
Must be ableto respond quicklyto change
Daily Change Management
20 © PrettyGoodTesting 2010
Tracebility from Requirements -> TC´s involved
Change only the involved Primitives
1:1 manual TC´s -> automated TC’s
* SCRUM = Dynamic Requirements!
When Requirement’s are changing*
21 © PrettyGoodTesting 2010
Easy indication of what has to be changed
Full traceability of what has been changed
1:1 traceability eases quicker validation*
Use of Primitives made the test suite scalable
Use of Primitives made the test suite flexible
* FDA requires full validation of:
Designs, Implementation, Tools, TestCases etc.
What was reached?
22 © PrettyGoodTesting 2010
Initial Test Analysis…still to be carried out* (no Quick-fix)
Change Impact Analysis…still required* (no Silver Bullet)
* Exploratory Testing and Model-based Testing...is still to be carried out (not a stand-alone thing)!
What was NOT reached?
23 © PrettyGoodTesting 2010
Equals 7-14 days of man. regression tests...nightly!
- September 2010, 7 days
- December 2010, 14 days (expectedly)
40% WorkLoad Reduction
Every team member plays an important role- Happy Testing!
And…Milestones are now met!
Outcome
24 © PrettyGoodTesting 2010
Too many Primitives (incorrect partitioning)
Improper mindset during review
- This seems possible! But is it?
Lack of review of Manual and Automated tests
Lack of review of Test Scripts
Perfecting / ”goldplating” test breakdown
Pitfalls
25 © PrettyGoodTesting 2010
Cooperation between Test Designer, Manual Test Team and Automation Test Team reduces effort, by:- Thinking Automation First- Reusing of Test Documentation/Primitives
Primitives (aka ActionWords, KeyWords etc.):- Promotes flexible & easy maintainable Test Suites
Exploratory Testing and Model Based Testing...still to be carried out (not a stand-alone thing)!
Summary
26 © PrettyGoodTesting 2010
Time for Questions and Comments
Thomas Kauders: (+45) 3163 [email protected]
PrettyGoodTesting ApSLautruphøj 1-3DK-2750 BallerupDenmark (+45) 3163 0200
www.PrettyGoodTesting.com
27 © PrettyGoodTesting 2010
Test Running
© PrettyGoodTesting 2010Additional Slide
Step 1: Using Primitives by calling with new param. values
Step 2: In making test, needed Primitives are implemented
Step 3: Putting it all together in a test script
Step 4: Making a test suite for Hudson
Capture Replay
© PrettyGoodTesting 2010Additional Slide
Continuous Integration
© PrettyGoodTesting 2010Additional Slide