Date post: | 07-Jul-2015 |
Category: |
Technology |
Upload: | addq-consulting |
View: | 206 times |
Download: | 1 times |
Testautomatiseringserfarenheter
Presentation
Michael Albrecht Lars Wahlberg (Aguil)
Vad innehåller boken?
• Michael Albrecht – Anecdote 29.12 Cooperation Can Overcome
Resource Limitations • Lars Wahlberg
– Chapter 18 Automated Tests for Marketplace Systems: Ten years and Three Frameworks
Nu blir det vissa slides på Engelska !!!
Våra bidrag
• Hur gick det till – Hur vi blev kontaktade
• Arbetet med att ta fram bok – Massor med reviewer & uppdateringar – Tekniska saker som att pillra med bilder
• Legal process – Skriva på kontrakt osv
• Känslan att få boken (tillslut)
Framtagning av en bok
Första stegen
Anecdote 29.12 Cooperation Can Overcome Resource Limitations
• Sit together • Daily status meetings • Learn some basic programming (same
language as the developers) • Select tools already used by the developers
for unit testing if possible • Create semi-automatic and automatic tests • Learn from the past • Get going!
Before the agile hype
• 55% a good tester • 35% development skills • 10% business or domain knowledge
Technical automater
• http://www.addq.se/nyheter/back-to-the-future-an-agile-story-from-the-past/
• http://www.automatedtestinginstitute.com
.and the rest of the story?
Chapter 18 Automated Tests for Marketplace Systems: Ten years and Three Frameworks
• Market Place Systems • Automated Tests • Three frameworks • ROI
Market Place Systems (MPS)
Gwy A
Gwy B
Gwy C
Gwy D
Matching Engine
Central Data
Query / History Server(s)
Trading Clients Protocol X
Trading Clients Protocol Y
Market Data
Admin Client
Primary Standby
Typical instruments: Equities, Commodities, Bonds, Derivatives …
Drivers for large nr of tests … • Complex functionality • Interference of functionality • Critical functionality
– Wrong auto match L – Wrong transparency L – Possible to “game” the system L
• High uptime requirements • Many customers (adaptations of a product)
Why automate?
• Large nr of tests • Development methods – many releases!
– Incremental (10 years ago) – Agile (now days)
• Time to market important (rapid deliveries) – Enhancements (new business opportunity) – Patch (maintenance)
Solution => automate tests ...
Development of Test Framework
Abstraction Layer
Framework for automated tests
System
Test Scripts
Test Engine
Configuration
(ex JUnit)
Test Script
Enter Buy Order (25 SEK)
Enter Sell Order (24 SEK)
Verify Trade (25 SEK)
Enter Buy Order(price)
Enter Order (price, True, ...)
Enter Order (price, isBuy, …)
• Create Enter Order Message
• Fill in Price X
• Fill in Buy or Sell
• Fill in …
• Send Transaction
Changes in the protocol can be handled in the abstraction layer, instead of rewriting ~100 of tests scripts …
Abstraction Layer – Example
Abstraction of Configuration
• Predicates can be used Test Script
myList.add(new PredicateInstrument(“Equity”) )
myList.add(new PredicateCurrency(“USD”) )
myList.add(new PredicateTrader(myTrader1) )
myList.add(new PredicateTrader(myTrader2) )
OrderBook myOb = TestConfig.getAnyOb(myList)
Hint: Use TDD to design the Abstraction Layer !
If not found =>
Test Skipped !
Not same as Failed !
Development of Test Tool
• Abstraction Layer is important – Thin / Thick? Requirements on people skills! – Who can modify? Who will maintain? – When tests increased problems will be revealed!
• Abstraction against test data – Hard coded in a file, version controlled with tests – Predicates may be used to increase robustness
Tip: Very good reference go to => www.stickyminds.com Search for “mexus” (author Lars-Ivar Sellberg)
Development of Tests Automatic Tests
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
Week
Test
s
ActualPlanned
Test Execution (then)
Test Execution (then)
TestCase written before implementation
Now it works ! :)
Oops, it fails! :( Regression tests are good !
Oops, vacation!
Now it works again! :)
Framework - 1 • Developed under much time pressure 1998 • Tests written in Java
– New language, attracted developers to test – Thin Abstraction Layer
• Own Test Engine (JUnit not available) • Predicated used • Simulations used (later more on that) • After ~1 year: 2000 tests, ~3 year: 15 000 tests • A real success !!!
Framework - 2 • Developed 1999 based on success of -1 • Development took some time …too much
– Another system => new abstraction layer, still thin – Predicates not used – Switched to JUnit – Focus on Tool instead of tests … few tests L
• Heavy investment during 2002 – Partly due to one senior Manager J – Larger team => 50 000 tests after 3 years!
Framework - 3
• Developed around 2003 • Based on ideas from mostly from 1
– But should be better! Func & Non Func tests, Preds – Thick Layer – Tests in XML action language
• Test Tool Architects became bottlenecks – Difficult to do modifications of abstraction layer L
• Re-done, Thin layer & No Predicates • Now ~ 20 000 Tests
ROI - Savings Saving of Automization
-350
-300
-250
-200
-150
-100
-50
0
50
100
150
500
700
900
1100
1300
1500
1700
1900
nr of tests in batch
savi
ngs
in % Yearly
MonthlyWeeklyDaily
Effort dev manual test 2 h/testEffort dev automatic test 2 h/testTest Tool Maintanence 3000 h/yearEffort Execute manual test 0,1 h / testEffort Execute automatic test 8 h/batch
Utlottning av böcker !