Date post: | 16-Aug-2016 |
Category: |
Documents |
Upload: | emarketinglmax |
View: | 108 times |
Download: | 1 times |
Testing within an agile environmentChris Gollop & Beyza Sakir
Overview
History Changing Testing Concerns Adaptations Future Conclusion Further Reading
History – what we do
Release every 2 weeks Trade $billions/day Average 0.4ms execution time for MTF members 10,000 orders second 1 tester per 5-6 developers Very low rate of production issues
History - company
Conceived as an Agile company Agile has matured but we’re still adapting Agile testing:
Acceptance Tests Integration Tests Exploratory Testing
History - company
• This has allowed us to experiment and adapt:• Time freed up• Quick feedback on changes• Historic data available to analyse
History – test team
4 test specialists with different strengths complement each other as a 'team‘ Rotate through all teams• No Test Manager• Part of development team though test interests have a separate iteration
meeting• Testing is responsibility of everyone
History – team structure
l
Team One Team ThreeTeam Two
Test Specialists
Business Specialists
Development Specialists
History - innovate
Keep changing, even if something appears to work there may be something better
“If I had asked people what they wanted, they would have said faster horses. I gave them a car” (Henry Ford)
History – measure and feedback
Measure To know if it is effective Many monitoring and measuring tools
Feedback Tying feedback to ideas Quicker this is the faster you can confidently adapt Can be numerical or social
Changing Testing Concerns Over Time
Traditional Testing Maturity Models start at “ad-hoc” and move through structured testing before finishing at “Continuous Improvement”
Some of our test concerns (over time) Decrease Manual Testing & increase automated Acceptance Tests Performance DSL Speed of Feedback Increasing number of Edge Cases as system becomes more complex
Changing Testing Concerns Over Time
9,000 Behavioural Tests 8,000 Acceptance Tests (UI, API) 1,000 Integration Tests (subsets of services using internal interfaces)
Acceptance Test would test a rejection Integration Tests would test the different rejection types
Changing Testing Concerns Over Time
Was right at the time as needed Acceptance Tests to get good coverage quickly, trade-off is they are slower to run repeatedly.
Focus is now quicker feedback.
Adaptations – Fitness Landscape
Adaptations – Fitness Landscape
Each square is similar to it's neighbour Higher it is the better the solution – valleys are bad, mountaintops are good
You don't fully know where you are on the landscape You feel your way by trying out new ideas and recognising failure You may be half-way up the mountain rather than at the top And the mountains move!
Some peaks move faster than others: Google v McDonalds
Evolution isn't just biological it's everywhere
Adaptations – Palchinsky Principles
Peter Palchinsky was a Russian engineer who advised on (against) Lenin Dam and Magnitogorsk – large projects which essentially failed due to being too big and unable to adapt.
Palchinsky Principles are shaped to encourage stronger innovation, better leadership and more effective policies:
Variation - seek out new ideas and try new ideas Survivability - when trying something new do it on a scale where failure is
survivable Selection - seek out feedback and learn from mistakes as you go along, avoid
an instinctive reaction of denial
Adaptations – Recognise failure quickly
Being able to recognize a failure means you can re-cast it into something more likely to succeed.
Accept something is failing, don’t deny or ignore it.
"Hard to see what changes would improve current process when they’re currently sweeping all problems under the ‘our context’ rug." (Anthony Green)
The essential first step is being willing to fail.
Adaptations - Google v Apple
Both considered as innovative companies.
Apple innovates in things that are core to its business Google innovates on things that indirectly contribute to its business
Apple releases products that are polished and well-scoped Google releases many different products (Buzz, Wave, Docs, AppEngine) early in
their development and lets the community develop and adapt them.
Apple create new mountains on the fitness landscape Google is surfing along on, or near, the moving fitness landscape mountain-top by
adapting its strategy and offerings.
Adaptations – a brief history of what we have tried
Adaptations – what worked
Production Data Testing in Live Big Feedback Intermittent Tests Tester Pairing
Adaptations – Sanitised Production Data
Raw data (database) Patterns of use Data profile patterns
Migration (automated) Real world data Larger than we can create Many edge cases Try and reincorporate into our Acceptance Tests Performance Staging
Progression in what and how we use the data
Adaptations – Testing in Live
Uses same hardware and code as it is just an isolated venue Runs a subset of the Acceptance Tests
When we expanded our Exchange by adding a multi-tenanted concept known as a venue we took advantage for testing
Verifies that the various components of the exchange are hooked up and configured correctly and that the deployment process worked correctly
Runs automatically throughout the day as one of our monitoring systems
Adaptations – Big Feedback
Adaptations – Big Feedback
Important CI pipeline builds at the top Red (failed) items rise to the top Continuously and easily expanded Useful features survive
Failure ownership/failures at a glance - AutoTrish Intermittent tests - Magic Eight Ball Auto Embargo Performance metrics
Build sheriff to keep check-in embargo
Adaptations – Intermittent Tests
Tests that can fail even if feature is working Have 1 or 2 intermittent tests per run (out of 8,200) Each about 1 test run in 20
Hides genuine failures Have previously missed a release as couldn't tell if genuine
Can result from Asynchronous messaging Tests changing global settings/lack of isolation End of day transitions Date/time
Adaptations – Big Feedback
anon
anon
Adaptations – Intermittent Tests
All Failures run twice Fail twice = failure Fail once, pass once = intermittent Accepted as something we need to deal with
Work to remove DSL Fix a genuine issue (race condition) End of iteration is given over to removing intermittency Build Sheriff also works on this
Adaptations – Intermittent Tests
Adaptations – Tester Pairing
Each tester has their own strengths
Noticed that at rotation handover the new tester generally found a bug Issue with the testers (denial) Something we can harness (acceptance)
Already do development pairing (Dev & Dev and Dev & Tester)
When do we pair? At the right time for the story
Initial brainstorming when creating a mindmap When we feel we need to (pull) When someone has time (push)
Adaptations – mindmap
Adaptations – what didn't work
Not many absolute failures
Although many reasons it can fail
Even if something fails you learn something
Natural selection will continue
You can't stay still, unless you are improving you are losing ground as others will innovate or adapt
Adaptations – facts
Of the top 25 industrial corporations in the United States in 1900, only two remained on that list at the start of the 1960s. And of the top 25 companies on the Fortune 500 in 1961, only six remain there today (IBM)
Average life expectancy of a Fortune 500 company has declined from around 75 years half a century ago to less than 15 years today, and heading towards 5 years (Deloitte)
Future
Expanding the way we test
Speed up feedback Increasing integration test coverage Use PhantomJS for UI Acceptance Tests
Improving feedback/communication Test team standups
Documentation What is just enough How to get people up to speed in a complex system
Future
Is about the people
Team make-up is varied and we recruit to keep it this way
“’Testing is a strictly formal process that by its nature should not vary’ - this is a typical belief by people who know nothing about testing.” (James Bach).
Conclusion
Need to experiment and adapt to stay ahead
Even if something works it does not mean there isn't something better
Make changes on a scale where failure can be survived
Do not be afraid of failure but know how to recognise it quickly
Further Reading
• Tim Harford• Richard Wiseman• Malcolm Gladwell• Daniel Kahneman• Dan Ariely• Richard Thaler
• LMAX Staff blogs - http://blogs.lmax.com/staff-blogs/