SQS Group Limited
Scott Armstrong
Belfast, February 2010
Test AutomationA Roadmap For Success
Agenda
Steps for Success
Automation Tool Selection
Test Selection and Prioritisation
Implementing the Strategy
SQS Case Studies
Questions
2© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010
Steps for SuccessApproaching Test AutomationPlanning the Framework
© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010
Approaching Test AutomationSteps for Success
AUTOMATION PROJECT
DESIGN
Build maintainable tests
Target and prioritise test areas
TOOL
Select with care
Pilot tool
Manage Implementation
TEST MANAGEMENT
Defect Management
Config Management
Information and Reporting Requirements
TEAM
Code reuse and collaboration
Partnership between test and dev
PLAN
Treat as a Development Project
Manage Expectations
Commitment at all levels
© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010
Planning the FrameworkSteps for Success
What are the Management Information Requirements Contents Format
What are the Execution Requirements? On-demand Automated Nightly Builds
What technologies are we testing? Embedded Operating System
What resources are available? Hardware Technical Expertise
Are there cultural or organisational considerations? Outsourced IT Management – no admin access Current partnership with tool vendor
© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010
Automation Tool SelectionSelection ProcessIdentify RequirementsBuy, Open Source or Build
© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010
Selection ProcessAutomation Tool Selection
Define Problem
Consider automation
as a solution
Make business
case
Define required features
Shortlist
Evaluate
Demo
Trial
Decide
Constraints
© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010
Identify RequirementsAutomation Tool Selection
Start by evaluating your requirements, not by investigating the tool market.
Look at the current problems – what has prompted the search for an automated solution: Manual testing problems (e.g. too time consuming, error
prone); No time for regression testing when small changes are made
to the software; Set-up of test data or test cases is error prone; Inadequate test documentation; Don’t know how much of the software has been tested; Testing is ineffective.
Not all of these problems are best solved by automation.
Rank your problems by importance or current cost to the organisation.
© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010
Buy, Open Source or BuildAutomation Tool Selection
•Quickly provides a high level of features
•Well supported in terms of documentation, help and training
•Often an attractive project to be involved with
•Added cost of license fees
Commercial Tool
•No license fee cost
•Limited to support from community forums
•Defects in tool may not be addressed
•Tool may lack a clear roadmap; may not be future-proof
Open Source Tool
•No license fee cost
•Development and maintenance can prove expensive
•Designed to meet project requirements
•Lacks full range of features
Build Bespoke
Tool
© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010
Test Selection and PrioritisationTest SelectionTest Prioritisation
© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010
Test SelectionTest Selection and Prioritisation
Some tests cannot be automated.
Some tests are not worth automating. May take more time to automate tests than to run them
manually
You may not have to automate everything to get significant benefits 10% of the tests may be run 90% of the time.
Some types of tests are amenable to automation, some not: Functional tests are often the most straightforward Performance tests are notoriously difficult to carry out
manually. These are a good candidate for automation. Non-functional tests can cover maintainability, portablility,
testability, usability, etc, and can be inappropriate for automation.
© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010
Test PrioritisationTest Selection and Prioritisation
Do not automate too much too soon. Best practices take time to develop.
Go for ‘quick wins’ – the areas where automation will have the largest impact most quickly.
Prioritisation factors: A set of breadth tests (sample each system area); Tests for the most important functions; Tests that are easiest to automate; Tests that will give the quickest payback; Tests that are run the most often.
© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010
Implementing the StrategyAutomation Team FitMaintenance Model
© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010
Automation Team FitImplementing the Strategy
Automation Team
Specialist Group/Tool
Support
Subject Matter Expert
Management Buy-in
Development Team
Test Team
© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010
Maintenance ModelsImplementing the Strategy
• Client representative with technical background is trained during automation project
• Maintenance of current scripts can be supported independently for future product releases
• Future new tests can be automated in-house and added to the regression pack
Knowledge Transfer
• Client does not have to be involved in the details of automation development, execution or maintenance
• The necessary technical skills do not need to exist in-house
• Assured best practice automation and quality
Managed Service
© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010
SQS Case StudiesLloyds TSB Wholesale BankingLocal Utility Company
© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010
Lloyds TSB Wholesale BankingSQS Case Studies
Manual Testing Challenges•600+ Tests and
growing•200 man days per
release•Concerns over
coverage•Regression pack
growth impacted by limited resources
Our Engagement• Innovative test automation solution
implementation• Highly maintainable and extensible solution• Successful solution handover to client test team
Customer Success
Return on Investment•Increased test
execution efficiency by 85%
•Reduction in deployed defects >50%
•Enabled daily smoke test execution
•Overall ROI of £1.9m over 6 years
Lloyds TSB
“Without automation,
we would not have gone
live”
“The automated regression suite
made my job more interesting”
© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010
Local Utility CompanySQS Case Studies
Manual Testing Challenges•100 test scripts with
multiple complex data-driven iterations
•120 man days regression testing per year
•Low test coverage – limited to ‘spot checks’
•Testing of new functionality impacted by large regression packs
Our Engagement• Collaborative design of framework specific to
client requirements• Full knowledge transfer to existing test team for
ongoing maintenance and development• Modular framework allows new tests to be built
with minimal effort
Customer Success• SQS are now engaged on further automation
projects with the parent company• The existing test team continue to expand and
maintain the framework to increase their ROI
Return on Investment•Saving of 96 man
days of execution per year
•Data-driven scripts increased test coverage by a factor of approximately 200
•Automation solution was ported to a further 2 business units, multiplying the ROI
Local Utility
© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010
Thank you for your attentionThank you for your attention
SQS Group Limited
The Concourse | Northern Ireland Science ParkQueen‘s Road | Queen‘s Island | Belfast BT3 9DTPhone: +44 (0) 2890 022 660 E-Mail: [email protected]: www.sqs-uk.com
© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010