Maximize ROI from Automation Budget
12-12-12
Sreekanth Singaraju
Vice President of QA & Testing Services
Allianceglobalservices.com
2 © Alliance Global Services 2012
Automation
3 © Alliance Global Services 2012
Promise of Test Automation
Accelerated Testing Cycle
Decrease time-to-market for new applications and products – especially for regression testing
Increased Coverage
Enable more extensive and detailed testing
Repeatable and Auditable
Implement formal testing processes that provide evidence of execution and are easy to execute
Accuracy in Execution and Reporting
Not prone to human judgment or errors
Enhanced Productivity
Execute repeatedly and often; Future test cycles will be more productive
Use of correctly implemented automated frameworks aligned with organization objectives, deliver substantial benefits to the organization.
4 © Alliance Global Services 2012
Reality of Most Test Automation
Implementation More expensive than planned
Takes longer
Fewer manual testing resources replaced
ROI is not met
Steady State Costs more to maintain than planned
Never seems to run successfully
Development loses buy-in
Out of sync with the application
Many automation projects unfortunately fail before they deliver projected ROI or substantially under-perform on the objectives.
5 © Alliance Global Services 2012
Impediments to Achieving Best in Class Automation
Objectives Non-existent or Unrealistic automation objectives
Focused around technical goals rather than business goals
Automation Strategy
Highest automation ROI areas not identified
Subscale initiatives or insufficient commitment
Start-Up Costs
Tool licenses and framework creation costs
Up front Labor cost for implementing test cases
Expertise & Assets
Manual test teams lack development expertise
Knowledge of test case lifecycle mgmt. missing
Measuring Effectiveness
Lack of a metrics program to measure alignment with overall objectives
Evaluate effectiveness of testing programs and fine-tune strategy
6 © Alliance Global Services 2012
FIVE PROVEN STRATEGIES FOR A
SUCCESSFUL TEST AUTOMATION
7 © Alliance Global Services 2012
Objectives
8 © Alliance Global Services 2012
Automation Objectives
Testing objectives need to be clearly defined and derived from business objectives.
These then serve as a guidepost for managing Automation programs.
• Execute a rapid Test Assessment for anyone planning to build or
refactor an automated regression test suite to identify business
objectives and current state of capabilities of the test organization
• Automation objectives are inherently easy to quantify, since a
baseline for costs, timelines and resources exists from comparable
manual testing
• Current state of capabilities and assets provides an understanding of
resources needed to implement automation
• Sets accurate expectations to key stakeholders
• Provides benchmarks for evaluating the progress of the automation
initiative
9 © Alliance Global Services 2012
Automation Business Objectives – Example
A high level business objective was provided to decrease cost of testing while
providing same or better level of quality control.
• Utilized Alliance’s Assessment
kit to perform a rapid
assessment in matter of
weeks
• Mapped business objectives
to specific Testing objectives
• Defined automation initiatives
that need to be implemented
to achieve Testing objectives
• Baselined existing metrics on
the key objectives to be used
for measuring ROI and course
corrections
10 © Alliance Global Services 2012
Automation Business Objectives – Example
A high level business objective was provided to decrease cost of testing while
providing same or better level of quality control.
Business Impact Area Current State
Quality
Documented Regression Test scenarios Less than 50
Regression testing for Mini Releases No Exhaustive Regression testing for Mini Releases
UAT Test Validations process Business and IT teams validate tests and is very expensive
Test data planning Test data setup after UAT start date extends the testing cycle time
Speed / Efficiency
Reduced # of Testing Cycles of Major Releases
4 Cycles
Duration of Test execution of Major Releases
25 calendar days
Internal business team effort reduction 221.75 days of effort (60% of IT (10) and 60% Biz (4) over 25 days)
External team effort reduction 13.25 days of effort (25% of 4 resources over 25 days)
11 © Alliance Global Services 2012
Automation Strategy - What To Automate?
12 © Alliance Global Services 2012
Automation Strategy - What To Automate?
Automation Candidates
High Business Value
Workflows based on large number
of test data combinations
Frequently used functionality
Long useful lifespan
End-to-End scenarios across
complex sub-systems and
integration points
Tedious and/or resource
intensive to conduct test
manually
What to Automate is the single most important planning step as it defines directly the achieved business value of the automation program
13 © Alliance Global Services 2012
Prioritization
•Obtain the candidates and assign them scores on Business Value, Expected Use of Scenario and Ease of Automation
•Interface with business and development team to obtain inputs
Code
•Cyclomatic complexity of modules and dependencies
•Interactions with internal external systems
Data & Realtime Tracing
•Volume of Data generated
•Importance of data
•End-user usage needs
What To Automate (cont’d.)
Analyse the applications source code, database and real-time tracing to highlight candidates that should be then prioritized by business value, longevity and ease of automation
14 © Alliance Global Services 2012
Upfront Costs
15 © Alliance Global Services 2012
High Upfront Costs
Tools Explore multiple tools strategy if feasible to decrease the cost of testing
Resources Explore jumpstart programs to get valuable expertise into your teams rapidly
Framework & Build-out Attempt the smallest but highest business
value modules first – outsource if necessary
Infrastructure Utilize cloud or virtual models for Test Environment and Automation Infrastructure
16 © Alliance Global Services 2012
Expertise - Frameworks In order for automated tests to be reusable, repeatable and maintainable, test development standards need to be defined and followed. These can be achieved by defining the framework and guidelines document has to be taken as the base and follow the development architecture finalized. There are some basic features that a framework is expected to have which include:
Application Independent
Easy to Use
Easy to Maintain
Is Scalable
Execute on Demand
17 © Alliance Global Services 2012
Automation Framework
• Automation dependent on an efficient framework and governance
• Utilize quality measurement and reporting to evaluate effectiveness
• Execute and automate all aspects of testing from Framework to TestLab
• Reusable test data and test environment management processes
RESOURCES Experience Expertise
Flexible Scalable
ASSETS Accelerators
Frameworks Templates
TDM & STRATEGY
E2E TEST ENVIRONMENT
METRICS DASHBOARD
AUTOMATED TESTING FRAMEWORK
Successful automation frameworks focus on all aspects of automation and plan for resources and reusable assets
18 © Alliance Global Services 2012
Titanium xAFT – Enterprise Test Automation Framework
Titanium xAFT (eXcel Automated Functional Tester) is
a robust enterprise test automation framework – Single
Script can be executed against Multiple execution
engines and browsers
Supports HP
QTP, Selenium,
TWIN execution
engines
Flexible &
Maintainable
framework
Based on Open
and proven
technologies
Titanium xAFT is a Open-Source Test Automation Framework to develop Automation
rapidly across the enterprise while creating highly maintainable suites.
30% productivity
improvements
over Selenium
automation
Collaborative
script
development
by BAs and
Testers in
Excel & Open
Office
Cross-browser Mobile - Web WebServices
19 © Alliance Global Services 2012
Measuring Effectiveness
20 © Alliance Global Services 2012
Measuring Effectiveness
Automation initiatives need to have metrics, reporting and analysis as an integral
part of the initiative to ensure success.
• Metrics should capture both business level and execution level values
to ensure efficiency is accurately captured and reported
• Metrics should not be left for the end of the project to implement
• Metrics measurements make implementation level course corrections
and expectations management easier
• Business Objectives identified should be directly tied to the metrics
program and measured throughout the initiative
• Assumptions and approximations can be made at the beginning of the
initiative
21 © Alliance Global Services 2012
Business Metrics
Value Add Before Titanium Testing
Services After Titanium Testing
Services
Quality
Documented Regression Test cases
Less than 50 1900+ implemented in test management tool
Enabled Mini Support releases and Regression testing for Hot fixes
No Exhaustive Regression testing for Mini Releases
Enabled 2 Mini releases and 4 Hot fixes into release calendar
UAT Test Validations Business and IT teams validate tests
Testing team validates tests while Biz, IT approve results
Test data planning Test data setup after UAT start date
Test data setup and approval prior to UAT start date
Speed / Efficiency
Reduced # of Testing Cycles of Major Releases
4 Cycles 2.5 Cycles
Duration of Test execution of Major Releases
25 calendar days 13 calendar days
Internal business team effort reduction
221.75 days of effort (60% of IT (10) and 60% Biz (4) over 25 days)
74.5 days of effort (40% of IT (10) and 40% of Biz (4) over 13 days)
External team effort reduction 13.25 days of effort (25% of 4 resources over 25 days)
0 days of effort (Regression Test bed also supports Releases)
22 © Alliance Global Services 2012
Execution Metrics Dashboard
• Should provide risk-based actionable information
with drill down capabilities
• Aggregates information from all aspects of testing
• Metrics should include
– Testing Confidence
– Testing Design and Execution Velocity
– Design Effectiveness
– Automation Coverage
– Automation Backlog
– Defect Identification Efficiency
– Cost of Quality
23 © Alliance Global Services 2012
Thank You Sreekanth Singaraju
Title – VP of QA & Testing Services
Email - [email protected]
Website - www.allianceglobalservices.com