Post on 23-Mar-2022
transcript
STeP-IN SUMMIT 2008
5th International Conference On Software Testing
Delivering Defect Free High Customer Experience Solution through Agile driven Test Automation and
Innovative Defect Prevention Practices in Multi OEM Vendor/SI Environment for a European Telecom
Giant
by Amandweep Goel,
Subramanian Thangavelu & Venu Kozhummal
Infosys Technologies Limited
Amand_Goel@infosys.com SubramanianT@infosys.com
Venu_Kozhummal@infosys.com
Copyright: STeP-IN Forum and Quality Solutions for Information Technology Pvt. Ltd.
Published with permission for restricted use in STeP-IN SUMMIT 2008 in agreement with full copyrights from owner(s) / author(s) of material. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise without
the prior consent of the owner(s) / author(s). This edition is manufactured in India and is authorized for distribution only during STeP-IN SUMMIT 2008 as per the applicable conditions.
Practices Experience Knowledge Automation
Produced By Hosted By
www.stepinforum.org www.qsitglobal.com
STeP-IN SUMMIT 2008 Delivering Defect Free High Customer Experience Solution through Agile
Paper Publication
driven Test Automation and Innovative Defect Prevention Practices in Multi OEM Vendor/SI Environment for a
European Telecom Giant
http://www.stepinforum.org/
http://www.qsitglobal.com
1. Abstract
Customer Experience has multiple dimensions to it, of which most important is the density of
delivered defects. High Customer Experience derives itself as a result of close partnership
between the participating organizations through and through, but is never complete without a
Defect Free Delivery.
While cross team discipline and strict process adherence might seem like a wholesome recipe to
achieve precise control over Defect Injection Rate [DIR] and a 100% Defect Removal Efficiency
[DRE] thus enabling Defect Free Delivery, it cannot ensure success when
1. The project requires very high effort and is expected to be executed in an agile manner
to enable real-time re-alignment with the constantly changing scope, not leaving any
waste for the client in the whole churn.
2. The system delivered interfaces not just with another software system, but also a real,
large, next generation nation-wide telecom network that itself is evolving hand-in-hand
with the system.
3. The delivering organization has to interact not only with its client but also needs to
manage, for its delivery, alignment with other suppliers to that client, which themselves
are some of the largest Consulting Firms, System Integrators, Network Equipment
Manufacturing Giants and Software Product Vendors across the globe.
4. Every defect gets attention of likes of the CIO’s in the client organization, given that the
initiative, to which the project contributes, is actively followed by analysts and hence
affects the shareholder confidence.
The authors here, attempt to present their experiments and success with continuous innovation in
management of one such project, resulting in exceeding the promised measures of customer
experience and 100% defect free delivery.
STeP-IN SUMMIT 2008 Delivering Defect Free High Customer Experience Solution through Agile
Paper Publication
driven Test Automation and Innovative Defect Prevention Practices in Multi OEM Vendor/SI Environment for a
European Telecom Giant
http://www.stepinforum.org/
http://www.qsitglobal.com
2. Target Audience
This experience share would prove useful to IT Project Managers who have committed to respond
actively to change, but are facing issues in controlling the quality of their deliverables. Also this can
be a thought provoking excerpt to those who have implemented Agile Models in their projects and
are looking for ways to mature further in that unexplored area. Finally the whole IT community
would benefit by extrapolating best practices established through this example.
3. Application Area
The project to which these practices were applied, aims to deliver under extremely challenging
deadlines & multi-vendor environment, a defect free (functional, performance & user experience)
web-based OSS (Operating Support Systems) solution which can provide flow through provisioning
(Service Fulfillment) and activation for the next generation network of the client.
4. Achieving 100% Defect Free Delivery and High Customer Experience
4.1 Need
Competitiveness in the telecommunications space asks for bringing new services to customer and
bringing them first in the market! To keep such a pace, the systems enabling that telecom company
should always service the customer orders ‘right in the first try’ and that too with less ‘Order Cycle
Time’ of the competitor.
For a System Integrator, this translates into an objective to match the pace by delivering the
solution right in the first go (defect free, both from the functional, performance and user-
experience perspective) without compromising the resilience to multi-magnitude of scope change or
the delivery timelines. Constant innovation is the order of day in such circumstances, especially
when agile working patterns are followed to achieve resilience to change, which by their basic
nature induce flatulence to the project execution.
4.2 Execution Details
A combination of the following was employed:
1. Continuous Integration through 100% Test Automation & Iterative Implementation to ensure
resilience to constant scope changes and elimination of verification and validation
overheads that come coupled with the growth of the application
2. UAT during Requirement Analysis Phase which enabled the software to be delivered
without any functional, performance and user experience defects
STeP-IN SUMMIT 2008 Delivering Defect Free High Customer Experience Solution through Agile
Paper Publication
driven Test Automation and Innovative Defect Prevention Practices in Multi OEM Vendor/SI Environment for a
European Telecom Giant
http://www.stepinforum.org/
http://www.qsitglobal.com
3. Performance Engineering and Test Driven Development that ensured the application to be
really ready for deployment anytime
4. Co-Location of a core system engineering team with E2E test and UAT teams that
reduced the communication wastage to nil and leveraged the collective knowledge of all
the teams to thrash out inappropriate defect candidates faster, resulting in reduced
delivery time.
5. WAR Room – Virtual combat of defects through constant analysis and optimization. Can also
be called as ‘Frequent Cyclic Retrospective’ [FCR] and ‘Active Defect Prevention’ [ADP].
4.2.1 Continuous Integration
Rapid realignment of the project scopes required breaking down the project scope into smallest
possible chunks and aligning the delivery in accordance, enabling the customer to have a fully
tested, working piece of software at any given point of time. Therefore, realigning the scope of the
project became feasible without generating any waste in the development and testing process. This
demanded for releases on much lesser intervals as small as a week.
Continuous Integration is a software development practice where members of a team integrate
their work frequently. Each time the integrated piece of functionality is verified by an automated
build-deploy-test process enabled by tools to detect software variances as quickly as possible and
get them fixed immediately.
A method of churning out weekly releases with fully tested working functionality was realized
through deployment of tools and methodologies that help automate the repetitive phases like build,
deploy, test, report and release.
Each tool that enabled CI was used to target a specific area of implementation:
# Tool Name Tool Description Used to 1 JUnit Open Source framework
for automation of unit
testing
Used to facilitate the development of automated
unit test cases for the Java classes employed in
the application
2 Apache ANT Freeware for automation
of build and deploy of
enterprise archives
Used for automated build and deploy of the
application. Removed person dependency and
zeroed the scope of manual errors in the build /
deploys
STeP-IN SUMMIT 2008 Delivering Defect Free High Customer Experience Solution through Agile
Paper Publication
driven Test Automation and Innovative Defect Prevention Practices in Multi OEM Vendor/SI Environment for a
European Telecom Giant
http://www.stepinforum.org/
http://www.qsitglobal.com
# Tool Name Tool Description Used to 3 Compuware
Test Partner
Third Party tool to
automate GUI based
functional testing
Used to develop scripts for automating the
functional testing of the application from end-to-
end concentrating more on the GUI and
automated interface
4 AutoIt Freeware Windows
Automation Language
Used to invoke Test Partner Scripts on Windows
after deployment is completed on Unix platform
thus enabling E2E automated execution
5 Cruise Control Used to co-ordinate the execution of build, Unit
Tests and AutoIt for functional test execution and
report from E2E process of release automation.
Open Source framework
for a continuous build and
test process
These tools together facilitated the build, deploy, regression test (unit tests and functional tests),
report and release process at the offset of a single mouse click, thereby achieving 100% automation.
Automated Build-Deploy-Test-Report-Release Process – How does it work?
The Automated Build-Deploy-Test-Report-Release Process is realized in the following manner:
1. Developer checks in unit tested code and test classes into the CVS
2. Release Manager schedules a build in Cruise Control
3. At pre-scheduled time Cruise Control checks out source code, JUnit test classes, build and
install scripts from CVS
STeP-IN SUMMIT 2008 Delivering Defect Free High Customer Experience Solution through Agile
Paper Publication
driven Test Automation and Innovative Defect Prevention Practices in Multi OEM Vendor/SI Environment for a
European Telecom Giant
http://www.stepinforum.org/
http://www.qsitglobal.com
4. Cruise Control runs JUnit test classes on the checked out code, and makes available for
viewing the report generated by JUnit in its dashboard
5. Cruise Control proceeds with the build if all JUnit test cases ran successfully; else halts
the build at this stage, which needs correction before proceeding further
6. Cruise Control builds and installs the Application using ANT build and install scripts
7. Cruise Control invokes AutoIt scripts on the Windows PC designated for functional testing
8. The AutoIt scripts perform post installation steps on the PC designated for functional
testing using the COTS Product Admin Client
9. The AutoIt scripts then initiate the Test Partner automated functional test scripts
10. Test Partner executes the functional tests on multiple environments in parallel to reduce
test time
11. Test Partner generates a report with the status of the functional test cases run, which can
be viewed from the Test Partner Admin Client
This completes the Build-Deploy-Test-Report-Release process without any manual intervention at all
and hence enables the project to have weekly releases without adding any overheads.
STeP-IN SUMMIT 2008 Delivering Defect Free High Customer Experience Solution through Agile
Paper Publication
driven Test Automation and Innovative Defect Prevention Practices in Multi OEM Vendor/SI Environment for a
European Telecom Giant
4.2.2 UAT during Requirement Analysis Phase
To enable the software to be delivered without
any functional, performance and user experience
defects, three-fold virtual UAT comprising of
Virtual Integration Testing, Performance &
Process Review and Continuous Usability Check
was launched that started right from the
Requirement Analysis Phase.
Virtual Integration Testing – For the system in context here, integration tests with interfacing systems and
the network [vended by world’s largest OEMs] used to be performed by the client teams themselves as a
part of UAT. This testing phase typically used to take a quarter for each quarter’s delivered
functionalities, since the defects identified in that late stage caused a big impact to by then hardened
system and resulted in increased turnaround times.
To compress this testing phase and reduce the
resultant high rework, the requirement team
while identifying the requirements, worked
alongside requirements team of other
components [delivered by other international big
consulting firms and SI partners] to virtually test
the interfaces through exchanging hand-crafted
XMLs for interface design and development
rather than just agreeing on interface
specification documents. These XMLs were then
used with stubs by all the interfacing system
development teams to validate the interface.
http
http://www.stepinforum.org/
http://www.qsitglobal.com
STeP-IN SUMMIT 2008 Delivering Defect Free High Customer Experience Solution through Agile
Paper Publication
driven Test Automation and Innovative Defect Prevention Practices in Multi OEM Vendor/SI Environment for a
European Telecom Giant
Performance & Process Review – With the new
evolving service and network design, the Solution
Design Teams [vended by a world renowned IT
consulting company] were concentrating on the
functional requirements to feed into component
delivery to meet the stated deadlines by the
business. This is a typical scenario for any new
initiative being rolled out. The common thing
that gets ignored is the non-functional
requirements and also the process change that
the transformation would cause. This void gets
detected late in the process post-deployment of
the solutions and causes a huge flux before it
gets settled thereby increasing the time to
actual-use-of-systems.
To tap these delays early in the implementation
lifecycle, the team involved itself with the users,
network, service and solution designers to arrive
at the expected real life loads that the
application would be subjected to and also other
non-functional aspects like security. In addition
to this, applicability of the assumed process was
checked for through close collaboration with the
users. The results were fed into the solution
design and major acceptance defects were
avoided.
Continuous Usability Check – Rather than
allowing the users to UAT the delivered system
and report back the deviations from the
expected, the requirements team and project
management involved users partly through the
above stated Process Review, but mostly also
organized active feedback sessions with users,
where the application was shown to users at
frequent intervals (almost once every month),
feedback was taken and preferences were
implemented after getting the solution design
tuned. This made users pretty comfortable with
the application and also avoided any UAT defects
that might have been reported even due to
incomplete solution design, something whose
delivery we did not even participate in.
http
http://www.stepinforum.org/
http://www.qsitglobal.com
STeP-IN SUMMIT 2008 Delivering Defect Free High Customer Experience Solution through Agile
Paper Publication
driven Test Automation and Innovative Defect Prevention Practices in Multi OEM Vendor/SI Environment for a
European Telecom Giant
4.2.3 Performance Engineering and Test Driven Development
Running into issues with the performance of the application and solving them, developing the
application and then testing it are the typical ways projects are executed. This pattern was
reversed and applied to get better control and early detection of defects.
Performance Engineering – A small team of two was set up to find the memory leaks in the
application that caused a threat to its sustainability, to perform load testing on the application and
report back to the users to get agreement or expected figures, experiment concurrent access and
http
http://www.stepinforum.org/
http://www.qsitglobal.com
STeP-IN SUMMIT 2008 Delivering Defect Free High Customer Experience Solution through Agile
Paper Publication
driven Test Automation and Innovative Defect Prevention Practices in Multi OEM Vendor/SI Environment for a
European Telecom Giant
object contentions through the available 100% automated test suite and proactively tune the
application making it ready for real live deployment. This made sure that not only the application
provides the functionality to the users; it can also sustain the live conditions and give extended up-
times that were required.
Test Driven Development - Test-Driven
Development (TDD) is a software development
technique consisting of short iterations where
new tests covering the desired improvement or
new functionality are written first, then the
production code necessary to pass the tests is
implemented, and finally the software is re-
factored to accommodate the changes. The
availability of tests before actual development
ensures rapid feedback after any change.
Practitioners emphasize that test-driven
development is a method of designing software,
not merely a method of testing.
This method of functioning achieves multiple
benefits
1. The tests are written before the
application is developed, this ensures
that the tests performed on the
application are user / requirement
centric rather that being design /
implementation specific. Thus the
typically unmet validation phase gets
pronounced as required.
2. Periodic tests performed help reducing the waste because there is always an option to
revert back to the previous fully tested version
http
http://www.stepinforum.org/
http://www.qsitglobal.com
STeP-IN SUMMIT 2008 Delivering Defect Free High Customer Experience Solution through Agile
Paper Publication
driven Test Automation and Innovative Defect Prevention Practices in Multi OEM Vendor/SI Environment for a
European Telecom Giant
http
http://www.stepinforum.org/
http://www.qsitglobal.com
4.2.4 Co-Location of a core system engineering team with E2E test and UAT teams
E2E Test and UAT Phase comprised of testing a stack of applications together end-to-end along with
the network downstream out of which the most critical application was the application in context
here. This was because the application was at the bottom of the aforesaid application stack
touching the network and translating a very high level order given somewhere near the top of the
application stack to complex inter-interlinked activations and also synching up the inventories of
the systems all along this system stack.
The complexity experienced therefore was to wait for the orders to start flowing through from the
topmost system in the stack and making its way through various systems, sorting out their
functionalities and interfaces in the process. In very obvious circumstances of these upstream
systems taking a lot more time to straighten themselves than what should have been
proportionately taken up, the system in context was just expected to work right every time the E2E
Test Team could achieve pushing an order through to it.
While the means discussed in the previous and the following sections ensured that the system in
context was robust and 100% defect free, any valid error even in the case of an upstream system
faltering its expected behavior was initially recognized as a defect in this system lying at the
bottom of the stack and touching the network. Since the timelines left for this last systems were
almost nil, escalations went to the likes of CIO’s of the client putting the credibility of the
delivering organization into question.
Rather than being reactive to this course of actions, and attempting to analyze the issues faster,
getting into face-saving discussions with senior managers at the client site, the project management
took a proactive step by co-locating a part of its engineering team with the E2E test team and the
UAT teams.
What resulted thereof was a combination of localized End to End tests between systems as a result
of recommendation of this engineering team thus giving equal breathing time to every system to
straighten itself with a clear understanding into the E2E and UAT test teams of the expected
behavior of various systems thus enhancing their accuracies in localizing the defected system.
STeP-IN SUMMIT 2008 Delivering Defect Free High Customer Experience Solution through Agile
Paper Publication
driven Test Automation and Innovative Defect Prevention Practices in Multi OEM Vendor/SI Environment for a
European Telecom Giant
http
http://www.stepinforum.org/
http://www.qsitglobal.com
This also helped the UAT teams to ask for minor confidence-building tweaks in the systems. One
such example for providing for a gate in the system touching the network which the user needs to
open before the system actually starts activating the network. This gave an opportunity to the UAT
teams to check everything that was going to the network before it went, acting as a huge
confidence building measure. And well all the other procedures and measures discussed here
ensured that it was always right!
4.2.5 WAR [Weekly Active Retrospective] Room
FCR [Frequent Cyclic Retrospective] & ADP [Active Defect Prevention] are the two terms coined by
the management of this project. While Agile Practices, cross-team discipline and strict adherence
to processes was helping the project sailing through very low Defect Injection Rates [DIR] and very
high Defect Removal Efficiencies [DRE], the retrospectives and Defect Analysis Meetings identified
that root causes of defects that slipped into further stages of SDLC were different every time and
that the processes set up for the project needed constant fine-tuning. Comparing this to the vision
of Defect Free Delivery that the management had set for the project, there was always a gap that
realized itself and broadened in between any two delivery cycles, thus allowing some dusty corners
to getting carried over into further stages.
Lateral retrospection of this strange pattern evolved an understanding of the way client was
reacting to the business and changing its processes every quarter and hence its effect to the
relevance of the set processes of the project in context here. This change was not just a quarterly
change, but a continuous one, primarily because the new processes identified every quarter took a
whole quarter almost to settle down thus causing continuous ripples to the project processes.
As an experiment, a weekly active retrospective
was scheduled to check the relevance of the
current processes and suggest changes. Also a
mini Defect Prevention exercise was undertaken
to discuss just the defects raised within the
previous week, identifying the root cause of each
one of them and setting process changes and
initiatives to combat the same. This exercise
involved every participant from the team that
delivered the functionality in which the defect
was identified indirectly building high
conscience, sensitivity and discipline in the
team.
STeP-IN SUMMIT 2008 Delivering Defect Free High Customer Experience Solution through Agile
Paper Publication
driven Test Automation and Innovative Defect Prevention Practices in Multi OEM Vendor/SI Environment for a
European Telecom Giant
The result was drastic reduction in the defects carried over to the further stages making it a defect-
free customer experience eventually [no defects in the E2E test and UAT phased and zero live
incidents].
5. Issues and Resolutions
1. Integration testing with other systems and network asked for tremendous manual regression
testing effort thus hitting the delivery time - Automated the tests even when testing with
the network and other systems!
2. Unavailability of test network and interfacing systems during requirements phase since
these were getting evolved in parallel - Worked with the respective co-vendors to get the
tests executed in their development labs and developed stubs to combat the absence of
interfacing systems!
3. Project Processes getting outdated with constantly changing Delivery Environment –
Implemented ‘Frequent Cyclic Retrospective’ and channeled the findings into ‘Continuous
Process Optimization’.
4. Developers and Testers mind-set as a blocker – Constant motivation of team members,
facilitation and encouragement of un-ending innovation opportunities.
5. In a multi-vendor scenario, aligning the deliveries with the program expectations was highly
difficult - Early identification of all dependencies, cross-alignment with co-vendor delivery
http
http://www.stepinforum.org/
http://www.qsitglobal.com
STeP-IN SUMMIT 2008 Delivering Defect Free High Customer Experience Solution through Agile
Paper Publication
driven Test Automation and Innovative Defect Prevention Practices in Multi OEM Vendor/SI Environment for a
European Telecom Giant
http
http://www.stepinforum.org/
http://www.qsitglobal.com
roadmaps, continuous progress tracking and developing a congenial environment where each
vendor contributes to the early integration testing thus resolving issues to a great extent.
6. Benefits Achieved
Following benefits were realized by applying these innovative practices:
1. Time to deliver: Software delivery cycle time brought down by 55%
2. Defect reduction: 100% DRE – No Defects in E2E tests, UAT and production
3. Effort reduction: Regression test effort maintained at zero even with linear growth of the
application
4. Increased Return On Investment: Tests automation in all areas saved 12% of effort which
was used to deliver more features to business at the same cost
5. Rework effort: Reduced from 5% to 1% of the total effort
Glossary
ADP - Active Defect Prevention EMS -Element Management System
ANT - Another Neat Tool FCR - Frequent Cyclic Retrospective
CI - Continuous Integration IT - Information Technology
CIO -Chief Information Officer OSS - Operation Support System
CLI - Command Line Interface PC - Personal Computer
COTS -Commercial Off The Shelf SDLC - Software Development Life Cycle
CVS -Concurrent Versioning System SI - System Integrators
DIR - Defect Injection Rate TDD - Test Driven Development
E2E - End To End UAT - User Acceptance Testing
XML - Extensible Markup Language
ALL RIGHTS RESERVED Copyright in whole and in part of this document ‘Delivering Defect Free High Customer Experience Solution through Agile driven Test Automation and Innovative Defect Prevention Practices in Multi OEM Vendor/SI Environment for a European Telecom Giant’ belongs to the authors. This work may not be used, sold, transferred, adapted, abridged, copied or reproduced in whole or in part in any manner or form or in any media without the prior written consent of the authors.
STeP-IN SUMMIT 2008 Delivering Defect Free High Customer Experience Solution through Agile
Paper Publication
driven Test Automation and Innovative Defect Prevention Practices in Multi OEM Vendor/SI Environment for a
European Telecom Giant
About the Author(s)
Amandweep Goel - The author is a Project Manager working with the ‘Communications Service Provider’ division of Infosys Technologies Limited (www.infosys.com), Bangalore - India.
He holds Master of Science, Birla Institute of Technology & Science, Pilani, India; Bachelor of Technology, National Institute of Technology, Punjab, India.
He has more than seven years of experience in implementing various software projects in Operational Support Systems (OSS) domains for major OEMs and Telco’s across the globe.
Currently he is responsible for delivering Service Provisioning and Assurance solutions to a major telecom giant in Europe using agile best practices and re-usable component development.
Subramanian Thangavelu, PMP - The author is a Senior Project Manager working with the ‘Communications Service Provider’ division of Infosys Technologies Limited (www.infosys.com), Bangalore - India.
He holds Bachelor of Engineering Degree from College of Engineering, Guindy, Chennai (Madras) - India.
He has more than nine years of experience in implementing various software projects in Network Management Systems (NMS) and Operational Support Systems (OSS) domains for OEM and Telco’s in America and Europe.
Currently he is responsible for delivering Service Provisioning and Assurance solutions to a telecom giant in Europe using agile best practices and re-usable component development.
Venu Kozhummal - The author is a Test Manager working with the ‘Independent Validation Solutions’ group of Infosys Technologies Limited (www.infosys.com), Bangalore - India.
He holds Master of Technology Degree from College of Engineering, Trivandrum - India.
He has more than eight years of experience in implementing various CAD-CAM Engineering, Banking and Telecom OEM software projects for clients across the globe.
Currently he is managing Integration Testing of multiple OSS Solutions
http
http://www.stepinforum.org/
http://www.qsitglobal.com
STeP-IN SUMMIT 2008 Delivering Defect Free High Customer Experience Solution through Agile
Paper Publication
driven Test Automation and Innovative Defect Prevention Practices in Multi OEM Vendor/SI Environment for a
European Telecom Giant
http
http://www.stepinforum.org/
http://www.qsitglobal.com
delivered by multiple System Integrators for planning, building and activating the next generation telecom network of a leading service provider in Europe.