1© 2011 Infosys Ltd.

ERP Testing Challenges– Need for a Stable Test Environment

Presented At STEP AUTO 2011 – ERP testing Conference – Dec

8, 2011

byA. Raghunath, Senior Project Manager – Package testing

Pinaki Sen Sarma, Project Manager – Package testing

2

Table of Contents

• Nuances of ERP Testing & related discussion

• Typically how QA environment is provided

• Environment transition activity

• Types of environment issues

• Environment Issue Impacts

• Recommended activities for improved environment stability

• Conclusion

3

Nuances of ERP Testing

Integration of

multiple modules

But are these enough?

What are the critical success factors other than skill set?

Do we take anything for granted?

There may be quite a few contributing factors,

but for our discussion, we will focus on THE

NEED FOR A STABLE TEST ENVIRONMENT

The bigger picture – End

to end perspective

Niche Skill set

4

Typically how QA environment is provided

• The instances for QA environment (Stage environment) could be created in multiple ways:

• Copy of past release environment • Copy of production

• Hardware of the QA environment is almost never at par with the production environment because of the cost factor involved!

5

Environment transition activity

• Environment related verifications done during transition phases are: i. Pre-flow activity

ii. IT Verification

iii. Smoke Test

iv. Ramp-up

6

Types of environment issues and associated impacts

a. Concurrent Program (CP) Issue

b. TIBCO Issue

c. Data-sync Issue

d. Performance Issue

e. Unplanned Environment Downtimes

Teams

Issue QA Development Infra DBAs

Concurrent Program Issue Yes Yes Yes Yes (Occasionally)

TIBCO Issue Yes Yes No No

Performance Issue Yes Yes No Yes (Occasionally)

Data-sync Issue Yes Yes No Yes

Environment Downtime Yes Yes Yes Yes

7

Environment Issue Impact (Contd..)

Environment issues account for the most number of defects in a majority of the releases we have handled this far. Following is the Pareto analysis of one such release:

EnvironmentCode Related

Config/SetupSystem/

Backend data

Not a defectDocumenta-

tion

Design'Invalid Test

Script

Reqt not in Pjt Scope User-Error data

Enter

0

20

40

60

80

100

120

140

160

180

200

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%184

140

98

37 35 3220

5 4 2

32.97%

58.06%

75.63%82.26%

88.53%94.27%

97.85% 98.75% 99.46% 99.82%Pareto Analysis

Total Cum %

Around one-third (33%) of the total defects were environment related!

8

Environment Issue Impact (Contd..)

There can be a wide range of issues which fall under the environment issue category :

CDB & Concurrent Program issues constitute 26% of the environment related defects

Many of the CDB issues were pertaining to replication of data from DB1 -> DB2

Less frequent sub root-causes (like Cross Functional Issue, Invalid Trigger, SABRIX Issue etc ) have been clubbed under Misc

category.

9

Environment Issue Impact (Contd..)

It’s not just the environment issues per-se that affects testing, but also the frequency of recurrence of the same. While some of them are one-time issues, there are quite a few which recur very frequently, resulting in more effort wastage than originally anticipated:

10

Environment Issue Impact (Contd..)

We had captured data for effort loss across 2 consecutive releases and following were our findings:

Converting the above effort loss into dollar values shows the significant impact of an unstable environment on the overall cost.

Quarterly Release Total Available Hrs Actual Impacted Hrs

Release 1 – Core Projects 6050 610Release 1 – Project B 3778 880

Release 1 – Regression 1144 110Total Release 1 10972 1600

Total % Effort Lost for Release 1 15%Release 2 – Core Projects 2100 320

Release 2 – Project B 1900 472Release 2 – Regression 1900 309

Total Release 2 5900 1100Total % Effort Lost for Release 2 19%

Release Total QA Cost Actual Loss (%)

Actual Loss (‘000s of $$)

Release 1 700 K 15 % 105 KRelease 2 400 K 19 % 76 K

We need a stable test environment!

11

Recommended activities for improved environment stability

• Strengthening the environment transition process

• Data-sync verification during instance copy

• Dedicated Infra/DBA support to monitor environment issues

• More hardware support for testing instances

• Robust resolution mechanism for QA issues (w.r.t environment)

• Creation of focused teams for environment monitoring

• Separate performance test environment

• Looking at new paradigms like Cloud computing / Virtualization as options for creating the test instance

12

Conclusion

Initial drivers for cost reduction – Offshoring

Looking for additional efficiencies – one key area is a stable QA environment

Adequate attention, proactive monitoring and other factors can reduce cost between 15 – 20%

Newer Paradigms like Cloud computing / virtualization of QA environment will help in driving additional efficiencies.

THANK YOU!

© 2011 Infosys Technologies Ltd. 13