Wim Demey - Regression Testing in a Migration Project

Copyright © CTG, Inc.

Organize regression testing in companywide migration project

Challenge or nightmare?

Eurostar 2009 – Wim Demey


Agenda

Context Project Challenges Lessons learned Questions


Context

Similar project in timeframe of 2 years Customers within same sector Organisation

Business Unit A

Business Unit B

Business Unit D

ITS

Infrastructure ApplicationsDivision A

Business Unit …

Division …

Project


Context

Project mode -> service mode

Project mode

Initiate Define Design Build Implement Close

- Project methodology needs to be followed

- A lot of deliverables (project + specific test deliverables)+ Test strategy+ Test requirements matrix+ Test scenarios+ Test reports (intermediate – progress – final)+ Defect management

Project leader

Test coordinator

Test team

Service mode

Business Unit B

Service Delivery Manager

Business Unit …

Business Unit A

SLA based support for application(s)

Maintenance budget


Context

Test awareness within organisation Pool of test resources // separate team

Mainly test coordinator profiles Methodology forces projects to use pool/test team

Especially for large projects Being compliant with methodology requires a lot of time Internal/external audits

Difference in focus on testing Infrastructure <-> application development

Separate test environments

Development Validation Production


Context

Work station concept -> standard environment By default no admin rights Default software for every user Additional software/rights on demand and dependent of business unit

Application software

Default software

Operating System(SP’s, hotfixes)


Agenda

Context Project

Content Test approach

Challenges Lessons learned Questions


Project >> Content

Initiate

WHO HAS INITIATED PROJECT?

-> Infrastructure-> Similar project methodology as application development

WHAT IS PROJECT SCOPE?

-> Default software upgrade (e.g. Office2007, IE7, SapGui 7.2)-> Front-end project-> No upgrade of OS

STRUCTURE OF PROJECT TEAM?

-> Infrastructure architect team has lead-> Organized in tracks-> Involvement of test coordinator (build <> define phase)

WHAT IS IMPACT FOR USERS?

-> All business units (company wide)-> All workstation types (12.000 – 3.400)-> Way of migration (upgrade <> re-install)

Define Design Build Implement Close


Project >> Content

What is new in Office2007?

Ribbon (with tabs -> menus)

Office button

New and xml based formats-> .docx, xlsx, .accdb -> possibility to save as PDF


Project >> Content

What is impact of software upgrades? Huge number of Office related applications

Mainly Access applications Heavy Excel files (VBA, macro,…) New behaviour of Word <> regulations Compatability of third party software with new Office formats

New software versions as prerequisite for ongoing projects Design decisions

Out of the box installation <-> Office configured to needs of BU’s Increase security-level (VBA, macro) in Office and IE

What about applications in project mode?


Project >> Test approach

WHO PERFORMS THE TESTS?

-> SDM + team -> Business -> (support by internal test team)

HOW IS TESTING ORGANIZED?

-> Mini test cycle (ad hoc, on preliminary build)-> Major test cycle (3m -> 6 months) (frozen build)

WHICH TEST ENVIRONMENT IS USED?

-> Sandbox idea (=virtual pc)-> SDM is responsible for installation of application + apps/test data

WHAT IS ROLE OF TEST COORDINATOR?

-> Follow up testing by SDM’s -> Follow-up defects/issues-> Ensure tight schedules are followed-> Deliver required test documents

MAIN GOAL REGRESSION TESTING

What works today, still works tomorrow on migrated software

SDM = Service Delivery Manager


Project >> Test approach

Overview test environment Sandbox contains clean installation of components

CORP LCORP: XPLV90000xx

- Internet Explorer

- Outlook RepositoryInstall applications

admin

Remote Desktop Connection

Production account

Test account

ValidationProduction


Agenda

Context Project


Challenges How to apply test approach? How to coordinate/motivate SDM organisation? How to gather/report information -> project? Does test environment fit the needs? Can we keep business out of scope?

Lessons learned Questions


Challenge 1: Test approach

Late involvement of test coordinatorLate involvement of test coordinator

Very rough estimations of test effortVery rough estimations of test effort

Confusion about what is in the project budgetConfusion about what is in the project budget

No official/central list of applications to testNo official/central list of applications to test

Project methodology / test deliverables not 100% fit for project

Project methodology / test deliverables not 100% fit for project

“Tour of SDMs”

-> Public - individual-> Get overview of portfolio-> Planning-> Roles + responsibilities-> Deliverables-> Support

Methodology adapted

-> Pragmatic but structure is essential-> Risk application test matrix-> Test scenario matrix-> MoSCoW principle

Clear out objections

-> Refine estimations -> Provide enough information-> Escalation to steering committee


Challenge 1: Test approach

Risk & Application Test Matrix

Sap GUI 7.0 IE 7.0 Word 2007 Excel 2007 PowerPoint 2007 Outlook 2007 Access 2007

0001 APPLICATION A SDM X N/A X N/A N/A N/A N/A X0002 APPLICATION B SDM X X N/A X X N/A N/A N/A

COMPONENTSID Application SDM

Impact Severity Risk exposure

Priority Overall Status

1 - Low 3 - High 3 Should Not started3 - High 2 - Medium 6 Must Preparation

Link TSM Comment


Challenge 2: Test coordination

No dedicated test resources available for this project

No dedicated test resources available for this project

No test cases availableNo test cases available No Office2007 knowledge to adapt application(s)

No Office2007 knowledge to adapt application(s)

Daily activities interfere with test executionDaily activities interfere with test execution

For some applications business has to perform tests

For some applications business has to perform tests

Technical support

-> Analysis of Office2007 issues-> Document solutions-> Assist developers during implementation-> Functional mailbox for business

Pragmatic follow up

-> Planning on weekly basis-> Network driven approach per SDM-> Test scenario template-> Ad hoc meetings-> Simple web based defect form


Test scenario matrix

Challenge 2: Test coordination


How detailed must info be collected/reported?How detailed must info be collected/reported?

Main focus on planning/milestonesMain focus on planning/milestones

No time to read long reportsNo time to read long reports

Which feedback do we give to SDM/business?Which feedback do we give to SDM/business?

Several documents as input for reportingSeveral documents as input for reporting

Challenge 3: Info gathering & reporting

Dashboard

-> Overview of progress (total, per SDM)-> Overview of defects (type, severity, trends)-> Small VBA code to gather automatically status update-> All information in one sight-> Basis for status meeting with project team

Test news letters

-> Only sent out if relevant & necessary-> Information about general issues & solutions-> Changes in test strategy-> Status about builds-> Keep it as short as possible


Challenge 3: Info gathering & reporting


Mismatch between theory and practiceMismatch between theory and practice

Test machines needed in production environment

Test machines needed in production environment

Teams create own test environmentTeams create own test environment

Some applications could not be tested remoteSome applications could not be tested remote

Need to travel around with test environmentNeed to travel around with test environment

Security causes a lot of problemsSecurity causes a lot of problems

Physical machines

-> Test accounts-> Separate container in Active Directory-> Availability managed by test coordinator-> Production like install of applications-> Staged by project team-> Pick up & return concept

Security level reviewed

-> Decreased to current level-> Alternative of exception policy doesn’t work-> Digital signing/increased security outscoped to separate project

Challenge 4: Test environment


Unknown number of applications developed by business

Unknown number of applications developed by business

Often very critical applicationsOften very critical applications

No support by ITS/helpdeskNo support by ITS/helpdesk

No experience / knowledge to solve issues

No experience / knowledge to solve issues

Problem of resources even higher than SDM teams

Problem of resources even higher than SDM teams

Not always clear what is expectedNot always clear what is expected

Challenge 5: Business

Active involvement

-> Same roadshows as for SDM-> Inventory made + prioritisation-> Critical applications are taken over by SDM-> Technical support-> Test scenario template provided-> Same defect tool + procedure-> Business representatives are SPOC


Agenda

Context Project


Challenges Lessons learned

Some facts & figures Conclusions

Questions


Lessons learned >> Facts & figures

Applications 1052 ITS supported applications identified (420)

650 to test (Must, Should, Could) 402 rest (Won’t) -> not impacted by components

All applications tested in major test cycle (3 months) One team had 6 months time due to number of applications (420)

Issues 180 issues reported (both test cycles)

Mainly Excel + IE7 related (70% of all issues) Compatibility issues with third-party software (e.g. SAS/SAP) REMARK: only for ITS supported applications No need to rebuild applications from scratch

Some issues led to delayed migration of users/divisions


Lessons learned >> Conclusions


Lessons learned >> Conclusions


Questions / Remarks

Wim Demey ([email protected])

mailto:[email protected]

Date post:	17-Jul-2015
Category:	Software
Upload:	eurostar-software-testing-conference
View:	61 times
Download:	0 times

Wim Demey - Regression Testing in a Migration Project

Software