Test Automation Infrastructure - STeP-IN Forum · Q&A. STeP-IN SUMMIT 2008 ... An Intuit Case Study...

STeP-IN SUMMIT 2008 Test Automation Infrastructure

STeP-IN Forum & QSIT Copyright Page 1 of 42

1

- Sudhir Patnaik, Narendran Bhojan

Test Automation Infrastructure

2

Agenda

Test Automation Infrastructure – Introduction

The Need

Setting up a Test Automation Infrastructure (Test Center of Excellence – TCoE)

TCoE OfferingsTCoE Architecture, other technical aspectsThe Silk frame

Benefits/Metrics

Challenges

Next steps

Conclusion

Q&A



3


It is NOT:The only solutionA radically different approach to development/ QA

It is:A process that enables the team members to take firm control of the automated testing and product qualityA framework that allows execution of automated test suitesA vehicle to collect “automation” metrics to provide insight and opportunities for improvement

4


GUI Applications

Non-GUI Applications

Web-based Applications

Multiple platformsUnix/Linux/Windows



5

Need for Automation Infrastructure

1. Building Quality vs. testing qualityCurrent Methods

All types of testing

Testing Quality

Emerging TrendsBuilding Quality instead of testing quality

2. Lack of process DisciplineStructured vs. unstructured

Test Execution

Test Metrics

Test Standardization (scripts)

3. Quality compromise due to environment dependencies

6

Requirements for the Infrastructure

Improve Quality -> Higher Productivity

Predictability

Visibility

Efficiency

Continual Improvement

Self-directed teams

Mindset change



7

An Intuit Case Study

8

What the TCoE team offers?The TCoE team builds and maintains an infrastructure that supports:

Automatic execution of tests (BATs/FSTs) for daily builds

On-Demand execution of tests

Centralized test set creation/test execution

Centralized results management

Developer build tests

Efficient usage of hardware resources

User Permissions

Administration

Ability to add any new tools into the infrastructure

An infrastructure that supports multiple geographic sites

A standard that can be followed across the company for all automation needs



9

Some definitions

BATs – Basic Acceptance Tests:A set of basic tests that is run on all builds. Only when these tests pass, other tests are run on the build (Approximately 2-3 tests)

FSTs – Feature Sanity Tests:A set of more detailed tests specific to any area/feature (Approximately around 20 tests)

Smokes:A set of more detailed tests than the FSTs

Regression:The complete set of tests

10

Automatic execution of BATs

Runs automatically when a new build is available

Runs daily on a predefined set of branches

Tests need to be run is completely configurable

Machines are locked for triaging purposes

Configurable E-mail notifications

Results are viewable on the web

Results can be updated based on triage

Supports multiple products/branches

Configurable OS and sku’s

Note: All the test files are part of the build



11

BATs processing

1. Request Generated

-Build started by SCM (.FIN)

2. Allocate resources

-Machines allocated-Copy build to lab

3. Execute request-configure machine

- install build-Test configuration (e.g.

install Office)-Execute tests

4. Generate Results

5. Post process-If pass, then other

scheduled requests are kicked off

-If fail, machine is locked for triage

6. Cleanup-Image OS

-Free machine

Request

Process

Post-Process

12

BATs/FSTs Web center– results page



13

Automatic execution – tests configuration

Tests needed to be run after the BATs pass for every build can be configured through a simple text file

Every site and every branch has a separate config file

Task preferences can be set as to what is a primary test. This is used to update the SCM database

Developers can sync to builds that have only passed the primary test for further development

Config files are all under source control.

taskPrefs file for different sites:

14

On Demand test executionAll automated tests written by QA are integrated to the system (made available through the web pages)

These tests are available as test sets for all the engineers

They are grouped under three categories - FSTs, Smokes and regression

Only builds that have passed primary tests are made available inthis page

Request confirmation e-mail is sent to the requestor along with a link to the test status page:

Follows the same process as BATs except that the request comes from the web

We can abort the task in between if required



15

On Demand test execution contd…On Demand Page

16

On Demand test execution contd…Request Status Page



17

On Demand test execution contd…

Once the test is complete an email notification is sent to the requestor

If the test has passed the machine is freed after copying the result files to the network/file server

If the test has failed the machine is kept locked for a period of 6 hours for the engineers to triage

The lock time may be extended if required through the web

18

On Demand test execution contd…On demand ID and task ID

On Demand ID uniquely identifies the request you submitted

Task ID is assigned to each task or test suite that is executed

A single On-Demand ID may have multiple Task ID’s

Each task can run on multiple machines in parallel

If the test is taking very long, we can convert it to run on two machines in parallel and decrease the test execution time depending on the machine resources available



19

Centralized test set creation/test execution

20

Creating new test set

1. Enter test set name:

2. Select the applicable projects/products for the test:

3. Choose test type, and other options:



21

Creating new test set contd…

Edit the XML template and save it:

22

Centralized test execution

Once a test is added to the system it is accessible for all the engineers across all sites

Engineers have the option to execute tests On-demand, use the Task Builder or through the Developer Build Test (DBT) requests

QA Engineers use On-Demand for their automated testing

QA Engineers use Task Builder for their manual testing needs and cloning the task

Developers use the DBT’s for verifying their changes before checking-in their code



23

Task Builder

Two types:

QA task Builder – Used by the QA engineers for setting up the systems for performing some manual testing

Automation Task builder – Used to clone tasks that had failed already in order to triage again

24

QA task builderSimilar to the On-demand systemUses a separate pool of machines called the QA poolNo automated tests are available hereMachines can be locked for longer periods of time (weeks together)The important usage is to have a clean machine and install the product in it automatically so that they can perform manual testing



25

Automation task Builder

Similar to the QA Task builder

But the test sets are available here to be run

Is mainly used to clone tasks for triage purpose

26

Centralized results management

Results of all the tests are available on the web

Ability to search for results using task ID or Request ID (On-Demand ID)

Results filtering options

Separate results (called BATs/FSTs Web center) page for tests run automatically on the builds

Results show details of tests and sub-tests run and their status

Also the results link has a link to the directory on the file server where the result files are copied

Also has a link to open the machine using VNC viewer



27

Results Page

28

Developer Build testsDeveloper Build Test (DBT) is a tool that allows a developer to run automated tests against their local QuickBooks build. This way an engineer can verify that their changes do not adversely affect the build prior to checking in their changes Helps to verify bug fixes to BATS, FST or other test areasVerify changes on different Operating SystemsVerify changes in a non-developer environmentGive Tools/QA a heads up at coming UI changes which may adversely affect the existing tests, allowing time for automation engineers to update effected areas prior to the change appearing in the build Multiple DBT tests can be run in parallel



29

How DBT works

30

The DBT Request web request form



31

Tracking the DBT request

32

Results page



33

Efficient usage of hardware resources

The automation lab consists of both physical machines and virtual machines

Currently there are 15 physical machines and 65 Virtual machines

5 ESX servers are installed each with 13 virtual machines

Each virtual machine has 1 GB RAM and 40GB hard disk space

Few machines are also configured with 512MB RAM

34

Automation lab setup

The lab is split into QA pool and automation pool of machines

QA pool of machines is used by QA engineers for manual testing purposes

Automation pool is used for all the automated runs

Automation Pool QA Pool



35

Machine Usage

Machines are re-imaged after every test. The entire process is automated

The Operating systems that every machine can install can be configured through the “machine operating system Configuration page”:

36

Machine Usage contd…Adding a new machine:

Purpose can be QA or GeneralMachines marked as QA will be available in the QA poolMachines marked as General will be available in the automation poolMachines “permalock”ed are permanently locked and not usable



37

Machine usage contd..

Machines in automation pool cannot be locked for more than 6 hours

In order to lock for more time engineers contact the TCoE team through the ticketing system

An admin user can lock the machines for any number of days

Machines in QA pool can be locked for longer periods of time

38

Admin pages



39

Tools integration

Currently we have integrated the following tools in to the infrastructure:

Silk TestPowershellNunitAnd few in house tools like Dart, Qite and automator

All GUI tests are written in Silk Test using a Silk Frame built by the TCoE teamA coding standard is followed in the Silk tests so as to make the integration effort easierAny tool integration is done by the TCoE team who wrap the tools around scripts( mostly Perl) and execute tests so that the infrastructure understands the test failures and reporting back on the web pagesAny tool can be integrated to the system with ease

40

Support for multiple sites

Web server is the same for all the sites

Each site can have it’s own automation lab setup

Test sets are common and available in all sites

Products and Projects are common and available in all sites

Admin pages are common for all sites

Depending on which site we are in, tests are scheduled in that sites automation lab using the sites local builds and other test related files from the local sites local file server



41

A single point of access for all automation across the company

All Automation efforts across a company can be ultimately integrated into the infrastructure

A common coding standard

Better utilization of hardware resources in the automation lab

Common point of access for all historic results

Better management automation suites

A centralized way to share tests across multiple sites

A framework setup for continual improvement

Scope for integration into other test case management tools like Quality Centre or Test Link

42

Architecture and other technicalAspects

Distributed system architecture

The application server

The web server

The database server

The backend server scripts

Client side scripts

Machine cleaning/re-imaging

The test execution flow – Automatic BATs

The test execution flow – On-Demand

The test execution flow – DBT



43

Architecture

43

QA/Dev Application ServerAcquires Resourcesand Executes Test

UserEnters

Request

Display Results & Notify User

Record Results

Acquire Physical Machine

Resources

Execute Teston AcquiredResources

Acquire Physical Machine

Resources

Execute Teston AcquiredResources

Mountain View Bangalore

Build Sys

44

Architecture

DBserver

Web server

Appserver

MTV Autolab

Bangalore app server

Bangalore autolab



45

Application server

All the backend scripts run on the application server

There is one application server in every site

The application server takes care of the following things:

Adding new test requests to the DBChecking for new requests and allocating machinesStarting the tests on the clientsChecking for new builds and copying them to file serverProcessing automatic execution of BATs based on task preferencesCleaning machines/re-imaging services

46

Web server

There is a single web server (Apache) residing in MountainView

All pages are built using PHP

Provides the user interface for the following:Triggering tests On-DemandTracking requestsViewing the resultsRunning DBTsGeneral AdministrationViewing metrics Locking machines for manual testingTest sets managementRelated documents



47

DB Server

A common DB server is used located in MountainView

MS SQL Server 2000 is used

Stores all the related data such as:TestsResultsTasksMachine infoUser infoProducts and projects infoBuild infoGroups and permissionsOS info

48

Client Side Scripts

Listener.vbs

Host_run.vbs

Clean_image.vbs

Mount.vbs

Master.wsf

WinPEStartup.vbs

WinPEImage.vbs

TestExecutionEngine

TestResultWriter



49

Listener

Start

Look for command files in the commands share on app server

NewCommand File

available

Run the command on local machine

No

Yes

Delete the command on the commands share

50

Machine Cleaning/Re-imaging

Ghost images are taken with the following configurations:

All drivers installedNecessary user accounts created and passwords setNecessary folders created and permissions setScreen resolutions set and screen savers disabledRequired software’s like Perl, Silk Test, ultra VNC and WinZipStartup scripts added and enabledSysprep is done: Sysprep is the process of creating hardware independent images provided by MicrosoftWinPE boot images are created on this machine which is used by clean_image.vbs during re-imaging



51

Machine cleaning/Re-imaging contd…

StartUpdate the DB with the OS to re-image

for this machine

Copy the ISO image to local drive;Modify boot.ini to boot using the ISO image file

Restart the machine

Check IP address and map network driveTrigger WinPEImage

Check OS to image from the DBMap drive to Images directory on file server

Invoke Ghost (XP) or ImageX (Vista) with appropriate arguments to re-image

and restart the machine

End

Clean_Image.vbs

WinPEStartup.vbs

WinPEIMage.vbs

52

Test Execution Engine (TEA)

This is the execution engine that will run on the client machine while executing the tests

It is a C-Sharp application

The processTask.pl starts this application on the client machines

It executes the tasks specified in the Tasks file created by processTask.pl

It takes care of fetching the test details like the actual path to test script files to execute and calls the appropriate applications to run the tests

It passes on the test results to the Test Results Writer



53

Test Results Writer (TRW)

This is another application written in C-Sharp

It communicates with the TEA through sockets

It gets the results of the tests being currently run

The results are written in an XML format and stored in the fileserver

ProcessResultQueue.pl puts these results in the database

The web parses this data and displays in the correct format when the results page is accessed for any task or request

54

The test execution flow – Automatic BATs



55

The test execution flow – AutomaticBATs contd…

56

The test execution flow – On-Demand



57

The test execution flow – On-Demand contd…

58

Remote On-Demand User



59

The test execution flow – DBT

60

The test execution flow – DBT contd…



61

Remote DBT User

62

The development environment setup

There is another replicated setup of the infrastructure with a separate app server, web server, file server, DB server and few test clients

All the infrastructure scripts are first tested in this Dev environment before they are “integrated” to Live

For web code we can setup the development environment on any machine and then integrate the changes to Live

Both the Dev and Live code are under source control and are kept in sync across all sites



63

Silk Frame

“Building test cases around a powerful test frame is the heart of what we do with SilkTest.”

The test frame components are:Product windows and Child object definitionsReusable functions for common features in the productGlobal variables for use by all the test cases

E.g.: product install path, version, build etc

RecoveryCleanup after tests

Results reporting

Application states handling

Unexpected windows handling

Functions to let the infrastructure know about test results and report in the web pages

64

Recovery System

What the recovery system does?

• Launch the Application if not running

• Close extra windows; Report which ones were closed

• Set the application active. Report which other window was active

• Execute user-defined BaseState

Report errors to the results file at conclusion of each test



65

Benefits…

66

Benefits…

Test Automation is finding more defects earlier and more cost effectively

Enabled by higher test coverage, reduced execution effort and increased test execution frequency

4 key automation metrics are captured:Automated Test coverage – The percentage of tests automatedEffort spent on automation execution -Test throughput – Number of tests executed over a period of timeDefects found through automation



67

Automated Test Coverage

0%

10%

20%

30%

40%

50% Automated Coverage Level

Lava Kona

Automated Code Coverage increased from 19% to 42%

68

Effort spent on Automation execution

0

20

40

60

Overall Main St Payroll Platform AIIR

Full Regression Test Execution Effort (FTE Months)

Lava Kona

Reduced full regression testing effort required by 44%



69

Test Throughput

050000

100000150000200000250000300000350000

Throughput

Automated Test Throughput

Lava Kona

Testing throughput has doubled

70

Defects found through Automation

9% 15%

0%10%20%30%40%50%

% of Defects Found by Automated Tests

Lava June 2006 Kona June 2007

More defects are being found by Automated Testing



71

Lessons Learnt

Problems faced when going distributed:Single central database server or multiple database servers in each siteSingle web server or multiple web serversIf going for single centralized database and web server, the WAN link needs to be really fastIf going for multiple database servers, the method to sync data between the two databases needs to be identifiedArchitecture was rigid and it was difficult to extend it to multiple sites. This was mainly because the infrastructure was not planned to be used in distributed environment.The web forms needs to be designed in such a way that they do not do lot of database query which returns large amount of data at a single stretch. These pages take extremely long time to load over the WAN All the back-end scripts which connect to the database server remotely, loose the connection often

72

Lessons LearntDatabase :

We decided to stick with a single centralized database as we found that keeping multiple databases and synching data selectively was a maintenance overhead

Web server:Decision was made to have a single centralized web server after figuring out that having a local web server which will be talking to the remote Database server will even worsen the performance of the web server to a greater extent

Architecture:Necessary integration points defined and scripts written to handle multiple sites

Web pages:Moving from PHP towards Adobe Flex

Database connection problems:Changing scripts to use Restful services



73

Next Steps…

74

Automation Lab - Virtualization

Use VMWare Lab manager/ VMLogix LabManager to effectively use the Virtual machines in the ESX server

Lab manager allows saving machine states and takes care of allocating machine resources dynamically

Machines will be created On-demand and resources allocated on the fly

Many OS configurations will be available as part of the images for tests to chose from

This will ensure better utilization of hardware resources

Also machines wil not be kept locked for triaging; When triaging is required the machine state will be saved which can be later used for triaging



75

Reduce time spent on setup

Through Virtualization there can be many number of images with different configurations

The tools allow us to store the base image and only the difference in a separate snapshot

That way the image size also remains less

When ever a new build is available the product to be tested will be installed automatically on the required OS’s and an image will be made available with the installed setup

This will reduce the time required to re-image to an OS and also the time required to install the product for all tests

76

Better Results ManagementCurrently the results for a requested task are only available after the task has completedThis is because we store the results as a single XML in the database and parse this when displaying the results for the taskWith this approach it is only possible to show results after the test is completed as the results will not be available till the xml is loaded in to the database by the TestResultsWriterSo we are working on replacing the XML approach for results and storing results in different database tablesThe results would be updated in the database when every subtest has completedThe web pages would query the different results tables to display the results rather than just parsing a single XML file



77

Reduce Triaging Effort

Currently when a test fails the reason to figure out the actual failure is sometimes difficult

The failure may be caused because of product failure, product change, script failure or infrastructure issues

The time spent on triaging is sometimes high because of insufficient logging, non-availability of resources, triage engineers knowledge on the infrastructure etc

This can be improved through better logging, automated mechanisms to narrow down the failure and better training for the triage engineers

The next step would be to have the system automatically take action like logging a defect in to the defect tracking database or even fix the issues if possible

78

Integration of test case management and automation

Currently there is no tracking between test cases that are automated and tests that are not automated.

Ideally we should track all the tests in a test case management repository like Mercury Quality Centre or Test Link

Also integrate the current automation infrastructure to the test case management tool to enable the following:

Be able to select group of tests from the Test case management tool and run it in the labBe able to view results of the tests in the test case management toolAutomatically schedule the automated tests on builds based on the test plan created on the test case management tool on different milestones an publish the results



79

Non-Windows Support

Currently the test execution engine supports only Windows

In order to support for non-Windows clients only this engine needs to be re-written to work on different Operating Systems

Remaining mechanisms will remain the same

80

Next Steps - Summary

Improving the automation lab resource usage

Reduce time spent on test set-up

Better results management

Reduce triaging effort

Integration of test case management and automation

Support for tests running on non-Windows



81

Conclusion

Key points:In order to achieve a successful automation infrastructure, think in long termBe confident and be patient to get the ROIInvest enough on resources and get the design and architecture rightHave a dedicated Automation development teamKeep on evaluating new tools which would improve the infrastructureThink in terms of company’s future roadmaps and the products that you will need to supportThese steps will ensure that we are on track for the next generation of test automation requirements

82

Q & A



83

Thank You..

Sudhir PatnaikGroup Manager-QA

[email protected]

Narendran BhojanSoftware Engineer - TCoE

[email protected]

Date post:	01-May-2018
Category:	Documents
Upload:	duongdien
View:	221 times
Download:	3 times

Test Automation Infrastructure - STeP-IN Forum · Q&A. STeP-IN SUMMIT 2008 ... An Intuit Case Study...

Documents