8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 1/67
© 2007, Cognizant Technology Solutions. All Rights Reserved.The information contained herein is subject to change without notice. C3: ProtectedC3: Protected
Fundamentals of Test Execution
Session 1: Fundamentals of Test Execution
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 2/67
2
About the Author
Created By: Rajarajan(225558); Balasubramanian(226466);
Elangovan(208633); Venkatakrishnan(102083);Rohit(158977);Vishnu Priya(179360); Mathuram(105686)
CredentialInformation:
3-12 years of Experience in Software Testing
Version and
Date:
Version 5.0 and 22 Feb 2011
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 3/67
3
Icons Used
Questions
Contacts
Reference
Try it Out
Hands onExercise
CodingStandards
Test YourUnderstanding
Tools
AWelcomeBreak
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 4/67
4
Fundamentals of Test Execution Session[1]:Overview
Introduction: This session explains thevarious activities performed during the testexecution phase and the process to log thetest results and the defect managementmethodology
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 5/67
5
Fundamentals of Test Execution Session[1]:Objective
Objective:After completing this chapter, you will be able to:
» Describe various activities involved during TestExecution
» Describe and interpret a Test Execution Plan
» Describe Test Environment and the need for it
» Explain Build Management and Verification Process
» Describe the process of Test Execution
» Describe Triaging and Defect Management
» Differentiate Test Execution for Fresh developmentand Maintenance Testing
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 6/67
6
Test Execution: An Introduction
Fig: Test Execution Process
Planning
Activities
for
TestExecution
The Test Execution involves:
» Initial Planning for TestExecution
» Build Verification Process
» Test Case Execution
» Test Log Creation
» Defect Reporting and Tracking» Defect Triaging
» Updating Traceability Matrix
» Re Testing of Defects
» Regression Testing
» Test Execution Status reporting
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 7/677
Follows the Test Design phase in STLC
Process of executing a series of test cases asdefined in the test plan to validate if theapplication meets the requirements
The application is tested and the defects arereported to the development team
The fixes provided by the development teamare retested and ensured that they do notrecur, the fix does not introduce any new
defects Can be manual or automated using a tool
Test Execution: AnIntroduction(Contd.)
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 8/678
Below activities need to be taken care beforethe start of Test execution» Test Cases Review and Sign Off
• All the test cases that are identified for the specific level andtype of testing need to be approved and signed off by therespective stake holders
» Test Cycles• The number of rounds the testing will be conducted and the
scope of testing at each level need to be identified in the Testplan and agreed upon by all the stake holders
Planning for Test Execution
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 9/679
» Test Lab set up
• The methodology for deploying the build in the respective testenvironment need to be devised and approved by thedevelopment, testing teams and the customer
• Test Lab set up also includes creation of a test suite with the testcases which need to be executed in a specific sequence
– Creation of Cycles for each level of testing
– Identification of Smoke/Build verification test cases– Identification of Test Suite(collection of test cases) per cycle
– Frequency of Build delivery per cycle
» Task allocation / Run plan
• The allocation of the test cases to be executed need to be donebased on the skill set and the availability of the testers
Planning for TestExecution(Contd.)
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 10/6710
Planning for Test Execution:Execution Run Plan
Execution Run Plan
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 11/6711
Sample Execution Run Plan
Execution Run Plan - Sample
Execution Run
Plan Sample
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 12/6712
» Risks and Mitigation Process
• The potential risks that are identified for Test execution duringthe course of the project are revalidated and the availability of the mitigation and contingency plan for each of these risksneed to be ensured
» Escalation Process
• The escalation process if an unforeseen event occurs during
test execution need to be arrived and agreed upon by thestake holders
Planning for TestExecution(Contd.)
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 13/6713
Testing an application hosted in the desktops/Servers
used by the developers for coding purposes is notrecommended for the below reasons
» Both testers and developers will have access to the code
» Behavior of the application may be unstable if thedevelopers work on the code while the testers are testing
» The developers may have used simulated programs duringcoding and the testers (being unaware of this) may alsouse the same for testing. This may result in huge failurewhen the application is deployed in the productionenvironment
» Parallel or overlapping data update by various teams
Need for a Test Environment
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 14/6714
Mirror the production environment as closely as possible
The availability of the test environment exclusively forvarious types and levels of testing is always recommendedbut is subject to every individual project like
» Entire test environment is exclusive
» Database is the same but the hardware used is different
» A sub set of the data will keep changing for every cycle andthe master data remains the same
Need for a Test Environment(Contd.)
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 15/6715
Configuring of Test Environment can includesetting up of the following:» Software required(Ex. Tools like Eclipse, Crystal Reports)
» Hardware required(Ex. Special interfaces like HandheldDevice, Touch Screen)
» Server to host the application in the desired configuration» Database setup/Server Setup to run the application
» Appropriate level of access for test team to accessDatabase, server and application
» Appropriate level of access to external application if thetesting scope covers for the same
Test Environment Configuration
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 16/6716
» Application to be tested
» Test management tool set up
» Access to the remote desktops and the connectivity tothem
» Contact Details of the customer’s helpdesk should beavailable during the testing if the environment is slow
» Environment availability – All the environments may notbe available full time during test execution. There may beexceptions like
• An application will be running only during US business hours
• An application will be down for maintenance during US publicholidays
Test Environment Configuration(Contd.)
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 17/6717
Test Management Tool is a software used to plan and
execute testing activities like capturing requirementsand test cases, managing defects, collecting metricsand analyzing the reports
Version of the tool or the instance agreed upon in theTest Plan needs to be installed in the test environment
desktops Project has to be created in the tool to manage the test
activities and right level of access should be given to alltest and development users
All requirements should be uploaded into the tool by
Business Analyst / Testing team
Test Environment: Test ManagementTool Set Up
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 18/67
18
Cycles for execution needs to be created in the tool
under the project» Cycles agreed upon in the Test Plan is considered and may
vary during the test execution
» Cycle period or dates should be mentioned and it shouldbe mapped with the test cases that we plan to execute
The tool may also be integrated with the third party tool
Test Environment: Test ManagementTool Set Up (Contd.)
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 19/67
19
The test environment should be as close to the production
environment in terms of » Database(DB) set up
» Levels of users defined and the no. of users defined for eachlevel
» Volume of data in the database
» Type of data Detailed Environment Requirement has to be addressed and
tracked in a separate document like
If there are multiple test environments, the abovementioned points need to be ensured in all theenvironments
Test Environment: Database SetUp
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 20/67
20
Test Users set up
» The Appropriate roles identified as test data for testingneed to be created and obtained from the customer for allthe instances of test environments
» Once the test data is received, it is always better to ensureif the credentials provided works and has the appropriate
levels of access» In some cases, the credentials provided may be valid only
for a limited period of time. This may require us toreschedule the run plan for execution
Batch Job dependencies (if there are any) need to beidentified and planned accordingly
Test Environment: Database andTest User Set Up
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 21/67
21
Test Environment: Test User set up
21
Testers
Rights to accessDBServer
Application
Server
ExternalApplication
h
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 22/67
Test Data Set Up in the TestEnvironment
22
Primary / Secondary Data
» Primary Data is generally referred to the master data and is obtained fromcustomer. Example for Primary data tables are Employee Data, Countriesapplicable for the application
• Secondary / Transaction data
» Secondary data can also be obtained from the customer if the application(previous version) already exists in Production
» If it is a new version of application, Secondary data can be created and
customized during testing» Test team is responsible for ensuring the test data availability before test
execution
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 23/67
23
Test Data Management in TestEnvironment
23
Test data should be available before the test
execution
The data used for every cycle of testing mayvary like» The entire database is refreshed
» A subset of data is retained and reused
» Only the master data is retained
Tools can also be used for creating test data
Batch queries should be created for insertion/
Updation / Deletion of data in the tables Ownership of data has to be clearly defined
among and within the teams
T D M i T
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 24/67
24
Test Data Management in TestEnvironment (Contd.)
24
Sample Queries for cleaning up and inserting
the data» Delete from empdetail
» Delete from emp
» Insert into emp(table columns)(table Values)
» Insert into empdetail(table columns)(table Values)» Update can be done as required
Note: While doing deletion, secondary/transaction data should bedeleted first and then primary but it is vice versa during insertion
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 25/67
25
For every cycle of test execution, the
development team shares a controlled versionof software (Build) which needs to be deployedin the respective test environment and tested
Each build is associated with a version
number and a release note attached The release notes of the build primarily
includes information of the features of thesoftware included in the build and the
deployment details
Build Management
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 26/67
26
Frequency of build delivery to the project team
is as per the Test plan and in consensus withthe stake holders of the project
The files/DB files/Batch Programs used togenerate a build are controlled by a
configuration tool/ team assigned for it Build Manager is responsible for
communicating to the stake holders on therelease of the build to test with release notes
(In Some Projects, Program Manager) The version of the build is incremented for
every change in the build
Build Management (Contd.)
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 27/67
27
Build Verification Process
Build Manager
Test Team
BVT Failed
BVT Pass
Release Notes
DevelopmentTeam
Build the compiledcode into software
Adds the releasenotes
BuildVerification
usingSmoke /
Sanity Test
Test ExecutionTest Team
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 28/67
28
The units of software developed is integrated, compiled
and built as a single unit by the developers
The Build Manager will add the release notes to thisversion of the software build and release it for testing
The Testing team runs the identified Smoke and Sanitytest cases on the build, once it is ready
If the smoke/sanity test cases pass with the desiredresults, build will be accepted and the testing team willproceed with the Test Execution
If the smoke/sanity test fails with critical defects that
the team is not able to proceed with testing, the buildwill be rejected
Build Verification Process (Contd.)
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 29/67
29
Release notes includes details on» Version of the build
» Features included / not included in the build
» Defects that are fixed / not addressed in the build
Testers need to test only those features thatare mentioned clearly in the release notes
Importance of Release Notes
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 30/67
30
Test Your Understanding
Test YourUnderstanding
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 31/67
Questions on Build Management
BVT means ?
a) Smoke b) Regression c) SIT
What is the percentage of BVT test caseshave to be passed ?
a) 85% b) 90% c) Depends on the project
Release notes should be sent by ?a) Build Manager b) Program Managerc) Test Manager
31
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 32/67
32
Test Execution
Test Case Test Log document to log actual results
Identification of test cases assigned to each tester forthe day
Does theApplicationbehave asdesired ?
Log Test Results withproof of execution
Tester understands the dependency between testcases and identifies the sequence among them
Tester ensures the prerequisite of each test case ismet before execution
Executes the test case / Retest if the defect is fixed
Log defect / Updatestatus of defect
Update TraceabilityMatrix
Tester uses the test data identified for the test caseduring test execution
Test Case Execution and Logging
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 33/67
33
Test Case Execution and LoggingResults
Execute Step 1 of a test case
Does theApplication behave
as desired ?
Execute next test case
Log defect / Updatestatus of defect
Execute the next step of the same test case
Log Test Results for the test step withproof of execution
Is the next test stepavailable and
executable with thedefect raised for theprevious step in this
test case?
Test Case Execution and Logging
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 34/67
34
Each test case is executed step by step
After executing the test step, the actual behavior of theapplication is logged in the “Actual result” section of thecorresponding test step
If the actual result is the same as the “Expected result”, theapplication behaves as desired and hence the test step ismarked as “Pass”
If the actual result is the not same as the “Expected result”, theapplication is not behaving as desired and hence the test stepis marked as “Fail”
While logging the “Actual result” the behavior of theapplication, needs to be mentioned clearly. This will help the
following:» Other testers in the team to understand the impact of failure of this
test case on the test cases that they would execute
» Development team to identify the exact issue and addressing it
» Retrospection of the cause to introduce this defect
Test Case Execution and LoggingResults
Test Case Execution and Logging
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 35/67
35
If a particular test step fails, a defect needs to be logged in the
defect log and it is recommended to execute the consecutivetest steps if possible
This may not be possible in all cases, then the tester mayproceed to execute the next test case
A test case is considered to be pass only if all the test steps inthat particular test case pass
Test Case Execution and LoggingResults (Contd.)
ExampleIn the Leave Management system, While applying leave, if the “Type of Leave” combo box does not have the label name as expected, a defect needs to beraised and the testing can be continued for the consecutive test steps of thesame test case
ExampleIn the Leave Management system, While applying leave, if the “From” and “To” dates are not editable
Logging of the Actual Results in the
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 36/67
36
The actual result of a test case and the details of execution are
logged in a document called Test Log/ Test Management tool,irrespective of the behavior of the application
Test log sheet is a copy of test case document with additionaldetails like Actual results, Pass/Fail, Defect ID and the Testername who executed it
Test Log is a chronological record of the all the informationabout the test case execution
» What are the test cases executed
» In What order it was executed
» Who executed those test cases(Tester)
» Status of the each step of the test cases (PASS/FAIL)
» This can be created manually or using test management tool For some specific applications it is mandatory to log the test
results physically in the test case document, to be compliantwith the regulations
Logging of the Actual Results in theTest Log
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 37/67
37
Sample Test Log
Sample Test Log Sheet
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 38/67
38
Sample of the Test Case run status
Sample Test Case Run Status
Test Case Run Status
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 39/67
39
Testers raise defects when the expected result
and the actual behavior of the application donot match
The defect management methodology definedfor every testing project would be different
The testers need to follow the one defined inthe Test Plan of that particular project
The defect logging can be done in a simpleexcel document or any defect management /
test management tool adopted by the team
Logging a Defect
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 40/67
40
Key Elements of a defect includes:
» Defect Id• A unique Id is assigned to each defect raised during testing.
Usually, a naming convention decided by the testing team isfollowed. This can be auto generated if a test management /defect management tool is used
» Defect Description
• A textual description provided to describe the defect to theother stakeholders of the project
» Steps to reproduce the defect
• The list of steps to be followed to reproduce the defect
» Test Case Id
•The Id of the test case executed to find the defect is providedfor reference
• The Test data used to reproduce the defect may also beprovided here for reference
Logging a Defect (Contd.)
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 41/67
41
» Assigned to
• Each defect will be assigned to a developer to fix, the name of the developer is provided here. This is usually assigned duringthe defect triage meeting
» Module Name
• The module of the application in which the defect was found ismentioned here
Logging a Defect (Contd.)
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 42/67
42
Key Elements of a defect include
» Severity
• The impact of the defect in the software is called Severity
• Every project will define the severity levels of defects in the Test Plan
• Severity of a defect is generally set by the Testing team while logging defects
• A few samples of the severity levels of a defect are
Logging a Defect (Contd.)
Examples for Defects–Severity:
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 43/67
Examples for Defects–Severity:Critical
43
All pages areblank whilelogged into
theapplication
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 44/67
Examples for Defects–Severity: High
44
“Half a Day” option isnot available for
“Leave Start Date”
Examples for Defects–Severity:
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 45/67
Examples for Defects Severity:Medium
45
The calendaricon does not
allow to click
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 46/67
Examples for Defects–Severity: Low
46
The label isincorrect
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 47/67
47
Key Elements of a defect include
» Priority• Priority of a defect is generally determined by the
delivery manager of the project as defined in the TestPlan based on the following
– Severity of the defect
– Capacity management of the development team– Time required to fix a defect
– Impact of fixing a defect on the other defects –sometimes, fixing one defect may rectify many of therelated defects
• Each of these priority levels will have a time associated
with it, to enable the developers to fix within that time-– For Example, Critical defect may have a target resolution
time of 2-6 hrs, less critical defects can have the targetresolution time anywhere between 8-48 hrs
Logging a Defect
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 48/67
48
» Capture screenshot of the defect
• Though the details of the defect are documented, there may be cases
where the defect is not repeatable• A clear screenshot of the defect with the relevant area highlighted will
help the development team and the business to understand the defectbetter
• It is always better to highlight the area where the defect had occurredin the screen shot
» Defect Status
• The status of a defect would change from stage to stage during thedefect life cycle
• The defect life cycle is defined in the Test Plan for a project
» Test data used to identify the defect
• This will help the other stake holders to
– Reproduce the defect
– Identify if the issue is with choosing the test data
» Version of the build in which the defect is identified
» Test Environment details in which the defect occurred
Logging a Defect (Contd.)
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 49/67
49
Sample Defect Life Cycle
49
Sample Defect Life Cycle (Defect Status and the process may vary from Project to Project as per Test Plan)
Is ValidDefect?
NEW
OPEN
FIXED
CLOSED
REJECTED /CANCELLED
REOPEN
Dev Team
Test Team
If Defectfound?
Fix of defects
TestExecution
Fix in
thisrelease?
DEFERRED
Defectfound inRetest?
TriageMeeting
Best Practices While Logging
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 50/67
Best Practices While LoggingDefects
Log defects with:
» Clear description of the defect – as is in the application» Appropriate tone of the language
» Screenshot(s) with
• Clear marking of the defect area
• Call outs
• Timestamp
• Sequence of screen shots for each step if required
» Highlight if there are any similar/ related defects earlier in theremarks
» Mention the other areas/features of testing affected due to thisdefect in the remarks
» Even if the defect is observed once and is not reproducible, log it
and the development team and the business will decide toconsider it as a defect or just an observation
» In the above case, log the defect with more details on when thedefect occurred and when it is not reproducible
50
d h b l
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 51/67
51
The traceability matrix created during the
Requirements Analysis and the Test Designphase is updated with the defect details
The defect id is mapped against thecorresponding failed test case id
This will help in tracking which are therequirements that are misinterpreted /misunderstood by the teams
Only the defect id is mapped here and the
details of the defects are obtained from thedefect log
Updating the Traceability Matrix
R i f d f
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 52/67
52
Defects that are fixed by the development team are
retested by the testing team to ensure that theapplication behaves as desired after the fix
The status of the defect is changed to Re-Open and isassigned to the developer if the application does notbehave as desired, even after the fix
Otherwise, the defect status is changed to closed andthe test log is updated appropriately
In some cases, fixing of a defect would introduce newdefects in the same module or any other related moduleof the application. Hence the related modules also need
to be retested
Re-testing of defects
R i T ti
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 53/67
53
If the application under test is a maintenance
application, the enhancement to theapplication may introduce defects in theexisting functionality of the application
Needs to be done to ensure that the
application is in tact and behaves as beforeeven after the enhancement/ defect fix
There will generally be a set of regression testcases identified or automated test suite
already available
Regression Testing
Best Practices During Test
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 54/67
Best Practices During TestExecution
Identify the business critical/ error prone areas and
focus on those. These can be identified from/using:» Knowledge of the application
» Business knowledge
» Programming knowledge
» End user experiences
Test the application as an end user
Try and relate to similar scenarios while testing
Collaborate with other testers/leads/SMEs in the projectto understand
» Implications of the defects on other modules» Behavior of the defect if tested with other credentials
» Behavior of the defect in various environments – Entiretest environment/hardware/software
54
T t E ti R t
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 55/67
55
Types of Reports
» Daily Execution Report – End of Day» Test Summary Report (No / No Go Decision) – End of
Execution
These reports will be circulated to the stakeholdersincluding
» Onsite Test Manager» Test Leads
» Dev Manager
» Program Manager
» Onsite Dev Managers
» Dev Module Leads» Client
Data for reports can be prepared manually or taken fromthe tool if a test management tool is used
Test Execution: Reports
Test Execution Status – Daily
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 56/67
56
The Status of test execution needs to be communicated
to the stakeholders of the project in the agreedfrequency as per the Test Plan
Below are some of the key information which will beshared with the stakeholders during test execution
» % of test cases executed
» No. of test cases passed Vs No. of test cases executed» No. of test cases failed Vs. No. of test cases executed
» Defect report• By Status
• By Severity
• By Priority
• By Module• By age of the defects
Test Execution Status DailyExecution Report
E d f T t R t
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 57/67
57
At the end of every test cycle, the testing team will sharethe end of test report with the stake holders
This will help the other stake holders of the project tounderstand
» Status of test execution
» Summary of defects found grouped by• Severity
• Status
• Age of defects
• Modules of the application in which the defects occur
» Summary of the modules that are impacted due to the fixesmade to the defects
» Changes introduced in the business due to the defects found
by the testing team (if any)
» Stability of the application
» Risks involved in moving to the next cycles of testing or UATwith the current state of the application
End of Test Report
Sample Daily Execution Status
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 58/67
a p a y u o a uReport
Daily Execution Report consists of Defect Status, Execution
Status, Defect List and the Downtime Tracker
58
Daily Execution
Status Report Sample
Test Execution: Comparison ‘App
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 59/67
p ppDev’ Vs ‘App Maintenance’
59
Test Environment Test Data Tools & Process Test Execution
Application
Development
The Test Environment has to be setafresh
Test Data has to be identified for all the test cases, as applicable
Test Management tool needs to be set up for the project
Defect Management related setup needs to be done afresh
Focus is more on SystemTesting, System Integration
Testing and retesting of defects
ApplicationMaintenance
The server set up for the Test
Environment would generally beavailable Only the new build of the applicationhas to be deployed in the existingenvironmentIf there are any specific changes
based on the enhancement, those needsto be taken careThe access to the DB, App server andother external interfaces may beavailable already
Only those that are additionallyrequired need to be obtained beforeexecution
Test data may be identified from
the existing production environmentOnly the additional need for thetest data required need to beidentified and obtained beforeexecution
Project would have been
already set up in the TestManagement tool New folders and other essentials need to be set for thecurrent release onlyDefect Management and the
process is already set and hence just need to be followed
Focus is more on
Regression testing of theapplication to ensure theenhancement does notintroduce any defects intothe existing functionality
Test Your Understanding
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 60/67
Test Your Understanding
When the Test Summary Report will besend?
a) End of SIT b) End of Week c) Both
Which tracker is used for down times ?
a) Down time b) Environment c) Defect
Who will take decision to go for UAT ?
a) Dev Manager b) Client c) TestManager
60
Fundamentals of Test Execution Session
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 61/67
61
1: Summary
Process of executing a series of test cases as defined in
the test plan to validate if the application meets therequirements
Can be manual or automated using a tool
The following need to be planned before the TestExecution
» Test cases review / Sign Off
» Test cycles
» Test Lab set up
» Task Allocation / Run plan
» Risks and mitigation process» Escalation process
Fundamentals of Test Execution Session
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 62/67
62
1: Summary (Contd.)
Configuring of Test Environment includes the following:
» Software required(Ex. Tools like Eclipse, Crystal Reports)
» Hardware required(Ex. Special interfaces like HandheldDevice, Touch Screen)
» Server to host the application in the desired configuration
» Database setup/Server Setup to run the application
» Appropriate level of access for test team to accessDatabase, server and application
» Appropriate level of access to external application if thetesting scope covers for the same
» Identifying and accommodating the batch job
dependencies
Fundamentals of Test Execution Session
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 63/67
63
1: Summary (Contd.)
The release notes of the build primarily includes information of
the features of the software included in the build and thedeployment details
Build verification Test validates if the build is good to proceedwith testing and does not have any major showstoppers
Test Log is a chronological record of the all the informationabout the test case execution
» What are the test cases executed
» In What order it was executed
» Who executed those test cases(Tester)
» Status of the each step of the test cases (PASS/FAIL)
» This can be created manually or using test management tool
Fundamentals of Test Execution Session
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 64/67
64
1: Summary (Contd.)
Testers raise defects when the expected result and the actual
behavior of the application do not match The defect management methodology defined for every
testing project would be different
The testers need to follow the one defined in the Test Plan of that particular project
The defect logging can be done in a simple excel document orany defect management / test management tool adopted bythe team
Fundamentals of Test Execution Session
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 65/67
65
1: Summary (Contd.)
Key Elements of a defect includes:» Defect Id
» Defect Description
» Steps to reproduce the defect
» Test Case Id
» Assigned To
» Module Name
» Severity
» Priority
» Screenshot of the defect
» Defect Status
» Test data used to identify the defect
» Version of the build in which the defect is identified
» Test Environment details in which the defect occurred
Traceability Matrix has to be updated with the defect details post the test execution
Defects that are fixed by the development team are retested by the testing team toensure that the application behaves as desired after the fix
Regression Testing needs to be done to ensure that the application is in tact andbehaves as before even after the enhancement/ defect fix
During and post the Test execution, various reports are generated to shareinformation on the status of the execution and the application defects with the stakeholders of the project.
Fundamentals of Test Execution Session
8/3/2019 Fundamentals of Test Execution V1.0
http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 66/67
66
1: Source
Disclaimer: Parts of the content of this course is based on the materials available from the Web sites
and books listed above. The materials that can be accessed from linked sites are not maintained by
Cognizant Academy and we are not responsible for the contents thereof. All trademarks, service marks,
and trade names in this course are the marks of the respective owner(s).
CSTE CBoK
ISTQB
Project experience