Test Effort Estimation What is Estimation? 6th Sep, 2011 Chemuturi Consultants, 1.

Test Effort Estimation

• What is Estimation?

6th Sep, 2011Chemuturi Consultants,

www.chemuturi.com1


• Definition of Estimation– Estimation is defined as the intelligent anticipation of

the quantum or work that needs to be performed and the resources (namely, human resources, monetary resources, equipment resources and time resources) required to perform the work at a future date in a defined environment using specified methods.


www.chemuturi.com2


• Definition of Test Effort Estimation– Test Effort Estimation is that estimation of the testing size,

testing effort, testing cost and testing schedule for a specified software test project in a specified environment using defined methods, tools and techniques.


www.chemuturi.com3


• Now the key words are – – Estimation – defined earlier– Testing Size – the amount (quantity) of testing that

needs to be carried out– Testing Effort – the amount of effort in either person

days or person hours necessary for conducting the tests – Testing Cost – the expenses necessary for testing,

including the expense towards human effort– Testing Schedule – the duration in calendar days or

months that is necessary for conducting the tests


www.chemuturi.com4


• EstimationStandard Time – is defined as the amount of time taken to accomplish

a unit of work in a defined environment by a qualified worker, after adjustment, at a pace that can be maintained day after day without any harmful physical effects

Estimated Time - the time / effort that is expected to be consumed for accomplishing a defined amount of work in a defined environment

Actual Time - the time / effort that is actually consumed for actual amount of work in the actual environment


www.chemuturi.com5


• Remember, an “exact/accurate estimate” is an oxymoron (“credible estimate” is a better phrase)

• Estimate how long will it take you to place to another place– On what basis did you do that?– Experience right?– Likely as an “average” probability– For most software projects there is no such ‘average’

• Most estimations are off by 25-100%


www.chemuturi.com6


• Why testing?– Uncover all defects– Assure the customer that the product is built

conforming to his/her specs/requirements– Certify the product – IV&V

• No virus / No spyware / No malware • Works as specified• Safe & secure• Any other objective


www.chemuturi.com7


• Test Effort Estimation– Issues in Test Effort Estimation– Estimate testing project size as a percentage of

Software size to be tested and derive effort– Convert Software Size into Testing Project size and

then convert it into testing effort– Test Case Enumeration– Task Based Estimation – STU Estimation– Delphi– Analogy


www.chemuturi.com8


• Issues in Test Effort Estimation– Who Needs Test Effort Estimation? - There

are four categories of people who would like to carry out test estimation –

• The project Team who obviously have to carry out testing to ensure that their work is satisfactory and they can certify it so and pass it on to professional independent testing team. This set would carry out all the tests necessary


www.chemuturi.com9


• Issues in Test Effort Estimation– Who Needs Test Effort Estimation?

The in-house Software Quality Assurance (SQA) Team that would test and certify the product for customers. Their objective is to unearth all defects and make the product as defect proof as possible while ensuring that the product meets all the specified functionality.


www.chemuturi.com10



Testing Organizations, whose mainline of business isto test other’s software and certify the products.

Their objective is to ensure that the product meets the end-user expectations. This set would carry out –

a. Mainly Black Box testingb. Functional Testingc. System Testingd. Negative Testinge. Regression Testing


www.chemuturi.com11



Customers who entrusted their software development to a vendor. Their objective is to ensure that they are getting what they are paying for. This set would carry out

a. Acceptance Testing / Positive Testingb. Regression Testing


www.chemuturi.com12


• Issues in Test Effort Estimation– The type of test effort estimation carried out by

sets would be different.1. Project Team – since they carry out testing as part of

development work – it may be taken as a percentage of overall development work. Their purpose would be mainly to estimate resource requirement and meeting the delivery schedule

2. In-house SQA Team – they need to estimate mainly for the purpose of resource estimation and meet the delivery schedule

3. Testing Organizations – they need to estimate so that they can offer a fixed price to the customers and manage a healthy profit for their organizations besides resource estimation and meeting delivery schedule. For them “Size” of testing project would have significant appeal.

4. Customers – They really are supported in this activity of acceptance testing by the vendor organization.


www.chemuturi.com13


• Issues in Test Effort Estimation– Types of Tests conducted

1. Unit Testing2. Integration Testing & Incremental Integration

Testing3. System Testing4. User Acceptance Testing5. Volume Testing6. Load Testing7. Functional Testing8. End to end Testing


www.chemuturi.com14



9. Parallel Testing10. Concurrent Testing11. Stress Testing12. Positive Testing13. Negative Testing14. Deployment Testing15. Sanity Testing16. Regression Testing


www.chemuturi.com15



17. Security Testing18. Performance Testing19. Usability Testing20. Install / Uninstall Testing21. Compatibility Testing22. Intuitive Testing23. Comparison Testing


www.chemuturi.com16



24. Alpha Testing25. Beta Testing26. GUI Testing / navigation Testing

• How of Testing• White Box Testing• Black Box testing• Product Testing• Project Testing


www.chemuturi.com17


• Testing Project– Testing as part of Development projects

• Embedded Continuous Testing• Testing at the end of the development

– Independent Verification and Validation Testing

• What tests are included in Testing project?– Often specified– Often not specified !!– Quantum of work depends upon the tests included


www.chemuturi.com18


• Test Strategy– First Step in Test Effort Estimation– Includes

• Testing Objectives– Unearth all defects– Unearth all possible defects within the given time– Obtain Customer acceptance sign-off– Certify the product

» Virus & Spy-ware free» Function, security, and so on

– So on


www.chemuturi.com19


• Test Strategy– Includes

• Types of testing to be carried out• How testing is done

– Intuitive– Test Plan and Test Case based– White box or Black Box– Manual Testing or Tool based Testing

• Testing Environment– How many?

• Criteria for completing Testing• Regression Testing


www.chemuturi.com20


• Test Strategy– Includes

• Defect Closure and Escalation mechanism• Progress Reporting• Defect Analysis

– ABC analysis– Defect Category Analysis

• Now we are ready for test effort estimation !


www.chemuturi.com21


• Test Effort Estimation Conversion from Software Size– One way is to compute software size to Test Effort

• Let us say one FP needs 2 hours of testing– If the project size is 100 FP then Testing Effort is 200 Person Hours

• Let us say one UCP needs 4 hours of testing– If the project size is 100 UCP then Testing Effort is 400 Person Hours

– Simple to use– There are no benchmarked data available – therefore,

organizational data needs to be developed and maintained– Norms to be developed and maintained–

• No of FP (or UCP) for one hour of User Acceptance Testing• No of FP (or UCP) for one hour of Integration Testing• No of FP (or UCP) for one hour of System Testing• Etc.


www.chemuturi.com22


• Test Effort Estimation Conversion from Software Size– Demerits

• Too simplistic• Testing Size is not available in Software Test Units – hence deriving

productivity is based on software size• Amount of testing differs from project to project – the variance can be very

high• Organization has to keep meticulous records and strictly adhere to process

to obtain reliable figures• Time sheet and data collection have to be rigorous

– Merits• Facilitates arriving testing effort for any individual or combination of tests• Very quick• Can be surprisingly accurate – if the norms are well derived and maintained


www.chemuturi.com23


• Test Effort Estimation Conversion from Software Size– One way is to convert software size to Software Test Units (STU)

• Let us say one FP results in 5 STU– If the project size is 100 FP then Testing Size is 500 STU

• Let us say one UCP results in 8 STU– If the project size is 100 UCP then Testing Size is 800 STU

• Now convert the STU into effort– Let us say each STU consumes 30 minutes then the project consumes -

» 500 * 0.5 = 250 Person Hours» 800 * 0.5 = 400 Person Hours


www.chemuturi.com24


• Test Effort Estimation Conversion from Software Size– We need to define a STU– Maintain meticulous records to derive and establish relationship

between Project in FP / UCP and TP


www.chemuturi.com25


• Test Effort Estimation Conversion from Software Size– Demerits

• Too simplistic• Need to compute the software size first (for IV & V projects) in FP

or UCP• If the software size is not computed in-house – then the software

size itself may be in question• Organization has to keep meticulous records and strictly adhere to

process for deriving relationship• Time sheet and data collection have to be rigorous


www.chemuturi.com26


• Test Effort Estimation Conversion from Software Size– Merits

• Testing Size is defined in STU – hence facilitates deriving Testing Productivity

• Very quick• Can be surprisingly accurate – if the norms are well derived and

maintained


www.chemuturi.com27


• Test Effort Estimation – Test Case Enumeration– Enumerate the test cases– Estimate testing effort for each test case– Use Best Case, Normal Case and Worst Case scenarios for

estimating effort needed for each test case– Compute Expected Effort for each case using Beta Distribution

• Best Case + Worst Case + (4 * Normal Case) / 6

– Sum up the –• Expected times to get Expected effort estimate for the project• Best Case times to obtain best case effort estimate• Worst Case times to obtain worst case effort estimate• Normal Case times to obtain normal case effort estimate


www.chemuturi.com28


• Test Effort Estimation – Test Case Enumeration– Best Case estimate and the worst case estimate give the range

of effort estimate for the project– We can use Excel / MS Project to define predecessors and

successors, and build a network diagram – this facilitates arriving at the project duration and computation of probability


www.chemuturi.com29


• Test Effort Estimation – Test Case Enumeration– Demerits

• No Testing size – hence productivity can not be derived• All test cases and attendant overheads need to be enumerated

– Merits• Auditable estimate• Fairly accurate• Probability can be computed• Progress monitoring is facilitated


www.chemuturi.com30


• Test Effort Estimation – Test Case Enumeration– Best Case estimate and the worst case estimate give the range

of effort estimate for the project• Standard Deviation for a test case is –

– (Worst Case - Best Case)/ 6

• Variance for an a test case is the square of its standard deviation

• Standard deviation (ST) for the entire project is –– Square Root of (sum of variances of all critical a test case)

– We can use these for computing probability of success of project


www.chemuturi.com31


• Estimation – Activity/Task -based Estimation– This is concerned with identifying the tasks (Activities) to be

performed to complete a task and estimating effort required to complete each of them

– Steps are –• Identifying the component tasks in each phase• Estimating the effort for each task• Calculating the effort for each phase• Consolidating the effort for the entire project


www.chemuturi.com32


• Estimation – Activity/Task -based Estimation• Phases in a Testing project

– Project Initiation– Project Planning– Specifications Analysis– Test Strategy Design– Test Planning– Test Case Design– Testing and Defect reporting – Regression Testing and defect closure– Documentation– Project closure


www.chemuturi.com33


• Estimation – Activity/Task -based Estimation• List out all the tasks in each of these phases• Eg – Specifications Analysis

• Study proposal and Order• Document understanding gaps• Obtain Clarifications

– Test Strategy Design• Define Testing Objectives• Decide the type and number of tests• Decide the Test Environment• Define Testing Completion Criteria• Define defect reporting and closure mechanism• Define Regression testing mechanism• Define Tools usage


www.chemuturi.com34


• Estimation – Activity / Task-based Estimation– Assign durations for each of the Tasks– Use three time estimates – Best Case, Worst Case and Normal Case– Compute the expected time using formula – (Best Case + Worst Case + 4* Normal Case)/6– Make adjustments for project complexity, familiarity with the platform, skill of

the developers, tools usage– Make adjustments using heuristics – (e.g. Requirements 10%, Design 25%,

Construction 40%, Testing 25%)– Use Delphi technique to validate the estimate– Sum up the total effort estimate of the project


www.chemuturi.com35


• Testing Size Estimation– Why is size necessary?

• Productivity computation• Measure performance• Agree with Client about the quantity of work being performed• Monitor progress


www.chemuturi.com36


• Testing Size Estimation– What are the units of measure available for measuring testing

size?• STU are being discussed• But no universally accepted definition is available on what

Constitutes a STU• The definition is just escaping STU !!


www.chemuturi.com37


• What is my definition of STU?– STU should cater to all combinations of Tests– Be it a unit test or be it acceptance test, be it system test – or

any other test – STU should be able to measure its size– What is common to all types of tests is – the Test Case

• STU is a size measure for measuring the size of a software-testing project and that a STU is equivalent to a normalized test case.


www.chemuturi.com38


• What is my definition of STU?– What is a test Case?

• Test Case provides an input to the system which would produce a response from the system. The response could be –

– Store the data– Produce derived / un-derived data– A message


www.chemuturi.com39


• STU Estimation approaches– Derive from the size Estimate– Bottoms Up approach


www.chemuturi.com40


• STU Estimation approaches – EstimatorPal Approach– The Method of computing STUs

1. This method uses an existing software size in FP/UCP etc2. It converts development project size into Unadjusted STUs (UTP)

using a conversion factor which can be modified by the user to suit his environment

3. It computes a Composite Weight Factor (CWF)a. Sum up all individual weights of included testsb. Multiply it by the weight of the application weightc. Multiply it by the language weight if Unit Testing is selectedd. Multiply it by Tools Weight if Tools Usage is selected


www.chemuturi.com41


• STU Estimation4. UTP are multiplied by CWF to obtain the testing size in STUs

size5. The Productivity Factor indicates the number of person hours

per one STU6. Testing Effort in Person hours is computed by multiplying

STU Size by the Productivity Factor.


www.chemuturi.com42

Common Pitfalls in Estimation

• Inexperienced people carrying out estimation activity• Lack of training• Lack of historical data• Absence of tools• Over-estimation and under-estimation• Inadequate time for estimation• Lack of review


www.chemuturi.com43

Common Pitfalls in Estimation

• Lack of causal analysis for deviations in actual Vs. estimates

• Productivity Figures - no focus• Application of one productivity figure for conversion

from size to effort


www.chemuturi.com44

Over and Under Estimation

• Over estimation issues– The project will not be awarded

• Conservative estimates guaranteeing 100% success may mean winning probability of zero.

– Parkinson’s Law: Work expands to take the time available for its completion

– Danger of feature and scope creep– Be aware of “double-padding”: team member + manager

• Under estimation issues– Quality issues (short changing key phases like testing)– Inability to meet deadlines– Morale and other team motivation issues


www.chemuturi.com45

Estimation Good Practices

• Experienced people carry out estimation• Process is defined for estimation• Estimation guidelines are defined and available• Productivity figures are derived and maintained up-

to-date• Estimators are trained• Estimation tools are available


www.chemuturi.com46

Estimation Good Practices

• Repository of estimates are maintained and are made available to estimators when needed

• Expert guidance is available• All estimates are subjected to peer review as well as

managerial review• Actual Vs. Estimate analysis (postmortem) is always

carried out and causal analysis is carried out and norms of estimation updated


www.chemuturi.com47

Estimation Guidelines

• Estimate iteratively!– Review two heads are better than one– Make your best estimates at each planning stage– Refine estimates and adjust plans iteratively– Plans and decisions can be refined in response– Balance: too many revisions vs. too few


www.chemuturi.com48

Estimation “Presentation”• How you present the estimation can have huge

impact• Techniques

• Plus-or-minus qualifiers• Ranges• Risk Quantification

– +/- with added information– +1 month of new tools not working as expected– -2 weeks for less delay in hiring new developers

• Cases– Best / normal / Worst cases

• Confidence Factors– April 1 – 10% probability, July 1 – 50%, etc.


www.chemuturi.com49

Other Estimation Factors

– Account for resource experience or skill• Up to a point• Often needed more on the “low” end, such as for a

new or junior person

– Allow for “non-project” time & common tasks• Meetings, phone calls, web surfing, sick days

– There are commercial ‘estimation tools’ available


www.chemuturi.com50

Other Estimation Factors

– Please download our tool TestPal• Test Case Design assistance• Test Result logging• Test Effort Estimation

– STU– Task Based– Test Case Enumeration

– www.chemuturi.com/TestPal-Setup.exe

–Thank you Very much


www.chemuturi.com51

http://www.chemuturi.com/TestPal-Setup.exe

Date post:	26-Dec-2015
Category:	Documents
Upload:	janel-jefferson
View:	216 times
Download:	0 times

Test Effort Estimation What is Estimation? 6th Sep, 2011 Chemuturi Consultants, 1.

Documents