Test Estimation Techniques - STeP-IN Forum

STEP-IN SUMMIT 2007 Estimation Techniques for Testing Projects

Estimation Techniques for Testing Projects

V.K.Kripanand


Table of ContentsEstimation

OverviewComponents of EstimationFactors for EstimationEstimation ModelTechniques in UseA discussion on techniquesKey Points and Future forward

Test Projects Estimation Why now?How is it Different?Why is it Important?Future Forward


Table of ContentsEstimation Techniques for Testing Projects

Principles of EstimationBlack BoxWhite BoxPerformance TestingStress Testing Other TestingInfluencing Factors Team Dynamics

Worksheets to useTest Estimation WorksheetsOther Supplemental Worksheets

Summary


Estimation

Topics OverviewComponents of EstimationFactors for EstimationEstimation ModelTechniques in UseA discussion on techniquesKey Points and Future forward


Estimation

OverviewCurrent Estimation techniques are aplentyNo subscribed method adoptedExperience, plays a crucial role“Plug and Pray” – “Predict and Pray”Why do we need estimation at all?


Components of EstimationManpower costCost of infrastructure Operational costs

Special softwareCommunicationTravel Training

OverheadsShared facilitiesInfrastructure overheadsCost of moneyCost of Electricity/welfare/overtime etc


Factors for estimationMarket opportunity

Who the competition is?What is the opportunity now and in the future?

Contractual termsAre there penalty clauses?Is it phase wise delivery?

Volatility of requirementsHow standard is the requirement?What could be the change in requirements over time?

Past ExperienceHave we had a better or bitter experience?Have we delivered something similar in the past?


Factors for estimationOrganizational strengths

Management supportStrength and expertise of the team

Skill setTalent pool of engineers and the expectations of the projectIn house training capabilityAbility to learn quickly

Ability to ramp up Can we ramp up our team in case we win the project?Option and flexibility to outsource?

TechnologyComplexityEnvironment – Onsite/OffsiteVirtual Testing


Estimation Models

What is it?Estimation model is the combination of the technique and the factorsA predefined framework or template Adaptable with modifications

Build or use Models?Using existing models is a safe bet Derive and build your own models


Techniques in usePricing to Win

What the customer wants to spendBased on budget not on functionality

Parkinson’s lawExpand to fill timeNo objective assessment

AnalogyDomain basedHugely dependent on past completed projects.

Expert JudgmentExperts of the domain and development practice are consultedAgreement on final estimate is an iterative process

Algorithmic cost modellingBased on historical cost informationSoftware metric information and a model determine the effort


Some more…Pricing to win, Parkinson’s law, Expert judgment

May not be relevant in today’s contextAnalogy and Algorithmic models

Widely in useCOCOMO and its variants are a great example

Algorithmic approachBased on metrics and formulaeRepeatable and Re-usable

Function points, object pointsProvide a framework based input


Some more…Estimation is not only about effort

Model reveals the importance of staff experience in software development As well as effort estimation, managers must estimate the calendar timeThe time required is independent of the number of people working on the projectStaff required <> (development time/required schedule)Phase of the project decides the people working on the project

Cost/schedule risk managementPerform computations with and without the risk being realized and see the size of the impact best/worst case estimates


A discussion on techniques…Each method has strengths and weaknessesEstimation should be based on several methodsIf these do not return approximately the same result, there is insufficient information availableSome action should be taken to find out more in order to make more accurate estimatesChallenges

When RS is incompleteComplete Details are not availableToo frequent changes to requirements


Key Points and future forwardWhich Model to use?

Because no model is right, all models can be usefulDifferent techniques of cost estimation should be used

Factors that influenceProductivity, Individual aptitudeDomain experienceProject size, Tool support and Technology challenges

Use of specific modelsAlgorithmic cost estimation is difficult because of the need to estimate attributes of the finished productThe COCOMO model takes project, product, personnel and hardware attributes into account when predicting effort requiredAlgorithmic cost models support quantitative option analysisThe time to complete a project is not proportional to the number of people working on the project


Key Points and future forwardFuture Forward

Put the best models in use and derive a standard templateLearn from past experiences and document the sameCollect metrics to use in future, not for name sakeJunk in – Junk out (Actual effort data is very crucial)With over 2 decades of collective experience, we should now retrospect with past dataCollaborate and co-operate to have a win-winCut across competitive lines….Adapt the same for different nature of projects – More about this later……


Test Projects Estimation

Topics The three W’s!Why now?How is it Different?What is Important?Future Forward


Test Projects Estimation

The three W’sWhy?

Why the focus now on Test Projects?

How?How is this different from normal projects?

What?What are the important parameters to consider?


Why Now? The market

Gartner estimates the worldwide software testing market at $13 BillionIndia is expected to touch $1 Billion by 2007

What we need now?Established Estimation TechniquesBuild large and skilled teamsRemove the fad of ‘Testing’ as a non-interesting job!Build frameworks and process methods specific to testingUse automation effectivelyLeverage the large talent poolIncubate talent with the relevant skills


How is it Different?Similar but not same!

Estimation techniques can be similarNature of Effort estimation is differentModels can be customized for Test ProjectsOverheads and factors are different

Test Projects have to consider theseAnalysis of test typesFactors for estimation are based on Development environment and test environment Demands on Skill, expertise and automation Technology and infrastructure requirements Domain knowledge


What is Important?Parameters to consider

Estimation methodsCan we derive from software estimation techniques

FrameworkIs there a framework that can be built?

Knowledge and know-how?Past Experience and know-how is crucialCan we learn from the past?Intelligence (metrics and data) from Projects is very important

TeamTeams play a crucial roleThey are the backbone of the executionWho else can predict accurately but themEstimation is not a individual effortIt is a team effort!


Future ForwardLearn from the past

Put metrics to best useBuild a estimation model for testing projects

Use existing techniques to interpolate for testing projectsProcess Maturity

Build a model for quality of testing Build reusable assetsLeverage existing maturity models of the organization

People skill and talent poolIncubate new talentIdentify methods and techniques


Estimation Techniques for Testing Projects

TopicsPrinciples of EstimationBlack BoxWhite BoxPerformance TestingStress Testing Other TestingInfluencing Factors Team Dynamics


Principles of EstimationPrinciple of Estimation for testing projects shall be

Based on Software requirementsPrevious projectsMetrics

Estimation shallNever forget the pastBe recordedBe supported by toolsBe always verifiedConsider automation needsConsider people skills

Other Inputs to consider# of test cases/scenarios,# test cases per scenario, # of builds (for regression)


Black BoxPointers for Estimation

Business CaseFunction pointsObject pointsRequirement SpecificationsDevelopment TimeEstimate versus ActualAutomationTest Bed ConditionsTechnology requirementsTest ReportingSkills


Business CaseBusiness cases

Normal Scenario Test prescribed functionalityComplete Details are available

Domain specificDifferent types of domains require different calculationsSometimes a combination is to be usedNot all testing can be completedScope of testing could be restricted based on the domainCertain assumptions would have to be made

Challenges – Abnormal ScenarioDetails are not availableSpecifications are incomplete!


Function PointsFunction points

Measure the size of a software application or project based on the number and complexity of user functions requested or deliveredThe calculation is objective.Function points are independent of the technology The calculation can be used early in the development cycleProductivity history on completed projects with similar characteristics can be usedThe measurement is meaningful to non-technical users.


Function PointsFunction points and Testing

Can be used to estimate test effortEach function point can help in developing relevant test casesNature of Testing (Black Box, White Box) decide the test cases and their complexitiesFunction point complexity mapped with test case complexity help in deriving test estimationAre a good starting pointCombined with Requirement Specifications, Design and Analysis documents, help in arriving to near to actual test effortCould be the best starting point for “Test Case development”


Object Points

Object PointsObject points are an alternative function-related measure to function points when 4Gls or similar languages are used for developmentThe number of raw objects are estimated, the complexity of each object is estimated, and the weighted total (Object-Point count) is computedHave nothing to do with objects in Object Oriented Programming


Object PointsObject Points and Testing

Can be used to estimate test effortEach object point can help in developing relevant set of test cases or test suitesRe-usability, modularity can be built as against the FP based approachNature of Testing (Black Box, White Box) decide the test cases and their complexitiesObject points are most often than not used as indicators of estimationAccuracy on estimation is an iterative processIdeally suited if you have more than one testing project in the same domain


Requirements SpecificationsRequirements Specifications

Depict the functionality of the systemDetails the various components and their desired functionalityA good reference point to build test cases for both test to pass and test to failTest requirements are a good starting point if availableA well documented RS will have a good structure and a mature traceabilityCould be the basis for estimation and project planning of Test Projects


Development TimeDevelopment of the System Under Test (SUT)

Time TakenTime taken to develop the SUT is keyIndicates the trend on overrun, complexityIf actual time is available, it validates the estimation further

Interviews If possible with development team members will provide indicators

Domain expertisePrior experience of Testing team and Development Team – a great PlusPast experience in testing similar projects

What to focus on – Test to pass or Test to fail?Indicator that decides estimation largely


Estimate versus Actual

Estimate versus ActualActual effort spent is a key indicator

If each RS item can indicate the time taken, and time allocated, variance can be effectively usedMore about this in worksheet discussion

Actual metrics can provide indicationsComplexity can be arrived atTechnology obstacles can be previewedTricky and Sticky code portions and implementations can be exposedNature of effort required to break can be judged


AutomationAutomation

A key indicator in estimatingAutomation helps in regressionBut it always takes more time to build automation suitesTrade off – Automation to Individual non-repeatable testsAvoid automating when not deemed fitIncorporate estimations for automation based on repeatability, reusability factorsTest Harness creation, maintenance

Key to identifying estimatesOne time effort, but useless if it has no repeatabilityIf mandated, accommodate estimate to ensure that the overall estimate is not hurt


Test Bed ConditionsComplimentary to Automation

Test bed needs more inputs if automation is involvedAutomation decides the nature of test bed creationTest Bed creation estimates have to consider

Initial and repeatable costsEnvironmental considerationsRemote testing Testing on live environmentsTechnology considerationsHardware and software pre-requisitesTime to execute


Technology requirements

Technology challenges are always newNew technology, old domain has hidden challengesSupport from Development teams crucial for estimationChallenges

Training timeLearning curveAvailability of experienced handsTechnical know-how is a prime factor


Test Reporting

Test Reporting is a major activityEstimates usually do not account for reportingReporting needs are based on process maturity and methodologiesInvolve your QA Team to identify reporting needs and evolve estimationsBuilding new reports or Bug DB involves development effortKeep the reporting simple if feasibleUse ‘Expert Judgment’ extensively


SkillsBuild your team skills continuously

Keep hiring the best mindsAllow good transition from projectsBuild Domain expertise and prepare your team for the same

Your estimate should allow time for skill buildingDelivering all the time is not funLearning, delivering and re-learning is the way forwardKeep the human factor on top-priority

Use an updated matrix of skillsUse an matrix to map your teams skillsUpdate the matrix regularlyAlways Interpolate your estimate with skill set matrix Build your own matrix that is suitable and “Keep it simple”


Using a MatrixThe skill matrix can contain the following

ExperienceWork Experience, Roles, Domains worked on

SkillsSoft skillsTesting skills on standards, process, SDLC, Testing maturityTechnologyExecution skills on Manual testing, automated testing, script building, automated test executionAutomation tools expertise, functional, structural and others

Process MaturityTesting skills on planning, documentation, bug isolation, trackingDesign and development skills on black-box, white-box, static, dynamic testing, load testing, stress testing, etcConfiguration management knowledge


Black box - SummarySummary

Use a combination of Function/Object points and Requirement SpecificationsArrive at test cases and the relevant test suitesFind about development time estimated and actualInterview development team to identify critical paths to testUnderstand automation strategy and build plans for test bed creation/maintenanceTest harness creation/maintenance should be accounted forFinalize your test results reporting techniques and account for the same in your estimateReview your skill matrix and map it to the estimate


White boxPointers for Estimation

Business CaseLines of CodeTypes of white box testing

StructuralCode ReviewCoding Standards

Language factorsTechnology considerationsSkillsTest ReportingAutomation


Business Case

Business casesScenario for White Box TestingAdherence to StandardsToo buggy codeContractual requirementLegacy codeReverse or Re-Engineering


Lines of Code

A major contributorLines of code is a decisive factor for estimation

Other contributorsComplexity, requirementAdherence to StandardsStructure of the codeTechnology consideration Programming Language


Types of white box testingStructural

ContentReusabilityComponentsMaintainability

Code ReviewWalkthroughsDiscussionsPeer Review

Coding StandardsRegulatory StandardsInternal StandardsRules set by QA/Process teams


Language factorsProgramming Language

Key factors to look forComplexityRulesAdherence to specificationsSupport of various StandardsA key indicator for building the estimation based on skills availableConventional, 3GL, 4GL and machine specific use of language

Object oriented languagesIncreases complexityComponent based indices are to be verified


Technology considerations

TechnologyPlays a crucial roleIdentifies what can be testedNon Testable portions or grey areas should be investigated and documentedChallenges

Training timeLearning curveAvailability of experienced handsConsider using technology experts if skill set is not available internally


SkillsDifferent needs from Black box Testing Skills

Usually developers with testing prowess is a safer betProgramming language expertise is a must.Good exposure to Software Engineering, metrics, quality indices are absolutely essentialExcellent analytical skills to avoid common pitfalls of misinterpreting code.Good knowledge about standards

Especially true for embedded/ Mission critical projectsAware of standards and its importance (E.g. DO178B)


Test Reporting

Test Reporting is a major activityIdentify methods of reporting white box testing resultsAdhere to standards of reporting, if required

Very critical when testing against standards

Identify critical estimates for buildingSpecial reportsReporting templatesComprehensive detailing of white box errors


Automation

AutomationProven to be at least 10 times more effectiveStatistical analysis can be much fasterReporting is usually much faster and effectiveCan reduce the need for

Variety of language skills Skilled individuals with knowledge of Adherence to standardsReporting mechanisms and templates


White Box - SummaryLines of Code plays an important roleProgramming Language is a key indicatorComplexity increases when the testing types are more than oneSkill demands are different from Black Box TestingTest results reporting is different Adherence to Standards and rules are of prime importanceAutomation reduces time


Performance TestingPointers for Estimation

Business caseDomainNature of end users/usage patternsTest Case development is an ongoing processFramework based approach is best advisableUse of automation tools an important factorAutomation frameworks provide a lot of estimation built inTechnology challenges are highTest Bed Requirements are highTest Data requirements are highPreamble and post amble conditions are very stringent and extra focus is required


Stress Testing

Pointers for EstimationBusiness CaseEstimation is very similar to performance testingDomain is crucialNature of end users/usage patternsCritical tests take precedenceTest Data requirements are highTechnology challenges are highTest Bed Requirements are highRepeatability is a key issue to handle


Other Testing

“OPEN HOUSE” discussion


Influencing FactorsHigh ComplexityStakeholders are manyExternal DependenciesLacking Skills in Domain, TechnologyHave to build new tools/frameworksExperimenting…Requirement of Hardware/Other infrastructureTest Data Legal contractual obligations


Team DynamicsPeople, People and People

The ones who make Good test EngineersAnticipating the worst – Attitude hard to get, but should strive to find such attitudeShould be creative, testing is a creative work, however destructive the approaches are!Curiosity is important, but should not kill the cat!

Idea is to find bugs, isolate and report them

Skill set mix, is quite importantTechnical skills, inter-personal skills, soft skills, communication etc, all play a crucial role

Domain Experience, would be crucial

The team is likely to dilute, so keep the search on all the time


Worksheets

Worksheets to useTest Estimation WorksheetOther Supplemental Worksheets


Test Estimation Worksheet

Assimilates inputs from various estimation models to provide

Black Box testing estimationWhite Box testing estimationA basic building block A Collaborate platform to build a comprehensive worksheet


Other Supplemental Worksheets

Skill MatrixTest Bed ChecklistTechnology challenge checklistDomain expertise and past project inputs


SummaryUse existing models to suit your test project requirementsEstimations could be imprecise – but it’s ok!Wrong Estimations should be controllable!Re correct your estimate, if you have made a mistake – but do it earlyCollecting metrics and creating ‘Intelligence’ is crucial for success in the futurePrepare for changes and challenges on a continuous basisStart Early and Stay focused


SummaryMake your team understand dynamics of Test projects – all the timeAllow team participation wherever possibleCollaborate, retrospect and co-operateExperiment only on a small scaleBuild contingency and risk plans – but keep them trimmedGain Confidence of management with methodologies and processesPut your team first and not yourself – it is better in the long run


References and Acknowledgments

Acknowledgments

Team – QSITTeam – SAMGRAM, ChennaiTeam – RESOLVERS, Bangalore

References:

Software EngineeringIan Sommerville

Metrics and Models in Software EngineeringStephen H Kan

Critical Testing ProcessRex Black

And all the “Software Testing Minds” around the Globe

Date post:	03-Feb-2022
Category:	Documents
Upload:	others
View:	3 times
Download:	0 times

Test Estimation Techniques - STeP-IN Forum

Documents