Testimation User Instructions€¦ · What does using our Product Suite mean for you? Users Uses 1....

www.testimation.com

Sunday,November5,2017 1

ApplicatorSuiteUserInstructions |v1.0forUseCase/UserStory |TEstimator1 |TEstimator2 |Testimator3 |REstimator |@RiskAuthor/s |TestimationTeamEngineeringtoolsforthescienceofestimation&riskmanagement

www.testimation.com


ApplicatorSuite......................................................................................................................................11. Introduction................................................................................................................................42. TheBigPicture...........................................................................................................................53. TheApplicatorProductSuite................................................................................................6A. Composition&Architecture.............................................................................................................6B. FeatureList...............................................................................................................................................81. TEstimator1...........................................................................................................................................................82. TEstimator2...........................................................................................................................................................93. TEstimator3........................................................................................................................................................104. REstimator..........................................................................................................................................................105. @Risk....................................................................................................................................................................116. AEstimator..........................................................................................................................................................117. PEstimator..........................................................................................................................................................12

4. HowtoexecuteaTEstimator1Estimate..........................................................................15A. Applicability...........................................................................................................................................15B. EstimationProcess..............................................................................................................................15C. HowtoexecuteaQuEstimateorLEstimate.............................................................................151. Inputs....................................................................................................................................................................152. Configuration.....................................................................................................................................................15

D. HowtoexecuteanEstimateinBasicView...............................................................................16E. HowtoexecuteanEstimateinAdvancedView......................................................................16

5. HowtoexecuteaTEstimator2Estimate..........................................................................17A. Applicability...........................................................................................................................................17B. EstimationProcess..............................................................................................................................17C. Procedure................................................................................................................................................171. Inputs....................................................................................................................................................................172. Configuration.....................................................................................................................................................183. Analysis|Historical&ProjectedSolutions..........................................................................................194. QualityOverride...............................................................................................................................................205. RiskOverride.....................................................................................................................................................216. GeneralConsiderations.................................................................................................................................23

6. HowtoexecuteaTEstimator3Estimate..........................................................................24A. Applicability...........................................................................................................................................24B. EstimationProcess..............................................................................................................................25C. TEstimator3UserScenarios.............................................................................................................25

7. Definitions................................................................................................................................27A. CalibrationFactor................................................................................................................................27B. Configuration.........................................................................................................................................27C. ConversionRatio..................................................................................................................................27D. CriticalValue..........................................................................................................................................28E. Defect-FreeConfidence.....................................................................................................................28F. DIT’sperTestCase..............................................................................................................................28G. DynamicInformationTests(DIT’s).............................................................................................28H. ExecutedTestCases...........................................................................................................................28I. Framework...............................................................................................................................................28J. FrameworkProbabilities...................................................................................................................28K. FunctionalProcess..............................................................................................................................29L. ProgressionTesting............................................................................................................................29M. QualityAssuranceArchitecture....................................................................................................29N. QualityAssuranceStrategy.............................................................................................................29O. QualityOverrideSimilarity.............................................................................................................29P. RegressionTesting..............................................................................................................................30Q. RequiredTestCases...........................................................................................................................30R. RiskExposure........................................................................................................................................30S. RiskMitigation.......................................................................................................................................30

www.testimation.com


T. RiskOverrideSimilarity....................................................................................................................30U. RiskQuotient.........................................................................................................................................30V. Scope.........................................................................................................................................................31W. SystemUnaffected..............................................................................................................................31X. UseCase....................................................................................................................................................31Y. UserStory................................................................................................................................................31Z. TestCase..................................................................................................................................................32AA. TestCase&DefectComplexity...................................................................................................32BB. TestScenario.......................................................................................................................................32

8. ApplicatorFunctionalProcesses.......................................................................................33A. FunctionalProcessList.....................................................................................................................33B. UseCase(UserStory)List................................................................................................................34C. HowtoCalculate(orestimate)theConversionRatio..........................................................34D. UserWorkFlows(FunctionalProcessMap)..............................................................................35

www.testimation.com


1. Introduction

****ImportantNote****ThepurposeofthisdocumentistoactasacompaniontotheApplicatorSuite.Pleaserefertoitasrequired.Forbrevity&practicality,mostimages&screenshotswithintheApplicatorSuiteofToolshavebeenomittedfromthisdocumentduetotheintendedcompanionrelationship.****EndNote****

Have you ever wondered why your computer crashes so frequently but your car alwaysstarts?Theanswerissimple.Humanshavebeenmakingthingsformanythousandsofyears,butthe commercial software industry is only decades old. We have leveraged off this hardwaremanufacturing experience to develop a suite of products that are predictive & true riskmanagement tools. At Testimation, we believe that risk management should rarely be aboutqualitative high-level statements, rather, it should be about hard quantitative numbersexpressed as a percentage value. For example; “it's too risky” is a verydifferent statement to“there'sa23%probabilityofsuccess”.Theformeriscommonpracticeinriskmanagement,thelatterisTestimation.

Wearepassionateaboutsoftwarequality&engineeredsolutions.WebelievethatforfartoolongintheInformationTechnology(IT)sector,emphasishasbeenondevelopmentcostsratherthan product quality. This emphasis has arisen, not because organizations are mean withmoney, but because a scientific approach to inform business sponsors of risk has not beenmainstreaminIT.However,intheengineeringsector,thespecificationsoffailureprobabilitiesare commonplace. Stories appear too often in themedia about catastrophic software failuresinconveniencingorendangeringcustomers.Webelievethattheprimarycauseofthisisduetobusinesssponsorssimplynotbeingawareoftheriskoffailurebecauseithas(probably)neverbeenpresentedtothemasapercentagevalue.Typically,only“words”areusedtodescribetheriskofsoftwarefailure.Theproblemwiththisapproachisthatitishighlysubjective&dependsas much upon the person receiving the information, as it does upon the person sending it.Imagineifapharmaceuticalcompanyreleasedexperimentalcuresbasedupontheconceptof“itshouldbe fine”, rather thanstatisticallymeaningful clinical trials;howreadywouldyoube togivetheirdrugstoyourchildren?At Testimation, we want to change things ......

When risk is presented as a numerical probability of failure, human reaction to thisinformation is farmore sober thanwhen it is presented in qualitative forms only. If your ITprovidersorprocurementadvisersaregivingyou“words”&notnumbers,thenyoushouldturntoTestimation.“Words”areeasytoproduce,theyrequirelittleeffort&onedoesnotneedtobean expert to “talk” (e.g. sales people); but only experts will give you hard numbers from theapplication of engineering methodologies. If your software development estimates are beingdelivered to you without the number of tests being specified, or without the probability offindingDefectsbeingexpressedwithhardnumbers,thenyouareplayingRussianroulette&youmay need to re-define success throughout the Project Life-Cycle to make it appear tostakeholdersthatyouhavedeliveredsuccessfully;thisoccursfarmorecommonlythanyoumayrealise.However,withourtools&services,youcandesigntheQualityAssurancestrategythatsuits your budget & time constraints, with confidence that you're aware of the risk & thepotentialimpactuponyourcustomers,users&business.

The Testimation Team brings formalized & verified engineering techniques to thedisciplinesofestimation&riskmanagement;specificallytothesoftwaredevelopmentindustry.It utilizes the same Quality Assurance principles as the manufacturing sector & highperformance industries such as aerospace, commercial aviation & medical research.Pharmaceutical companies do not release products to themarket unless the potential failurerate is numerically understood& the riskmitigated. Sowhy is society soprepared to releasesoftwaretouserswhenthedefect-freeconfidenceisonlydescribedwithwords(ifatall)&nothard numbers? ….We have addressed this question by developing engineering tools for the

www.testimation.com


science of estimation & risk management. Our suite of tools empowers clients & serviceproviders to present scientific arguments supporting their Quality Assurance & riskmanagementprofiles.Otherstalkofriskinsubjective&salesmanshiplanguage;wequantifyitwithhardscience&hardnumbers.Ourproducts&servicesofferfourkeyfeatures:

1. Theworldsfirstsuiteofscientificallyformulated,web-deliveredestimationtools

2. ThenumberoftestsrequiredtobeexecutedforthelevelofQualityAssurancespecifiedbyprojectgovernance

3. Theprobabilityoffindingdefectsinaccordancewithprojectgovernancespecifications

4. Development,testing&projectmanagementeffort

2. TheBigPicture

The Testimation Team has decades of experience in Hardware + Software QualityAssurance, Process Re-Engineering & Risk Management across multiple industries, such asInformation Technology, Business Technology, Manufacturing, PetroChem, Utilities, Food,Government,Financial, Insurance,Construction,Aluminium&Retail.Fromourexperience,wehave learned that some industries possess significantly greater levels of Quality Assurance&Risk Management controls than others. For example, in 1969, the United States of AmericaplacedmenupononthemoonwithlesscomputingpowerontheApollo11commandmodulethanexistsonyoursmartphonetoday.Thishistoricalfactspeaksvolumesabout“whatcanbe”& “what is” within the IT sector. The harsh reality of 21st century IT is that, although ourtechnologyhasdramaticallyadvancedoverrecenthistory,good&healthyengineeringpracticeshave (largely) not been introduced or totally ignored, sector wide, across IT. This fact issurprisingly easy to prove; for example, howmany software development projects have youseen thathaveever referenced therelevant international standards in relation to theproductbeingbuilt,orthedevelopmentprocessbeingundertaken?Ifyou’rehonestwithyourself,you’llprobablyanswer“never”,oratleast,“veryrarely”.Manyinternationalsoftwarestandardsexist&manymorearebeingdevelopedasyoureadthisdocument.

Whilst it is exceedingly common practice in themainstream engineering sector to designsolutionsaccordingtogeneral&specificstandards,withintheITsectorthisdoesnotseemtooccur with any significant frequency. The primary reason for this may be because softwaredevelopersareoftennot formally trainedorqualifiedengineers& therefore, lacksomeof thefundamental design skills acquired during the university indoctrination process. One is notrequiredtobeuniversityeducated&examinedtobetitled-“adeveloper”,butoneisrequiredtopossesstheseattributesinordertobetitled“anengineer”(globally).Infactasageneralization,in many places around the world, the only professions whose signatures constitute a legaldocument are doctor, lawyer & engineer; all of which require formalized & recognizeduniversity training. So why is this important? It is important because repeatable processesrequire standardization, & standardization comes from national & international bodies ofacceptance such as the International Organization for Standards (ISO). For example, imaginethatyouhadtostartyourcar inadifferentmannereachtimeyou intendedtouse it. Imaginethatonetimeyouusethekey,thenexttimeyoutaponarearwheel,&afterthatyoumustdosomethingdifferenteachtimeyouwishtostartthecar.Thelessonhereisthateffectivedesign,development,delivery&UserAcceptancemodelsrequirestandardization,&standardizationisa hallmark characteristic of a formal engineering design process. A process involves theexecution of actions & one of the very first required actions is estimation. The SoftwareDevelopmentLife-Cycleisnoexception;asignificantportionofitinvolves;TheSoftwareTestingLife-Cycle....Itallbeginswithanestimate

1. Whyisestimationimportanttoaclient?• Becausethisiswhenaclientsetsthebudget

www.testimation.com


2. Whyisestimationimportanttoaserviceprovider?• Becausethisiswhenprofitorlossisdefined

3. WhencanourProductSuitebeused?

• Insituationswherethequalitycapabilityshouldbeknown1. Bids/Tenders,Projects,Audits,Operations,UAT2. AssessmentsofServiceProviders&VendorSoftware

4. WhatdoesusingourProductSuitemeanforyou?

Users Uses1. BoardofDirectors2. CEO's,CIO’s,CTO’s,CPO’s3. TM's,DM's,PM's,RM's4. BusinessSponsors5. Procurement6. SystemIntegrators7. SecurityAgencies8. SoftwareHouses9. Corporations10. Government11. Military

1. Savemoneybyforecastingaccurately&efficiently2. Define,manage&mitigateriskproperly3. DesignQualityAssurancestrategiesfittingyourbudget4. Conformity,repeatability&transparencyofprocess5. Thesubjectivityplaguingestimationhaslargelyevaporated6. EstimatesareEngineeringbased7. FacilitatesContinuousImprovementProcesses8. Respondrapidlytochangesinscope9. ScalefromanHistoricalSolutiontoaProjectedSolution10. CalculatetheprobabilityoffindingDefects11. Calculatetherequirednumberoftests

3. TheApplicatorProductSuite

A. Composition&Architecture

The Applicator Product Suite incorporates five (5) key Use Cases (User Stories). Each ofthese five (5) User interactions executes a distinctly different process. The Product Suite iscomposedofthefollowing:

1. TEstimator1|estimatestheminimum&maximumnumberoftestsi

2. TEstimator2 | estimates the exact number of tests when scaling from an HistoricalSolutiontoaProjectedSolution

3. TEstimator3 | estimates the exact number of tests when scaling from an Historical

SolutiontoaProjectedSolution

4. REstimator(StandAlone)|estimatestheminimumnumberofRegressiontests5. @Risk(StandAlone)|UserAcceptanceTesting(UAT)RiskVisualisation

Additionalembeddedcapabilitiesare:

1. AEstimator|comparestwoestimates(sidebyside)

2. PEstimator|estimatesthetesteffortinMan-Days

ii.e.Thelower&upperlimits

www.testimation.com


KeydifferencesbetweenTEstimator1,2,3are:

1. TEstimator1requiresone(1)input• Scopeonly

2. TEstimator2,3requirethree(3)inputs

1. HistoricalScope2. HistoricalTestCases3. ProjectedScope

KeydifferencesbetweenTEstimator2,3are:

1. TEstimator2 can scientifically assess the level of Quality Assurance of Historical

Solutions,butislessflexibleiithanTEstimator3

2. TEstimator3 provides greater flexibility than TEstimator2, but cannot scientificallyassessthelevelofQualityAssuranceofHistoricalSolutionsiii

Applicator Product Suite architecture & solution intelligence may be represented by thefollowingVenndiagram;

iii.e.TheQualityOverridefunctionisoptionallyactivatediiii.e.TheQualityOverridefunctionismandatorilyactivated

www.testimation.com


B. FeatureList

WhatisaFeature&Why?

AFeature isany InputorOutput.TheyareFeaturebecause theydonotoccurnaturally;a

person is required to physically design, build & implement them. Even if the Features arerepeated,ittakeshumanwill&efforttorepeatthemondifferentscreensorreports.Everything,withoutexception,inallApplications,requireLines-of-Codetobecreated,copiedormodifiedtoperform the functions we desire. The choice of Input & the form of Output requires humandecision-makingintermsofneed&suitability.Forthesereasons,Inputs&OutputsareFeaturesbecausetheyrequirethemindofahumanbeingforthemtobecometangible.

1. TEstimator1

A. Userinterfaces1. QuickEstimateView2. LeanEstimateView3. BasicView4. AdvancedView

B. UserInputMetric5. FunctionalProcesses6. UseCases7. UserStories8. TestScenarios

C. Userscanspecifyprioritization&riskaspartoftheirtestingstrategy9. IntermsofCalibrationFactoriv10. IntermsofRiskQuotientv

D. ProgressionTestingTheminimumnumberofrecommendedProgressionTestCases11. IntermsofDynamicInformationTests12. IntermsofTestCasesThemaximumnumberofrecommendedProgressionTestCases13. IntermsofDynamicInformationTests14. IntermsofTestCases

E. RegressionTesting15. TheoptimalnumberofmanualRegressionTestCases16. TheoptimalnumberofautomatedRegressionTestCases

F. RiskManagement17. RiskExposurevi{β}18. RiskMitigationvii{α}19. TestCoverageofDevelopment(%)20. SystemUnaffectedbyDevelopment(%)21. TestCasecomplexity(%)22. Defectcomplexity(%)

G. General23. GraphicallydisplaysQualityAssurancearchitecture&strategy{α,β}24. RandomconfigurationgeneratorforUsertrainingpurposes

ivi.e.TestCasepriorityvi.e.ThelevelofAcceptableRiskvi%System-wideimpactofDevelopment{β}vii%ProbabilityoffindingDefects{α}

www.testimation.com


2. TEstimator2

H. UserInputMetric25. FunctionalProcesses+TestCases26. UseCases+TestCases27. UserStories+TestCases28. TestScenarios+TestCases

I. TestCaseCeilingGuide HistoricalSolutions

29. 99%CoverageConfidence30. 99.6%CoverageConfidence31. 99.84%CoverageConfidence32. 99.93%CoverageConfidence33. 99.97%CoverageConfidence

ProjectedSolutions34. 99%CoverageConfidence35. 99.6%CoverageConfidence36. 99.84%CoverageConfidence37. 99.93%CoverageConfidence38. 99.97%CoverageConfidence

J. ProgressionTesting39. TheminimumnumberofrecommendedProgressionTestCases40. ThemaximumnumberofrecommendedProgressionTestCases

K. RiskManagement HistoricalSolutions

41. RiskExposureviii{δ}42. RiskMitigationix{γ}43. TestCoverageofDevelopment(%)44. SystemUnaffectedbyDevelopment(%)45. TestCasecomplexity(%)46. Defectcomplexity(%)47. TestApproachundertakenintermsofCalibrationFactorx48. TestApproachundertakenintermsofQuotientxi49. AssessmentofthelevelofQualityAssuranceoftheHistoricalSolution

ProjectedSolutions50. RiskExposurexii{ζ}51. RiskMitigationxiii{ε}52. TestCoverageofDevelopment(%)53. SystemUnaffectedbyDevelopment(%)54. TestCasecomplexity(%)55. Defectcomplexity(%)56. TestApproachtobeundertakenintermsofCalibrationFactorxiv57. TestApproachtobeundertakenintermsofQuotientxv58. AssessmentofthelevelofQualityAssuranceofHistoricalSolutions

viii%System-wideimpactofDevelopment{δ}ix%ConfidencethatallDefectswerefound{γ}xi.e.TestCasepriorityxii.e.thelevelofAcceptableRiskxii%System-wideimpactofDevelopment{ζ}xiii%ConfidencethatallDefectswillbefound{ε}xivi.e.TestCasepriorityxvi.e.ThelevelofAcceptableRisk

www.testimation.com


L. General59. Users canspecifyQualityOverrideaspartof their interpretationof the testing

strategyintheHistoricalSolutionxvi60. UserscanspecifyRiskOverrideaspartoftheirtestingstrategyfortheProjected

Solutionxvii61. GraphicallydisplaystheQualityAssurancearchitecture&strategy,ofProjected

Solutions&HistoricalSolutionssidebyside{δ,γ,ζ,ε}62. Intelligence preventing unrealistic or impossible User-Defined Historical

SolutionsinfluencingProjectedSolutions(viaWorkFlowpreventionlogic)xviii63. RandomconfigurationgeneratorforUsertrainingpurposes

3. TEstimator3

M. UserInputMetric64. ThirteenvariantsofUserinputmetricavailablexix65. User-DefinedNotesfield(HistoricalSolution)66. User-DefinedNotesfield(ProjectedSolution)

N. ProgressionTesting67. The number of recommended Progression tests for new Projected Solutions

baseduponHistoricalSolutionsO. RiskManagement

HistoricalSolution68. Defect-FreeConfidence:see{η}

ProjectedSolution69. Defect-FreeConfidence:see{θ}

P. General70. GraphicallydisplaystheQualityAssurancearchitecture&strategy,ofProjected

Solutions&HistoricalSolutions(overlaid):see{η,θ}71. Users canspecifyQualityOverrideaspartof their interpretationof the testing

strategyintheHistoricalSolutionxx72. UserscanspecifyRiskOverrideaspartoftheirtestingstrategyfortheProjected

Solutionxxi73. Intelligence preventing unrealistic or impossible User-Defined Historical

SolutionsinfluencingProjectedSolutionsxxii74. RandomconfigurationgeneratorforUsertrainingpurposes

4. REstimator

Q. UserInputMetric75. TestCaseRepository(TCR)Size(i.e.theoverallnumberofTestCases)76. %EstimatedimpactupontheTestCaseRepositorybytheProject77. User-DefinedSamplingConfidence%78. AveragenumberofDynamicInformationTests(DIT’s)perTestCase

xvii.e.thelevelofExecutedRiskxviii.e.ThelevelofAcceptableRiskxviiiUsersarepreventedfrommovingtotheAEstimatorscreenxixe.g.Requirements,linesofcode,securitythreats,businessscenariosetc.xxi.e.thelevelofExecutedRiskxxii.e.ThelevelofAcceptableRiskxxiii.e.viaNormalizationoftheestimateatthePEstimatorscreen

www.testimation.com


R. RiskManagement SamplingDistribution

79. Number of Functional Processes associated with the User-Defined SamplingDistributionConfidence

80. Number of Test Cases associatedwith the User-Defined Sampling DistributionConfidence

81. Risk Mitigation: i.e. system-wide coverage of the User-Defined SamplingDistributionxxiii{ν}

S. General82. Number of Functional Processes associated with the existing Test Case

Repository83. NumberofFunctionalProcessesimpactedbytheProjectchanges84. NumberofexistingTCRtestsimpactedbyProjectchanges85. Number of TCR tests associated with the User-Defined Sampling Distribution

Confidence86. System-widecoverageofexistingTCR:see{κ}87. System-widecoverageofexistingTCRimpactedbyProjectchanges:see{λ}88. Graphically displays Quality Assurance& Risk profiles of theTCR & the User-

DefinedSamplingDistributionsub-set{ν,κ,λ}89. RandomconfigurationgeneratorforUsertrainingpurposes

5. @Risk

T. UserInputMetric90. FunctionalProcesses91. UseCases92. UserStories93. TestScenarios94. UserAcceptanceTests(i.e.TestCases)

U. RiskManagement95. Defectcomplexity(%)96. %ProbabilityofUserAcceptanceTesting(UAT)findingDefectsxxiv{ρ}97. %FunctionalProcessesimpactedbyProjectchangesxxv{β}98. %FunctionalProcessesimpactedbyProjectchangesxiii&coveredbyUATxxvi

V. General99. Graphicaldisplay{α, β}100. RandomconfigurationgeneratorforUsertrainingpurposes

6. AEstimator

W. UserInputMetric101. TestCases

X. RiskManagement102. TEstimator1RiskQuotient103. TEstimator2RiskQuotient104. AlternativeEstimateRiskQuotient105. TEstimator1CalibrationFactor106. TEstimator2CalibrationFactor107. AlternativeEstimateCalibrationFactor

xxiii%ProbabilityoffindingDefects{ν}xxivi.e.System-wide{ρ}xxv%System-wideimpactofDevelopment{β}xxviTestCoverageofDevelopment

www.testimation.com


108. TEstimator1DIT’sperTestCase109. TEstimator2DIT’sperTestCase110. AlternativeEstimateDIT’sperTestCase111. %TEstimator1RiskExposure:see{β}112. %AlternativeEstimateRiskExposure:see{β}113. %TEstimator1RiskMitigationxxvii{α}114. %AlternativeEstimateRiskMitigationxxviii{ι}115. %TEstimator2RiskExposure{ζ}116. %AlternativeEstimateRiskExposure{ζ}117. %TEstimator2RiskMitigationxxix{ε}118. AlternativeEstimateRiskMitigationxxx{ι}119. TestCoverageofDevelopment(%)byTEstimator1120. TestCoverageofDevelopment(%)byAlternativeEstimate(TEstimator1screen)121. TestCoverageofDevelopment(%)byTEstimator2122. TestCoverageofDevelopment(%)byAlternativeEstimate(TEstimator2screen)123. SidebysidecomparisonbetweenTEstimator1&theAlternativeEstimate124. SidebysidecomparisonbetweenTEstimator2&theAlternativeEstimate125. %SimilaritybetweentheUserdefinedestimate&thealternativesourcebyArea126. %SimilaritybetweentheUserdefinedestimate&thealternativesourcebyTest

CasepopulationY. General

127. GraphicallydisplaysQualityAssurancebetweentheUserdefinedestimate&thealternativesource

128. RandomconfigurationgeneratorforUsertrainingpurposes

7. PEstimatorZ. UserInputMetric

129. ProgressiveTestCaseDesignThroughput130. ProgressiveTestCaseModificationThroughput131. ProgressiveTestCaseExecutionThroughput132. RegressiveTestCaseExecutionThroughput133. QualityAssurancePersonnelResourcing134. %ManualProgressiveTesting135. %AutomatedREstimatorTestCases136. %AnticipatedTestCaseFailures

AA. RiskManagement TEstimator1

137. FunctionalProcesses138. TestCases139. RiskExposurexxxi{β}140. RiskMitigationxxxii{α}141. RiskMitigationxxxiii{ν}

TEstimator2142. FunctionalProcesses143. TestCases

xxvii%ProbabilityoffindingDefects{α}xxviii%ProbabilityoffindingDefectsrelatingtoTEstimator1{ι}xxix%ProbabilityoffindingDefects{ε}xxx%ProbabilityoffindingDefectsrelatingtoTEstimator2{ι}xxxi%System-wideimpactofDevelopment{β}xxxii%ProbabilityoffindingDefectsviaProgressionTesting{α}xxxiii%ProbabilityoffindingDefectsviaRegressionTesting{ν}

www.testimation.com


144. RiskExposurexxxiv{ζ}145. RiskMitigationxxxv{ε}146. RiskMitigationxxxvi{ν}

TEstimator3147. FunctionalProcesses148. TestCases149. RiskExposurexxxvii{ξ}150. RiskMitigationxxxviii{ο}151. RiskMitigationxxxix{ν}

REstimator152. FunctionalProcesses153. TestCases

BB. QualityAssuranceBreakdown TEstimator1

154. ProgressiveTEstimatorTestCasestobeDesigned155. RegressiveTEstimatorTestCasestobeModified156. TEstimatorTestCasestobeExecuted157. REstimatorTestCasestobeExecuted158. AnticipatedDefects159. TotalExecutableTestCases

TEstimator2160. ProgressiveTEstimatorTestCasestobeDesigned161. RegressiveTEstimatorTestCasestobeModified162. TEstimatorTestCasestobeExecuted163. REstimatorTestCasestobeExecuted164. AnticipatedDefects165. TotalExecutableTestCases

TEstimator3166. ProgressiveTEstimatorTestCasestobeDesigned167. RegressiveTEstimatorTestCasestobeModified168. TEstimatorTestCasestobeExecuted169. REstimatorTestCasestobeExecuted170. AnticipatedDefects171. TotalExecutableTestCases

CC. ProjectManagement TEstimator1

172. Man-Days173. Man-Weeks174. Man-Months

TEstimator2175. Man-Days176. Man-Weeks177. Man-Months

TEstimator3178. Man-Days179. Man-Weeks180. Man-Months

xxxiv%System-wideimpactofDevelopment{ζ}xxxv%ConfidencethatallDefectswillbefoundviaProgressionTesting{ε}xxxvi%ProbabilityoffindingDefectsviaRegressionTesting{ν}xxxvii%System-wideimpactofDevelopment{ξ}xxxviii%ConfidencethatallDefectswillbefoundviaProgressionTesting{ο}xxxix%ProbabilityoffindingDefectsviaRegressionTesting{ν}

www.testimation.com


DD. General181. GraphicallydisplaysprojectQualityAssurance&Riskprofiles{α,β,ν}182. GraphicallydisplaysprojectQualityAssurance&Riskprofiles{ζ,ε,ν}183. GraphicallydisplaysprojectQualityAssurance&Riskprofiles{ξ,ο,ν}184. RandomconfigurationgeneratorforUsertrainingpurposes

EE. TotalFeatureCount1. NumberofTEstimator1Reportfeatures =242. NumberofTEstimator2Reportfeatures =393. NumberofTEstimator3Reportfeatures =114. NumberofREstimatorReportfeatures =155. Numberof@RiskReportfeatures =116. NumberofAEstimatorReportfeatures =287. NumberofPEstimatorReportfeatures =568. TotalNumberofReportfeatures =1849. NumberofScreenfeatures=TotalNumberofReportfeatures=18410. TotalNumberofApplicatorProductSuitefeatures =368+

www.testimation.com


4. HowtoexecuteaTEstimator1Estimate

A. Applicability

Uses UserInterfaces Users1. Bids2. Tenders3. RiskAssessment

1. QuEstimateViewQuickEstimate

2. LEstimateViewLeanEstimate

3. BasicView4. AdvancedView

1. BoardofDirectors2. ChiefExecutiveOfficers3. ChiefTechnologyOfficers4. SecurityAgencies5. ServiceProviders6. SystemIntegrators7. ProjectManagers8. DevelopmentManagers9. QualityAssuranceManagers10. Corporations11. Government

B. EstimationProcess

C. HowtoexecuteaQuEstimateorLEstimate

1. Inputs

1. SetupScope1. SelectFunctionalProcesses,UseCases,UserStoriesorTestScenariosfromthe

dropdownmenu2. Specifythedesiredvaluebyusingtheslider

2. Configuration

1. SpecifyorConfirmConversionRatio1. ThisistheaveragenumberofFunctionalProcessesyouexpecttohaveperUse

Case,UserStoryorTestScenario2. Thedefaultvalue forUseCases&UserStories is5, thedefaultvalue forTest

Scenariosis13. IfyoudonotknowtheConversionRatio,usethedefaultvalue

2. SpecifyorConfirmDynamicInformationTests(DIT’s)perTestCase1. ThisistheaveragenumberofDynamicDataFieldsyouexpecttocheckperTest

Case2. IfyoudonotknowtheDIT’sperTestCase,usethedefaultvalue

SetupScope

•  AllViews

SpecifyorCon�irmConversionRatio

•  AllViews

SpecifyorCon�irmDIT's

•  AllViews

SpecifyorCon�irmCalibrationFactor

•  BasicView•  AdvancedView

SpecifyorCon�irmRiskQuotient

•  BasicView•  AdvancedView

OutputEstimate

•  AllViews

www.testimation.com


D. HowtoexecuteanEstimateinBasicView

1. See“HowtoexecuteaQuEstimateorLEstimate”2. Select a Calibration Factor Set-Point. The Calibration Factor (CF) is part of the

System-WideTestApproach,not justthecodebeingDevelopedorModifiedfortheProject

1. CF=0.25TargetsCriticalPriorityTests2. CF=0.5TargetsCritical&HighPriorityTests3. CF=0.75TargetsCritical,High&ModeratePriorityTests4. CF=1TargetsCritical,High,Moderate&LowPriorityTests5. CF=1.25TargetsallTests+25%Redundancy6. CF=1.5TargetsallTests+50%Redundancy7. CF=1.75TargetsallTests+75%Redundancy8. CF=2TargetsallTests+100%Redundancy9. SeeFrameworkProbabilitiesInformationtoassistwithdecisions

3. SelectRiskQuotientSet-Point1. TheRiskQuotientallowsaUsertotargetanacceptablelevelofriskinorder

tominimiseexpenditure2. SeeFrameworkProbabilitiesInformationtoassistwithdecisionsxl

E. HowtoexecuteanEstimateinAdvancedView

1. See“HowtoexecuteaQuEstimateorLEstimate”2. See“HowtoexecuteanEstimateinBasicView”3. ClicktheCalibrationFactorExpandbutton(ifrequired)

1. TheUserhasaccesstotheCalibrationFactorApplyCriticalValuebutton2. ThisfeaturemaybeinvokedtocomputetheminimumnumberofTestCases

providing100(%)TestCoverageofDevelopment.3. This featureprovidesaDevelopedorModifiedcode-onlyspecificTestCase

sizingsolutionwhenα=β4. ClicktheRiskQuotientExpandbutton(ifrequired)

1. TheUserhasaccesstotheRiskQuotientdial2. TheUserhasaccesstotheRiskQuotientApplyCriticalValuebutton3. ThesefeaturesprovidetheUserwiththeabilitytoreduceTestCasesizingby

controllingrisk4. These features may be invoked to compute the minimum number of Test

Casesproviding100(%)TestCoverageofDevelopment5. ThesefeaturesprovideaDevelopedorModifiedcode-onlyspecificTestCase

sizingsolutionwhenα=β

xlParticularly:TheprobabilityoffindingatleastoneDefect

www.testimation.com



A. Applicability

1. The Projected Solution is scaled from an Historical Solution via the Defect-FreeConfidence

2. The Defect-Free Confidence is calculated scientifically by default, or may beoverridden&assignedavaluebytheUser

Uses Users1. In-houseProjects2. Bids3. Tenders

1. BoardofDirectors2. ChiefExecutiveOfficers3. ChiefTechnologyOfficers4. SecurityAgencies5. ServiceProviders6. SystemIntegrators7. ProjectManagers8. DevelopmentManagers9. QualityAssuranceManagers10. Corporations11. Government12. Military


C. Procedure

1. Inputs

1. SetuptheHistoricalSolution1. Select Functional Processes, Use Cases, User Stories or Test Scenarios from

thedropdownmenu2. Specifythedesiredvaluebyusingtheslider3. InputthenumberofExecutedTestCases

2. SetuptheProjectedSolution1. Specifythedesiredvaluebyusingtheslider

SetupScope

•  HistoricalSolution•  ProjectedSolution

SpecifyorCon�irmConversionRatio

•  HistoricalSolution

SpecifyorCon�irmDIT's

•  HistoricalSolution


•  HistoricalSolution•  QualityOverride

•  ProjectedSolution•  RiskOverride

OutputEstimate

•  ProjectedSolution

www.testimation.com


2. Configuration

1. SpecifyConversionRatio1. This is the average number of Functional Processes you expect to have per

UseCase,UserStoryorTestScenario2. ThedefaultvalueforUseCases&UserStoriesis5;thedefaultvalueforTest

Scenariosis13. IfyoudonotknowtheConversionRatio,usethedefaultvalue

2. SpecifyDynamicInformationTests(DIT’s)perTestCase1. This is theaveragenumberofDynamicDataFieldsyouexpect to checkper

TestCase2. IfyoudonotknowtheDIT’sperTestCase,usethedefaultvalue

3. ConfigureQualityOverride1. The default configuration at page load is a scientific assessment of the

Historical Solution. At this juncture, the User must decide to override thescientificassessment(ornot)

2. If the User decides to utilise the scientific assessment; no further action isrequired

3. IftheUserdecidestooverridethescientificassessment,theUsermustassignaDefect-FreeConfidence to theHistorical Solutionbyutilising theSet-PointSliderorRiskQuotientDialxli

4. IftheUserwishestoreturntothescientificassessment,itmaybeachievedbyclickingthe“ApplyCriticalValue”buttonunderneaththeRiskQuotientDial

4. ConfigureRiskOverride1. Byinspectionofthegraphs&generalProjectconsiderations,theUserdecides

theappropriateconfigurationfortheProjectedSolution2. IftheProjectisconstrainedbytimeorbudget,theUsermaywishtobalance

theappetiteforTestingEffort(TestCases)withanacceptablelevelofDefect-FreeConfidence

3. The default Risk Quotient value is 0% (yielding a Defect-Free Confidence of99%). If the User wishes to focus on the Projected Code Changes only &minimise the Test Effort, the “Apply Critical Value “ button underneath theRiskQuotientDialmaybeutilised;theeffectofthisactionisreflectedinthegraphs

4. IftheCriticalValuefortheRiskQuotientappearsnegative,thisindicatesthatthe default scientific assessment of the Historical Solution has beenoverriddenbytheUser&it is likelythat insufficientTestingwasperformed.In this case, the User should consider utilising an alternative HistoricalSolutionasaframeofreferencefortheProjectedSolution

xliExamples1. TheUser believes that theHistorical Solution yielded an 80.22%probability that no

Defects exist (system-wide); therefore, the User selects a Risk Quotient Set-Point of50%.Thisrelatestoan80.22%probabilitythatallsystem-wideCriticalPriorityTestswereexecuted&passedsuccessfully

2. The User believes that the scientific assessment of Defect-Free Confidence is toocautious & recalls that the Historical Solution was implemented without significantincident. The User believes that the system-wide Defect-Free Confidence isapproximately95%.Consequently,theUserselectsaRiskQuotientSet-Pointof23%

3. The User believes that the scientific assessment of Defect-Free Confidence is toocautious & recalls that the Historical Solution was implemented without significantincident. The User believes that the system-wide Defect-Free Confidence isapproximately99%.Consequently,theUserselectsaRiskQuotientSet-Pointof0%

www.testimation.com


3. Analysis|Historical&ProjectedSolutions

1. TheGraphsgeneratedcommunicate1. RiskExposure

1. HistoricalSolution1. Theproportionofsystem-wide functionality thatwasDevelopedor

Modified2. ProjectedSolution

1. The proportion of system-wide functionality to be Developed orModified

2. RiskMitigation1. HistoricalSolution

1. TheprobabilitythattheapproachtoTestingfoundallDefects2. ProjectedSolution

1. The probability that the approach to Testing will find at least oneDefect

3. TestCoverageofDevelopment1. HistoricalSolution

1. The proportion of Development that was covered by the TestApproach

2. ProjectedSolution1. TheproportionofDevelopmenttobecoveredbytheTestApproach

4. TheTargetedDistributionfortheHistoricalSolutionxlii5. TheTargetingDistributionfortheProjectedSolutionxliii

2. TestExecutionApproachcommunicates1. HistoricalSolution

1. ViaCalibrationFactor1. The composition of Critical, High, Moderate & Low Priority Test

Cases,whichshouldhavebeenExecuted2. ViaQualityOverride

1. The composition of Critical, High, Moderate & Low Priority TestCasestheUserclaimswereExecuted

2. ProjectedSolution1. ViaCalibrationFactor

1. The composition of Critical, High, Moderate & Low Priority TestCaseswhichshouldbeExecuted

2. ViaQualityOverride1. The composition of Critical, High, Moderate & Low Priority Test

CasestheUserclaimswillbeExecutedasaScaledSolution

xliiDescription1. AvisualrepresentationofthecombinedeffectoftheCalibrationFactor&RiskQuotient2. Communicates the effect of the Calibration Factor& RiskQuotient upon the Targeted

DistributionofFunctionalProcessesthatwereDevelopedorModified3. CommunicatesthattheUserfocusedTestingaroundthemeannumberofDataFieldsper

DatabaseRecordorthemostcommonlyusedsystem-wideFunctionsxliiiDescription1. AvisualrepresentationofthecombinedeffectoftheCalibrationFactor&RiskQuotient2. Communicates the effect of the Calibration Factor&RiskQuotient upon the Expected

(Targeting)DistributionofFunctionalProcessestobeDevelopedorModified3. CommunicatesthattheUserexpectstofocusTestingaroundthemeannumberofData

FieldsperDatabaseRecordorthemostcommonlyusedsystem-wideFunctions

www.testimation.com


4. QualityOverride

1. This section allows theUser to Override the default (scientific) Analysis of theHistorical Solution by specifying the perceived level of Risk Quotientretrospectively

2. After entering the appropriate information in the Scope section, theUsermustdecideifthedefaultAnalysisisindicativeoftheactualresultsassociatedwiththeHistoricalSolutionxliv

3. IftheRiskQuotientsliderisselectedortheUserappliesthedial,thenthedefaultAnalysis has been overridden by the User in favour of his / her own opinion.Consequently, the Projected Solution utilises theOverriddenUser-SpecificationtocalculatetheRequiredTestCasesfortheProjectedSolution

4. CriticalValuexlv1. Unlike TEstimator1, the “Apply Critical Value” button does not graphically

illustratethevalueofRiskQuotientpreciselycorrespondingto100(%)TestCoverage of Development. It is not permitted to do this because it isconstrainedbytheactualnumberofTestCasesexecutedhistorically

2. If the “Apply Critical Value” button cannot be clicked, or has been clicked,then the Projected Solution utilises the default Analysis of the HistoricalSolutionbaseduponStatisticalMechanics&Probabilities

5. Defect-FreeConfidencexlvi1. Definition:TheprobabilityoftheHistoricalSolutionbeingDefect-Freebased

uponUserchoices2. ByselectingaRiskQuotientpre-setbuttonon theslider,orusing theRisk

Quotient dial to specify a value anything other than the Critical Valueindicatedbelowthedial,theUserclaimsthattheDefect-FreeConfidenceoftheHistoricalSolutionwasdifferenttothescientificvaluepredictedxlvii.Forexample,byselecting0%RiskQuotient,theUserclaimsthatthenumberofTest Cases Executed for the Historical Solution yielded a Defect-FreeConfidenceof99%.Similarly, if theUser selects50%RiskQuotienton theslider, the User claims that the Historical Solution had a Defect-FreeConfidenceof80.22%

xlivDescription1. Atpage load,orwhentheUserclicksthe“ApplyCriticalValue”button, theapplication

invokes the default Analysis; i.e. the scientific assessment of the Historical SolutionbaseduponStatisticalMechanics&Probabilities

2. The default Analysis determines the number of Test Cases, which should have beenexecuted if 99% of all functionality (system-wide) was Quality Assured; i.e. at aCalibration Factor of unity & Risk Quotient of 0%. The result of this calculation issubsequentlyutilised todetermine the actualCalibrationFactor (seeIndicator)&RiskQuotient(displayedastheCriticalValue)

xlvDescription1. Executedasabackgroundcalculation&representstheleanestTestingSolutionformost

situations(i.e.whenγ=δ)2. RelatestotheFunctionalProcessesDevelopedorModifiedfortheProjectexclusivelyxlviNote1. The scientific assessment regarding the probability of having found all Defects is

displayedasRiskMitigation2. When a Defect-Free Confidence value is enforced, the User overrides the scientific

assessmentinfavourofUserassessmentxlviiDisplayedasthe“CriticalValue”

www.testimation.com


6. RiskMitigation1. Definition: The probability that all Defects were found in the Historical

SolutioniftheactualQualityAssuranceStrategyfollowedthescientificbasis(defaultAnalysis)definedabove

7. QualityOverrideSimilarity1. Definition: The Proportional similarity between Defect-Free Confidence &

RiskMitigationwithintheHistoricalSolution2. Description: The Proportional similarity between User perception &

scientificassessmentwithintheHistoricalSolution

5. RiskOverride

1. The Projected Solution utilises the Historical Solution as a frame of reference,scaling the Required Test Cases accordingly. After entering the appropriateinformationintheScopesection,theUsermustdecideifthedefaultAnalysis(i.e.at page load) is desirable. This section allows the User to override the defaultvalueofRiskQuotientintheProjectedSolutionbyusingtheslider&dial.xlviiiTheProjectedSolutionhastwopossiblepreconditions

1. TheHistoricalSolutionutilisesthedefaultAnalysisxlix2. TheHistoricalSolutionutilisesaUserSpecifiedValueofRiskQuotientl

2. CriticalValue1. Ifthe“RiskOverrideApplyCriticalValue”buttonisclicked,thevalueofRisk

Quotientpreciselycorrespondingto100(%)TestCoverageofDevelopmentis applied. The result relates to the Functional Processes Developed orModifiedfortheProjectedSolutionexclusively.ThisimplementstheleanestTestingSolution formostsituations{ε=ζ}.Hence,whenε>ζorε<ζ, the“Risk Override Apply Critical Value” button enforces the value of RiskQuotientsuchthatε=ζ

2. The Risk Override Critical Value may become negative (depending uponUser-Specifications);indicatingthattheHistoricalSolutionwasunder-testedfromtheperspectiveofscientificassessment

3. ExampleEstimate1. HistoricalSolution

1. Setup:UserStories=61,ExecutedTestCases=583,ConversionRatio=5,DIT’s=10,RiskQuotientSet-Point0%isappliedli

2. ThedefaultAnalysisindicatesthataninsufficientdegreeofTestingwasperformed(viatwomethods)lii

xlviiiThe scientific prediction of the Projected Solution based upon Statistical Mechanics,Probabilities & Engineering principles of Similitude, such that the default Analysisdetermines the number of Required Test Cases which should be executed if 99% of allfunctionality(system-wide)wasQualityAssured; i.e.ataCalibrationFactorofunity&RiskQuotientof0%xlixi.e.TheCriticalValueoftheRiskQuotienthasbeenappliedinQualityOverridelThe scaled result is subsequently utilised to determine the actual Calibration Factor (seeIndicator)lii.e.InsistingthattheHistoricalSolutionincorporated99%Defect-FreeConfidenceliiMethods1. TheTestExecutionApproach(viaCalibrationFactor)=CriticalPriorityTests;meaning

thatonlyaproportionofCriticalPriorityTestswereexecuted2. Graphicallybyγ<δ

www.testimation.com


3. However, the User rejects the default Analysis & insists that the TestExecutionApproach(viaQualityOverride)was(inreality)Critical,High,Moderate&LowPriorityTests.Consequently,theUserwishestoapplythe Historical Solution as a baseline frame of reference to bereproducedintheProjectedSolution

2. ProjectedSolution1. Setup:UserStories=61,RiskQuotientCriticalValue=0%2. Because the User has overridden the default Analysis, the Projected

SolutionisconstrainedbytheHistoricalSolutionsuchthattheRequiredTest Casesmust equal the Executed Test Cases because the ProjectedSolution intends to mimic the perceived success of the HistoricalSolution

4. Defect-FreeConfidenceliii1. Definition:TheprobabilityoftheProjectedSolutionbeingDefect-Freebased

uponUserchoices2. ByselectingaRiskQuotientpre-setbuttonon theslider,orusing theRisk

Quotient dial to specify a value anything other than the Critical Valueindicatedbelowthedial,theUserclaimsthattheDefect-FreeConfidenceoftheHistorical Solutionwasdifferent to the scientific valuepredictedliv. Forexample, by selecting 0% Risk Quotient, the User seeks a Defect-FreeConfidenceof99%.Similarly, if theUser selects50%RiskQuotienton theslider,theUserseeksaDefect-FreeConfidenceof80.22%fortheProjectedSolution

5. RiskMitigation1. Definition: The probability of finding at least one Defect in the Projected

SolutionattheRiskOverrideconfigurationspecified6. RiskOverrideSimilarity1. Definition: The Proportional similarity between Defect-Free Confidence &

RiskMitigationwithintheProjectedSolution2. Description: The Proportional similarity between User perception &

scientificassessmentwithintheProjectedSolution

liiiNote3. The scientific assessment regarding the probability of having found all Defects is

displayedasRiskMitigation4. When a Defect-Free Confidence value is enforced, the User overrides the scientific

assessmentinfavourofUserassessmentlivDisplayedasthe“CriticalValue”

www.testimation.com


6. GeneralConsiderations

1. The importance of knowing the Defect-Free Confidence & Risk Mitigationprobabilities (together) is for overall Risk Assessment & decision-making. Thesmaller the difference between these twoprobabilities, the lower theNetRisk&thegreaterthesimilaritybetweentheHistorical&ProjectedSolutions

2. Defect-FreeConfidenceDecisionTable

Defect-FreeConfidence TestCasesSuccessfullyPassed99%-97.43% Critical(all),High(all),Moderate(all)&LowPriority97.42%-93.15% Critical(all),High(all)&ModeratePriority93.14%-80.23% Critical(all)&HighPriority>0-80.22% CriticalPriority

3. TEstimator2UserScenarios1. TheUserapplieslvtheCriticalValueRiskQuotient to theHistoricalSolution&

the Risk Quotient of the Projected Solution is 0%. In this scenario, the Useracceptsscientificassessmentoverhumanrecollectionofhistoricalevents&theRequired Number of Test Cases for the Projected Solution is calculated at aDefect-FreeConfidenceof99%

2. TheUserappliesaNon-CriticalValueRiskQuotienttotheHistoricalSolutionlvi&theRiskQuotientoftheProjectedSolutionis0%.Inthisscenario,TheUseroverridesscientificassessmentoftheHistoricalSolution&declarestheDefect-FreeConfidencetobeaUserspecifiedvaluelvii

3. TheUserappliestheCriticalValueRiskQuotienttotheHistorical&ProjectedSolutions.Inthisscenario,TheUseracceptsscientificassessmentoverhumanrecollectionofhistoricalevents& intends toconfine theTestCoverage in theProjected Solution to the code under development only (testing the broadersystembeyondthecodeunderdevelopmentwillnotoccur).ThisimplementstheleanestTestingSolutionformostsituations{ε=ζ}

4. TheUserappliesaNon-CriticalRiskQuotientvaluetotheHistoricalSolution&theCriticalValueRiskQuotienttotheProjectedSolution

5. The User applies a Non-Critical Risk Quotient value to the Historical &ProjectedSolutions

4. TestCoverageofDevelopment1. Definition:TheproportionofDevelopmentcoveredbyTesting2. Description: The scientific assessment of the proportion of Development

coveredbyTesting.ThisdoesnotincludenorconsidertheperceivedcoveragebytheUser

lvThisisthedefaultsettingatpageloadlviTestCoverageofDevelopmentshouldbeconsideredpriortoinvokingthisactionlviii.e.TheDefect-FreeConfidenceindicatorontheleft

www.testimation.com



A. Applicability

Uses UsersIn-houseProjects

1. BoardofDirectors2. ChiefExecutiveOfficers3. ChiefTechnologyOfficers4. BusinessOwners5. SecurityAgencies6. Corporations7. Government8. Military

The TEstimator3 Tool is a powerful & flexible instrument, facilitating the scaled

reproductionof anHistorical Solution to aProjectedSolution. Itmaybeused for any formofQuality Assurance, including Hardware as well as Software. The technique is based upon afundamental Engineering technique termed Similitude (typicallyappliedtoThermodynamics&Fluid Mechanics). A milestone for the development of Similitude was introduced by EdgarBuckingham via Dimensional Analysis (now known as Buckingham Π Theory). The basicprincipleofDimensionalAnalysis& thederivationof theΠGrouping techniqueDevelopedbyEdgar Buckingham, is such that a User (a Scientist or Engineer) may formulate an equationincorporating any number of unknown multidimensional variables, preceded by anexperimentally determined factor (traditionally expressed as “K”). The value of “K” may be aconstantorfunction,butitcanonlybedeterminedexperimentally&cannotbemathematicallyderived,onlyassignedanassumedvalue:

• Forexample1. “E=mc2”isaspecificsolutionof“E=K•mc2”where“K=1”2. Forthisexampleinreality,“K”isnotassumedtobeanyothervalue,onlyunity

A key feature of this technique is that there is no “wrong or right” combination of

multidimensional variables to combine; it is entirely the personal choice of the individualformulating the governing equation. This becomes particularly important in the TEstimator3Tool because its application is influenced by the logic being employed by the User. For thisreason,anotessectionhasbeenaddedsothatassumptions&otherrelevant informationmaybe captured& recorded at the time of estimation. The examples shown in “TEstimator3 UserScenarios”demonstratethataUsermayselectaparameterfromthedropdownlist&relateittoaperceived levelofHistoricalQualityAssuranceviathe“QualityOverride-RiskQuotient”.Bydoingthis,theUsercombinesallotherfactorsintoanundefinedexperimentalvalue(“K”).Theexperimentalvaluemaybedependentupon(forexample):

1. Publicholidays2. Resourceutilisation3. Sickleave4. Skillsets&/orSkillLevels5. Anyreasonableinfluenceorcombinationofinfluencesetc.

BecausetheUserisscalingfromanHistoricalSolutiontoaProjectedSolution,thevalueof

“K” is not required to be known. The effect of this is such that all the benefits & problemsnaturally incorporated into the Historical Solution’s experimental value “K” is also factoredinvisiblyintotheProjectedSolution.ThismeansthatwhatevernegativeinfluencestranspiredintheHistoricalSolution,arepresumedtooccurintheProjectedSolution.

www.testimation.com



C. TEstimator3UserScenarios1. By example: Business Scenarios, Features, Integrated Systems, Interfaces, Number of

Screens,Reports,Requirements,Servers1. AUserwishestodetermineaRough-Order-of-MagnitudefortheQualityAssurance

EffortassociatedwithapotentialProject2. Aftersomeinvestigation,theUserdecidestouseanHistoricalSolutionasaframeof

Reference3. TheUserdecidestousethenumberofRequirementsastheirHistoricalEstimation

Basis4. The User selects Requirements from the dropdown list & inputs the number of

HistoricalRequirements&ExecutedTestCases5. The User inputs an estimate for the number of Requirements expected in the

ProjectedSolution;thisdoesnotneedtobeavaluewithhighconfidence(itmayberevisedatalatertime)

6. TheUserconsultswithrelevantstakeholders&assignsavalueof“QualityOverride-RiskQuotient”.Thevalueassignedisintendedtorepresentthesystem-wideDefect-FreeConfidenceoftheHistoricalSolution.ThisisnotjusttheDefect-FreeConfidenceof thespecificcode thatwasDevelopedorModified for theHistoricalSolution,butalsoitsimpactupontheexistingsystem&functionality.Apossibleindicatorforthisvaluemay be the standardRegression Suite; if the significantmajority of the TestCasesinthestandardRegressionSuitehavepassedinthenormalmanner,thentheDefect-Free Confidence of the Historical Solution should be high. If the standardRegression Suite experienced more failures than anticipated, then the “QualityOverride-RiskQuotient”valueshouldbesetlow.TheeffectofthisistoincreasetheTestingintheProjectedSolution.IftheUserbelievesthattheHistoricalSolutionwasashiningexampleofQualityAssurance,thenthe“QualityOverride-RiskQuotient”shouldbesetto0%

7. TheUserdecidestosetthe“RiskOverride-RiskQuotient”in-linewiththeavailabletime&budgetfortheProject;LowerRisk=moreTesting=greatercost

8. AProjectedValuefortheRequiredTestCasesiscomputed2. Byexample:Lines-of-Code,Modules,Objects

1. A User wishes to determine a Rough-Order-of-Magnitude for the number of UnitTests(TestCases)associatedwithapotentialProject

2. The User decides to the use Lines-of-Code of an Historical Solution as a frame ofReference

3. TheUser knows that “X” Lines-of-CodewereModified in theHistorical Solution&thattheProjectedSolutionwillcontainapproximately“Y”Lines-of-Code

4. TheUserknowsthat“X”Lines-of-Coderequired“Z”numberofUnitTests5. TheUserinterpretsthe“QualityOverride-RiskQuotient”(inthiscontext)toreflect

the system-wide coverage (influence) of the “X” Lines-of-Code upon the HistoricalSolution1. TheUserestimatesthatthe“X”Lines-of-Codetouchedapproximately20.33%of

system-widefunctionalityinsomeway(directlyorindirectly)

SetupScope

•  HistoricalSolution•  ProjectedSolution


•  HistoricalSolution•  QualityOverride

•  ProjectedSolution•  RiskOverride

OutputEstimate

•  ProjectedSolution

www.testimation.com


2. The User selects a “Quality Override - Risk Quotient” of 90% using the RiskQuotientDial

6. TheUserdecidestosetthe“RiskOverride-RiskQuotient”in-linewiththeavailabletime&budgetfortheProject(lowerRisk=moreTesting=greatercost)

7. AProjectedValuefortheRequiredTestCases(UnitTests)iscomputed3. Byexample:SecurityThreats(pings)

1. AUserwishestodetermineaRough-Order-of-MagnitudeforthepotentialnumberofSecurityThreatsassociatedwithanewProject

2. TheUserselectsanappropriateHistoricalSolutiontoutilizeasaframeofreferenceforscaling

3. TheUserdescribes the structureofhis /her reasoning for selecting theHistoricalSolutionintheNotessection.Inthisexample,theUserdecidestorelatethenumberofaccessibleportstothenumberofpings

4. TheUserconcludesthat“X”numberofportsresultedin“Y”numberofpingsintheHistoricalSolution

5. TheUserisinformedthattheProjectedSolutionwillcontain“Z”numberofports6. TheUserinterpretsthe“QualityOverride-RiskQuotient”(inthiscontext)toreflect

thesystem-widepotentialforundetectedSecurityThreatsintheHistoricalSolution7. The User assumes a “Quality Override - Risk Quotient” of 21%; consequently, the

Useris95.81%confidentthatnoundetectedSecurityThreatsexistintheHistoricalSolution

8. The User decides to set the “Risk Override - Risk Quotient” in-line with a ZEROThreatpolicy&selects0%

9. Intheexampleabove1. FortheHistoricalSolution

1. TheSecurityThreatsfieldispopulatedby“X”(numberofports)2. TheExecutedTestCasesfieldispopulatedby“Y”(numberofpings)

2. FortheProjectedSolution1. TheSecurityThreatsfieldispopulatedby“Z”(numberofports)2. ThepotentialnumberofpingsiscalculatedintheRequiredTestCasesfield

(i.e.theexistenceofonepingisoneTestCase)

www.testimation.com


7. Definitions

A. CalibrationFactor1. Dealswithsystem-widetests,notjustProjectscopetests2. This may include some or all of the Test Cases required to validate the

Functional Processes being Developed or Modified within the scope of theProject

3. Refer to the Quality Assurance Strategy Graph to determine if the selectedCalibrationFactorprovidestheappropriateTestCoverageofDevelopment

4. Set-Point0.25isselected1. AllCriticalPriorityTestsaretobeexecuted2. 80.22%probabilityoffindingatleastoneDefect

5. Set-Point0.5isselected1. AllCritical&HighPriorityTestsaretobeexecuted2. 93.15%probabilityoffindingatleastoneDefect

6. Set-Point0.75isselected1. AllCritical,High&ModeratePriorityTestsaretobeexecuted2. 97.43%probabilityoffindingatleastoneDefect

7. Set-Point1isselected1. AllCritical,High,Moderate&LowPriorityTestsaretobeexecuted2. 99%probabilityoffindingatleastoneDefect

8. Set-Point1.25isselected1. All Critical, High, Moderate & Low Priority Tests are to be executed +

25%Redundancy(25%moreteststhanSet-Point“1”requires)2. 99.6%probabilityoffindingatleastoneDefect





11. Set-Point2isselected1. All Critical, High, Moderate & Low Priority Tests are to be executed +


B. Configuration12. SeeConversionRatio13. SeeDIT’sperTestCase

C. ConversionRatio14. AppliestoUseCases,UserStories&TestScenarios15. DenotestheaveragenumberofFunctionalProcessesassociatedwitheachofthe

above16. Examples

1. One (1) Use Case maps to five (5) Functional Processes (on average);thereforetheConversionRatioisfive(5)

2. One (1)User Storymaps to five (5) FunctionalProcesses (onaverage);thereforetheConversionRatioisfive(5)

www.testimation.com


3. One(1)TestScenariomapstoone(1)FunctionalProcesses(onaverage);thereforetheConversionRatioisone(1)

D. CriticalValue17. ThevalueofCalibrationFactorpreciselycorrespondingto100TestCoverageof

Development(%)18. Relates to the Functional Processes Developed or Modified for the Project

exclusively19. Implements the leanest Testing Solution for most situations (unless otherwise

overriddendeliberatelybytheUser)20. CalculatestheprobabilityoffindingatleastoneDefectassociatedwiththeabove

E. Defect-FreeConfidence

21. HistoricalSolution1. Theprobabilityof theHistoricalSolutionbeingDefect-Freebasedupon

Userchoices22. ProjectedSolution

1. Theprobabilityof theProjectedSolutionbeingDefect-FreebaseduponUserchoices

F. DIT’sperTestCase

23. AnacronymforDynamicInformationTestsperTestCase24. The average number of fields per Test Cases containing dynamic information

whichmustbeverifiedwith“yesorno”confirmationbyahumanbeing25. Excludesallstaticcontent&fields

G. DynamicInformationTests(DIT’s)26. Themaximumnumberof“yesorno”Testsrequiredtobeexecutedbyahuman

being27. A “yes or no” Test may be described by the following question: is this field

correct,“yesorno”?28. Yes=TestPassed,No=TestFailed29. PartialorincompletePasscategorisationdoesnotapply30. These Tests are confined to Dynamic content only; Static content is not

applicable31. These Tests may include any combination of Positive & / or Negative Tests

required

H. ExecutedTestCases32. TheactualnumberofTestCasesExecuted

I. Framework

33. TestApproachfocusingonCalibrationFactor&RiskQuotient

J. FrameworkProbabilities

34. Avisual representationof the combinedeffectof theCalibrationFactor&RiskQuotient

35. Communicates the effect of the Calibration Factor & Risk Quotient upon theExpected (Targeting) Distribution of Functional Processes to be Developed orModified

www.testimation.com


36. CommunicatesthattheUserexpectstofocusTestingaroundthemeannumberofDataFieldsperDatabaseRecord

37. CommunicatesthattheUserexpectstofocusTestingaroundthemostcommonlyusedsystem-wideFunctions

K. FunctionalProcess

38. A train of FunctionPoints enactedby theUser to execute aworkplace activity(i.e.theUserdoingtheirjob;WorkInstructions)

1. CreatingorDeletingCustomerAccountsetc.2. PrintingWeekly,MonthlyorAnnualSalesReportsetc.3. BillingCustomers,loggingCustomerComplaintsetc.

39. A suite of one ormore appropriate Functional Processes is termed a BusinessProcess

1. Example1:On-Boardinganewemployee(i.e.theBusinessProcess)mayinvolvethefollowingFunctionalProcesses

1. CreatinganE-MailAccount2. CreatingaUserApplicationAccount3. CreatingaFinancialAccount(salary,wageetc.)4. CreatingaSecurityPass5. CreatingUserAccess(aPass)totheBicycleCage

2. Example 2: Logging Customer Complaints is one Functional Process inthebroaderBusinessProcessofHandlingCustomerComplaints

L. ProgressionTesting

40. ReferstotestsrelatingtotheProjectCodebeingModifiedorDeveloped

M. QualityAssuranceArchitecture41. Theoverarching(system-wide)intendedapproachtoTesting

N. QualityAssuranceStrategy42. Theoverarching(system-wide)resultantapproachtoTesting43. ItutilisestheCalibrationFactor&RiskQuotientparametersdefinedbytheUser

intheQualityAssuranceArchitecturesection44. The key difference between Architecture & Strategy is that the Architecture

expressestheintended(desired)approachtoTestingviatheCalibrationFactor&Risk Quotient as independent parameters, whilst the Strategy expresses thecombinedeffectoftheCalibrationFactor&RiskQuotientupontheapproachtoTesting

45. TheQualityAssuranceStrategyGraphcommunicates1. Risk Exposure: the proportion of system-wide functionality being

DevelopedorModified2. RiskMitigation:theprobabilitythattheapproachtoTestingwill findat

leastoneDefect3. TestCoverageofDevelopment: theproportionofDevelopmentcovered

bytheTestApproach

O. QualityOverrideSimilarity

46. The Proportional similarity between Defect-Free Confidence & RiskMitigationwithintheHistoricalSolution

www.testimation.com


P. RegressionTesting

47. ManualRegressionSuite1. Refers to a statistically random sampling (i.e. a subset) of the Manual

TestCasespredictedintheProgressionTestingsection2. Re-Execution of the statistical sample with a status of “Test Passed”,

yields 99% Confidence that the entire population of Manual Tests willpassiftheywereRe-Executed

3. Thisstatisticalapproach ispredominantlysuited tovery largeprojects;i.e.withthousandsofManualTestCases

4. Statistical approaches may be applied to provide cost effectivealternatives to the Quality Assurance of Software; particularly incompressedtimelines

48. AutomatedRegressionSuite1. Refers to a statistically random sampling (i.e. a subset) of the Manual

RegressionTestCasespredicted2. AutomatedRe-Executionof the statistical samplewith a statusof “Test

Passed”, yields 99% Confidence that the entire population of ManualRegressionTestswillpassiftheywereRe-Executed

3. Is appropriate if an intention exists to Re-Execute the subset multipletimes(typically:atleastfivetimes)

Q. RequiredTestCases49. ThenumberofTestCasesRequiredtobeExecuted

R. RiskExposure50. Istheproportionofsystem-widefunctionalitybeingDevelopedorModified51. Is governed by the number of Functional Processes under Development or

Modification(i.e.Scope)

S. RiskMitigation52. General:theprobabilitythattheQualityAssuranceStrategywillfindatleastone

Defect53. TheprobabilitythatallDefectswerefoundintheHistoricalSolutioniftheactual

QualityAssuranceStrategyfollowedthescientificbasis(defaultAnalysis)54. TheprobabilityoffindingatleastoneDefectintheProjectedSolutionattheRisk

Overrideconfigurationspecified

T. RiskOverrideSimilarity

55. The Proportional similarity between Defect-Free Confidence & RiskMitigationwithintheProjectedSolution

U. RiskQuotient56. Representstheacceptablelevelofrisk57. ReducesthecostoftestingastheRiskQuotientincreases58. ReducestheprobabilityoffindingDefectsastheRiskQuotientincreases59. CanbeappliedtomergetheTestApproach(CalibrationFactor)totheavailable

TestingbudgetinType1estimates.Thatis,thenumberofTestCaseswhichcanbeDesigned&Executedwithinthetime&financialconstraintsoftheProject

1. Decided by the User in isolation or consultation with the QualityAssuranceGovernancebody

2. Takestheform

www.testimation.com


1. QualityBiasedTesting:Range=0-49%2. NeutrallyBiasedTesting:Singlecondition=50%3. RiskBiasedTesting:Range=51-100%

3. CriticalValue4. The value of Risk Quotient precisely corresponding to 100Test

CoverageofDevelopment(%)5. Relates to the Functional Processes Developed or Modified for the

Projectexclusively6. Implements the leanest Testing Solution formost situations (unless

otherwiseoverriddendeliberatelybytheUser)7. Calculates the probability of finding at least one Defect associated

withtheabove

V. Scope60. The number of Functional Processes to be Developed or Modified within the

contextofaproject

W. SystemUnaffected

61. Istheproportionofsystem-widefunctionalitynotbeingDevelopedorModified62. IstheresidualoftheRiskExposure(governedbytheScope)

X. UseCase

63. AsuiteofoneormoreFunctionalProcesses64. The term applied in WaterFall Methodology (historically), referring to the

mannerinwhichaUsershallexecuteaBusinessObjective.Thatis,howtheUsershallinteractwiththeFunctionsbeingDevelopedorModified

65. Typically, a well-written Use Case contains approximately five (5) TestScenarios;however,thisdependsuponmanyfactorsincludingthewritingstyle&experienceoftheUseCasecomposer.

66. ApoorlyconstructedUseCasewillcontaingreaterthanten(10)TestScenarios.Ifthisoccurs,theUsershouldconsiderdecomposingtheUseCaseintomultiplesmallerones

67. TheWaterFallanalogueofUserStorywithinAgileMethodology

Y. UserStory

68. AsuiteofoneormoreFunctionalProcesses69. ThetermappliedinAgileMethodology,referringtothemannerinwhichaUser

shallexecuteaBusinessObjective.That is,howtheUsershall interactwiththeFunctionsbeingDevelopedorModified

70. Typically, a well-written User Story contains approximately five (5) TestScenarios;however,thisdependsuponmanyfactorsincludingthewritingstyle&experienceoftheUserStorycomposer

71. A poorly constructed User Story will contain greater than ten (10) TestScenarios. If thisoccurs, theUser should considerdecomposing theUser Storyintomultiplesmallerones

72. TheAgileanalogueofUseCasewithinWaterFallMethodology

www.testimation.com


Z. TestCase

73. ATestCaseiscollectionofDIT’stobeexecutedsimultaneously74. AnaverageTestCasecontainsanaveragenumberofDIT’sdefinedbytheUserin

theQualityAssuranceSection75. “Test Cases” refers to the maximum number of Test Cases that the Scope is

expectedtoyield76. “TestCase”referstomanualTestsonly;noautomatedTestingisapplicable

AA. TestCase&DefectComplexity

77. TestCaseComplexity1. Example: The average Complexity as configured by the User in the

QualityAssuranceArchitecturesection2. 10DIT’sperTestCaseconfiguredbyUser(DIT’s=DynamicInformation

Tests)3. 50DIT’sperTestCaseisthemaximumpermissiblevalueofthedial4. TestCaseComplexity=10/50=20%

78. DefectComplexity1. Refers to the complexity of its discovery, not the complexity of its

correction(fixing)bytheDevelopmentTeam2. It follows that the average complexity of a population of Test Cases is

directly proportional to the average complexity of the population ofDefectstheyraise

BB. TestScenario79. AsynonymforFunctionalProcess(typically)80. One(1)TestScenarioisequivalenttoone(1)FunctionalProcess(typically)81. A Use Case or User Story contains approximately five (5) Test Scenarios

(typically)82. Theconstitution&specificformofaTestScenarioissubjective&affectedbythe

experienceofthecomposer

www.testimation.com


8. ApplicatorFunctionalProcesses

A. FunctionalProcessList

AFunctionalProcessisatrainofFunctionPointsleadingtoarequiredactionoroutputviaapathway;typicallythistakestheformofUserWorkInstructions.TheFunctionalProcessesforthis Application Suite are shown belowlviii. Each Functional Process begins with a jump-onpointlix& completes with a jump-off pointlx. The jumping-on points are QuEstimate View,LEstimate View, Basic View, Advanced View, TEstimator2, TEstimator3, REstimator (StandAlone) & @Risk (Stand Alone); the jumping-off point is always Save to File. Hence, the totalnumberofFunctionalProcesseswithintheApplicatorSuite(thisproductsuite) isshowntobetwentyfive(25),asfollows:

1. TEstimator11. QuEstimateViewàpathways

1. SavetoFile2. AEstimatoràSavetoFile3. AEstimatoràREstimatoràSavetoFile4. AEstimatoràREstimatoràPEstimatoràSavetoFile

2. LEstimateViewàpathways1. SavetoFile2. AEstimatoràSavetoFile3. AEstimatoràREstimatoràSavetoFile4. AEstimatoràREstimatoràPEstimatoràSavetoFile

3. BasicViewàpathways1. SavetoFile2. AEstimatoràSavetoFile3. AEstimatoràREstimatoràSavetoFile4. AEstimatoràREstimatoràPEstimatoràSavetoFile

4. AdvancedViewàpathways1. SavetoFile2. AEstimatoràSavetoFile3. AEstimatoràREstimatoràSavetoFile4. AEstimatoràREstimatoràPEstimatoràSavetoFile

2. TEstimator2àpathways1. SavetoFile2. AEstimatoràSavetoFile3. AEstimatoràREstimatoràSavetoFile4. AEstimatoràREstimatoràPEstimatoràSavetoFile

3. TEstimator3àpathways1. SavetoFile2. REstimatoràSavetoFile3. REstimatoràPEstimatoràSavetoFile

4. REstimator(StandAlone)àSavetoFile5. @RiskàSavetoFile

lviiiTotalnumberofFunctionalProcesses=4+4+4+4+4+3+2=25lixAstartingFunctionPointlxAfinishingFunctionPoint

4Pathways

4Pathways

4Pathways

4Pathways

4Pathways

3Pathways

2Pathways

www.testimation.com


B. UseCase(UserStory)List

1. AUserexecutesaTEstimator1estimate2. AUserexecutesaTEstimator2estimate3. AUserexecutesaTEstimator3estimate4. AUserexecutesaStandAloneREstimatorestimate5. AUserexecutesaStandAlone@RiskVisualisationanalysis

• TotalnumberofUseCases(UserStories)=5

C. HowtoCalculate(orestimate)theConversionRatio

TheConversionRatiodenotestheaveragenumberofFunctionalProcessesassociatedwithUseCases,UserStories&TestScenarios.AlthoughUseCases,UserStories&TestScenarioshaveindustry definitions, no Universally accepted rigorously precise definition exists; Use Cases,User Stories & Test Scenarios are highly subjective & significant variation exists withincategories depending upon the composer. To overcome this practical impasse, theApplicatorProductSuitestandardisesontheuseofFunctionalProcessesinplaceofUseCases,UserStories&TestScenarios.ItachievesthisviatheConversionRatiosuchthattheUserassignsadefinitionofUseCases,UserStories&TestScenarios,intermsofthenumberofFunctionalProcesseachcategorypossesseslxi.ThedefaultvalueswithintheApplicatorSuiteare:

1. One(1)UseCasemapstofive(5)FunctionalProcesses(onaverage);thereforetheConversionRatioisfive(5)

2. One(1)UserStorymapstofive(5)FunctionalProcesses(onaverage);thereforetheConversionRatioisfive(5)

3. One(1)TestScenariomapstoone(1)FunctionalProcesses(onaverage);thereforetheConversionRatioisone(1)

4. Themathematicalrepresentationoftheapplicableformsareshownbelow

Inourcase:

𝐶𝑜𝑛𝑣𝑒𝑟𝑠𝑖𝑜𝑛 𝑅𝑎𝑡𝑖𝑜 =𝐹𝑢𝑛𝑐𝑡𝑖𝑜𝑛𝑎𝑙 𝑃𝑟𝑜𝑐𝑒𝑠𝑠𝑒𝑠

𝑈𝑠𝑒 𝐶𝑎𝑠𝑒𝑠 𝑜𝑟 𝑈𝑠𝑒𝑟 𝑆𝑡𝑜𝑟𝑖𝑒𝑠=255= 5

Or

𝐶𝑜𝑛𝑣𝑒𝑟𝑠𝑖𝑜𝑛 𝑅𝑎𝑡𝑖𝑜 =𝐹𝑢𝑛𝑐𝑡𝑖𝑜𝑛𝑎𝑙 𝑃𝑟𝑜𝑐𝑒𝑠𝑠𝑒𝑠

𝑇𝑒𝑠𝑡 𝑆𝑐𝑒𝑛𝑎𝑟𝑖𝑜𝑠=2525

= 1

lxiOnaverage,overapopulation

www.testimation.com


D. UserWorkFlows(FunctionalProcessMap)

Date post:	15-Jul-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

Testimation User Instructions€¦ · What does using our Product Suite mean for you? Users Uses 1....

Documents