RESEARCH AND ESTABLISHMENT OF QUALITY COST ORIENTED

SOFTWARE TESTING MODEL

YUYU YUANMicroprocessor R&D Center,

Computer Architecture Group ofComputer Science Department,PKUING UNIVERSITY, Beijing

100871, P.R. Chinayuanyuyu@mprc.pku.edu.cn

AbstractIn a competitive market, quality is essential to the software.

Software test plays an important role during the software-developing period Any saving on software test will greatlyreduce the total cost ofsoftware. The key point ofthis paper isto build a software test process with the cost controlmanagement and to make tradeoff between the quality and thecost.

Typical software testing model are analyzed and compared.A novel QC 3-D test model is introduced. The model combinedwith software test process, software quality cost and testingtype. The computational formula is defined to calculatesoftware quality cost. A software quality balance law isintroduced to balance control cost and failure cost. Byadjusting software test method and strategy, organization canreach the objective of the best balance between softwarequality and cost. An enterprise case is studied and the result isanalyzed to prove the basic law. The result shows an optimisticpoint that will prove the accurate ofthe model and the law.

Keywords. Quality Cost, Software Testing Model, SoftwareTesting Process, Testing Type

1. Introduction

Glenford Myers wrote in his classical work the Art ofSoftware Testing: "Software testing refers to the running of aprogram or a system to find the errors of software." [6] In 1983,Bill Hetzel wrote: "Testing refers to any kinds of activities aimat evaluating the properties of one program or system. Testingis a process to measure the quality of the software. "Myers'and Hetzel's definitions are still valid till now.

However, these definitions have limited scope. In theSystematic Software Testing written by Rick Craig and StefanJaskiel, testing has following definition: "Testing refers aconcurrency life circle process of engineering design,utilization and maintenance of testing software with an aim tomeasure and improve the quality of testing software." [5] Suchtesting is called "preventative test". .

Fundamentally, the quality of testing process determines thequality and effectiveness of software testing. Software testing

1-4244-0038-4 2006 24IEEE CCECE/CCGEI, Ottawa, May 2006

SHEN GUCollege ofSoftware Engineering,Graduate School ofthe ChineseAcademy ofSciences, Beijing

100049, P. R. ChinaShen.GU(gemplus.comShen_GU@hotmail.com

has its own life circle; each phase of the development progressis accompanied with an important testing process.

In order to cooperate the testing work well with the lifecircle of the software development, reasonable division of eachtesting phases, formulating reasonable criteria and followingthe criteria are important. The life circle models are introducedto address the problem. The models are called software testmodel. Currently, typical software test models are V model, Xmodel and proactive testing model.

With the continuous improvement of software quality,people began to pay special attention to the cost that spent toget high quality. Current software test models emphasize onlyon phase and process, the cost in software test process areneglected.

1.1. Definition of Quality Cost

The typical quality cost structure established by Feigenbaumprovides an ideology and is widely applied. Cost related tosoftware product quality is divided into two categories:Control Cost and Failure Cost.

1.1.1 Control cost. Control cost is expenditure spent to preventand find software bugs, and to reduce bugs to an acceptablelevel. Control cost is a kind of fixed cost and can be sub-divided into preventative cost, evaluation cost, andmanagement and control cost:

Preventative cost includes investment to infrastructure andsoftware quality. Preventative cost doesn't target at specialitem or system, and is taken as the general cost.

Evaluation cost includes the cost of activities to check theitems or bugs of software system.Management & control cost includes the cost of activities

for management of project plan and supervision of itsimplementation.

1.1.2 Failure cost. Failure cost refers to cost to repair thefailures of software. It belongs to variable cost. Function costcan be sub-divided into internal malfunction cost and externalmalfunction cost.

.10

Internal failure cost includes the cost for correction of bugsfound during design evaluation, software testing and productcheck and acceptance (completed by customers) before thedelivery of products.

External failure cost includes cost for correction of bugsfound by customers or maintenance personnel after delivery ofthe products.

1.2. Distribution of Quality Cost inSoftware Test Model

In respect of quality cost analysis, cost can be quantizedwith workload consumed. When analyzing the quality ofsoftware test model, it is helpful to understand the relationshipsbetween software test activities and quality cost.

Since preventive cost is not aiming at special project orsystem, each phase in the software test process will notconsume the cost, and the consumption of preventive cost shallnot be studied in the software test model. Appraisal cost isconsumed on check software bugs, and the related main testactivities are centralized in test case design, test deploymentand evaluation. Appraisal cost is a key cost type for softwaretest model study. The test activities of management and controlcost mainly include the processes for developing test planstrategy and management monitoring test. The internal failurecost is the expense created in correcting the bugs found indesign evaluation, software test and acceptance test before theofficial product delivery. Depending on different test activities,it mainly includes failure mode analysis, bug repairing test,regression test and test library updating as well as the final testevaluation. External failure cost refers to the expense for repairthe bugs detected by customers or maintenance personnel afterproduct delivery. External failure cost in typical test processincludes building reparation environment, failure modeanalysis, bug reparation and regression test and updating testlibrary.

1.3. Relationship between Control Costand Software Quality

In Figure 1-1, X-axis shows that software quality is fromlow to high while Y-axis shows the trend that control cost isfrom low to high.

Control cost consists of appraisal cost and management cost.It is the cost spent in preventing and finding out software bugsand minimized them to the acceptable level to ensure softwarequality. Control cost is a predictable, visible quality cost anddoes not vary with the change of test process. Figure 1-1 showsthat, when the cost of preventing and finding out bugs is less,the bugs found are less and the software quality is poorer.

With the increase of control cost and the cost input forpreventing and finding out bugs, the bugs found out insoftware test rises and the software quality also risescorrespondingly. As we know, the test is an endless process.The software quality can never reach an ideal status no matterhow much the control cost is. So the control cost investment

I

hCt

op

_-- -t

H'hSoftvaeQt

Low

Figure 1-1 Relationship between Control Cost and SoftwareQuality

for improving software quality is an important issue in qualitycost control.

1.4. Relationship between Failure Cost andSoftware Quality

In Figure 1-2, X-axis shows that software quality is fromlow to high while Y-axis shows that the failure cost is fromlow to high.

Failure cost includes the bug reparation and regression testcost due to failure of prevention or finding out software bugs.Un-detected bugs are not predictable, so the failures cost areunpredictable and invisible quality cost, which varies with thechange of test process. It is difficult for us to forecast andcontrol the consumption of failure cost.

Figure 1-2 shows that the failure cost of the test for repairingbugs is high when software quality is low and lots ofbugs exist.With the improvement of the quality of software and thereduction of bugs, the consumed failure cost also reducedcorrespondingly. But the number of bugs can never reach zero,and the software quality cannot achieve the best. So the failurecost can only tend to zero infinitely when the software qualitylevel rise to the infinite.

1.5. Balance Relationship of Control Costand Failure Cost

Through the analysis to control cost and failure cost, we seethat the control cost cannot be infinitely increased and thefailure cost is impossible to be reduced to zero. In actualprojects, cost expenditure is restricted by various conditions,such as market situation and customer contracts etc. Within therestriction of the cost, organization has to fully utilize theresource and optimize the software quality.Most measures adopted for analyzing quality and cost data

are based on the application of the software quality and costbalance concepts. Figure 1-3 illustrates the quality cost balanceconcepts and the relations between control cost and

2411

Figure 1-2 Relationship between Failure Cost and SoftwareQuality in Test

Figure 1-3 Software Quality Balance in Test Model

failure cost on all quality levelsAccording to the concept, the increase of predicted control

cost will result in software bug number decrease, softwarequality increase and corresponding failure cost decrease andvice versa. Based on Figure 1-3, software test total cost will bea fluctuating curve with the variance of control cost and failurecost. When control cost and failure cost tends to the same, theminimum of the total cost curve is reached, that is, theoptimum software quality level is reached at the minimumsoftware test cost.The variance effect of control cost may vary with the

predicted software quality level. (The blue lines in Figure 1-3).The increase of expected software quality will result in thecorresponding increase of control cost and failure cost and thetotal software cost. But the variance trend of control cost andfailure cost are stable. When both tend to be the same, the totaltest cost achieves the minima and the product reaches theoptimum software quality level.

Project manager normally pay more attention to theminimum test cost, but not detailed analyze the balancerelations between the project's control cost and failure cost.The basic ways are to balance the relations of both, to find out

the factors for inconsistent relations and to work outcorresponding measures based on inconsistency factors.

2. Study on Typical Software Test Model

Currently typical software test models include V Model, XModel and Pre-Model. Aiming at the application model ofquality cost in the software test model, the thesis emphasizeson the quality and cost of current typical software test models.

2.1. Introduction of Models

V Model [1] was firstly raised by the deceased Paul Rook inlate 1 980s and published in the literatures of UK StateComputing Center, aiming at the improvement of theefficiency and effect of software development.

The description of left side of X Model [2] is aiming at theseparated coding and testing for single progress segment, andthen frequent communication shall be done and finally it willbe combined into executable programs through integration(upper part of the right side). The finished products qualifiedintegrated test can be sealed and delivered to the user or be one

part of an integration with larger scale and scope.

Proactive testing model [3] is a model closely combining testand development. Pre-test absorbs advantages ofV Model andX Model and tries to correct their disadvantages. Pre-testmodel integrates the lifecycles of development and test,identifies the key activities from the start to the end of project'slifecycle and shows the value of these activities in the projectlifecycle.

2.2. Feature Comparisons of TypicalSoftware Test Model

We made comparisons to the results of analysis and study ofthree typical test models mentioned above. Throughcomparison, we find out that the test quality of V Model is ofhigh risk due to the adoption of waterfall development and testflow, the quality of X Model is increased due to a partlyimprovement of the bug of V Model, and the Pre-teststrengthens the test of each delivered product in the test phaseand has tests from different aspects of the products and theproduct quality is greatly improved.

The three models have not clearly defined the measuringmechanism, quality cost estimation, quantification andmeasuring mechanism.

Evaluation mechanism will check the projectimplementation status during test. Official evaluationmechanism may effectively control and manage quality costand software product's quality. However, there is not cleardefinition to the quality cost and software quality in threemodels.

2.3. Problem Analysis

According to the feature evaluation and comparison, we findthat current software test models cannot offer a powerful

2412

Failai Cost

n1I

HighSoflwam Qait

SoflNaale Quality

OpirmiuS ofvare Qu*k

......... . .........

ThtalQouSiC ost

solution to following problems and are unable to provide acomplete solution to the current software test situation andproblems.

- Lack of the mechanism for the estimation, measuring,control and analysis of software quality.

- Lack of a measuring mechanism to quality.- Lack of the evaluation activities for monitoring cost and

quality.- To test the process system with the software from test-

driven development and close combination of test anddevelopment.

3. Quality Cost 3D Model

By analyzing current typical software test models, we foundthat current software test models lack the control capability toestimate and measure the quality cost. To solve the problems, anew software test model will be introduced.

3.1. Principles

3.1.1 Define the activities in the software test model basedon quality cost type.

Quality cost is divided into control cost and failure cost.During software test, control cost and failure cost need to bemeasured and quantified respectively for balancing therelations between them and achieving the optimum of costquality. According to this principle, control cost and failurecost are respectively defined.

3.1.2 Closely integrate the quality cost and software testprocesses.

Combination the application of quality cost in software testmodel will: first, clearly define the test activities of quality costconsumption in each phase of the software test process; second,provide the quantified calculation of each phase in the softwaretest process; thirdly, achieve the targets of measuring,supervising and controlling the execution of software test.

3.1.3 Build the measuring mechanism to software quality inthe process of software test.

Cost consumption is built on the basis of quality rating. First,the measuring index to the quality of software shall be defined.Second, the detailed indexes needed to be measured in eachphase of the software test process shall be clarified in the testplanning process. Finally, the effectiveness of test shall bemeasured and verified in the deployment of the software andthe consequent test shall be adjusted based on the measuredresults.

3.2. Quality Cost 3D Model SystemStructure

Quality Cost 3D Model (QC 3D Model, see Figure 3-1 QC3D Model System Structure) is a 3-dimensional modelcombining quality cost and software test process. All

components of the model can be trimmed, altered and extendedto adapt to each kind of specific test.

Dimension X indicates software test phase consisting of fiveactivities (general plan, plan, analysis design, measuring andregression) and three milestones (plan evaluation, designevaluation and test evaluation).

Dimension Y indicates software quality cost that is put intocorresponding quality cost sub-category based on the activitiesof different test phases.

Dimension Z is divided into different test ratings based onunit test, integration test, system test and acceptance test.

4. Balance software quality and quality cost

4.1. Control cost activity and failure costactivity in QC 3D model

QC 3D model defines relative activities of control cost andfailure cost according to the type of quality cost. Control costactivities include general testing strategy, plan and evaluation,plan, analysis and design, technical examination, andmeasurement. Regression, testing evaluation and maintenanceare failure cost activities. (see Table 4-1 Quality cost activitiesin QC 3D model)

4.2. Computation of testing cost

4.2.1 Computation of control cost. QC 3D model includessix control cost activities, and they are general testing strategy,plan evaluation, plan, analysis and design, technical evaluationand measurement.QC 3D model may define different testing grades, and each

testing grade includes four control cost activities of plan,analysis and design, plan evaluation, and measurement.

General testing strategy cost and plan evaluation cost haveno relation to testing grade, and are set to Al and A2. Thenumber of testing phases is set to N, the number of testinggrades is set to K, and the control cost in each testing phase ateach testing grade is set to Xi, k. The computational formula ofcontrol cost is: n,k

Ccontrol -Al+ A2 + Xi, ki=3,k=l

4.2.2 Computation of failure cost QC 3D model has threefailure cost activities of regression, testing evaluation andmaintenance. QC 3D model may define different testing grades,each of which includes two failure cost activities of regressionand testing evaluation.

Maintenance has no relation to testing grade, and is set to B3.The number of testing phases is set to N, the number of testinggrades is set to K, and the failure cost in each testing phase ateach testing grade is set to Yi, k. The computational formula offailure cost is: m,k

Cfailue = Yj,k + B3j= ,k=1

4.2.3 Computation of total testing Co$kCcos t = Ccontrol + Cfaiiue = Ai + A2 + Xi,k +

i=3,k=l

m,k

XYj,k+B3j=l ,k=l

2413

4.3. Balancing control cost and failure cost

The control cost of a project's quality cost is relatively fixedand easily evaluated and controlled, while the failure costchanges. At initial phase of the project, it is very difficult for usto predict how many bugs will be found during testing and howmany regression testing need to be executed. Estimation offailure cost is a difficult point in predicting quality cost, andthis difficulty can be overcome by software quality balancingprinciple of the testing model. According to software qualitybalancing principle of testing model analyzed in Chapter 3,minimum quality cost can bring best software quality whencontrol cost gets closely equal to failure cost. (see Figure 4-1 )On the one hand, quality cost is limited. At this premise,

best software quality grade is expected, and control cost ismade equal to failure cost. According to quality balancingprinciple of software testing model, Ccontrol CfailueC ost /2 can be concluded, and then specific quantitativecontrol cost and failure cost can be obtained. In accordancewith the definition of control cost activity of QC 3D model,general testing strategy, testing grade, plan, analysis and design,and measurement activities can be arranged and coordinatedfrom top to bottom at the premise of limited control cost. In thesame way, in accordance with the definition of failure costactivity of QC 3D model, regression testing and maintenanceactivities can be arranged and coordinated from top to bottomat the premise of limited failure cost.On the other hand, software quality grade is limited. At this

premise and according to the requirement of software qualitygrade, control cost activities, including general testing strategy,testing plan, testing grade, analysis and design, andmeasurement, are planned. Control cost can be predictedthrough plan and computation of control cost activities that arerelatively fixed in QC 3D model. According to qualitybalancing principle of software testing model, expectedsoftware quality grade is achieved. Minimum software qualitycost Ccost 2X Ccontrol , and Cfailue Ccontrol . Hence,prediction and control purposes of failure cost and totalsoftware testing cost are realized.

Quality cost needs to be quantified at first during testing. Atthe same time, testing activities need to be adjusted at any timein order to balance control cost and failure cost.

Ccontrol Cfailue: If control cost is equal to or close tofailure cost, cost management and control is in very goodcondition.

Ccontrol > Cfailue: If control cost is more than failure cost,too much cost is consumed in plan, analysis and design, andmeasurement phases. Although high-quality software productis procured through testing, performance-price ratio of qualitycost is high, and optimal value and quality cost is not achieved.This will not cause subsequent problems of software product,and only wastes cost in an extent. Therefore, activities of plan,analysis and design, and measurement should be reduced todecrease control cost.

Ccontrol < Cfailue: If control cost is less than failure cost,

Figure 3-1 QC 3D Model System Structure

Table 4-1 Quality cost activities in QC 3D modelTesting Testing activitiesphase of QC

_________3D model ___________Control cost General Making general testingactivities testing strategy and plan_____________strategy __________________

plan Making detailed testing___________ ~~~strategy and plan

Plan Plan evaluationevaluationAnalysis and Analyzing and evaluating

__________design demandAnalyzing and designing

~~testing ware

Technical Technical evaluationevaluation

___________Measurement First-time testing executionManaging and monitoringimplementation of testing plan

____________ and testing strategyFailure cost Regression Failure mode analysisactivities ___I_I______I __I

Regression Repair and regression testingTable_4-1_QuaRegression Updating testing library

Testing Testing evaluationevaluation

___________Maintenance Creating repair environment____________ 1Maintenance Failure mode analysis|_Controlcost Maintenance Repair and regression testing|________ pl_Maintenance Updating testing library

initial investment during software testing is not enough, so toomany bugs are found in the later phase of software testing. Thefailure cost that is used to repair one bug in the later phase ofsoftware testing will increase in times, so the failure costincreases in times, and software quality cannot be assured. Thisis the typical problem occurred in software testing at thepresent time. Because cost control is out of regulation,software quality and other concerning problems occur.Accordingly, cost investment in plan, analysis and design, and

2414

4-1 Relation between software cost and softwarein software testing model

measurement activities need to be increased to prevent thequality problem extends to later phase.

5. Case study ofQC 3D Model

Here we present a real case to prove the theory of balancesoftware quality and quality cost inside QC 3-D model. First itintroduces the project background, analyzes results of case

implementation and cuts out relative activities based on

definition of QC 3D model of quality cost activities. Itcalculates the control cost and failure cost with the calculationformulas of the QC 3D model. It analyzes data of control costand failure cost and validates the analysis result of cost datawith case implementation result.

5.1. Project background

The case presents a tool to develop and test S@T andMiniS@T card services. The main functions include deckinstallation, deck uninstall, managing Attachment Point andSTK Menu Entry, transferring XML files into card-readableByte Code and testing of local and remote services of the card,etc.

The difficulty of the project is lack of professionalknowledge related to technical information of S@T andMiniS@T.

5.2. Evaluation of project implementation

In the project implementation, a great deal of time isspending on analyzing and evaluating requirement of thecustomer. As a result the project team has better understandingof the project requirement and made adequate estimation torisks of the project. In the whole process of software testing,there is no obvious deviation found comparing with thebaseline. And the product is delivered on time. The bug ratio ison a steady trend after delivery. The product does not costmuch external failure cost. It has relative high customersatisfactory degree.

Total F ire Cobs Total QualiCst

Figure 5-1 Case 3 Software quality-cost relationships

5.3. Cost data analysis

Control cost calculationn,kCcontrol=Al+A2+ 7Xi,k 77.5 m/dFailure costnoalculation3,k=1Cfailue = Y,,k+B3= 74m/d

Control cost,4wid failure cost analysis:The calculation of the control cost and failure cost of the

project shows that the failure cost is close to the control cost.Such a fact indicates that the cost management and control issuccessfully carried out. The quality cost is close to the bestvalue.

6. Conclusions

With the real case to prove the basic law of balance betweencontrol cost and failure cost, and established QC 3-D model toreflect their relation and to adjust the inter-dependent factorsaiming to achieve the optimistic point.

References[1 ]Robin F. Goldsmith and Dorothy Graham, "The Forgotten

Phase[R]. WWW.SDMAGAZINE.COM," Articles, 2002,August

[2]Robin F. Goldsmith. "This or That, VorX?[R].WWW.SDMAGAZINE.COM", Articles, 2002, August.

[3] Robin F. Goldsmith. Proactive Testing[R].WWW.SDMAGAZINE.COM, Articles, 2002, August.

[4] ((Capability Maturity Model Integration (CMMISM),Version 1.1))

[5] Rick D.Craig Stefan P.Jaskiel. Systematic Software Testing,2003.10

[6] Glenford J.Myers. The Art ofSoftware Testing[7] Capers Jones. "Software Assessments, Benchmarks, and

Best Practices ", 2003.4[8] Rex Black. "Managing the Testing Process ", 2003.10[9] David Astels, Ron Jeffries. "Systematic Software Testing",

2003

2415

T^lControl C 6st

SdbwraeQ*Ltoptiunum 3ce are Qualitr