The Effect of Highlighting Error Categories in FSM Training on the Accuracy of Measurement

ALI MERT ERTUGRUL*, GOKCEN YILMAZ*, MURAT SALMANOGLU*, ONUR DEMIRORS*

* Department of Information Systems, Middle East Technical University – Ankara / TURKEY

October 6-8, 2014

International Workshop on Software Measurement and International Conference on Software Process and Product Measurement – IWSM / MENSURA 2014Rotterdam, the NETHERLANDS

3

OUTLINE

• Introduction to FSM• Related Work on FSM accuracy• Motivation• Research Methodology and Implementation• Results & Discussion• Conclusion & Future Work

07/10/2014

4

Functional Size Measurement

• Functional Size Measurement (FSM) is a technique for measuring software in terms of the functionality it delivers.• IFPUG, MkII, COSMIC, etc.• FSM accuracy is crucial since based on the size of

the software, the cost and effort estimation are done.

07/10/2014

5

Variance in FSM

• Two main reasons:• Different assumptions and interpretations

• Misunderstanding of measurement standards• Solution: COSMIC Training

Turetken et al. 2008

Ungan et al. 2009

07/10/2014

6

FSM Training on FSM Accuracy

• Educational framework for the teaching and learning of software measurement topics at the undergraduate level.• Guidelines to teachers and instructors for

promoting the learning of software measurement topics.• Besides FSM, productivity, customer satisfaction,

etc.

Villavicencio et al. 2013

07/10/2014

7

FSM Training on FSM Accuracy

• Conducted two experiments:• Standard COSMIC training• Standard COSMIC training with practical cases

• The latter increased the reliability and accuracy of FSM more.

• Errors were not the prior concern in practical examples.

Ungan et al. 2010

07/10/2014

8

Definition of Error Categories

• Errors made during COSMIC FSM are categorized.

• Proposed tool detects the measurement errors automatically.

• Does not aim to increase FSM accuracy via training.

Yilmaz et al. 2013

07/10/2014

9

Motivation

• COSMIC FSM accuracy increases with training.

• Frequent error categories are defined.

• We argued, whether highlighting the error categories during FSM training increases the accuracy of FSM.

07/10/2014

10

Error Categories (EC) Name

EC1 Duplicate Functional Process (FP)

EC2 Lack of List FP before Update FP

EC3 Lack of List FP before Delete FP

EC4 Lack of Retrieve FP before Update FP

EC5 Lack of Data Movement (DM) type Write (W) in Add, Delete and Update FPs

EC6 Redundant DM type W in List FPs

EC7 Multiple occurrences of the same DM within the same FP

EC8 Each FP should be composed of at least 2 DMs

EC9 Each FP should contain at least 1 Write (W) / Exit (X) DM

EC10 Each FP should contain at least 1 Entry (E) DM

EC11 List FP might be included in Update/Delete FPs

EC12 Create/Delete/Update operations might be combined

EC13 Data Group (DG) Duplication

EC14 User interface components and System users are considered as DG / Object Of Interest (OOI)

EC15 OOIs are named wrong

07/10/2014

11

Data Acquisition

• We analyzed COSMIC FSM assignment data:• Obtained from two courses (SM502 and IS529)• Four consequent years (2010, 2011, 2012 and 2013)• Given by the same instructor for four years and two

courses• Similar MIS projects• 44 measurement results (10, 11, 11 and 12)

07/10/2014

12

Highlighting Error Categories

• Measurer teams were:• Unexperienced about COSMIC FSM before the given

courses.• Given COSMIC FSM training without being warned about

Error Categories in 2010 and 2011.• Given COSMIC FSM training in which Error Categories

were highlighted with detailed examples in 2012 and 2013.

07/10/2014

13

Expert Review Process

• Reference keys were prepared for each project.

• Two experienced measurers evaluated the results.

• An iterative review process was followed.

07/10/2014

14

Total and Average # of Errors 2010 – 2013EC

Total number of errors for years

Average number of errors for years

2010 2011 2012 2013 2010 2011 2012 2013EC1 60 18 19 14 6 1,6 1,8 1,2EC2 17 57 5 1 1,7 5,2 0,5 0,1EC3 23 69 6 5 2,3 6,3 0,5 0,4EC4 20 20 6 10 2 1,8 0,5 0,8EC5 26 11 3 6 2,6 1 0,3 0,5EC6 3 0 0 0 0,3 0 0 0EC7 19 81 4 6 1,9 7,4 0,4 0,5EC8 0 0 0 1 0 0 0 0,1EC9 19 2 1 0 1,9 0,2 0,1 0

EC10 7 4 2 2 0,7 0,4 0,2 0,2EC11 25 38 4 9 2,5 3,5 0,4 0,8EC12 3 1 0 7 0,3 0,1 0 0,6EC13 N/A N/A N/A N/A N/A N/A N/A N/AEC14 330 63 13 7 33 5,7 1,2 0,6EC15 66 33 7 10 6,6 3 0,6 0,8

07/10/2014

1507/10/2014

16

Results & Discussion

• Majority of the ECs has a significant decrease in total and average number of errors.• Related to List FP:• EC2: Lack of List FP before Update FP• EC3: Lack of List FP before Delete FP• EC11: List FP might be included in Update/Delete FPs

• Related to Naming:• EC14: User interface components and System users

are considered as DG / Object Of Interest (OOI)• EC15: OOIs are named wrong

• Also, EC5 and EC7

07/10/2014

17

Results & Discussion• Some of the ECs do not have a significant decrease in

total and average number of errors.• Core measurement rules:

• EC8: Each FP should be composed of at least 2 DMs• EC9: Each FP should contain at least 1 Write (W) / Exit

(X) DM• EC10: Each FP should contain at least 1 Entry (E) DM

• Naturally correct:• EC6: Redundant DM type W in List FPs• EC12: Create/Delete/Update operations might be

combined• Analysis on the errors does not contain EC13.

• Students were not responsible for EC13 during measurements

07/10/2014

18

Conclusion

• Frequent errors during COSMIC FSM were analyzed.• In the first two years COSMIC FSM training did not

include Error Categories.• Error categories were highlighted in detail during

COSMIC FSM training in the second two years.• Sharp decrease was observed in ECs in the second two

years.• Proposed study shows that highlighting error categories

in the course curriculum increases the FSM accuracy remarkably.

07/10/2014

19

Future Work

• Increase sample size• Apply enhanced statistical analysis (e.g. variance)• Combine with error prevention and identification

methods and tools• Categorize errors to be prevented via training or

tools

07/10/2014

20

References• O. Turetken, O. O. Top, B. Ozkan, and O. Demirors, “The Impact of Individual Assumptions on

Functional Size Measurement,” IWSM/Metrikon/Mensura, volume 5338 of Lecture Notes in Computer Science, pp. 155–169, 2008.

• E. Ungan, O. Demirörs, and Ö. Ö. Top, “An Experimental Study on the Reliability of COSMIC Measurement Results.”, IWSM '09 /Mensura '09 Proceedings of the International Conferences on Software Process and Product Measurement, pp. 321-336, 2009

• O. O. Top, O. Demirors, and B. Ozkan, “Reliability of COSMIC Functional Size Measurement Results: A Multiple Case Study on Industry Cases,” 35th Euromicro Conf. Softw. Eng. Adv. Appl., pp. 327–334, 2009.

• E. Ungan, Ö. Ö. Top, B. Özkan, and O. Demirörs, “Evaluation of Reliability Improvements for COSMIC Size Measurement Results,” IWSM/MetriKon/Mensura, 2010.

• The Common Software Measurement International Consortium (COSMIC): 2011 Guideline for Assuring the Accuracy of Measurements, Version 1.0.

• M. Villavicencio and A. Abran, “Towards the Development of a Framework for Education in Software Measurement,” 2013 Jt. Conf. 23rd Int. Work. Softw. Meas. 8th Int. Conf. Softw. Process Prod. Meas., pp. 113–119, Oct. 2013.

• G. Yilmaz, S. Tunalilar, and O. Demirors, “Towards the Development of a Defect Detection Tool for COSMIC Functional Size Measurement,” 2013 Jt. Conf. 23rd Int. Work. Softw. Meas. 8th Int. Conf. Softw. Process Prod. Meas., pp. 9–16, Oct. 2013.

07/10/2014

21

Thank You!

07/10/2014