Mistakes, Errors and Defects

04/21/23 Mistakes, Errors, Defects , Copyright M. Smith,

ECE, University of Calgary, Canada

2

Basic Concepts You are building a project. The project is made up of “phases”.

You complete a phase of the project and then move onto the next phase

In the Lab., some people would consider each “task” as a “phase”

Some people would call each part of a task a phase The key element is – you decide a “phase” for your project,

and then you decide whether you have completed that “phase” and can move onto the

next stage, or whether you have to keep working to make this phase work

04/21/23 Mistakes, Errors, Defects , Copyright M. Smith,

ECE, University of Calgary, Canada

3

Mistakes in a project with 3 tasksTask 1 – 100 LOC – Lines of Code 7 Task 1 mistakes are made (developer does not know this) BUT ONLY 3 Task 1 mistakes are discovered before moving onto Task 2

Task 2 -- 100 LOC – Lines of Code 5 Task 2 mistakes are made (developer does not know this) BUT ONLY 3 Task 2 mistakes (errors) are discovered before moving onto Task 3 1 mistake (defect) made in Task 1 is discovered and fixed

Task 3 -- 200 LOC – Lines of Code 2 Task 3 mistakes have been discovered and fixed – at this point in time in the project 1 mistake from Task 1 has been discovered and fixed – at this point in time in the project 1 mistake from Task 2 has been discovered and fixed – at this point in time in the project ? mistakes have been made ? Defects will be fixed?

PRODUCT IS RELEASED AFTER THE DEVELOPER “COMPLETES” T3HOW MANY PROBLEMS LEFT IN THE PROJECT

04/21/23 Mistakes, Errors, Defects , Copyright M. Smith,

ECE, University of Calgary, Canada

4

Questions to ask How many mistakes is it likely that the programmer

will make during Task 3? How many mistakes made during Task 3, will the

programmer find in Task 3 (errors) before deciding to releasing the project to the customer?

How likely is it that the programmer will, during Task 3, find any mistakes (defects) left behind from Task 2?

How many mistakes (defects) can be expected to be left behind after the product has been released?

04/21/23 Mistakes, Errors, Defects , Copyright M. Smith,

ECE, University of Calgary, Canada

5

Errors and Defect Mistake – anything you do wrong (including misreading the manual)

Error – are mistakes you make in “ this phase” of the project and FIND before moving on to the “next phase” of the project.

Defects – are mistakes you make in “ this phase” of the project and DON’T FIND before moving on to the “next phase” of the project

There is no mention of what constitutes an error or a defect – it is anything you do wrong! Includes misreading the manual

Errors and defects can both be syntax errors or logical errors. The difference is one of time -- “when (in the project) did you find the mistake”

04/21/23 Mistakes, Errors, Defects , Copyright M. Smith,

ECE, University of Calgary, Canada

6

How many errors and defects in this code?

Can’t answer the question, until you define “phase of a project” and also know the code-review habits of the programmer.

Humphreys suggests – code + assembler + link = 1 phrase

04/21/23 Mistakes, Errors, Defects , Copyright M. Smith,

ECE, University of Calgary, Canada

7

How many errors and defects in this code?

Editor is suggesting syntax errors in Po (key-word colour coding). Would be found by assembler

Formatting is suggesting an error in the loop as there should be 4 non-indented lines

There are in fact 8 other mistakes in the code. How many would be caught by assembler?

04/21/23 Mistakes, Errors, Defects , Copyright M. Smith,

ECE, University of Calgary, Canada

8

Better – after a code-review

There are still 2 mistakes in code. A warning will be issued by the assembler and will then

cause a fatal error with the linker A mistake that will not be caught by the assembler or

linker

04/21/23 Mistakes, Errors, Defects , Copyright M. Smith,

ECE, University of Calgary, Canada

9

Errors and defects recognizedWhen should project be released?Task 1 – 100 LOC – Lines of Code 7 Task 1 mistakes were made (we don’t know this) BUT ONLY 3 Task 1 mistakes (errors) are discovered before moving onto Task 2

Task 2 -- 100 LOC – Lines of Code 5 Task 2 mistakes were made (we don’t know this) BUT ONLY 3 Task 2 mistakes (errors) are discovered before moving onto Task 3 1 mistake (defect recovered) made in Task 1 is discovered and fixed

Task 3 -- 200 LOC – Lines of Code 2 Task 3 mistakes (errors) have been discovered and fixed – at this point in time 1 mistake from Task 1 (defect recovered) has been discovered and fixed – at this point in time 1 mistake from Task 2 (defect recovered) has been discovered and fixed – at this point in time ? mistakes have been made ? Defects will be fixed?

PRODUCT TO BE RELEASED AFTER THE DEVELOPER “COMPLETES” TASK 3!

04/21/23 Mistakes, Errors, Defects , Copyright M. Smith,

ECE, University of Calgary, Canada

10

Should the developer release the project? Can’t really tell with this small amount of

data – metrics should be gathered over many projects.

Personal Software Process

Will demonstrate the principles

04/21/23 Mistakes, Errors, Defects , Copyright M. Smith,

ECE, University of Calgary, Canada

11

Based on the available data Developer does not know whether a mistake has been made Developer only knows about errors recognized and defects recovered

Task 1 – 100 LOC – Lines of Code 3 Task 1 mistakes (errors) are discovered before moving onto Task 2

Task 2 -- 100 LOC – Lines of Code 3 Task 2 mistakes (errors) are discovered before moving onto Task 3 2 mistake (defect recovered) made in Task 1 is discovered and fixed 1 mistake (defect recovered) made in Task 2 is discovered and fixed

Based on this data – developer makes 9 mistakes per 200 lines of code developed – 45 / kLOC -- Actually making 60 / kLOC but doesn’t know that till later.

Task 3 (assuming same level of complexity as Task 2 and 3) has 200 LOC and there fore the developer is probably making 9 +- 2 mistakes

I reckon that I am having a good day if I can get below 80 kLOC, but that is because I am not spending enough time fixing my development process. What’s your real mistake rate

04/21/23 Mistakes, Errors, Defects , Copyright M. Smith,

ECE, University of Calgary, Canada

12

Mistake Recovery Efficiency -- MRE How to calculateMistake Recovery Efficiency Errors found for a = -------------------------------------given project phase Mistakes found (Errors + Defects)Can’t really be calculated for THIS project till the project is finished (Must have past history

available)

Task 1 3 mistakes are discovered before moving onto Task 2 3 ERRORS FOUND 1 defect found in Task 2 and another in Task 3 2 DEFECTS FOUND Mistake Recovery Efficiency = MRE = 3 / 5 = 60% recovery calculated

Actually (but we can’t measure that ) True MRE is 3 / 7 – lower than 60%

Task 2 3 mistakes are discovered before moving onto Task 3 3 ERRORS 1 mistake made in Task 2 is discovered and fixed 1 DEFECT RECOVERY Mistake Recovery Efficiency = MRE 3 / 4 = 75% recovery

Average MRE = (60 + 75) / 2 = 67.5% (2/3)

04/21/23 Mistakes, Errors, Defects , Copyright M. Smith,

ECE, University of Calgary, Canada

13

Task 3 Problems with final customer codeHow many? – based on MRETask 1 – 100 LOC 7 mistakes are made – expecting only 4.5 in this amount of code 3 mistakes are discovered before moving onto Task 2 3 ERRORS

Task 2 – 100 LOC 5 mistakes are made – expecting only 4.5 in this amount of code 3 mistakes are discovered before moving onto Task 3 3 ERRORS 1 mistake made in Task 1 is discovered and fixed 1 DEFECT RECOVERY

Task 3 – 200 LOC -- Based on LOC developed – 9 errors in 200 LOC 2 mistakes have been identified 2 ERRORS 1 mistake from Task 1 has been discovered 1 DEFECT RECOVERY 1 mistake from Task 2 has been discovered 1 DEFECT RECOVERY

PRODUCT RELEASE HOW MANY PROBLEMS?

This programmer averages MRE = 67.5% in each phase – therefore expect that of the 9 mistakes probably made in task 3 – 9 * 0.675 of those will be found (6.3). Has found 2 errors already

Can expect to find another 4 errors based on past performance3 defects left will probably be left in customer code from Task 3

04/21/23 Mistakes, Errors, Defects , Copyright M. Smith,

ECE, University of Calgary, Canada

14

Problems with final customer codeFrom Defects left over from T1 and T2Task 1 – 100 LOC Expect 4.5 mistakes to be made 3 mistakes are discovered before moving onto Task 2 3 ERRORS 2 defects recovered (Task 2 and 3) -- probably no more defects left

Task 2 – 100 LOC Expect 4.5 mistakes to be made 3 mistakes are discovered before moving onto Task 3 3 ERRORS 1 mistake made in Task 2 is discovered and fixed in T3 1 DEFECT RECOVERY

PRODUCT RELEASE HOW MANY PROBLEMS?

This programmer averages MRE = 67.5% in each phase – therefore expect that of the 9 mistakes probably made in task 1 and 2 – 9 * 0.675 of those will be found (6.3). Has found 9 errors already

Based on the statistics from this project – developer has found all problems in Task 1 and probably most in Task 2

04/21/23 Mistakes, Errors, Defects , Copyright M. Smith,

ECE, University of Calgary, Canada

15

If you are going to do this sort of metrics seriously The programmer needs to know their own MRE and Defect Recovery Rate.

Must collect “metrics” from past experiences The programmer needs to know need to know whether the tasks were of equal

difficulty A more difficult task can be expected to cause more than the average number of defects

Is this a safety critical task Might spend more time looking for possible defects (Perhaps write more tests (TLD))

Time estimation – how long will the product take to develop? How big is the product? Learn estimation skills What is your efficiency (lines of debugged code / hour)? Errors / 1000 lines of code? Defects / 1000 lines of code? How long does it take you to fix a “typical” error or defect? When fixing errors and defects – how many new mistakes do you introduce for each

one you fix? (Industry about 1 in 3) What are your most common mistakes? And what are you doing to fix them in time for

final exam and post-lab 3 quiz

04/21/23 Mistakes, Errors, Defects , Copyright M. Smith,

ECE, University of Calgary, Canada

16

Example on Errors / Defects / MistakesPossible Final Exam question

A student is doing 5 tasks – all roughly the same size and difficulty.You know that the students MRE = 60% and DRE = 20%

Task 1 10 errors discoveredTask 2 5 errors discovered 2 defects from Task 1 foundTask 3 10 errors discovered 1 defects from Task 1 + 2 defects from Task 2 foundTask 4 12 errors discovered 1 defects from Task 1 + 2 defects from Task 2 + 2 defects from Task 3 found Task 5 10 errors discovered 2 defects from Task 2 + 2 defects from Task 3 + 1 defect from Task 4

Explain both qualitatively (reasonability argument) and quantitatively (number argument) the following issues

1) The programmer moved onto Task 3 from Task 2 at the correct time – true or false2) The programmer should stop working on Task 5 and release the product3) Why is impossible in Task 5 to discover a defect from Task 5

This programmer has been using test first development (test driven development). 4) Explain why you might think that this student does not know how to write TDD tests (some people are good at testing and some are

not)

5) Looking back at your previous classes – make an estimate of your MRE and DER rates6) Make a comment on what you feel is your MRE and DER rates now7) Has TDD helped you or hindered you?8) Some people feel that writing tests before coding takes longer that writing tests after the coding. In theory – since you need the same

tests – this is not true. But in practice it often is true – explain the discrepancy9) What is the difference between a unit test and a system test? Using one of your labs as an example give details of a unit test and a

system test?

04/21/23 Mistakes, Errors, Defects , Copyright M. Smith,

ECE, University of Calgary, Canada

17

Some more practice on loops and arrays.Indexing through arrays array(count) = 2;

How many errors and defects if you define errors as being “found before assembler is used”

04/21/23 Mistakes, Errors, Defects , Copyright M. Smith,

ECE, University of Calgary, Canada

18

04/21/23 Mistakes, Errors, Defects , Copyright M. Smith,

ECE, University of Calgary, Canada

19

04/21/23 Mistakes, Errors, Defects , Copyright M. Smith,

ECE, University of Calgary, Canada

20

Array(count) = count

04/21/23 Mistakes, Errors, Defects , Copyright M. Smith,

ECE, University of Calgary, Canada

21

Hardware loop – 11 syntax errors

04/21/23 Mistakes, Errors, Defects , Copyright M. Smith,

ECE, University of Calgary, Canada

22

What defects are present?

04/21/23 Mistakes, Errors, Defects , Copyright M. Smith,

ECE, University of Calgary, Canada

23

Getting ready to go “really fast”

04/21/23 Mistakes, Errors, Defects , Copyright M. Smith,

ECE, University of Calgary, Canada

24

04/21/23 Mistakes, Errors, Defects , Copyright M. Smith,

ECE, University of Calgary, Canada

25

Blackfin in “full parallel operation”