| Date post: | 15-Jan-2016 |
| Category: |
Documents |
| View: | 216 times |
| Download: | 0 times |
Testing
• Note, first, testing is not a distinct phase of production but should be performed in parallel with each phase. Otherwise, for example, a long specification document might be developed over several months, only to find, during a separate test phase, it has errors.
• Types of testing: Execution-based testing & non-execution based testing.
What is the point of testing?
• The 90-90 rule. Programmers start saying they are 90% complete when in fact they are only half done (ie., There is actually another 90% left to do). Why do you think this is? For one thing, programmers have not calculated test costs into their estimates.
• We know the software contains errors. A well designed test finds errors.
• A test which finds no errors may not have been well-designed and is a waste of time.
• Each test set should target a single class of errors. A test that exercises many functions in a module is not a good test since it may be difficult to determine which function caused the observed error.
The cost of things
• Errors can occur at any phase of the software life-cycle: requirements gathering, specifications, high or low-level design work, coding, even in testing itself.
• Since each phase builds on the previous ones, which types of errors are most expensive?
Example: A requirements omission
• A user reports a problem which reveals an error in our requirements document. We should have been able to store all the student’s grades. What is the cost?
1. We need to change specifications, including the accompanying user manuals (training aids, etc) for this software. Cost= S.
2. We need to go back to our designs, adding a data structure to hold student grades and determine what operations need to be performed on this data structure. (Is it a link list, a table, an array, etc?) We need to change design documents. Cost = D.
3. We may need to implement a new class for this structure. Cost=C.4. We need to change our implementation, which impacts other features of
our code and documentation. Cost =I5. At each step 1-4 above, we need to re-test. Cost =T(S)+T(D)+T( C)+ T(I) 6. We need to package for rerelease, update all related files, etc. Cost=R7. Total cost= S+D+C+I+T+R
Relative cost of errors
• Relative cost to fix errors:
1. In requirements: $1
2. In specs: $3
3. In design: $4
4. In code: $10
5. In integration: $30
6. In maintenance: $200
Failure to find faults early costs money
• Studies indicate 60-70% of faults in large programs are in specification or design.
0
50
100
150
200
250
300
350
400
450
500
Phases:Requirements/Specs/Design/Code/Integrate/Maintenance
cost
A budget decision?
• You might think of testing as a budgetary decision. If no money (time) is allocated to a testing phase, then the end user will report the faults and your money (time) will be spent to repair them in the maintenance phase.
• How does this approach affect, for example, the reputation of your organization?
• Would this be a good policy to follow for human-rated software? (Software on which lives depend, like medical systems, life-support for a space station, and so on.)
• Does this approach save money in the long run, or cost more?• When should testing be done?
Metrics: What should we measure and why?
• First: WHY? For the purposes of generating budgets, bidding new products, justifying new hires, allocating staff to projects underway, estimating time to develop software, time to fix software, size and speed of software. Without numbers to justify decisions and calculations, all this is just guesswork. Bad guesswork means: product does not meet specification, product is badly designed and consequently hard (expensive, time-consuming) to maintain, product is delivered late, product is poorly tested, staff is not prepared to fix bugs in a timely fashion.
• So what should we measure? Since products developed by a company are often similar, features of current and past products might help in estimating features of future products.
Cost of requirements given that 300 hours of programmer time was spent
Cost of meeting requirements R for T fixed
-200
-100
0
100
200
300
400
500
0 1 2 5 10 15 20 25 30
Requirements
co
st
V&V
• Schach avoids terms “V&V” preferring to use the word “testing”.
• Verification is the testing done at the end of each phase to insure the phase was correctly implemented (eg., “Did we design what we set out to design?”)
• Validation is typically performed before delivery. It’s purpose is to insure the product built was the product the client asked for. (ie., “Does this product meet the original specifications?”)
Execution vs non-execution-based testing
• Sometimes called static and dynamic.• Running source code on a sample input is an
example of dynamic a testing mechanism.• Code walkthroughs or inspections are static
testing.• Compiling probably needs to be regarded as
static.• Typically, non-execution-based testing uncovers
as many faults as dynamic testing.
terminology
• A “fault” is injected by human error. A single mistake may cause one or many faults. A number of mistakes may cause a single fault.
• A sw “failure” is observed incorrect behavior due to a fault.
• The error is the amount by which the result deviates from correct performance.
• “defect” refers to fault, failure or error.
Quality
• High quality software is typically not what is hoped for… For most sw production just getting the code to meet specifications is all that is desired and more than is attained.
• Quality can be measured in terms of conformance to specs, but also in terms of portability, reuse, maintainability, clarity, and many other “ities”.
• A SQA group is assigned the task of insuring quality sw production.
• SQA may also develop standards, and monitoring or testing mechanisms.
• It is importance to have independence between SQA and development teams. Neither team should trump the other.
SQA• How do you test a specification or design document? It can’t be executed,
yet we know it would be a mistake to proceed to implement code if these documents contain errors.
• Formal document reviews typically reveal as many errors as execution-based tests.
• Quality: this doesn’t mean “excellence” when applied to software, it simply means the software does what it is supposed to do.
• The SQA group must ensure that the product (of each phase of development) is correct.
• SQA also applies to the process as well as the product. This might include development of sw standards and inspections to assure compliance with standards. SQA group and development teams should ideally be under different managers with neither able to overrule the other.
• Whether to deliver a faulty product on-time or deliver a corrected product late should be decided by a more senior manager. (Clearly either option could cause a client to lose confidence in this sw provider.)
Non execution based tests Reviews: Walkthroughs & inspections
• Staff other than the product author must be involved in the review. (At IBM, a review moderator must “sign off” on a review, indicating that corrections suggested by the reviewers have in fact been effected by a sometimes reluctant author.)
• Walkthrough: 4-6 people involved: one rep from the team producing the artifact, the team manager, a client (for specs walkthrough), a rep from the next team (design team, if this is a spec walkthrough) and a rep from SQA. The SQA rep chairs the walkthrough since this group has the most at stake. (Client or spec team reps may be inclined to approve the workflow despite errors.) It is not the task of the team to correct faults, only to find them. See bottom pg 153 in text for explanation.
• Inspections are more formal and lengthy than walkthroughs
Walkthrough:1
• Walkthrough: 4-6 individual participants including: 1 representative from the development of the document, the manager responsible for this phase of production, a client, and a representative from the team which must use this document in the next production phase.
• All members should be experienced senior personnel. Before the walkthrough each member reviews the document in question and draws up two lists: a list of items that are not understood, and a list of items believed to be incorrect.
• An SQA member should chair the walkthrough since SQA has the most at stake. Clients may believe overlooked errors will be found later, members of the authoring team want to move on to something else.
• During the walkthrough it is not the goal to design corrections, only to uncover faults. (Why?)
• Walkthrough should last 2 hours.
Walkthrough:2
• Walkthrough can be participant driven or directed (document-driven).
• Document-driven walkthroughs typically uncover more faults. Partly, this is because the walkthrough does not jump through the document from point to point to isolate inspectors’ queries. Instead, the document as a whole is reviewed. Partly also, in the process of leading the group through the document, the presenter (author) will detect faults. This later method is recommended by IEEE.
• The leader should not be the manager responsible for personnel evaluations. If this is the case the presenter will try to minimize faults found and other participants will try to score points rather than find faults.
Inspection
• Inspections have 5 steps:1. An overview of the document is presented by an author. The
document is distributed.2. In the preparation phase, participants review the document.
Lists of types of faults found in recent inspections, ranked by frequency, are useful aids.
3. In the inspection, one participant leads a walkthrough making sure each item is covered and each branch taken at least once. Fault finding occurs here. (Not fault correction.) Within 1 day the inspection moderator must produce a written report of the inspection.
4. A rework of the document is produced by the authoring team which addresses all items in the report.
5. In follow-up, the moderator must insure every issue raised in his report has been resolved.
Inspection: continued
• If more than 5% of the material in the original document required
rework, then the team must reconvene for a 100% inspection.• Inspection should have 4 members including a member of the
authoring team, a member of the team responsible for the next phase, a tester (who should also be a member of the SQA group).
• IEEE recommends 4-6 members with special roles played by the moderator, the (document) reader, and the recorder.
Faults
• Faults/module can be compared with similar modules
• If too many faults are found in several inspections, careful inspection of all code artifacts is needed as well as other management corrective action
• Too many faults in a given artifact make a strong case for redesign of that module.
• Information regarding numbers and types of faults found is useful in preparing for future inspections. Similarly, faults found in a design artifact should be reviewed before inspecting the resulting code artifact.
Value of Inspections • Inspections do find faults: studies show that over 2/3 of total development
faults may be found during inspection and that inspection (rather than walkthrough) may reduce post-release faults by 30-40%
• Inspections reduce programmer time in unit testing. One study found an overall saving of 25% resulting from thorough artifact inspection despite the programmer time devoted to the inspections.
• Other studies indicated percentages of 70-90% of total faults found were detected in inspections. Inspections generally decreased product fault detection by as much as 90%production cost –perhaps counter-intuitively.
• JPL found 4 major faults and 14 minor faults on average per 2-hour inspection at a cost SAVINGS of $25K per inspection.
• Note that faults found in inspection do not have to be repaired before inspection proceeds, unlike execution-based test faults.
• Walkthroughs are a fairly casual 2-step process, inspections are a rigorous 5-step process.
Limitations of reviews
• Large pieces of code can not be effectively reviewed in a walkthrough or inspection.
• OOP classes facilitate the inspection process.
Metrics
• Inspection rate= pages per hour of specs or design inspected
• Inspection rate= LOC per hour of code inspected• Fault density is faults per page or faults per KLOC, which
can be broken into major and minor faults.• fault detection rate is the number of major and minor
faults detected per person/hour.• Of course, variations in these numbers may be
accounted for by a variety of factors including inspection team efficiency, product complexity or “newness”, code quality.
On execution-based testing
• In execution-based testing faults must be inferred from product performance given known inputs and expected outputs.
• Testing is always limited (ie., we probably can’t check all paths).
• We’d like to determine product quality in terms of utility, reliability, robustness, performance and correctness (and maybe also reuse, portability)
• See examples pg 162-163 and anecdote about correctness.
• Generally the author is NOT the tester.• Text discusses program-proving, bounds checking (in
java but not in C/C++), use of asserts.
Execution-based testing
• Utility: Does it meet user’s needs? Is it easy to use? Is it cost effective? If sw fails these test there is no point in going on, no one will use it.
• Reliability: What is the mean time between product failures? How bad are the effects of failure? How long does it take to fix it? (If fixes are time consuming or expensive then even if mean time between failures is long, the product is unreliable.)
• Robustness: A robust product always performs according to specifications, and does not perform too badly when subjected to operating conditions outside the range of the specifications.
• Performance: What are response-time & spacial constraints on the software?
• Correctness: Product conforms to specifications. (Does not mean that it is acceptable, since specs may be in error.)
Execution-based testing: overview
• Programmer creates software, checking as she goes that it seems to be functioning correctly (using sample test data to check correctness ad robustness).
• Another individual performs systematic and thorough module testing.
Some kinds of tests
• Black box: User manual or specification document provides guidelines for testing. These indicate product interface, functionality and appropriate input/output performance. Systematically test features as indicated in user manual. (Also called data-driven, input/output-driven, behavioral and functional testing).
• White box: software is “visible”, ie., made available to the testers. Design tests which exercise loops and conditional structures and test object values/states. (Also called glass-box, logic-driven, path-oriented, structural testing).
Testing to specification/Testing to code
• Exhaustive testing to specs is not possible. Every possible data value can not be tested.
• In testing to code, each path should be executed once. This is also not feasible, even a simple flow diagram can have millions of paths.
• There are 5 paths through the central boxes so the total paths including looping are
5+52+ 53+…+ 518 = 4.77*1012
Loop up to 18 times
Testing to code may not find errors
• Consider the erroneous fragment for determining if 3 values are equal:
– Code performs correctly when these two cases are run.
– if((x+y+z)/3==x)cout<<“values equal”;else cout<<“values not equal”;• Test case 2: x=y=z=2 • Test case 1: x=1,y=2,z=3
• Consider two fragments for calculating a quotient. If programmer did not test for runtime error (divide-by-zero), tester may not find it either:
1. x=n/d;
2. if(d!=0)x=n/d;else throw DivideByZeroException;
Testing to code:Equivalence classes
• Suppose specifications for a product are that it handles from 1 to 32767 (that is, 215-1) records correctly. Processing 1, 506, 12000 or 32767 records will likely work (or not) with the same probability: These values are all taken from the same equivalence class. Other test equivalence classes are less than 1 record, more than 32767 records. Testing on or past a boundary typically has a higher probability of finding a fault. Tests for this product should include 0 records, 1 record, several tests of legal record numbers, 32766 records, 32767 records and 32768 records.
• Generating boundary output values for a given product should also be tested.
Glass box testing
• Identify each item of functionality in a module. (These might be operations like getNextRecord(), calcStudentCreditLoad() and so on.)
• Devise test cases to test each function separately. • In the case that a function conditionally calls other functions, these
latter might be tested individually and the higher level function then tested using branch coverage.
Testing to code: functional testing
• Consider the hierarchy chart below. Drivers and stubs can be used to test the functionality. Stubs for e,f,g,h,and i would allow testing of b,c,d. Stubs for b,c,d would allow testing a.
• A driver to replace a would allow testing of b,c,d. Drivers to replace b,c,d would allow i,h,g,e,f to be tested.
• These techniques correspond to bottom-up and top-down module testing. A combination of bottom-up and top-down can also be used. a
b dc
e f g h i
Testing to code: functional testing
• Sample driver for a double-valued function f which requires 3 double parameters:void f_driver(void){
double x,y,z;
cout<<“enter test parameters for f”;
cin>>x>>y>>z;
//call the actual function
cout<<f(x,y,z);}
• Sample stub for function f as above:double f_stub(double x,double y,double z){
cout<<“arrived in function f\n”;
cout<<“parameter values are”<<x<<y<<z;}
Sandwich testing• If modules a,b,c and d are
logic/control modules and e,f,g,h, &i are operational modules (performing I/o, calculations, actual work) then a,b,c,d would be tested top-down.
• If modules b,c,d,e,f,g,h,i are operational, then they would be tested bottom-up.
a
b dc
e f g h i
Testing to code: functional testing
• Problems: – Calling sequences between functions may not be as simple as a
conditional structure.
– functions themselves may overlap module boundaries (possibly due to bad module design – low cohesion) so that it is hard to isolate and test them.
glass-box testing
• Structural testing: statement, branch, path tests.• statement testing: Each statement should be executed at least
once. This technique fails to properly test conditionals and paths. Consider the code fragment below:
if(subscript>limit && found==0)
errorcode=2;
Test case: subscript=limit+1, found=0. Does not reveal programming error (&& should be replaced by ||)
An improvement is branch coverage. Each branch should be covered at least once. The most powerful structural testing is to test every branch at least once.
Number of paths to test• We looked at this flow chart earlier- and
discovered we can’t test all five paths for each possible number of loop iterations.
• One way of reducing the number of tests needed is to restrict testing to linear code sequences, that is, sequences without return, goto, and conditional structures (like ifs and loops).
• Another way is to test all paths between definition of a variable and each use. A definition is eg.,
• int x=5;• A use is eg.,• if(x>6)… or y=x*4;• Although the upper bound on paths is 2d
where d is the number of decision structures, in practice the number of tests is actually likely to be proportional to d, so that the number is not too large.
Loop up to 18 times
Which modules contain errors?
• Typically, 70% or more of sw errors are located in 30% or fewer modules.
• Can you name several reasons why this might be so?– Very difficult (complex) modules probably make up a small proportion of
the package. These are likely to contain control errors.
– Errors might be more likely to be found in the small proportion of modules written by novice programmers.
– As mentioned earlier, a high proportion (60-70%) of errors are related to requirements, specification and design flaws. Those modules implementing flawed features from these phases would contan faults.
Testing
• Note, first, testing is not a distinct phase of production but should be performed in parallel with each phase. Otherwise, for example, a long specification document might be developed over several months, only to find, during a separate test phase, it has errors.
• Types of testing: Execution-based testing & non-execution based testing.
Non execution based tests• How do you test a specification or design document? It can’t be executed,
yet we know it would be a mistake to proceed to implement code if these documents contain errors.
• Formal document reviews typically reveal as many errors as execution-based tests.
• Quality: this doesn’t mean “excellence” when applied to software, it simply means the software does what it is supposed to do.
• The SQA group must ensure that the product (of each phase of development) is correct.
• SQA also applies to the process as well as the product. This might include development of sw standards and inspections to assure compliance with standards. SQA group and development teams should ideally be under different managers with neither able to overrule the other.
• Whether to deliver a faulty product on-time or deliver a corrected product late should be decided by a more senior manager. (Clearly either option could cause a client to lose confidence in this sw provider.)
Reviews: Walkthroughs & inspections
• Staff other than the product author must be involved in the review. (At IBM, a review moderator must “sign off” on a review, indicating that corrections suggested by the reviewers have in fact been effected by a sometimes reluctant author.)
• Inspections are more formal and lengthy than walkthroughs
Walkthrough:1
• Walkthrough: 4-6 individual participants including: 1 representative from the development of the document, the manager responsible for this phase of production, a client, and a representative from the team which must use this document in the next production phase.
• All members should be experienced senior personnel. Before the walkthrough each member reviews the document in question and draws up two lists: a list of items that are not understood, and a list of items believed to be incorrect.
• An SQA member should chair the walkthrough since SQA has the most at stake. Clients may believe overlooked errors will be found later, members of the authoring team want to move on to something else.
• During the walkthrough it is not the goal to design corrections, only to uncover faults. (Why?)
• Walkthrough should last 2 hours.
Walkthrough:2
• Walkthrough can be participant driven or directed (document-driven).
• Document driven walkthroughs typically uncover more faults. Partly, this is because the walkthrough does not jump through the document from point to point to isolate inspectors’ queries. Instead, the document as a whole is reviewed. Partly also, in the process of leading the group through the document, the presenter will detect faults. This later method is recommended by IEEE.
• The leader should not be the manager responsible for personnel evaluations. If this is the case the presenter will try to minimize faults found and other participants will try to score points rather than find faults.
Inspection
• Inspections have 5 steps:1. An overview of the document is presented by an author. The
document is distributed.2. In the preparation phase, participants review the document.
Lists of types of faults found in recent inspections, ranked by frequency, are useful aids.
3. In the inspection, one participant leads a walkthrough making sure each item is covered and each branch taken at least once. Fault finding occurs here. (Not fault correction.) Within 1 day the inspection moderator must produce a written report of the inspection.
4. A rework of the document is produced by the authoring team which addresses all items in the report.
5. In follow-up, the moderator must insure every issue raised in his report has been resolved.
Inspection: continued
• If more than 5% of the material in the original document required
rework, then the team must reconvene for a 100% inspection.• Inspection should have 4 members including a member of the
authoring team, a member of the team responsible for the next phase, a tester (who should also be a member of the SQA group).
• IEEE recommends 4-6 members with special roles played by the moderator, the (document) reader, and the recorder.
Execution-based testing
• Utility: Does it meet user’s needs? Is it easy to use? Is it cost effective? If sw fails these test there is no point in going on, no one will use it.
• Reliability: What is the mean time between product failures? How bad are the effects of failure? How long does it take to fix it? (If fixes are time consuming or expensive then even if mean time between failures is long, the product is unreliable.)
• Robustness: A robust product always performs according to specifications, and does not perform too badly when subjected to operating conditions outside the range of the specifications.
• Performance: What are response-time & spacial constraints on the software?
• Correctness: Product conforms to specifications. (Does not mean that it is acceptable, since specs may be in error.)
Execution-based testing: overview
• Programmer creates software, checking as she goes that it seems to be functioning correctly (using sample test data to check correctness ad robustness).
• Another individual performs systematic and thorough module testing.
Some kinds of tests
• Black box: User manual or specification document provides guidelines for testing. These indicate product interface, functionality and appropriate input/output performance. Systematically test features as indicated in user manual. (Also called data-driven, input/output-driven, behavioral and functional testing).
• White box: software is “visible”, ie., made available to the testers. Design tests which exercise loops and conditional structures and test object values/states. (Also called glass-box, logic-driven, path-oriented, structural testing).
Testing to specification/Testing to code
• Exhaustive testing to specs is not possible. Every possible data value can not be tested.
• In testing to code, each path should be executed once. This is also not feasible, even a simple flow diagram can have millions of paths.
• There are 5 paths through the central boxes so the total paths including looping are
5+52+ 53+…+ 518 = 4.77*1012
Loop up to 18 times
Testing to code may not find errors
• Consider the erroneous fragment for determining if 3 values are equal:
– Code performs correctly when these two cases are run.
– if((x+y+z)/3==x)cout<<“values equal”;else cout<<“values not equal”;• Test case 2: x=y=z=2 • Test case 1: x=1,y=2,z=3
• Consider two fragments for calculating a quotient. If programmer did not test for runtime error (divide-by-zero), tester may not find it either:
1. x=n/d;
2. if(d!=0)x=n/d;else throw DivideByZeroException;
Testing to code:Equivalence classes
• Suppose specifications for a product are that it handles from 1 to 32767 (that is, 215-1) records correctly. Processing 1, 506, 12000 or 32767 records will likely work (or not) with the same probability: These values are all taken from the same equivalence class. Other test equivalence classes are less than 1 record, more than 32767 records. Testing on or past a boundary typically has a higher probability of finding a fault. Tests for this product should include 0 records, 1 record, several tests of legal record numbers, 32766 records, 32767 records and 32768 records.
• Generating boundary output values for a given product should also be tested.
Glass box testing
• Identify each item of functionality in a module. (These might be operations like getNextRecord(), calcStudentCreditLoad() and so on.)
• Devise test cases to test each function separately. • In the case that a function conditionally calls other functions, these
latter might be tested individually and the higher level function then tested using branch coverage.
Testing to code: functional testing
• Consider the hierarchy chart below. Drivers and stubs can be used to test the functionality. Stubs for e,f,g,h,and i would allow testing of b,c,d. Stubs for b,c,d would allow testing a.
• A driver to replace a would allow testing of b,c,d. Drivers to replace b,c,d would allow i,h,g,e,f to be tested.
• These techniques correspond to bottom-up and top-down module testing. A combination of bottom-up and top-down can also be used. a
b dc
e f g h i
Testing to code: functional testing
• Sample driver for a double-valued function f which requires 3 double parameters:void f_driver(void){
double x,y,z;
cout<<“enter test parameters for f”;
cin>>x>>y>>z;
//call the actual function
cout<<f(x,y,z);}
• Sample stub for function f as above:double f_stub(double x,double y,double z){
cout<<“arrived in function f\n”;
cout<<“parameter values are”<<x<<y<<z;}
Sandwich testing• If modules a,b,c and d are
logic/control modules and e,f,g,h, &i are operational modules (performing I/o, calculations, actual work) then a,b,c,d would be tested top-down.
• If modules b,c,d,e,f,g,h,i are operational, then they would be tested bottom-up.
a
b dc
e f g h i
Testing to code: functional testing
• Problems: – Calling sequences between functions may not be as simple as a
conditional structure.
– functions themselves may overlap module boundaries (possibly due to bad module design – low cohesion) so that it is hard to isolate and test them.
glass-box testing
• Structural testing: statement, branch, path tests.• statement testing: Each statement should be executed at least
once. This technique fails to properly test conditionals and paths. Consider the code fragment below:
if(subscript>limit && found==0)
errorcode=2;
Test case: subscript=limit+1, found=0. Does not reveal programming error (&& should be replaced by ||)
An improvement is branch coverage. Each branch should be covered at least once. The most powerful structural testing is to test every branch at least once.
Number of paths to test• We looked at this flow chart earlier- and
discovered we can’t test all five paths for each possible number of loop iterations.
• One way of reducing the number of tests needed is to restrict testing to linear code sequences, that is, sequences without return, goto, and conditional structures (like ifs and loops).
• Another way is to test all paths between definition of a variable and each use. A definition is eg.,
• int x=5;• A use is eg.,• if(x>6)… or y=x*4;• Although the upper bound on paths is 2d
where d is the number of decision structures, in practice the number of tests is actually likely to be proportional to d, so that the number is not too large.
Loop up to 18 times
When to rewrite a module rather than debugging it
• When an SQA member finds faults a module is returned to the programmer for repair. If many faults have been found in a module then it is likely there are (many) more and the module should be rewritten. This is counterintuitive, but consider: Suppose the same person tests two modules N and M for the same amount of time. She finds 2 errors in N, 40 errors in M. Which likely contains more?
Probability of faults is proportional to faults found
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 5 10 15 20
faults found
prob
abili
ty o
f mor
e fa
ults
Which modules contain errors?
• Typically, 70% or more of sw errors are located in 30% or fewer modules.
• Can you name several reasons why this might be so?– Very difficult (complex) modules probably make up a small proportion of
the package. These are likely to contain control errors.
– Errors might be more likely to be found in the small proportion of modules written by novice programmers.
– As mentioned earlier, a high proportion (60-70%) of errors are related to requirements, specification and design flaws. Those modules implementing flawed features from these phases would contan faults.
