20
2011 The Cost of Quality: A Reason for Built‐In Quality QUEST Boston Wednesday, April 6 th , 2011 11:00 AM – 12:00 PM PRESENTER: Elaine Soat COMPANY: Cartegraph Systems
2011
The Cost of Quality: A Reason for Built‐In Quality
QUEST Boston
Wednesday, April 6th, 2011
11:00 AM – 12:00 PM
PRESENTER:
Elaine Soat
COMPANY:
Cartegraph Systems
This was page intentionally left blank
1
The Cost of Quality: Q yA Reason for Built-in
QualityQuest April 6, 2011Q p ,
Elaine Soat, CSQA, CSTECartegraph Systems
Dubuque, Iowa
Agenda• Past• Present• SDLC
Duration: 40 to 45 minutes
• SDLC• Testing• Future• Summary
Di i
Questions and Discussion Afterwards
The Cost of Quality: A Reason for Built-in Quality
• Discussion
01/04/2011
2
2
The Past: Some Background
Projects 1994 1996 1998 2000 2002 2004 2009Succeeded 16% 27% 26% 28% 34% 29% 32%Failed 31% 40% 28% 23% 15% 18% 24%
Challenged 53% 33% 46% 49% 51% 53% 44%
The lowest failure rating = 15% In 2002
The highest success rating = 34% In 2002
Check out 2009 ratings. We succeeded 32% and
The Cost of Quality: A Reason for Built-in Quality
01/04/2011
3
Resource:Software Project Failure Costs Billions. Retrieved October 11, 2010, from http://www.galorath.com/wp/software-project-failure-costs-billions-better-estimation-planning-can-help.php
gfailed 24%!
Why Do Failures Happen in Software Development?
Developer did not write the code correctly
Developer misunderstood what the customer wanted
Customer realized later that they misspoke the requirement
Customer spoke it properly, but realized they asked for the wrong thing once they saw what was delivered
01/04/2011The Cost of Quality: A Reason for Built-in Quality
4
Customer realized they got what they originally wanted, but they now have a better idea
3
What Can We Do About Failures?
Tell Developers to not write bugsImplore customers to be clearerFind better customersFind better customersHave people talk more clearly so we don’t have miscommunications!
Work HARDER!Work SMARTER!
This ignores the fact that most people are alreadyThis ignores the fact that most people are already working as hard and as smart as they can.
What are the real reasons for failure!!!!!!!!!!!!!!!!
01/04/2011The Cost of Quality: A Reason for Built-in Quality
5
We can miss the target for various reasons
Requirements FailureProcess Failure
There are Multiple Reasons for Failure
Process FailureDesign FailureCode FailureTesting FailureDeadlines Failure
What is the one constant above?
The Cost of Quality: A Reason for Built-in Quality
ALL DEFECTS!
Customers don’t care where it failed, it just FAILED
01/04/2011
6
4
Defects should NOT be Acceptable!Reasons why defects are accepted:
• Past performance• Reputation of the industry• External blame - “Not My Fault”
Consider this: Would you hire a building contractor knowing 1 out of 7 houses they build collapses?
Two groups of people expect to be wrong:
The Cost of Quality: A Reason for Built-in Quality
Two groups of people expect to be wrong:Weatherman and Developers
We Can Change That!
01/04/2011
7
What is the Cost of Defects
Phase in Which Fixed Relative Unit Cost (translates into hours)
Requirements 1Requirements 1Design 3-6Coding 10Development Testing 15-40Acceptance Testing 30-70Production – Released to Client 40-1000Production – Released to Client 40-1000
Reference:Guidelines for Successful Acquisition and Management of Software-Intensive Systems. RetrievedSeptember 19, 2010, from http://www.stsc.hill.af.mil/resources/tech_docs/gsam4/chap4.pdf
5
Let’s Do the Math for the Cost of a Defect
Translate the numbers into money
Each unit is equal to1 hourEach unit is equal to1 hour
Base pay of $50/hour
The Cost of Quality: A Reason for Built-in Quality
01/04/2011
9
The Cost of a Defect In DollarsPhase in Which Fixed
RelativeUnit
(Translated into Hours)
Cost($50/hour)
Requirements 1 $50Design 3-6 $150-$300Coding 10 $500Development Testing 15-40 $750-$2000Acceptance Testing 30-70 $1500-$3500Production – Released to Client
40-1000 $2000-$50000Credibility not Measured
The Cost of Quality: A Reason for Built-in Quality 01/04/2011
10
Client Credibility not Measured
This Could be the Cost of a Single Defect
6
Let’s Think about it
01/04/2011The Cost of Quality: A Reason for Built-in Quality
11
What Happens When an Issue Isn’t Found Early
DesignCode R
ele
Analysis
Design
Code
Code
Test
ease to Custom
er
Code
The Cost of Quality: A Reason for Built-in Quality 01/04/2011
12
Code
Simple Process with Errors
7
The Present - More Challenges - Less Time
• Software is bigger, more complex, and more widely distributed
• Significant challenges maintaining quality now and in the future
• New platformsNew platforms• Internet• Mobile Devices• Servers
• Different database types• Different configurations
W t it i d ti t d
The Cost of Quality: A Reason for Built-in Quality 01/04/2011
13
• Want it in production yesterday
• It isn’t going to stop
Our Goal
Find and eliminate defects as early as possible.
How do we accomplish this?Dedicated effortUnderstand how quality fits into the current
The Cost of Quality: A Reason for Built-in Quality 01/04/2011
14
q ydevelopment processImplement an effective and efficient quality process
8
Making it work in any Development Process
WaterfallRUP (Rational
Unified Process)
Prototype
Kanban
Scrum
Iterative
The Cost of Quality: A Reason for Built-in Quality 01/04/2011
15
Iterative
Agile
How Many of You Work in an Agile Shop?Our Company’s SDLC includes our style of AGILE
Clarification: What Agile is and isn’t in a nutshellRemove misconception:Remove misconception:
Agile DOES use SDLC
Agile Manifesto is:•Working software over comprehensive documentation•Individual and interactions over processes and tools
The Cost of Quality: A Reason for Built-in Quality01/04/2011
16
•Customer collaboration over contract negotiation•Responding to change over following a plan
9
Where Does Quality Fit in any Type of SDLC ?
System Development Life Cycle• Analysis• Requirements
Where Does it fit?
Requirements• System Design• Program Design• Program Implementation (Coding)• Unit Testing• Integration Testing• System Testing
The Cost of Quality: A Reason for Built-in Quality 01/04/2011
17
• System Testing• System Delivery• Maintenance
The AnswerEVERYWHERE!!!
Quality should be in every part of the SDLC•AnalysisAnalysis•System Requirements Specifications•Software Architecture Document•Program/software/system•Documentation•Testing Plans
Good job you knew the answer!
The Cost of Quality: A Reason for Built-in Quality01/04/2011
18
g•Delivery / Release•Maintenance Plan
10
How do you Build-in Quality?
01/04/2011The Cost of Quality: A Reason for Built-in Quality
19
TESTING
Two Kinds of Testing • Attempting to find defects — is WASTE
Two Kinds of Testing • Attempting to find defects — is WASTE
• Attempting to prevent defects — is ESSENTIAL• Attempting to prevent defects — is ESSENTIAL
As QA / Testers • Our job should not be to just find defects
As QA / Testers • Our job should not be to just find defects
01/04/2011The Cost of Quality: A Reason for Built-in Quality
20
• Our job is to prevent defects.
• A quality process builds quality into the code
• Our job is to prevent defects.
• A quality process builds quality into the code
11
The Job of Tests
The job of tests, and the people that develop and run tests, is to prevent defects, not to find them. A quality assurance organization should champion processes which build quality i d f h h h li i linto code from the start, rather than test quality in later.
This is not to say that verification is unnecessary. Final verification is a good idea: it’s just that finding defects should be the exception, not the rule, during verification. If verification routinely triggers test-and-fix cycles, then the development
01/04/2011The Cost of Quality: A Reason for Built-in Quality
21
process itself is defective.
Implementing Lean Software Development: From Concept to CashBy Mary Poppendieck, Tom Poppendieck, Published Sept 7, 2006
Test-Later vs. Test-First The DifferenceTest-Later
• Design the code “in your head” or using design artifacts
Test-First • Analysis/design is done in the context of acceptance tests and acceptance criteria, written for each
• Verify the structure mentally, if you think of it, and have the time. (how do you test your design?)
• Automated tests are hard to add (code not always designed for them)
• Intentions are recorded
VS.
story • Every aspect of the solution is verified - quickly and repeatable
• Code is designed to be inherently testable from the beginning
• Tests record intent and provide examples
01/04/2011The Cost of Quality: A Reason for Built-in Quality
22
separately from the code
• “Implementation-centric”
examples
• Tests give insight about design quality
• “Interface-centric”
12
What Can Be Tested?
Requirements Models Architecture Design
Everything that could possibly break
Valid and invalid inputs
Errors, exceptions, and
eventsBoundary conditions
Internal logic paths
Formulas and equations Communications Persistence Integration
The Cost of Quality: A Reason for Built-in 01/04/201123
Interfaces Logs User Interface Response time
The list goes on and on and on and on
RequirementsArchitecture and DesignCritical areas of the software
What Should be Tested?
• Interfaces and Public Methods• All complex and business logic• Input: valid, invalid and boundaries
Help / Technical Documents
InstallationTraining plansSupport plans
The Cost of Quality: A Reason for Built-in Quality
01/04/2011
24
13
How Much Effort Argument?
A Balance Has to be Established
Money &
Quality
Time &
Effort
The Cost of Quality: A Reason for Built-in Quality 01/04/2011
25
What Stage to Build-in Quality? (Test)
Requirement
• Verification approach• Adequacy of design• Functional test data
C i t f d iRequirement Definitions
• Consistency of design• Acceptance criteria• Initial use cases • Implementation adequacy• Code walkthrough and code inspection• Consistency of design
D i lk h h
The Cost of Quality: A Reason for Built-in Quality01/04/2011
26
Software Development
• Design walkthrough• Code standards• Unit testing• User interface analysis
14
Software Testing
• Integration testing Iteration testing• Configuration tests System test / Smoke test• Functional test User acceptance test • Load test User interface analysis
Acceptance tests Negati e testing
What Stage to Build-in Quality? (Test)
Testing • Acceptance tests Negative testing
Release Software
• Place system into production• Correct versions• Help files - Updates and enhancements• CD / Online test• User release announcements• Test install
The Cost of Quality: A Reason for Built-in Quality 01/04/2011
27
Test install
Maintenance• Verify reported issues• Approve the changes• Modify and retest the changes• Analysis where we should have caught the issue
Who Should Be Testing?
Everyone!Business Analyst - Is this what the client needs?Requirements Analyst Are the requirements correct?Requirements Analyst - Are the requirements correct?Stakeholders - Is this what we asked for?System Architects – Is it designed correctly? Is development following the design?Development team – Unit testsQuality Assurance Specialist Is the quality process
The Cost of Quality: A Reason for Built-in Quality 01/04/2011
28
Quality Assurance Specialist – Is the quality process working? Are the test plans effective?Clients – Does this meet my needs?
15
Building in Quality in Different WaysPair ProgrammingTest Driven DesignCode Reviews
Acceptance CriteriaProgramming TestsAcceptance Tests
Code StandardsReusable LibrariesStory OwnersTeam CollaborationUnit Tests
pAutomation TestsContinuous Integration BuildsGated Check-ins
The Cost of Quality: A Reason for Built-in Quality 01/04/2011
29
These are just some of the ways to build-in quality. There are many, many more.
A balance has to be found between time, effort, quality, and money
Building in Quality – How do we do it?
Requirements PhaseValidationAcceptance criteria
Test PlansUnit testingFunctional testingAcceptance criteria
Test scenariosTest casesUse cases
Functional testingIntegration testingRegression testingPerformance testingConfiguration testingUsability testing
The Cost of Quality: A Reason for Built-in Quality01/04/2011
30
Usability testingPair testingSystem Run Through
16
In the Future There Will Be• More platforms
• Larger, more distributed systems
• New user inputs/interactions• New user inputs/interactions
• New hardware
• More integration (Facebook)
• Different databases
• Different configurations
The Cost of Quality: A Reason for Built-in Quality 01/04/2011
31
Different configurations
• And so much more!
We need to get started now building in Quality
Think About It
Software is growing bigger, more complex
Increase demand on qualityq y
Increase demand on testing resources
Increase demand on efficiency
Must adapt!
The Cost of Quality: A Reason for Built-in Quality01/04/2011
32
Must adapt!
17
For Some Of Us
It Won’t Be Easy
• Some people won’t understand
• Some people won’t believe
• Some people won’t want to change
The Cost of Quality: A Reason for Built-in Quality 01/04/2011
33
Take Baby Steps –Show One Improvement at a Time
How do you Start Building in Quality
Sell the cost of building in quality vs. testing in quality
T k id itTake one idea propose it
Implement it
Measure your success
The Cost of Quality: A Reason for Built-in Quality 01/04/2011
34
Prove how it works
18
Questions and Discussion
Thank you for Listening
And
Discussion Time
The Cost of Quality: A Reason for Built-in Quality01/04/2011
35
