ASQ Lean andSix SigmaConference
Session M12
February 29, 2016
Jennifer WilsonDodd Starbird
Service Data Quality: When
99.7% Isn’t Good Enough
Learning Objectives
1. Discuss the value of using quality data for process and performance control, and understand the critical difference between defects per opportunity (DPO) and defects per unit (defective rate).
2. Discuss the change management challenges that a team overcame in creating visibility and team buy-in to improve performance, and learn how to create a positive culture of human error prevention.
3. Integrate key concepts of Lean (visible metrics) with key concepts of Six Sigma (sampling and process capability), all with the common foundation of Deming Quality to tie the solution together!
Differentiation: Data-driven Insights
1,000s of Utilities100,000s of Facilities1,000,000s of Households
ConsumptionDemand Expense Capital Impact
Data-driven insights(demand to impact)
Energy Carbon Water Waste Telecom Lease
Why We Needed a Data Quality ProjectData Entry QA data from December, 2014:
• 99.7% line item accuracy… only 3 defects per 1,000 opportunities (3,000 DPMO)• Data Entry team was only publishing this
number
• But only 88.3% bill-level accuracy!• 11.7% defective bills (1 or more defects)• Data Entry leaders had stopped using this
number “because it made people feel bad”
• Estimated impact of downstream scrubbing and fixing data: 200 FTE!
Sources: Dec 2014 QA files and data user survey Mar 2015
Survey Question #1How does your organization measure quality? Pick the closest answer…
A. Mostly from the customer perspective (output quality, final yield, complaints, etc.)B. Mostly from an internal perspective (defects per hundred or million, individual quality, etc.)C. Both an internal and an external perspectiveD. We don’t measure quality all that much!
Data Collection Plan• Designed two data collection spreadsheets (account & bill)
• Designed and ran proof of concept• Tested repeatability and reproducibility with 4 collectors• Re-designed and re-piloted
• Collected population stratification factors for all accounts (all channels of data entry were in scope)• Sampling 1,500 accounts, ~650 sites, with 3 months of
bills each• Estimation of defect rates to precision of +/- 3%
• Purpose:• Validate defect rate; correlate causal factors
• Timing:• Deployed: week of March 9 (starting with pilot) • Gathered data with 8 data collectors over a month• Results analysis: April 13–15, 2015
Measured Data Quality: Stratified Defects
89.6% Bill Data Entry
Quality overall
93.3% Bill Quality from
EDI (Automated) Data Entry
43% Bill Quality on Post-Audit resolved
Defect types
Data Entry Process Improvements• Turned off an automated edit resolution (computer
based data scrubbing) program that was allowing known defects to get deferred instead of fixed (52% of auto-resolved edits were defects!)
• Launched a project to use Optical Character Recognition (OCR) technology to expand on benefits of Electronic Data Interchange (EDI)
• Trended causes of defects using Quality Control data and delegated assignments to teams to discuss causes and solutions in daily huddles
• Example key cause and solution: • Open inactive accounts cause reporting errors downstream• Closing inactive accounts was perceived to be a lower-
priority task and was backlogged 30 days on some teams• Catching up on that backlog has avoided $75,000 in
rework labor costs annually
Data Entry Performance Improvements• Started to communicate “bill-level” quality scores• Quality Control caught up to real-time to produce weekly team quality scores v. sharing data just monthly• Added new quality metrics to team whiteboards (shown below) to balance efficiency and quality focus• Sharing “quality stories” in daily huddles• Quantified cost of quality in same terms as production… “We probably made 3,000 defects yesterday!”
Data Entry QC Scores
85.0%
87.0%
89.0%
91.0%
93.0%
95.0%
97.0%
99.0%
Monthly Data Entry Quality Scores
Data Entry Quality: Before and After
Process and Performance Improved!
89.5% Bill Quality in
May
93.8% Bill Quality inAugust
Shift in performance and training
Individual QC Scores
Nik Wallenda
Who is Nik Wallenda?
Nik Wallenda
Nik is accountable for his performance.
Quality Sampling Principles
All work should have some chance of being checked (to drive individual accountability)
But individuals should know the work is likely to go right out the door (no safety net!)
Sample sizes for checking should be statistically calculated to deliver an appropriate precision of resulting quality measurements by (at least) work type, client, team, & individual
Approach
Calculate the minimum sample size per month (usually…), in order to get:• Appropriate precision by individual for
purposes of an annual review• Appropriate precision by client for
quarterly reviews and/or required service-level reporting
• Appropriate precision by task typeevery month at a team level, for:- Trending of performance- Ongoing monitoring- Root cause analysis of defects
Finally!
Set a random generator to apply the selection percentage by task type; apply that test immediately to every completed task (systematic sampling).
Conduct regular checks to make sure we have selected enough samples for appropriate client and individual reporting, and over-sample (randomly) in stratified groups as needed.
Use the data for root cause analysis and performance management!
Deming’s Point
Survey Question #2
Where is your organization’s most significant opportunity to improve your quality measures?
A. Balancing internal and external quality measuresB. Gathering process (root cause) data to improve process quality at the sourceC. Collecting the right sample size to get the most effective data as efficiently as possibleD. All of the above!E. None of the above
Discussion of Learning Objectives
1. Discuss the value of using quality data for process and performance control, and understand the critical difference between defects per opportunity (DPO) and defects per unit (defective rate).
2. Discuss the change management challenges that a team overcame in creating visibility and team buy-in to improve performance, and learn how to create a positive culture of human error prevention.
3. Integrate key concepts of Lean (visible metrics) with key concepts of Six Sigma (sampling and process capability), all with the common foundation of Deming Quality to tie the solution together!
PresentersJennifer Wilson
Director, Business Process Improvement, Ecova Inc.
Dodd Starbird
Managing Partner, Implementation Partners LLC