44
Optimizing the Risk and Cost of Materials Quality Assurance Monday, November 6, 2017 2:00-3:30 PM ET TRANSPORTATION RESEARCH BOARD
Optimizing the Risk and Cost of Materials Quality Assurance
Monday, November 6, 2017
2:00-3:30 PM ET
TRANSPORTATION RESEARCH BOARD
The Transportation Research Board has met the standards and
requirements of the Registered Continuing Education Providers Program.
Credit earned on completion of this program will be reported to RCEP. A
certificate of completion will be issued to participants that have registered
and attended the entire session. As such, it does not include content that
may be deemed or construed to be an approval or endorsement by RCEP.
Purpose Discuss how state departments of transportation can identify and evaluate opportunities that may enhance their materials quality assurance (QA) programs.
Learning Objectives At the end of this webinar, you will be able to: • Describe how to optimize materials QA based on project delivery
methods, project/materials risks, and advancements in specifications and technology
• Understand how manufacturing industry models were used to develop the framework
• Describe how risk management tools have been used to optimize QA resources for product/materials management
• Understand how to apply the tiered optimization framework to optimize materials QA practices
NCHRP Research Report 838: Guidelines for Optimizing the Risk and Cost of Materials QA Programs NCHRP Project 10-92
NCHRP is a State-Driven Program
– Suggest research of national interest
– Serve on oversight panels that guide the research.
• Administered by TRB in cooperation with the Federal Highway Administration.
• Sponsored by individual state DOTs who
Practical, ready-to-use results • Applied research aimed at
state DOT practitioners • Often become AASHTO
standards, specifications, guides, syntheses
• Can be applied in planning, design, construction, operations, maintenance, safety, environment
Additional NCHRP Publication Available on this Topic
• NCHRP Ready Results: Guidelines for Optimizing the Risk and Cost of Materials QA Programs
You can learn more about this publication by visiting www.trb.org
Today’s Speakers
• Sidney Scott, HKA Global Inc. • Keith Molenaar, University of Colorado at
Boulder • Linda Konrath, HKA Global Inc.
NCHRP 10-92 Optimizing the Risk and Cost of Materials QA Programs SIDNEY SCOTT, KEITH MOLENAAR, AND LINDA KONRATH
1
Agenda
1. Introduction
2. Research Approach
3. Phase 1 Research: Current Practice ◦ Data Collection ◦ Findings
4. Phase 2: Developing Guidelines ◦ Level 1: Qualitative materials-based
assessment ◦ Level 2: Qualitative property-based
optimization ◦ Level 3: Quantitative cost-based optimization
5. Conclusions
2
NCHRP 10-92 Project Team
Sidney Scott (Co-PI) - HKA Linda Kornath – HKA Keith Molenaar (Co-PI) – U of Colorado @ Boulder Matthew Hallowell – U of Colorado @ Boulder Eric Oechler – U of Colorado @ Boulder Cecil Jones – Diversified Engineering Services, Inc. John D’Angelo – D'Angelo Consulting, LLC Gerald Huber – Heritage Research Group Jo Sias Daniel, University of New Hampshire
3
Why is it important? Materials represent 50% of Federal aid construction dollars
Asphalt represents 20% of the total infrastructure budget
DOTs’ budgets are shrinking Not enough inspectors Lack of qualified workforce
Accelerated delivery can incentivize contractors/suppliers to cut corners
The result of something going wrong can be catastrophic
4 Introduction Approach Phase 1 Phase 2 Conclusions
Res
ourc
es in
vest
ed
Nee
d fo
r opt
imiz
atio
n
Impa
ct o
f QA
Phase 1 Literature
Review Survey Interviews
Findings
Phase 2
Guidelines
5 Introduction Approach Phase 1 Phase 2 Conclusions
Approach outline NCHRP 10-92 Optimizing the Risk and Cost of Materials QA Programs
Data collection Research
Methodology Source of
Information Literature review
More than 80 relevant papers, manuals, schedules and reports.
Survey Responses from 58 people out of 37 DOTs.
Interview Maryland Washington Ohio California New Jersey Texas Virginia Florida
6 Introduction Methods Phase 1 Phase 2 Conclusions
Literature Review Current state of materials QA, considering practices used both inside and outside the U.S. highway construction industry
Materials Manuals
Current Materials QA Research Reports
Advanced test methods
Mature or advanced risk-based practices
QA practices outside of Transportation Industry (i.e. manufacturing)
7 Introduction Methods Phase 1 Phase 2 Conclusions
Survey 37 DOTs
Current state of DOT materials QA, considering practices used both inside and outside the U.S. highway construction industry
• Identify trends related to materials QA • Identify different ways DOTs identify, assess, and mitigate
quality-related risk • Obtain general data related to QA costs and probability of
failure (non-conformance) • Assess state DOT materials management systems • Identify what additional information would have to be
collected through the interviews
8 Introduction Methods Phase 1 Phase 2 Conclusions
Interviews Captured a deeper level of insight than could be obtained through a simple online questionnaire,
Eight DOTs, and industry reps selected in part due to their leading QA practices in response to the survey. Caltrans, Florida, Ohio, Maryland, New Jersey, Texas, Virginia, and Washington State
Probe for specific data regarding risk-based QA methods and costs of QA
9 Introduction Methods Phase 1 Phase 2 Conclusions
What are DOTs doing? Material’s QA practices based on legacy practices
QA approaches change from DOT-to-DOT and sometimes even within departments
Informal hierarchy based on materials type: ◦ Project produced ◦ Plant produced ◦ Standard manufactured material
10 Introduction Methods Phase 1 Phase 2 Conclusions
What factors influence DOT’s QA approach? DOTs approach based on factors: Factor 1. Material variability and level of control required for materials to meet specifications
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Project produced
Plant Produced
Standard Manufactured
Project produced
Plant Produced
Standard Manufactured
Sam
plin
g an
d Te
stin
gCe
rtifi
catio
n
Always Sometimes Never
11 Introduction Methods Phase 1 Phase 2 Conclusions
Factor 2: Criticality of materials or products – difficulty to repair or replace, safety, maintenance cost or cost of rework
Factor 3: Project characteristics – size and complexity or material quantity.
Factor 4: Industry experience
Factor 5: The use of alternative delivery methods 45% of the survey responses did not vary their QA approach on different project delivery methods
Vs
Research P3/DBOM contracts guidelines, schedules, and manuals
12 Introduction Methods Phase 1 Phase 2 Conclusions
What factors influence DOT’s QA approach?
Risk-Based Systems WSDOT’s Material Acceptance Rating Matrix
13 Introduction Methods Phase 1 Phase 2 Conclusions
Risk-Based Systems TXDOT’s QAP Implementation Guide: Categories for Owner Verification
14 Introduction Methods Phase 1 Phase 2 Conclusions
Levels for Analysis Level 1 Level 2 Level 3
EMBANKMENTS, SUBGRADES, BACKFILL, AND BASE COURSES
MATERIAL OR PRODUCT TEST FOR TEST NO. TxDOT RECOMMENDED
EMBANKMENT (CUTS & FILLS)
Liquid Limit Tex-104-E 2 Plasticity Index Tex-106-E 1
Linear Shrinkage Tex-107-E 2 Gradation Tex-110-E 2
Moisture/Density Tex-114-E 3 In-Place Density Tex-115-E 1
RETAINING WALL (NON-SELECT BACKFILL)
Liquid Limit Tex-104-E 2 Plasticity Index Tex-106-E 1
Linear Shrinkage Tex-107-E 2 Gradation Tex-110-E 2
Moisture/Density Tex-114-E 3 In-Place Density Tex-115-E 1
RETAINING WALL (SELECT BACKFILL)
Gradation Tex-110-E 2 Resistivity Tex-129-E 2
pH Tex-128-E 2 Soundness Tex-411-A 3
In-Place Density Tex-115-E 1
UNTREATED BASE COURSES
Liquid Limit Tex-104-E 2 Plasticity Index Tex-106-E 1
Linear Shrinkage Tex-107-E 2 Gradation Tex-110-E 2
Moisture/Density Tex-113-E 3 Wet Ball Mill Tex-116-E 2
Triaxial Tex-117-E 2 In-Place Density Tex-115-E 1 Moisture Content Tex-103-E 2
Thickness Tex-140-E 1
Risk-Based Systems California DOT’s Construction Quality Assurance Program Manual
15 Introduction Methods Phase 1 Phase 2 Conclusions
Tier Failure Category Consequence of Failure Example Items QA Requirements
1 Catastrophic Greatest consequence of failure. Failure is likely to cause loss of life or serious injury.
Structural steel, precast girders, pre-stressing
QA methods designed to provide the maximum level of confidence in the QC efforts of both the contractor and the producer.
2 Safety Failure creates a safety hazard for employees or the public.
Delineation, safety barriers, lighting, signal controllers
QA methods designed to provide a high level of confidence in the QC efforts of both the contractor and the producer through extensive use of pre-qualified materials from the authorized material list.
3 Interrupt Service
Failure or repair may cause an interruption in service, or environmental impact.
Pavements, bases, embankment, storm water pollution prevention plan-best management practice devices
QA methods based on 23 CFR 637 requirements for jobsite-produced items, applicable rules and regulations included in the contract for the environmental items; and certificates of compliance from the contractor or producer combined with intermittent inspection, sampling, and testing of in-progress work for drainage items.
4 Monetary Monetary loss only – consequence of failure is considered minimal in terms of project performance.
Grass seed, drainage and irrigation products, fencing
QA methods typically based on use of commercial quality products or extensive use of certificates of compliance from the contractor or producer combined with periodic random inspection of in-progress work.
Risk-Based Systems California DOT’s Office of Construction Materials, DSM Risk-Based Matrix
16 Introduction Methods Phase 1 Phase 2 Conclusions
Risk Assessment/Risk Factor
Prob
abili
ty
1. M
ater
ial &
Wor
kman
ship
A
sses
smen
t
High 3 Programmatic Assessment
(3)
Intermittent Inspection
(6)
Continuous Inspection
(12)
Continuous Inspection
(24)
Medium 2 Programmatic Assessment
(2)
Intermittent Inspection
(4)
Continuous Inspection
(8)
Continuous Inspection
(16)
Low 1 Programmatic Assessment
(1)
Programmatic Assessment
(2)
Intermittent Inspection
(4)
Continuous Inspection
(8) 1 2 4 8
Type I Projects
Projects with Regular Schedule
Loss of Funds to Repair
Item
Interruption in Service
Significant Safety
Concerns
Catastrophic Consequence
Type II Projects
Projects with Accelerated Delivery, Emergency, Significant Schedule or
Cost Impact
Loss of Funds to Repair
Item
Interruption in Service
Significant Safety
Concerns or Catastrophic Consequence
2. Project Assessment
Impact Legend
Low Risk Risk Factor < 3 Medium Risk 4 < Risk Factor < 7 High Risk Risk Factor > 8 Risk Factor = (Probability) x (Impact)
Risk-Based Systems InDOT’s Protocol for Inspection of Construction Activity
17 Introduction Methods Phase 1 Phase 2 Conclusions
Construction Activity Priority Macro- Consequences Due
to Missed/Reduced Inspection
Critical Items to Be Watched Frequency of Inspection
Clearing site Low ---- Areas to and not be cleared Randomly Clearing obstructions Randomly Removal to adequate depth Randomly
Identify wet spots Randomly
Aggregate base course High Functional failures, increased maintenance costs, decreased design life
Moisture and density control Frequently Compactor passes Constantly
Depth of each lift Constantly
Documentation Constantly
Obtain tickets for materials (depending on payment method)
Frequently
Embankment High Functional failures, increased maintenance costs, decreased design life
Quality of the soil being placed Constantly
Moisture content Constantly
Density Constantly
Measure embankment area Constantly
Lifts height and width Frequently
Costs of Quality
18 Introduction Methods Phase 1 Phase 2 Conclusions
1. Prevention costs: costs of activities to prevent defects in the design and development of a product;
2. Appraisal costs: costs incurred to inspect, test, and evaluate conformance to specifications;
3. Internal failure costs: costs incurred when a defect is detected prior to the sale of a product; and
4. External failure costs: costs incurred after a product is in the hands of the consumer
Cost of Quality Adapted from FTA 2012
19 Introduction Methods Phase 1 Phase 2 Conclusions
Category Description Example Components
Cost of Conformance (Prevention + Detection)
Prevention Costs Costs related to assuring the product or project meets requirements
Design analysis and reviews Constructability reviews Quality management systems
Appraisal Costs Costs related to determining the degree of product or project conformance
Inspection Sampling and testing
Cost of Non-Conformance (Defects or Failure)
Cost of Defects or Failures Costs associated with non-conforming materials
Repair/rework Schedule delays Road user impacts Reduced life
Cost of Quality Economics of Quality of Conformance [Adapted from Kirkpatrick 1970]
20 Introduction Methods Phase 1 Phase 2 Conclusions
So what did we learn?
QA Effort Level
Description Owner Contractor
Level 1 Visual Inspection Visually inspects manufacture Visually inspects placement
Process control
Level 2 Certification Verify that certification complies with specification requirements.
Certifies materials and installation meet specifications Performs testing and maintain data to support certification
Level 3 Certification with backup data attached
Verification of data (audit certification data for compliance including option to perform additional tests)
Performs testing and submits backup data to support certification (i.e. mill test or other tests attached to certification)
Level 4 Reliance on contractor data for acceptance with agency verification
Tests material on a reduced frequency and compares it to the contractor's results. Also responsible for IA.
Performs sampling and testing and provides results to owner
Level 5 Sampling and testing performed by agency
Performs sampling and testing and accepts materials using their results. Also responsible for IA.
Process control
**Inspection happens at all levels
21 Introduction Methods Phase 1 Phase 2 Conclusions
Phase 2
22 Introduction Methods Phase 1 Phase 2 Conclusions
Level 1: Qualitative material based optimization
23 Introduction Methods Phase 1 Phase 2 Conclusions
1. Identify goal of optimization
2. Identify materials of interest
3. Identify production mode (project-produced vs. fabricated vs. standard manufactured)
4. Assess the risk of non-conformance for each material of interest
5. Identify appropriate QA method given material tier
Level 1: Qualitative materials based optimization
24 Introduction Methods Phase 1 Phase 2 Conclusions
Numerical Rating
Adjectival Description Definition
1 Nonconformance is unlikely < X%
2 Nonconformance is somewhat likely
> X% to < X%
3 Nonconformance is likely > X% to < X%
4 Nonconformance is highly likely > X% to < X%
Numerical Rating Adjectival Description Definition
1 Minimal Impact Little if any impact to service life
2 Some Impact Earlier than planned maintenance needed
3 Significant Impact Earlier than planned major rehabilitation needed
4 Catastrophic Impact Immediate intervention needed
Risk Score Material Tier Description
Risk Score > 8 Tier 1 Materials having the greatest risk of failure
2 < Risk Score < 8 Tier 2 Moderate risk materials
Risk Score < 2 Tier 3 Low risk materials
Level 2: Property based optimization
1. Identify materials of interest a) Project-produced materials with greatest variability b) Identify material acceptance properties and acceptance
plan
2. Assess the extent that acceptance testing can be optimized based on significance of property and material use
3. How indicative of performance is the property for acceptance?
Primary indicator – highest residual risk (default acceptance/verification testing frequencies)
Secondary indicator – medium residual risk (reduced acceptance/verification testing frequency)
Observational indicator – low residual risk (observation of contractor QC with random verification testing)
25 Introduction Methods Phase 1 Phase 2 Conclusions
Density, AC content VMA
Need of better understanding of: ◦ Cost of QA ◦ Rationale behind acceptance
method ◦ Probability of non-conforming
material ◦ Impact of a non-conforming material
26 Introduction Methods Phase 1 Phase 2 Conclusions
Identify material and
property
Identify the factors
Quantify the cost of QA
Risk analysis
Optimize CoQ
Level 3: Cost-based optimization
Level 3: Cost-based optimization
𝐸𝐸𝐸𝐸 = I × PNC
Where: EV = Expected value of failure as a percentage of material cost (%) I = Impact of rework of a material non-conforming as a percentage of material cost. (%) 𝑃𝑃𝑃𝑃𝑃𝑃= Probability of non-conformance (%)
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
L E V E L 1 L E V E L 2 L E V E L 3 L E V E L 4 L E V E L 5
% O
F M
ATER
IAL C
OST
QUALITY ASSURANCE EFFORT
EV of non-conformance Cost of QA CoQ
QA Cost
EV CoQ
27 Introduction Methods Phase 1 Phase 2 Conclusions
Optimal point
28
Level 3: Cost-based optimization
Bridge members S-1 S-2 S-3 S-4 QA Effort\Scenarios 1 2 3 4 Visual inspection 80% 91% 75% 76% Certification 65% 75% 61% 55% Certification w/data 55% 67% 51% 42% Verification S&T 34% 31% 27% 26% Full S&T 22% 30% 27% 31%
Introduction Methods Phase 1 Phase 2 Conclusions
Spec\Scenario 1
2
3
4
1 Bridge members 110% 110% 110% 110% 2 Drainage structure 100% 100% 100% 100%
Bridge members
QA Effort\Scenarios 1 2 3 4 1 Visual inspection 70% 80% 65% 66%2 Certification 56% 65% 53% 47%3 Certification w/data 47% 58% 44% 35%4 Verification S&T 26% 33% 20% 19%5 Full S&T 11% 18% 10% 10%
Bridge members
QA Effort\Scenarios 1 2 3 41 Visual inspection 3% 3% 3% 3%2 Certification 3% 3% 3% 3%3 Certification w/data 3% 3% 3% 3%4 Verification S&T 5% 5% 5% 5%5 Full S&T 10% 10% 10% 10%
Given that: 1) The precast material is a bridge member; 2) The contractor has a high industry experience, the project has a large amount of material and is a complex or critical project, and the delivery method is a traditional design-bid-build; and 3) The agency performed verification sampling and testing. A) What is the probability that the material is non-conforming after the agency accepts it? B) What is the cost of performing that level of QA as a percentage of the material total installed or in-place cost?
Findings 1. DOTs have generally downsized (materials QA and
inspection staff) 2. Greater use of alternative delivery methods have
shifted quality management to industry 3. Greater reliance on Industry QA
a) Use of contractor QC Tests in acceptance decision b) Industry self-certification (fabricated, manufactured, and
constituent materials)
4. Several DOTs use a risk-based approach to optimize materials management and inspection (in manuals and specifications)
5. Key factors that influence level of QA include material variability, criticality of materials, project characteristics, industry experience, and delivery method.
29 Introduction Methods Phase 1 Phase 2 Conclusions
Conclusions 1. Agencies can significantly benefit from optimizing their
materials QA 2. Potential applications
a) Entire Program b) Project level c) All classifications of materials (but most beneficial for
jobsite produced materials)
3. Framework includes 3 levels of optimization a) Materials-based (qualitative risk-based material
assessment) b) Properties-based (qualitative risk-based property
assessment) c) Cost-based (analysis of total cost of QA)
30 Introduction Methods Phase 1 Phase 2 Conclusions
Future Work ◦ Model validation and/or correction with empirical data. ◦ Application of cost model to different materials. ◦ Further application of QA framework to a different
industries. ◦ More rigorous assessments of actual costs of non-
conformance (i.e. repair, replacement or maintenance) ◦ Better understanding of expert judgements if using
expert panels ◦ Correct probabilistic judgements from each expert. ◦ Expert judgement on very low probability but high impact events.
31 Introduction Methods Phase 1 Phase 2 Conclusions
32
Questions?
Today’s Participants
• Ed Harrigan, Transportation Research Board, [email protected]
• Sidney Scott, HKA Global Inc., [email protected] • Keith Molenaar, University of Colorado at Boulder,
[email protected] • Linda Konrath, HKA Global Inc.,
Panelists Presentations
http://onlinepubs.trb.org/onlinepubs/webinars/171106.pdf
After the webinar, you will receive a follow-up email
containing a link to the recording
Get involved with NCHRP
• Suggest NCHRP research topics • Volunteer to serve on NCHRP panels • Lead pilot projects and other
implementation efforts at your agency • For more information:
http://www.trb.org/nchrp/nchrp.aspx
Take Part in the Careers in Motion Networking Fair
http://bit.ly/CareersInMotionFair
