Date post: | 20-Jan-2018 |
Category: |
Documents |
Upload: | barnaby-atkins |
View: | 220 times |
Download: | 0 times |
1 of 48
The EPA 7-Step DQO Process
Step 6 - Specify Error Tolerances
3:00 PM - 3:30 PM (30 minutes)
Presenter: Sebastian Tindall
Day 2 DQO Training CourseModule 8
2 of 48
Terminal Course ObjectiveTo be able to define the decision errors, consequences of the errors, the null hypothesis, and the lower bound of the gray region for a specific project
3 of 48
Step Objective: To specify the decision
makers’ tolerable limits on decision errors, which are used for limiting uncertainty in the data– Since analytical data can only
provide an estimate the true condition of a site, decisions that are based on such data could potentially be in error
Step 6: Specify Error Tolerances
Step 4: Specify Boundaries
Step 2: Identify Decisions
Step 3: Identify Inputs
Step 1: State the Problem
Step 5: Define Decision Rules
Step 6: Specify Error Tolerances
Step 7: Optimize Sample Design
4 of 48
Step Objective: To specify the decision
makers’ tolerable limits on decision errors, which are used for limiting uncertainty in the data– Since analytical data can only
provide an estimate the true condition of a site, decisions that are based on such data could potentially be in error
Step 6: Specify Error Tolerances
Step 4: Specify Boundaries
Step 2: Identify Decisions
Step 3: Identify Inputs
Step 1: State the Problem
Step 5: Define Decision Rules
Step 6: Specify Error Tolerances
Step 7: Optimize Sample Design
Estimation Error
Inherent in the process of estimation is error (deviationfrom the true value).
That’s why it’s called estimation.
Error Mistake
Error = Deviation
5 of 48
Decision Error
Tolerances
Bounds of the
Gray Region
Assign probability limits on either side of the gray region
Information IN Actions Information OUT
From Previous Step To Next StepDecision
RulesStep 5
Determine the variability of the environmental variables
Step 6- Specify Error Tolerances
Choose the null hypothesis
Identify the decision errors
Specify the boundaries of the gray region
6 of 48
Decision Error Tolerances The goal of the planning team is to develop
a data collection design that reduces the chance of making a decision error to a tolerable level
Step 6 provides a mechanism for allowing the decision maker to define tolerable limits on the probability of making a decision error
7 of 48
Two Reasons Why Decision Makers Make Decision Errors
Sampling error occurs because the sampling design is unable to capture the complete extent of heterogeneity that exists in the true state of the environment
Measurement error occurs because analytical methods and instruments are not absolutely perfect
8 of 48
Where do errors occur?
Planning
Sampling
Analysis
Data Vs
Decision
9 of 48
Types of Decision Errors
Before we can talk about acceptable limits for making decision errors, we must first understand what correct decisions and decision errors look like
There are two types of correct decisions and two types of decision errors that can be made
10 of 48
Null Hypothesis:
Site is dirty. Site is dirty.Site is clean.
100
True State of Site
Alternative Action
Walk away from site. Clean up site.
75
Probability of deciding
that the site is dirty
0.0
0.5
1.0
Action LevelLower Bound of Gray Region
Typical Curve
Decision Performance
GoalDiagram
Sample MeanUCL
True Mean Sample
MeanUCL
Sample MeanUCL
True Mean
True Mean
11 of 48
Decision Error
Tolerances
Bounds of the
Gray Region
Assign probability limits on either side of the gray region
Information IN Actions Information OUT
From Previous Step To Next StepDecision
RulesStep 5
Determine the variability of the environmental variables
Choose the null hypothesis
Identify the decision errors
Specify the boundaries of the gray region
In order to calculate the number of samples needed(in DQO Step 7), an estimate of the population standard deviation is needed for each environmental variable.
• Compile a list of the “driver” COPCs• Use existing data (must pass Step 3 data assessments)• Establish the range based on historical information
– Existing data– Process knowledge– Professional judgment
• Estimate of the population standard deviation– Reference source – Method of calculating
Step 6- Specify Error Tolerances
12 of 48
Decision Error
Tolerances
Bounds of the
Gray Region
Assign probability limits on either side of the gray region
Information IN Actions Information OUT
From Previous Step To Next StepDecision
RulesStep 5
Determine the variability of the environmental variables
Choose the null hypothesis
Identify the decision errors
Specify the boundaries of the gray region
In order to calculate the number of samples needed(in DQO Step 7), an estimate of the population standard deviation is needed for each environmental variable.
• Compile a list of the “driver” COPCs• Use existing data (must pass Step 3 data assessments)• Establish the range based on historical information
– Existing data– Process knowledge– Professional judgment
• Estimate of the population standard deviation– Reference source – Method of calculating
Estimate the standard deviation by using the Deming approach of dividing the range by 2 or 3, depending on the frequency distribution.
Step 6- Specify Error Tolerances
13 of 48
Estimated Standard DeviationsEstimated Standard Deviations for NaI Example
Environmental VariableRange of
EnvironmentalVariable
Estimate of Population Standard DeviationDR#
Attribute Unit ofMeasure
LowerLimit
UpperLimit
StandardDeviation Source How Estimated?
1
True mean ofCs-137estimated by95% UCL ofsample mean
pCi/g 0.031 1.89 0.4052000HPGedata
From 116-H-1 CVP data
1
True mean ofCs-137estimated by95% UCL ofsample mean
cpm(convertedto pCi/g)
-5.17 99.92
7.48or
7.48/2=3.74
1999 to2000 NaIdata
From 116-H-1 CVP data
14 of 48
Estimated Standard DeviationsPerimeter Side-Slope Soil
CS
DR # EnvironmentalVariable
Range ofEnvironmental
Variable
Estimate of PopulationStandard Deviationa
Attribute Unit ofMeasure
LowerLimit
UpperLimit
StandardDeviation Source How
Estimated
Co-60 pCi/g 0.018U 0.49 0.1711
Cs-137 pCi/g 0.083 11.6 4.31
Eu-152 pCi/g 0.13 12.8 3.62
Sr-90 pCi/g 0.11U 5.8 1.56
RI/FSData
Computedfrom
RI/FS data
a The choice of an estimate of a standard deviation has a large impact on the number of samples required. Avoid underestimating thestandard deviation. Always be conservative when estimating the standard deviation.
U = Sample was undetected, value shown is the analytical detection limit.
15 of 48
Estimated Standard DeviationsTrench Footprint Soil
CS
DR # EnvironmentalVariable
Range ofEnvironmental
Variable
Estimate of PopulationStandard Deviationa
Attribute Unit ofMeasure
LowerLimit
UpperLimit
StandardDeviation Source How
Estimated
Cs-137 pCi/g 37.2 75 19.72
Pu-239/240 pCi/g 1.4 3 0.795
RI/FSData
Computedfrom RI/FS
dataa The choice of an estimate of a standard deviation has a large impact on the number of samples required. Avoid overly optimistic
estimates of the standard deviation. Always be conservative when estimating the standard deviation.U = Sample was undetected, value shown is the analytical detection limit.
16 of 48
Analytical +
Sub-sampling +
Natural heterogeneity of the site=
Total Uncertainty
Uncertainty is Additive!Remember the uncertainty is additive for
all steps in sampling and analysis
17 of 48
Decision Error
Tolerances
Bounds of the
Gray Region
Assign probability limits on either side of the gray region
Information IN Actions Information OUT
From Previous Step To Next StepDecision
RulesStep 5
Determine the variability of the environmental variables
Choose the null hypothesis
Identify the decision errors
Specify the boundaries of the gray regionDefine both types of decision error:Determine which one occurs above and which one occurs below the action level.
Two Types of Decision Error:• Cleaning up a clean site• Walking away from a dirty site
Step 6- Specify Error Tolerances
18 of 48
Decision Error
Tolerances
Bounds of the
Gray Region
Assign probability limits on either side of the gray region
Information IN Actions Information OUT
From Previous Step To Next StepDecision
RulesStep 5
Determine the variability of the environmental variables
Choose the null hypothesis
Identify the decision errors
Specify the boundaries of the gray region
For each Alternative Action:• Create a list of possible decision error(s) that may occur if an action
is incorrectly taken• Discuss the consequences of making each decision error• Rate the severity of the consequences of a decision error (i.e.,
low, moderate, severe) at a point:– Far below the action level– Below but near the action level– Above but near the action level– Far above the action level
• Indicate which decision error has the most severe consequencenear the action level
Step 6- Specify Error Tolerances
19 of 48
Decision Error Consequences CS
Severity of Consequences of Decision Error When TrueParameter Is …
(Rated as Low/Moderate/Severe)
DR # AA # PossibleDecision Error
Consequencesof Decision
ErrorFar Belowthe Action
Level
Below ButNear the
Action Level
Above ButNear the
Action Level
Far Above theAction Level
Decision ErrorThat Has More
SevereConsequences
Near the ActionLevel
1: Conductremedialaction
Remediating anuncontaminatedsite
Expense andscheduleimpacts ofremediating anuncontaminatedsite
Moderate Moderate None None11
2: Take nofurtheraction
Failing toremediate acontaminatedsite
Leaving a sitein place theposes a threat tohuman healthand safety
None None Moderate Severe
Not remediating acontaminated site
1: Conductremedialaction
Remediating anuncontaminatedsite
Expense andscheduleimpacts ofremediating anuncontaminatedsite
Moderate Moderate None None22
2: Take nofurtheraction
Failing toremediate acontaminatedsite
Leaving a sitein place theposes a threat tohuman healthand safety
None None Moderate Severe
Not remediating acontaminated site
1 Applies to the perimeter side-slope soil around the –20 level of the trench footprint.2 Applies to the bottom of the trench footprint after excavation.
20 of 48
Decision Error
Tolerances
Bounds of the
Gray Region
Assign probability limits on either side of the gray region
Information IN Actions Information OUT
From Previous Step To Next StepDecision
RulesStep 5
Determine the variability of the environmental variables
Choose the null hypothesis
Identify the decision errors
Specify the boundaries of the gray regionProvide rationale for rating the severity of consequences as low or severe
Step 6- Specify Error Tolerances
21 of 48
Rationale for Error Consequence Ratings
CS
Table 3b. Justification for Rating Severity of Consequences as “Severe”
DS/DR
AA Justification
#1a,#2b
#2 The decision errors for not remediating both perimeter side slope andtrench footprint soils when the concentrations are far above the action levelare also rated as “severe.” In these cases, soil with concentrations that maypose serious health risks would remain at the site.
a) Applies to decision statements and rules related to perimeter side-slope soil afterexcavation
b) Applies to decision statements and rules related to trench footprint soil afterexcavation
22 of 48
Decision Error
Tolerances
Bounds of the
Gray Region
Assign probability limits on either side of the gray region
Information IN Actions Information OUT
From Previous Step To Next StepDecision
RulesStep 5
Determine the variability of the environmental variables
Choose the null hypothesis
Identify the decision errors
Specify the boundaries of the gray regionDefine the null hypothesis (baseline condition) and the alternative hypothesis:The decision error that has the most adverse potential consequences should bedefined as the null hypothesis.
The null hypothesis should state the OPPOSITE of what the project hopes to demonstrate.
• Site is assumed to be contaminated until shown to be clean• Site is assumed to be clean until shown to be contaminated
Step 6- Specify Error Tolerances
23 of 48
Null Hypothesis
DR Null Hypothesis Statement (examples) IndicateSelection
The site is contaminated. #1-2 a
The site is uncontaminated.a Applies to both perimeter side-slope soils and trench footprint
Contaminated:H0 : > Action Level
Uncontaminated:HA : < Action Level
CS
24 of 48
Decision Error
Tolerances
Bounds of the
Gray Region
Assign probability limits on either side of the gray region
Information IN Actions Information OUT
From Previous Step To Next StepDecision
RulesStep 5
Determine the variability of the environmental variables
Choose the null hypothesis
Identify the decision errors
Specify the boundaries of the gray region
The gray region is a range of possible parametervalues within which the consequences of a decision errorare relatively minor.
Step 6- Specify Error Tolerances
25 of 48
Decision Error
Tolerances
Bounds of the
Gray Region
Assign probability limits on either side of the gray region
Information IN Actions Information OUT
From Previous Step To Next StepDecision
RulesStep 5
Determine the variability of the environmental variables
Choose the null hypothesis
Identify the decision errors
Specify the boundaries of the gray region
The gray region is bounded on one side by the action level, and on the other side by the parametervalue where the consequences of decision error beginsto be significant. This point is labeled LBGR, whichstands for lower bound of the gray region.
Step 6- Specify Error Tolerances
26 of 48
Decision Error
Tolerances
Bounds of the
Gray Region
Assign probability limits on either side of the gray region
Information IN Actions Information OUT
From Previous Step To Next StepDecision
RulesStep 5
Determine the possible range of the parameter of interest
Choose the null hypothesis.
Identify the decision errors.
Specify the boundaries of the gray region
Determine the variability of the environmental variables
Choose the null hypothesis
Identify the decision errors
It is necessary to specify the gray region because variability in the population and unavoidable imprecision in the measurement system combine to produce variability in the data such that a decision maybe “too close to call” when the true parameter value is very near the action level.
Step 6- Specify Error Tolerances
27 of 48
Decision Error
Tolerances
Bounds of the
Gray Region
Assign probability limits on either side of the gray region
Information IN Actions Information OUT
From Previous Step To Next StepDecision
RulesStep 5
Determine the variability of the environmental variables
Choose the null hypothesis
Identify the decision errors
Specify the boundaries of the gray region
Present the rationale of how the LBGR was calculated or determined.
Step 6- Specify Error Tolerances
28 of 48
Lower Bound of the Gray Region
Because the null hypothesis is that the site is contaminated, the upper bound of the gray region is set equal to the action level
The LBGR should be set at a value where the consequences of the decision error begin to be significant
29 of 48
How to Set the LBGR
LBGR = AL - (Analytical + Sampling Error) LBGR = AL - 1/2 Action Level LBGR = Decision-Maker “whim” - AL - 0.2 AL LBGR = Frequency Distribution method
30 of 48
The LBGR is often based on unavoidable variability in the concentration data– The LBGR may be estimated based on the
precision that the analytical methods allow plus an estimate as to the sampling variance
– LBGR = AL - (Analytical + Sampling Error) MARSSIM suggests the LBGR be set as:
– LBGR = AL - 1/2 AL
How to Set the LBGR (cont.)
31 of 48
The LBGR is often set at some other value– This is based on the decision makers’ choice and is not
scientifically based– LBGR = AL - (10 - 20% of AL)
How to Set the LBGR (cont.)
32 of 48
Use the Frequency Distribution method– The LBGR may be estimated based the Probability
Distribution Function (PDF)– Place the Action Level on the mean of the PDF– Ask: “Does a substantial amount of contaminant
concentration values exceed the Action Level?”– If yes, begin moving the PDF backwards along the
x-axis towards zero concentration– Pause and ask again– When the answer is no, you have set the LBGR
(e.g., where the mean of the PDF lies on the x-axis is now the LBGR)
• Use probability theory to show/prove this
How to Set the LBGR (cont.)
33 of 48
Show Probability Density Function Distribution
Demonstration
34 of 48
Methods for Evaluating the Attainment of Cleanup Standards - Volume 1: Soils and Solid Media
EPA, February 1989
PB89-234959
How to set the LBGR
212
1
211
25.0
Z
ALZZ
n
1 is a hypothetical “mean concentration where the site should be declared clean with a high probability”
35 of 48
“Normal” FD
“Skewed” FD
Computer Simulations: “Badly skewed” or Any FD• Evaluate and errors to select n
Using the LBGR to Estimate n
212
1
211
25.0
Z
ALZZ
n
212
1
211
25.016.1
ZAL
ZZn
36 of 48
Example 1 Balancing the width
of the gray region controls the cost of sampling and analysis
The closer the LBGR lies to the action level, typically the greater the number of samples, and thus the cost increases
Mean = LBGR AL
concentration
Width of the Gray Region
Mean = LBGR AL
concentration
Width of the Gray Region
37 of 48
Width of the Gray Region
GR = Analytical + Sampling Error– Estimated based on past data and general knowledge
GR = 1/2 of the AL – For each COPC, calculate and set LBGR
GR = 20% of the AL – For each COPC, calculate and set LBGR
GR = PDF method– Use PDF for worst COPC to set LBGR
38 of 48
Setting the GR Based on the (Analytical + Sampling Error)
CS
Environmental Variable
DR # Attribute Unit ofMeasure
ActionLevel LBGRa Gray
Region
Co-60 pCi/g 1.4 1.23 0.17
Cs-137 pCi/g 6.2 1.89 4.3
Eu-152 pCi/g 3.3 -0.32 3.62
1
Sr-90 pCi/g 4.5 2.94 1.56
Cs-137 pCi/g 500,000 500,000 19.82
Pu-239/240 pCi/g 718,600 718,000 0.795a For DR #1,2 the LBGR = AL – total error. Standard deviation of RI/FS sampling data was used
as the total error. Avoid overly optimistic estimates of the standard deviation. Always beconservative when estimating the standard deviation.
39 of 48
Setting the GR Based on Regulator Input (20% of the AL)
CS
Environmental Variable
DR # Attribute Unit ofMeasure
ActionLevel LBGRa Gray
Region
Co-60 pCi/g 1.4 1.11 0.28
Cs-137 pCi/g 6.2 5.0 1.24
Eu-152 pCi/g 3.3 2.64 0.66
1
Sr-90 pCi/g 4.5 3.6 0.9
Cs-137 pCi/g 500,000 400,000 100,0002
Pu-239/240 pCi/g 718,600 574,880 144,000a For DR #1,2 LBGR = AL – 80% of the AL. Avoid overly optimistic estimates of the standard
deviation. Always be conservative when estimating the standard deviation.
40 of 48
Setting the GR Based on ½ of the Action Level
CS
Environmental Variable
DR # Attribute Unit ofMeasure
ActionLevel LBGRa Gray
Region
Co-60 pCi/g 1.4 0.7 0.7
Cs-137 pCi/g 6.2 3.1 3.1
Eu-152 pCi/g 3.3 1.65 1.65
1
Sr-90 pCi/g 4.5 2.25 2.25
Cs-137 pCi/g 500,000 250,000 250,0002
Pu-239/240 pCi/g 718,600 359,300 359,300a For DR #1,2 LBGR = AL - 50% of the AL. Avoid overly optimistic estimates of the
standard deviation. Always be conservative when estimating the standard deviation.
41 of 48
Setting the GR Based on the PDF Method
CS
Environmental Variable
DR # Attribute Unit ofMeasure
ActionLevel LBGR Gray Region
Co-60 pCi/g 1.4 TBD: FAM/DWP TBD: FAM/DWP
Cs-137 pCi/g 6.2 TBD: FAM/DWP TBD: FAM/DWP
Eu-152 pCi/g 3.3 TBD: FAM/DWP TBD: FAM/DWP
1
Sr-90 pCi/g 4.5 TBD: FAM/DWP TBD: FAM/DWP
Cs-137 pCi/g 500,000 TBD: FAM/DWP TBD: FAM/DWP2
Pu-239/240 pCi/g 718,600 TBD: FAM/DWP TBD: FAM/DWP
42 of 48
Decision Error
Tolerances
Bounds of the
Gray Region
Assign probability limits on either side of the gray region
Information IN Actions Information OUT
From Previous Step To Next StepDecision
RulesStep 5
Determine the variability of the environmental variables
Choose the null hypothesis
Identify the decision errors
Specify the boundaries of the gray region
Assign probability values that reflect the decision maker’s tolerable limits for making an incorrect decision.
• At the action level• At the other bound of the gray region• At a point far below the action level• At a point far above the action level
Step 6- Specify Error Tolerances
43 of 48
Site is dirty.Site is clean.
100
True State of Site
Alternative Action
Walk away from site. Clean up site.
75
Probability of deciding
that the site is dirty
0.0
0.5
1.0
Action LevelLower Bound of Gray Region
Typical Curve
Null Hypothesis:
Site is dirty.
44 of 48
Decision Error Consequences CS
Probability of Deciding the Site is Dirty When the Mean is…DR #Far Belowthe Action
Level
At the LBGR At the ActionLevel
Far Above theAction Level
11, 22 0%Error = 0.0
20%Type II (beta)Error = 0.2
95%Type I (alpha)Error = 0.05
100%Error = 0.0
1 Applies to the perimeter side-slope soil around the –20 ft level of the trench footprint.2 Applies to the bottom of the trench footprint after excavation.
45 of 48
Step 6 Summary Define the two types of error
– Incorrectly cleaning a dirty site or– Incorrectly cleaning a clean site
Evaluate severity of the incorrect decisions both below, above, and near the action level
Select the null hypothesis Specify the error rates decision makers are willing
to accept and provide rational for the rates
46 of 48
Establish a LBGR based on one of the four methods shown previously
Step 6 Summary
Provide the basis for selecting the LBGR Remember the closer the LBGR is to the action level,
the more samples are needed More samples can mean real-time measurements or
traditional laboratory measurements Assign probability limits on either side of the gray
region
47 of 48
Decision Error
Tolerances
Bounds of the
Gray Region
Assign probability limits on either side of the gray region
Information IN Actions Information OUT
From Previous Step To Next StepDecision
RulesStep 5
Determine the variability of the environmental variables
Choose the null hypothesis
Identify the decision errors
Specify the boundaries of the gray region
Step 6- Specify Error Tolerances
48 of 48
End of Module 8
Thank you