Date post: | 18-Jan-2018 |
Category: |
Documents |
Upload: | neil-elliott |
View: | 224 times |
Download: | 2 times |
1 of 31
The EPA 7-Step DQO Process
Step 6 - Specify Error Tolerances
60 minutes(15 minute Morning Break)
Presenter: Sebastian Tindall
DQO Training CourseDay 3
Module 15
2 of 31
Step Objective: To specify the decision
makers’ tolerable limits on decision errors, which are used for limiting uncertainty in the data– Since analytical data can only
provide an estimate the true condition of a site, decisions that are based on such data could potentially be in error
Step 6: Specify Error Tolerances
Step 4: Specify Boundaries
Step 2: Identify Decisions
Step 3: Identify Inputs
Step 1: State the Problem
Step 5: Define Decision Rules
Step 6: Specify Error Tolerances
Step 7: Optimize Sample Design
3 of 31
Objectives
To be able to define, for a specific project:1. The variability for each COPC,2. the decision errors, 3. the consequences of the errors, 4. the null hypothesis,5. the error rates (Alpha and Beta) 6. the Lower Bound of the Gray Region & Delta
4 of 31
Decision Error
Tolerances
Bounds of the
Gray Region
Assign probability limits on either side of the gray region
Information IN Actions Information OUT
From Previous Step To Next StepDecision
RulesStep 5
Determine the variability of the environmental variables
Step 6- Specify Error Tolerances
Choose the null hypothesis
Identify the decision errors
Specify the boundaries of the gray region
5 of 31
Decision Error Tolerances The goal of the planning team is to develop
a data collection design that reduces the chance of making a decision error to a tolerable level
Step 6 provides a mechanism for allowing the decision maker to define tolerable limits on the probability of making a decision error
6 of 31
Two Reasons Why Decision Makers Make Decision Errors
Sampling error occurs because the sampling design is unable to capture and control the complete extent of heterogeneity that exists in the true state of the environment
Measurement error occurs because analytical methods and instruments are not absolutely perfect
7 of 31
Decision Error
Tolerances
Bounds of the
Gray Region
Assign probability limits on either side of the gray region
Information IN Actions Information OUT
From Previous Step To Next StepDecision
RulesStep 5
Determine the variability of the environmental variables
Choose the null hypothesis
Identify the decision errors
Specify the boundaries of the gray region
In order to calculate the number of samples needed(in DQO Step 7), an estimate of the population standard deviation is needed for each environmental variable.
• Compile a list of the “driver” COPCs• Use existing data (must pass Step 3 data assessments)• Establish the range based on historical information
– Existing data– Process knowledge– Professional judgment
• Estimate of the population standard deviation– Reference source – Method of calculating
Step 6- Specify Error Tolerances
8 of 31
Decision Error
Tolerances
Bounds of the
Gray Region
Assign probability limits on either side of the gray region
Information IN Actions Information OUT
From Previous Step To Next StepDecision
RulesStep 5
Determine the variability of the environmental variables
Choose the null hypothesis
Identify the decision errors
Specify the boundaries of the gray region
In order to calculate the number of samples needed(in DQO Step 7), an estimate of the population standard deviation is needed for each environmental variable.
• Compile a list of the “driver” COPCs• Use existing data (must pass Step 3 data assessments)• Establish the range based on historical information
– Existing data– Process knowledge– Professional judgment
• Estimate of the population standard deviation– Reference source – Method of calculating
Estimate the standard deviation by using the Deming approach of dividing the range by 2 or 3, depending on the frequency distribution.
Step 6- Specify Error Tolerances
9 of 31
Estimated Standard DeviationsEstimated Standard Deviations for NaI Example
Environmental VariableRange of
EnvironmentalVariable
Estimate of Population Standard DeviationDR#
Attribute Unit ofMeasure
LowerLimit
UpperLimit
StandardDeviation Source How Estimated?
1
True mean ofCs-137estimated by95% UCL ofsample mean
pCi/g 0.031 1.89 0.4052000HPGedata
From 116-H-1 CVP data
1
True mean ofCs-137estimated by95% UCL ofsample mean
cpm(convertedto pCi/g)
-5.17 99.92
7.48or
7.48/2=3.74
1999 to2000 NaIdata
From 116-H-1 CVP data
The choice of an estimate of a standard deviation has a large impact on the number of samples required. Avoid underestimating the standard deviation. Always be conservative when estimating the standard deviation.
10 of 31
Decision Error
Tolerances
Bounds of the
Gray Region
Assign probability limits on either side of the gray region
Information IN Actions Information OUT
From Previous Step To Next StepDecision
RulesStep 5
Determine the variability of the environmental variables
Choose the null hypothesis
Identify the decision errors
Specify the boundaries of the gray regionDefine both types of decision error:Determine which one occurs above and which one occurs below the action level.
Two Types of Decision Error:• Cleaning up a clean site• Walking away from a dirty site
Step 6- Specify Error Tolerances
11 of 31
Decision Error
Tolerances
Bounds of the
Gray Region
Assign probability limits on either side of the gray region
Information IN Actions Information OUT
From Previous Step To Next StepDecision
RulesStep 5
Determine the variability of the environmental variables
Choose the null hypothesis
Identify the decision errors
Specify the boundaries of the gray region
For each Alternative Action:• Create a list of possible decision error(s) that may occur if an action
is incorrectly taken• Discuss the consequences of making each decision error• Rate the severity of the consequences of a decision error
(i.e., low, moderate, severe) at a point:– Far below the action level– Below but near the action level– Above but near the action level– Far above the action level
• Indicate which decision error has the most severe consequencenear the action level
Step 6- Specify Error Tolerances
12 of 31
Decision Error
Tolerances
Bounds of the
Gray Region
Assign probability limits on either side of the gray region
Information IN Actions Information OUT
From Previous Step To Next StepDecision
RulesStep 5
Determine the variability of the environmental variables
Choose the null hypothesis
Identify the decision errors
Specify the boundaries of the gray regionProvide rationale for rating the severity of consequences as low or severe
Step 6- Specify Error Tolerances
13 of 31
Decision Error
Tolerances
Bounds of the
Gray Region
Assign probability limits on either side of the gray region
Information IN Actions Information OUT
From Previous Step To Next StepDecision
RulesStep 5
Determine the variability of the environmental variables
Choose the null hypothesis
Identify the decision errors
Specify the boundaries of the gray regionDefine the null hypothesis (baseline condition) and the alternative hypothesis:The decision error that has the most adverse potential consequences should bedefined as the null hypothesis.
The null hypothesis should state the OPPOSITE of what the project hopes to demonstrate.
• Site is assumed to be contaminated until shown to be clean• Site is assumed to be clean until shown to be contaminated
Step 6- Specify Error Tolerances
14 of 31
Decision Error
Tolerances
Bounds of the
Gray Region
Assign probability limits on either side of the gray region
Information IN Actions Information OUT
From Previous Step To Next StepDecision
RulesStep 5
Determine the variability of the environmental variables
Choose the null hypothesis
Identify the decision errors
Specify the boundaries of the gray region
The gray region is a range of possible parametervalues within which the consequences of a decision errorare relatively minor.
Step 6- Specify Error Tolerances
15 of 31
Decision Error
Tolerances
Bounds of the
Gray Region
Assign probability limits on either side of the gray region
Information IN Actions Information OUT
From Previous Step To Next StepDecision
RulesStep 5
Determine the variability of the environmental variables
Choose the null hypothesis
Identify the decision errors
Specify the boundaries of the gray region
The gray region is bounded on one side by the action level, and on the other side by the parametervalue where the consequences of decision error beginsto be significant. This point is labeled LBGR, whichstands for lower bound of the gray region.
Step 6- Specify Error Tolerances
16 of 31
Decision Error
Tolerances
Bounds of the
Gray Region
Assign probability limits on either side of the gray region
Information IN Actions Information OUT
From Previous Step To Next StepDecision
RulesStep 5
Determine the possible range of the parameter of interest
Choose the null hypothesis.
Identify the decision errors.
Specify the boundaries of the gray region
Determine the variability of the environmental variables
Choose the null hypothesis
Identify the decision errors
It is necessary to specify the gray region because variability in the population and unavoidable imprecision in the measurement system combine to produce variability in the data such that a decision maybe “too close to call” when the true parameter value is very near the action level.
Step 6- Specify Error Tolerances
17 of 31
Width of the Gray Region () :UBGR - LBGR or AL - LBGR
= Analytical + Sampling Error– Estimated based on past data and general knowledge
= 1/2 of the AL – For each COPC, calculate and set LBGR
= 10 - 30% of the AL – For each COPC, calculate and set LBGR
= PDF method– Use PDF for worst COPC to set LBGR
18 of 31
Decision Error
Tolerances
Bounds of the
Gray Region
Assign probability limits on either side of the gray region
Information IN Actions Information OUT
From Previous Step To Next StepDecision
RulesStep 5
Determine the variability of the environmental variables
Choose the null hypothesis
Identify the decision errors
Specify the boundaries of the gray region
Present the rationale of how the LBGR was calculated or determined.
Step 6- Specify Error Tolerances
19 of 31
Lower Bound of the Gray Region
Because the null hypothesis is that the site is contaminated, the upper bound of the gray region is set equal to the action level
The LBGR should be set at a value where the consequences of the decision error begin to be significant
20 of 31
How to Set the LBGR
LBGR set by the Analytical + Sampling Error LBGR set to 1/2 Action Level LBGR set to ~ 50 to 90% of AL (Decision-
Maker “whim”) LBGR set by the Probability Density Function (PDF)
method
UBGR - GR = LBGR
AL - = LBGR
21 of 31
The LBGR is often based on unavoidable variability in the concentration data– The GR may be estimated based on the precision
that the analytical methods allow plus an estimate as to the sampling variance
– LBGR = AL – GR (Analytical + Sampling Error) 100 ppm – (10 ppm + 31 ppm) = 59 ppm
MARSSIM suggests the LBGR be set as:– LBGR = AL – GR (1/2 AL)
100 ppm – 50 ppm = 50 ppm
How to Set the LBGR (cont.)
22 of 31
The LBGR is often set at some other value– This is based on the decision makers’ choice and
is not scientifically based– LBGR = AL – GR (20% of AL);
100 ppm – 20 ppm = 80 ppm
How to Set the LBGR (cont.)
23 of 31
Use the Probability Density Function (PDF) method– The LBGR may be estimated based the Probability Density
Function– Place the Action Level on the mean of the PDF– Ask: “Does a substantial amount of contaminant
concentration values exceed the Action Level?”– Ask: “Is there a reasonably high probability that data from a
small sample size will result in declaring the site clean?”– If “yes” to the first question and “no” to the second question,
begin moving the PDF backwards along thex-axis towards zero concentration
– Pause and ask both questions again– When the answer is “no” to the 1st and “yes” to the 2nd, you
have set the LBGR(e.g., where the mean of the PDF lies on the x-axis is now the LBGR)
• Use probability theory to show this
How to Set the LBGR (cont.)
24 of 31
Methods for Evaluating the Attainment of Cleanup Standards - Volume 1: Soils and Solid Media
EPA, February 1989
PB89-234959
How to set the LBGR
2
121
211
2
5.0
ZAL
ZZn
1 is a hypothetical “true mean concentration where the site should be declared clean with a high probability”. (1 = LBGR)
25 of 31
Show Probability Density Function Distribution
Demonstration
Show VDT File:Moving Gray Region
26 of 31
Decision Error
Tolerances
Bounds of the
Gray Region
Assign probability limits on either side of the gray region
Information IN Actions Information OUT
From Previous Step To Next StepDecision
RulesStep 5
Determine the variability of the environmental variables
Choose the null hypothesis
Identify the decision errors
Specify the boundaries of the gray region
Assign probability values that reflect the decision maker’s tolerable limits for making an incorrect decision.
• At the action level (Alpha error)• At the other bound of the gray region (Beta error)• At a point far below the action level• At a point far above the action level
Step 6- Specify Error Tolerances
Note: EPA QA/G9 recommends that you set both Alpha and Beta error rates to 1% to start.
27 of 31
Site is dirtySite is clean
100
True State of Site
Alternative ActionWalk away from site Clean up site
75
Probability of deciding
that the site is dirty
0.0
0.5
1.0
Lower Bound of Gray Region
Four Decision-MakerError Tolerance
Locations
Null Hypothesis:
Site is dirty.
True mean COPC Concentration
Action Level
The Gray Region
28 of 31
Step 6 Summary
Determine the variability for each COPC Define the two types of error
– Incorrectly walking away from a dirty site, or– Incorrectly cleaning a clean site
Evaluate severity of the incorrect decisions both below, above, and near the action level
Select the null hypothesis
29 of 31
Step 6 Summary Establish a LBGR based on one of the four
methods shown previously Provide the basis for selecting the LBGR Remember the closer the LBGR is to the
action level, the more samples are needed Assign probability limits on either side of the
gray region (Delta)– Specify the error rates (Alpha and Beta) decision
makers are willing to accept and provide rational for the rates
30 of 31
Decision Error
Tolerances
Bounds of the
Gray Region
Assign probability limits on either side of the gray region
Information IN Actions Information OUT
From Previous Step To Next StepDecision
RulesStep 5
Determine the variability of the environmental variables
Choose the null hypothesis
Identify the decision errors
Specify the boundaries of the gray region
Step 6- Specify Error Tolerances
31 of 31
End of Module 15
Thank you
Questions?
We will now take a 15 minute break.Please be back in 15 minutes.