Infectious Disease Testing: QC and Risk ManagementSt. John’s, Newfoundland, CanadaMike Toyoshima, BSMT(ASCP)SC, CLS; [email protected]
2 |
http://www.youtube.com/watch?v=FX30K9alQiU
“…sail in a little bituncharted waters of QC in semi-quantitative assays in immulogy and serology”
What They Do, And Their Goals
4 |
Even though 1/2 are S/CO quantitative and1/2 are qualitative, can use QC principles because qualitative tests have “a number behind it”
Why Their Lab Uses Software
Why They Use Software
• Results for their QC has been ‘online’ for chemistry for a long time but for immunology ‘has been a big improvement’
• Can capture, in a single place, different QC & Levels• Can set reject and warning rules for those combos• Remarks recorded for maintenance, et cetera• Same units between multiple, same instruments• Comparison of their values with peers, using same units• Able to set analytical goals, different for each test• Documentation and reports available (review and inspections)
What Rules Do They Use?
• Use 1-3s, 7Trend for most tests• 1-2s or 2-2s for some others• ‘No benefit to use sophisticated rules’• Capture all data for retrospective review/Inspections
Special Aspects of QC in Serology
• Existence of negative values; two measurements, QC below cutoff
• Definition of cutoff values, leading to false pos/neg values, which are different from just a number reported to clinician.
• Lack of primary standards, leading to variation in values• Use of arbitrary units by manufacturers = no comparison• No specifications for biological variation
• Example: If cutoff for CMV IgG is 7 IU/L, patient’s first visit value is 10 and subsequent visit is 15, is this value reflective of re-infection, reactivation or just biological variation?
Lack of Standardization
• Technical and cost issues (cross-reactivity, donors)• Regional variation of antigens• Some kits use old standards• Standards should use relevant isotypes – IgM, IgA• No standardization across platforms• Though molecular testing was proposed as a ‘standard’ for
serological testing, test values are not interchangeable
Different Control Materials Necessary
• In-Kit Controls• 3rd Party Commercial• If values of above are far from medical decision level, may
need to use Pooled patient sera, with drawbacks• Specific ‘important’ sera or seroconversion panels
So, no single standard, no reference analyzer….should one judge by most prevalent instrument/kit mean or compare to all methods mean?
Quantitative Values, Yet Qualitative Report
Another Example, Values & Interpretation
EQAS sample results forPertussis IgM: 2+, 2_!
Serological Testing and Diagnosis
ID from culture or +PCR for diagnosis
Reagent variation, day to day or by lot, can affect patient outcome
Patient positives, two reagent lots, over time
Using patient pop. values:
1650=20%
1914=10%
She indicated that variation could be due to the season
QC Materials Matrix
Mira’s Conclusion:
“….quality control in immuno-nology-serology remains a bit of art and not so science.”
http://www.youtube.com/watch?v=A7tWVUzmOic
“Risk Management”
Not QC, but Risk Management in ID Testing
QC is based on risk in your lab, your context…If averse to risk, may miss opportunities
Steps in Risk Management
Risk Management (Potential of wrong/delayed/no result)• Identification (What)• Analysis (Probability/likelihood, Impact, Proximity/Time)• Risk Response (Actions – corrective or preventative)• Monitoring Actions (Complaints, repeat tests, et cetera)
Risk should be documented in the form of:• Condition: “There is a risk that….”• Cause: “It is caused by….”• Consequence: “That may result in….”• Context: “….”(for patient population affected)
Fishbone Chart, Testing
What parts of the testing conditions may result in anincorrect test result?
Analytical Phase
Flow of testing, Screen to Confirmation
Close S/CO Ranges for Interpretation
A quantitative value, S/CO, is interpretedas NEG, IND, or POS
Since NEG/POS values are so close,if avoidance of false + important, more QC at lower level
Sample Outcomes After Confirmation
Green = confirmed POSRed = confirmed NEGYellow = confirmation unknown
Savings by Eliminating Confirmation?
To save cost, may simply report initial > 11 as positive,but risk is that a few may actually be negative
Why False Positive Might be ‘Better’
To reduce risk, may run QC in the 11 range, to ensure thatis correct; may compare with other labs to verify that >11 = POS
False pos less damaging, as confirmation follows
Antibody levels may vary over time
Case Study #1: Same Patient, Variable Results Over Time
Same kit, same patient, over timeread as POS, NEG, IND
Tough, because hematology wanted CMV IgG status for transplantation
What Happened?
What Happened?
IgG after infection
IgG level waned
after IgGtransfusion
IgG level waned
‘A man with two watches doesn’t knowhow late it is’ = single point measurement unreliable
Case Study #2, Variable Patient Results
Identification of condition: Patients appeared to changeCMV Ab status, lab conclusion = serostatus indeterminate
Note: Hematology worked closely with clinicians -phone calls daily, weekly meetings
What They Discovered in the Process
CAUSE: •Donor samples were submitted to the lab under the ID#of the patient that was to receive unit•CMV IgG results went directly into patient registration database•Latest results overrode previous ones•Though note on reports indicated ‘Indeterminate Results’, clinicians did not read it
What Actions Were Taken
Consequence/Actions:• Went to a patient ID system that differentiated from donors• Disconnected PID from registration system• For ‘indeterminate’ results, lab talks directly to patient’s clinician;in an academic setting, this can be difficult; note of this in chart• In a general clinical lab, single result seen, no serial values=risk
Risk Assessment May Be Subjective
Risk analysis affected by emotions,
Dependent on test situation
“We never make mistakes in the lab” (analytical phase, shown to have low error component rate in studies)
A More Objective Way to Assess Risk
Talk With All Personnel to Determine Impactand Probability
Objectively balances‘quality, price, time’
Impact doubles, solow probability canstill be a ‘red’situation
Tools For Troubleshooting St. John’s, Newfoundland, CanadaMike Toyoshima, BSMT(ASCP)SC, CLS; [email protected]
Tools For Troubleshooting Out of Control Conditions
• How Do You Detect These?• Immediately - QC Out of Control Rule Violations• Upon review of QC Reports
68.0%95.5%99.7%
Arriving at Decision Limits
s = SD = variability
§99.7%
Gaussian Distribution – the Key
68%95.5%99.7%If +/- 1S ~ 2/3 in; 1/3 out
If +/- 2S ~ 19/20 in; 1/20 outIf +/- 3S ~ 99.7/100 in; 0.3/100 = 3/1000 ~ 1/333 out - rare
Troubleshooting and Westgard Rules, Recap
• 12S: Warning, initiate further testing• 13S: Rejection, large RE, or SE start• 22S: Rejection, SE• R4S: Rejection, RE• 41S: Warning, SE• 10X: Warning, SE
Immediate QC Failure, Troubleshooting Information
• Identify source of error• Consider if other analytes affected• Localized to a particular incubator, wavelength,
reagent/sample dispense • Identified and documented from a previous service or
problem log?
Immediate QC Failure, Random Error Investigation
• Look at charts to see if trending or one time• Consider if values out have been deleted• Related to a change in test conditions?
• Do a quick precision check• Compare with within run imprecision from IFU• Single occasion outlier?
Immediate QC Failure, Systematic Error Investigation
• Look at charts to see if trending or one time• Consider if values out have been deleted• Related to a change in test conditions?
• Verify the bias• Alternate Lot of QC material with values? • Proficiency material, thawed• Alternate Calibrator lot, not what you used previously• If related to new cal or rgt, repeat patient samples• CALL your QC program!
Immediate QC Failure, Systematic Error Investigation
• If mechanical/instrument related• Consider if other analytes affected (sample size, UV)• Electrical (too low or varied? Line voltage conditioner)
• Verify the bias• Alternate Lot of QC material with values? • Proficiency material, thawed• Alternate Calibrator lot, not what you used previously• If related to new cal or rgt lot, repeat patient samples
QC Report Interpretation,CVR and SDI to Troubleshoot
• CVR - Indicator of Random Error / Imprecision• Lab CV / Peer CV• Ideal CVR is </= 1.0
• SDI - Indicator of Systematic Error / Bias• [Lab Mean – Peer Mean] / Peer 1 SD• 3 variables = 3 sources of error• Ideal SDI is 0.0
Investigating• Look at your CV – compare month vs cumulative.
Are they the same or increased? Before / After:Lab CV / Peer CV = 5.0% / 5.0% = 1.0 CVR BEFORELab CV / Peer CV = 10.0% / 5.0% = 2.0 CVR AFTERLab CV / Peer CV = 5.0% / 2.5% = 2.0 CVR AFTER
• Did your CV increase or your peer decrease?
Increased CVR on Your Report
Possible Reasons for INCREASED CVR:• Instrument / Test System Example, Coag:
• Mechanical• Electrical
• New Reagent Formulation / New Calibrator Lot• Matrix Effects – mix of old and new QC
during a single month
Elevated CVR, Sources
Investigating an INCREASED CVR:• Look at your QC – any other related analytes high?• Mechanical• Weak pump tubing on MLA• Leaking sample syringe on small volume tests
• Electrical • Weakening UV bulb; light output affects analyte value
Increased CV, Mechanical or Electrical
Why does this INCREASE CVR?• If the change is small = no rejections, but monthly CV
is a mix of old & new values, may increase CV for crossover month
• If change is larger, and some/many values are rejected in your lab, again CVR will be/may be increased, again for crossover month
• What happens if other labs are switching too?
Increased CV, New Reagent or Calibrator Lot
Investigating • Consider your monthly versus cumulative CV. Same
or greater?• If the same, did your peer group decrease?• What happens to CVR if other labs are switching
too? What happens if over different times depending on distribution/release of new formulation or calibrator?
Increased CVR, New Reagent or Calibrator Lot
Investigating• Look at your CV – compare month vs cumulative. Is
it the same or decreased? Before / After:Lab CV / Peer CV = 5.0% / 5.0% = 1.0 CVR BEFORELab CV / Peer CV = 2.5% / 5.0% = 0.5 CVR AFTERLab CV / Peer CV = 5.0% / 10.0% = 0.5 CVR AFTER
• Did your CV decrease or your peer increase?• If your peer increased, how is this for other methods?
So, if your lab uses EIA, how about Immunoturbidimeric?
Decreased CVR on Your Report
Investigating • Consider your monthly versus cumulative CV. Did
you improve?• If the same, peer group got larger• Why would this happen?
Decreased CVR, Causes
Possible Causes & Call to QC Program• Peer labs may be submitting wrong level values or
have incorrect unit coded (Calcium, 8.6 mg/dL = 2.5 mmol/L)
• Mix of old & new reagent/calibrator values• For elevated peer CV, determine if limited to
particular lab or instrument family• Check another QC lot to see if group CV elevated
Increased Peer CV
Internal QC Data (ID=994) — One-Way ANOVA Plot
M T W T F M T W T F M T W T F M T W T F M T W T FWeek 1 Week 2 Week 3 Week 4 Week 5
1820
2224
2628
3032
X L
X122
3 Fe
rritin
, ng/
mL
-3-2
-10
+1+2
+3
Your lab’s CV
Peer CV
One Lab’s CV vs Peer CV
Possible Causes• Instrument or test system• Reagent Lot Change, Reformulation• Calibrator Lot Change with New Values
Increased SDI
Instrument / test system, Mechanical or Electrical• Look for changes in:• QC, Repeat Proficiencies, Alternate lots of Calibrator• Repeated patient samples
• Improve by:• Maintenance or repair• Decreasing electrical or environmental effect
Increased SDI
Calibrator Value Reassignment• Look for changes in:• QC, Repeat Proficiencies, Alternate lots of Calibrator• Repeated patient samples
• Reconcile by:• Change in reference interval (normal range)• Change in therapeutic range
Increased SDI
Reagent Reformulation/Matrix Effects• Look for changes in:• QC, Repeat Proficiencies • Repeat of stored proficiencies• Alternate lots of calibrator, run as patients• Repeated patient samples not affected
• Reconcile by:• Contacting QC Program to see if new code available, use it to segregate ‘different’ QC values
Increased SDI
• Lab called, L1 QC data rejected, ALT, SieDim Vista• Using new, calibrated kit, ALTI, as of 6/12 data• Their lab getting ALTI values of 28 / 183 U/L• Using Unassayed Multiqual, no values
• Steps to investigate• Look at insert, but of ASSAYED Product• Look at peer group data • Call QCProgram to view data & contact labs
Rejected Data Investigation (RDI)
• Look at control insert, Assayed control for hints:•
RDI – Control Insert Info from Different Lot
ALT, UV with P5P, L1/L3 = 28.4 / 197 U/LALTI, UV with P5P, L1/L3 = 26.7 / 191 U/L, IFCC Ref Procedure, Calibr.ALT, UV with P5P, L1/L3 = 21.5 / 180 U/L, Corr Factors to ~ IFCC Ref Proc
• Manufacturers Report, Dade (old name)•
RDI – Peer Group Data
If labs run 2 levels per day, and labs run 1 & 3, why so few L1?
• Manufacturers Report, Dade (old name)•
RDI – So, What Happened?
Preliminary 8/12 data, now L1/L3 = 27/184 U/L!
• Manufacturers Notification:•
Help From Manufacturer
• Manufacturer’s Notification, excerpts:•
Help From Manufacturer 2
• Looking at values:• UV with P5P group, Dedicated, L1 = 28.4 U/L• Applying correlation factors: (28.4 x 0.949) – 5.033 = 21.9
• QC Program actions:• Contacted labs with low values, OUS• Verified they were using correlation factors• Advised to change reagent code to Factored
RDI – Resolution
• Unity Bulletin Item for QC submissions:
••
RDI – Resolution 2
Coding Example & A Question
1. Instrument model breakout – Siemens Dimension RxL and Xpand, e.g. Urine Cl, UN, Creatinine
2. Can an SDI be too small?
Internal QC Data (ID=994) — One-Way ANOVA Plot
M T W T F M T W T F M T W T F M T W T F M T W T FWeek 1 Week 2 Week 3 Week 4 Week 5
1820
2224
2628
3032
X L
X122
3 Fe
rritin
, ng/
mL
-3-2
-10
+1+2
+3
Your lab’s CV
IncreasedPeer CV, larger peer SD
Same Slide, New InterpretationSince SDI =[Lab Mean – Peer Mean] / Peer 1 SD, IF peer SD larger, lab SDI gets SMALLER!
• Linear Regression for Method Comparison
• Hints from EP9-A2•
EP9A – Linear Regression
Accuracy Testing - CLIA
• Interpretive Guidelines: 493.1253(b)(1)(i)(A):
• “Lab is responsible for verifying that the method produces correct results. Verification of accuracy may be accomplished by:”
• Testing reference materials;• Comparing lab test values versus a reference
method;• Comparing split sample results obtained from a method
with clinically valid results
• CHM.13800 Phase II
• If the laboratory uses more than one instrument to test for a given analyte, the instruments are checked against each other at least twice a year for correlation of patient/client results.
•NOTE: This requirement applies to tests performed on the same or different instrument makes/models or by different methods. This comparison must include all nonwaived instruments/methods.
•CAP, Revised 6/15/09
Linear Regression - Uses
• Study of Two Methods to Describe their Relationship
• Compare new method to old• Compare two instruments• Compare new and old reagent formulations
• Two, New Errors – Constant & Proportional
Linear Regression - Uses
Linear Regression• Constant Error
Caused by Interferent. Affects accuracy (bias) throughout range, seen as change in Y-Intercept
0
250
500
750
1000
0 250 500 750 1000
EXPECTED VALUES
ACTU
AL
VA
LUE
S
}Constant Error
Actual results
Line of identitySlope, m = 1.00
Look for: Parallel lines, but one higher
Linear Regression• Proportional Error
Varies the Y value as the concentration of X increases. The results are seen as a change in the slope from 1.00
0
250
500
750
1000
0 250 500 750 1000
EXPECTED VALUES
ACTU
AL
VAL
UES
Line of identitySlope, m = 1.00
Actual results
}Proportional
Error
Look for: Change in slope of the line
Slope = 1.10,
10% higher
Positive Proportional Error
Slope = 0.90,
10% lower
Negative Proportional Error
Linear Regression –Comparison Studies
• Perform the ComparisonsOld versus New, Manual versus Automated• Population sample size should be at least 40• Split specimens, in duplicate• Distribution of sample values – at least 50% should
be outside the normal (reference) range• Use fresh specimens, when possible• Test in a narrow time window, daily• Extended testing period, minimum 5 days
Linear Regression
• Assumptions:
The original method is assumed to be precise and accurate, free from interference. So, variation from the line of identity due to proportional or constant error is assumed to be due to the new method
Linear Regression• X Axis
Old, Reference, “A”• Y Axis
New, Comparative, “B”
Old, Reference
New
, Com
para
tive
• Test Yields: Line equation Y = mX + bCorrelation N, number ofcoefficient, r tested pairsSy/x
Linear Regression
Linear Regression – 2 Tips• >50% points outside Reference Range
• All points same weight
Linear Regression –What You’ll See
Y = mX + b ErrorY = 1.00(X) + 0. Y = X, identical (None)Y = 1.05(X) + 0. Y is 5% > X (Proportional only)Y = 1.00(X) + 5. Y = X + 5 (Constant only)
• r = correlation between methods; 1.00 is best• S y/x = random error between methods; smaller is
better• Least squares regression
• Test Yields: Line equation Y = mX + bCorrelation N, number ofcoefficient, r tested pairsSy/x
Best
Fit
Residual
Linear Regression –“Least Squares”, Here’s Why
• Test Yields: Line equation Y = mX + bCorrelation N, number ofcoefficient, r tested pairsSy/x
Best
Fit
Residual
Deming Regression –Both Systems might have error
Consider Deming Regression
Residual is perpendicular to the line!
Method Comparison – Bias Plot
Method Comparison – Bias Plot
X axis the same; Y axis is the difference between Y and X
Y - X
X
Method Comparison – Bias Plot
Fluke, or more points needed at this concentration?
Linear Regression
Courtesy of Paul Durham,
Siemens (formerly DPC)
Popular Kit
% Change?
Higher or lower?
Linear Regression
‘Reference Method’
% Change?
Higher or lower?
Linear Regression
New vs Now
Look for this
% Change?
Higher or lower?
Linear Regression
Linear Regression Tips
• Suggestions:
• Adequate Number of Samples• Appropriate Range of Values• Watch for Outliers• Tested Range Must Fit Your Patients• CALL with any questions!
Linear Regression – Case StudyEstradiol II
• Initial notification – reformulation with:Improved Low End Precision Improved Low-End Sensitivity
• Crossover instructions:1. Run samples using original reagent2. Remove existing E2 assay from disk3. Install new reagent, recalibrate4. Analyze same previous samples
SLOPE = 1.00; What % proportional error?
Original Regression Statistics
• Estradiol II, Linear Regression• E2 II versus E2 1st Generation (from Manufacturer): • Y = 1.001(E2) – 10.25; R=0.981; N=120• Range (pg/mL): 0 – 600
• If E2 = 100, then Y = 1.001(100) - 10.25= 89.9• If E2 = 600, then Y = 1.001(600) - 10.25 = 590
• My lab called…..so I called Manufacturer
Manufacturer’s Supportive Data –Initial Linear Regression
1.1836X
Positive or Negative?
What % change?
Revised Linear Regression Data
All points have the same weight – why was it reformulated?
Original Regression, Revisited
See points >300
Remember the
CLSI suggestion:
At least 50% points outside ‘normal’?
Recall all points have the same ‘weight’
Original Regression Statistics
Remember My Tech’s Complaint?
What may really happen
Our Level 3
Consider Constant andProportional Errors
Linear Regression Tips
• Conclusions:• Systems must be comparable• Look for exact new vs. old regression• Compare reported change with yours• Compare well? Experiment successful• Don’t compare well? CALL Manufacturer. Ask for
revised regression data
If y = 0.94x, what percent change is this? Higher or lower than X?
Linear Regression Quiz
Linear Regression Update
New Dimension ALTI, DF143
Linear Regression Update
Linear Regression Update
Linear Regression Update
Important:Linear regressionsdone using humansamples; QC may differ
Linear Regression Update
All lots now FC2083+
•A graphic representation of the probability of rejection versus increasing error.
•Two types, one for RE, one for SE
• Program Variables• Number of control levels• Magnitude of error, SE or RE• Number of runs
Power Function Graphs andQuality Control Planning
Power Function Graph (SE)
0.00.10.20.30.40.50.60.70.80.91.0
0.00 1.00 2.00 3.00 4.00
Systematic Error (ΔSE, multiples of s)
Prob
abili
ty fo
r Rej
ectio
n (P
)
Probability of rejecting runs having systematic errors when using multi-rule procedures with N of 4 .
13S/22S/R4S/41S/10X 4 3
13S/22S/R4S /41S 4 1
N R
SDI
Power Function Graphs, In General
Power Function Graph (SE)
0.00.10.20.30.40.50.60.70.80.91.0
0.00 1.00 2.00 3.00 4.00
Systematic Error (ΔSE, multiples of s)
Prob
abili
ty fo
r Rej
ectio
n (P
)
Probability of rejecting runs having systematic errors when using multi-rule procedures with N of 4 .
13S/22S/R4S/41S/10X 4 3
13S/22S/R4S /41S 4 1
N R
SDI
Increasing Error
Increasing Rejection
Power Function Graphs, SE
Power Function Graph (RE)
0.00.10.20.30.40.50.60.70.80.91.0
1.00 2.00 3.00 4.00
Random Error (ΔRE, multiples of s)
Prob
abili
ty fo
r Rej
ectio
n (P
)
Probability of rejecting runs having random errors when using multi-rule
13S/22S/R4S 4 1
13S/22S/R4S 2 1
N R
CVR
Increasing Rejection
Increasing Error
Power Function Graphs, RE
• 12S, SEWhen SE = 0, RE = 1.0 and N = 1, there is a 5%chance of rejection when the system is in a steady state Power Function Graph (SE)
0.00.10.20.30.40.50.60.70.80.91.0
0.00 1.00 2.00 3.00 4.00
Systematic Error (ΔSE, multiples of s)
Prob
abili
ty fo
r R
ejec
tion
(P)
12S 4 112S 3 112S 2 112S 1 1
N R
When N=2 or 4, the probability of rejection rises to 9% and 19%, respectively.
What can be said about the 1-2s?Power Function Graphs, SE
• 12S, SEWhen SE = 0, RE = 1.0 and N = 1, there is a 5%chance of rejection when the system is in a steady state Power Function Graph (SE)
0.00.10.20.30.40.50.60.70.80.91.0
0.00 1.00 2.00 3.00 4.00
Systematic Error (ΔSE, multiples of s)
Prob
abili
ty fo
r R
ejec
tion
(P)
12S 4 112S 3 112S 2 112S 1 1
N R
When N=2 or 4, the probability of rejection rises to 9% and 19%, respectively.
It is sensitive, up to 20% false rejections!Power Function Graphs, SE
• 12S, REWhen SE = 0, RE = 1.0 and N = 1, there is a 5% chance of rejection when the system is in a steady state
Power Function Graph (RE)
0.00.10.20.30.40.50.60.70.80.91.0
1.00 2.00 3.00 4.00
Random Error (ΔRE, multiples of s)
Prob
abilit
y fo
r Rej
ectio
n (P
)
12S 4 112S 3 112S 2 112S 1 1
N R
When N=2 or 4, the the probability of rejection rises to 9% and 19%,
What can be said about the 1-2s RE?
Power Function Graphs, RE
• 12S, REWhen SE = 0, RE = 1.0 and N = 1, there is a 5% chance of rejection when the system is in a steady state
Power Function Graph (RE)
0.00.10.20.30.40.50.60.70.80.91.0
1.00 2.00 3.00 4.00
Random Error (ΔRE, multiples of s)
Prob
abilit
y fo
r Rej
ectio
n (P
)
12S 4 112S 3 112S 2 112S 1 1
N R
When N=2 or 4, the the probability of rejection rises to 9% and 19%,
Same Pfr so cannot distinguish RE from SE with this rule
QUESTION: If there is a 5% chance of rejection when the system is in a steady state for a single level, ~ 9% with a bilevel control, what does the Tech see when running a daily QC panel of >30 analytes on a chemistry instrument’s menu? What happens if the QC > 60 analytes?
Another look at false rejections with 1-2s….
Power Function Graphs
13S, SEWith SE = 0, RE = 1.0 and N = 1, the probability of rejection is low. Not sensitive to false error detection
Power Function Graph (SE)
0.00.10.20.30.40.50.60.70.80.91.0
0.00 1.00 2.00 3.00 4.00
Systematic Error (ΔSE, multiples of s)
Prob
abili
ty fo
r Rej
ectio
n (P
)
13S 8 113S 6 113S 4 113S 3 113S 2 113S 1 1
N R
Power Function Graphs, SE
13S, REWith SE = 0, RE = 1.0 and N = 1, the probability of
rejection is low. Not sensitive to false error detection
Power Function Graph (RE)
0.00.10.20.30.40.50.60.70.80.91.0
1.00 2.00 3.00 4.00
Random Error (ΔRE, multiples of s)
Prob
abili
ty fo
r Rej
ectio
n (P
)
13S 8 113S 6 113S 4 113S 3 113S 2 113S 1 1
N R
Power Function Graphs, RE
Comparison 12s vs 13s
Is 13s always better than 12s? Not if you must detect error where the (red) arrow is
12s
13s
Multirule, SEIf N is “fixed,” then the rejection rate increases with the number of runs on SE WHY?
Power Function Graph (SE)
0.00.10.20.30.40.50.60.70.80.91.0
0.00 1.00 2.00 3.00 4.00
Systematic Error (ΔSE, multiples of s)
Prob
abili
ty fo
r Rej
ectio
n (P
)
13S/22S/R4S/41S/10X 4 313S/22S/R4S/41S/10X 2 513S/22S/R4S/41S 4 113S/22S/R4S/41S 2 213S/22S/R4S 2 1
N R
Power Function Graphs, SE
Multirule, REWith RE graphs the rate increases with N
(levels of control) – WHY?Power Function Graph (RE)
0.00.10.20.30.40.50.60.70.80.91.0
1.00 2.00 3.00 4.00
Random Error (ΔRE, multiples of s)
Prob
abili
ty fo
r Rej
ectio
n (P
)
13S/22S/R4S/41S 4 1
13S/22S/R4S 2 1
N R
Power Function Graphs, RE
• Repeat the Control• If out 1 out of 20, then in 19 out of 20, right? • If rules chosen properly, less need to repeat• See next section
• Open a New Bottle of Control• Proper preparation & storage – training• Expensive
• Recalibrate• Introduces bias, may mask other problems
From “QC - The Out of Control Problem”, Elsa Quam, Westgard.com
Bad Habits of Quality Control
• View your charts for patterns• Relate the error pattern to possible causes
• Consider common causes on multichannel analyzers (same filter, small sample, etc)
• Relate the problem to recent changes• Verify the solution then document the remedy• Regularly review your quality system
From “Unity Real Time Reference Guide for Expert QC Data Management”
Good Habits of Quality Control
Tools For QC Planning St. John’s, Newfoundland, CanadaMike Toyoshima, BSMT(ASCP)SC, CLS; [email protected]
Tools For QC Planning –Analytical Goals
• Imprecision, using Biological Variation• Used to set the range if you provide the mean value• Uses published BV information• You select Minimum, Desirable, Optimal • You provide mean based on history or enter manually
Performance Goals-Definitions
• Optimum• Tests for which desirable performance limits may
be considered too liberal or easily obtained• Desirable
• Most widely used and generally accepted quality specification
• Minimum• Tests for which technology is not yet able to
achieve desirable limits
Controlling laboratory imprecision using biological variation: formulas for possible targets (choices) based on a performance goal selection
Optimum: CVA < 0.25 CVw
Desirable: CVA < 0.50 CVw
Minimum: CVA < 0.75 CVw
Better performance expected
Imprecision and BV, Use CVw (or CVi)
Tools For QC Planning –Analytical Goals
• Imprecision, using Biological Variation
Controlling laboratory bias using biological variation: formulas for possible targets (choices) for analytical bias
Optimum: BAA< 0.125 (CVw2 + CVb
2)1/2
Desirable: BAA<0 .250 (CVw2 + CVb
2)1/2
Minimum: BAA< 0.375 (CVw2 + CVb
2)1/2
Better performance expected
Bias and BV, Uses CVw (or CVi) & CVb (or CVg)
Definition of Total Error based on biological variation
• TE p<.01= (imprecision BV target)(2.33)+|bias BV target|• the imprecision BV target for the test• some fraction of within-subject biological variation
• the fraction is determined by choice of minimum, optimum or desirable performance
• multiplied by 2.33• plus the bias BV target for the test• some fraction of within-subject and between-subject
biological variation
KEY TOOL – Table of Area Under a Normal Curve
*2.33 = 0.01 = 1% probability of values outside of limit = 99% probability ‘in’
Z-value Area
0.00 0.5000000.50 0.3085381.00 0.1586551.50 0.0668072.00 0.0227502.50 0.0062103.00 0.0013503.50 0.000233
1.65 = 0.05 = 5%
Tools For QC Planning –Analytical Goals
• Total Error, Using Biological Variation• Based on published ‘within-subject’ and ‘between-
subject’ data• Used to set the range, using your bias & imprecision • Uses published BV information• You provide Min, Des, Optimal • You provide the TEa specification
Tools For QC Planning –Analytical Goals
• Total Error Using Biological Variation
Tools For QC Planning –Analytical Goals
• State of the Art• Based comparator group imprecision data
• Peer• Measuring Method• All labs • Or, you may manually enter the %CV
• Used to set the range if you select the mean • 30 day,60 day or cumulative• You may manually enter the mean
Tools For QC Planning –Analytical Goals
• State of the Art• Targets lab performance to the imprecision of a
comparator group – Instrument, Method, or All Labs• Review numbers of labs in each comparator group –
larger numbers at the peer level best
Tools For QC Planning –Analytical Goals
• State of the Art• Using your mean, apply CV of Peers, Method, or All Labs to
obtain range:• Lab mean = 100 mg/dL, Peer CV = 3%• Then, 1 SD = 3 and 2SD range is 94 – 106• Lab mean = 100 mg/dL, Method CV = 5%• Then, 1 SD = 5 and 2SD range is 90 – 110• Lab mean = 100 mg/dL, All Labs CV = 7%• Then, 1 SD = 7 and 2SD range is 86 - 114
Tools For QC Planning –Analytical Goals
• Imprecision Using State of the Art
Can select Peer, Method, or All
Can select period of comparator data
Tools For QC Planning –Analytical Goals
• Medical Relevance• Amount of error that would cause a clinician to change
a patient’s diagnosis, prognosis or therapy• Use to differentiate a statistical from a medically
important change • Must be determined by laboratory director• Expressed as percent or absolute values
Medical Relevance
• How Does It Work?• Sets a limit based on medical importance,
determined clinically• What Does it Mean?• A tool to help assess overall (total) error• Alert when TE exceeds limits; useful for tests where
statistical limits are more sensitive than medical requirements
Medical Relevance - Application
1. Take target value from lab’s 30 or 60 day rolling, OR cumulative mean
2. Set a Decision Limit, concentration or %3. Develop a range, example:
Lab mean = 100 mg/dL; Decision Level = 10%Range = 90-110 mg/dL
Medical Relevance – Sample Calculation
• Example: Potassium• Set a Decision Limit, clinically:
e.g. Absolute value = +/- 0.5 mmol/L• Develop a range
Lab mean = 5.0 mmol/L, thenRange = 4.5 – 5.5 mmol/L by MR
Medical Relevance – Sample Calculation & Comparison
Potassium example, continued:• Normally derived QC: 1 SD = 0.06 mmol/L. Then, 3 SD = 0.06 x 3 = 0.18 and range ~ 5.0 +/- 0.2 = 4.8 – 5.2 mmol/L• Using MR, from previous slide range = 4.5 – 5.5 mmol/L• Traditional QC narrower than that obtained by Medical Relevance, even though +/- 3 SD!
Example TEa (= TAE) for Glucose
Compare TEa for Glucose
• From BV, using 5% probability, TEa = 7.9%• At a mean value of 200 mg/dL, Limit +/- 16
• From CLIA TEa, limit is 6 mg/dL or +/- 10%, whichever is greater. Using 10% and a mean value of 200 mg/dL, Limit +/- 20
Article: Managing Quality in Networked Laboratories…Westgard QC 1/2012
• Conditions Within the Network:• 9 Siemens ADVIA Chemistry models• 7 Siemens Centaur XP• Initial Rule, 1-2s, N = 2• Chemistry run hourly• Immunoassay, 3 times / day
Testing and Evaluation Conditions
• Quality Requirements for 71 Tests• Blood and Urine Chemistries• Defined at clinical decision concentrations• In terms of Total Allowable Error (TEa), CVa (allowable
imprecision), Ba (allowable Bias)• Specs determined by Biological Variation (46), Pharmaco-
kinetics (9), and Expert/EQA (16)
Here’s What They Found
• QC Combinations:• 35 of 71 tests controlled by a single rule• 30 of those tests used BV for their TEa• 32 analytes now at 1-2.5s or 1-3s, N = 2• 8 of 71 had 1-3s/2-2s/R-4s/4-1s, N = 4• For 11 of 71, 8 mean rule added • 14 of 71 needed site-specific rules
• Establish comparability of laboratory QC results with other laboratories using the same method = evaluation of bias and imprecision
• Evaluate long term trends (if present) within a given method, instrument, reagent or control
• Detect performance changes in instruments/reagents
• Become Aware of “between - method variations”
• Educational aspects• Trace the effects of corrective actions• Access peer group data, educational material, discussion
forums, and product inserts via the Internet
Benefits of Daily QC and Interlaboratory Comparison