Post on 30-Dec-2015
transcript
Statistics & graphics for the laboratory 1
Content overview
Interactive part• Factors that influence the interpretation of a method comparison: qualitative
use of a difference (Bland & Altman) plotFinal remark
• When to use regression-based interpretation
Exercises• Case studies 1 - 5
Content
Statistics & graphics for the laboratory 2
Factors that influence the interpretation of a method comparison
Qualitative use of a difference (Bland & Altman) plot
How it works – The plot & the task
How it works – Example
Interactive part
PASSED!PASSED!
FAIL!FAIL!
DOUBT!DOUBT!
FOCUS: Total ErrorDecisions by use of the percentage of differences outside specifications for total error (TEspec):–Passed: 5% "Out" (95% "In": within ±TEspec)–Fail: >5% "Out" (<95% "In")–Doubt: no decision can be made: too close to the 5% criterium
Decision criterium Equals for pure random error"5% out" ["95% in"] 1.96 SDDifferences ≤TEspec
>Normal distribution: 95% of values are within ±1.96
-15
-10
-5
0
5
10
15
0 5 10 15 20 25
Reference (mmol/l)
Ro
uti
ne
(%
dif
fere
nc
e) n = x Total # of points
TEspec ± 6.3%
• Task –Count # of points "out"–Convert them into % of total–Decide (5% criterium): Pass, Fail, Doubt
• Plot
"Out"
-15
-10
-5
0
5
10
15
0 5 10 15 20 25
Reference (mmol/l)
Ro
uti
ne
(%
dif
fere
nc
e)
Decision criterium"5% out" ["95% in"]
n = 40 Total # of points
Total error limit ± 6.3%"1 Out"5%!
PASSED!PASSED!
Statistics & graphics for the laboratory 3
Glucose "biological criteria": Maximum deviation: 6.3%
Glucose “CLIA criteria”: Maximum deviation: 0.33 mmol/l or 10%
Glucose "Glucometer-criteria": Maximum deviation = 20%
1st observationData were the same for all Specifications were 6.3, 10, 20%
May depend on specification!
Interactive part
PASSED!PASSED!
FAIL!FAIL!
Decision………………………………………………………………………
Decision………………………………………………………………………
Decision………………………………………………………………………
Factors that influence the interpretation of a method comparison
-20
-15
-10
-5
0
5
10
15
20
0 5 10 15 20 25
Reference (mmol/l)
Ro
utin
e (
% d
iffe
ren
ce
) n = 50
-20
-15
-10
-5
0
5
10
15
20
0 5 10 15 20 25
Reference (mmol/l)
Ro
utin
e (
% d
iffe
ren
ce
) n = 50
-3
-2
-1
0
1
2
3
0 5 10 15 20 25
Reference (mmol/l)
Ro
uti
ne
(a
bs
. dif
fere
nc
e)
n = 50
-3
-2
-1
0
1
2
3
0 5 10 15 20 25
Reference (mmol/l)
Ro
uti
ne
(a
bs
. dif
fere
nc
e)
n = 50
-25-20-15-10
-505
10152025
0 5 10 15 20 25
Reference (mmol/l)
Ro
uti
ne
(%
dif
fere
nc
e)
n = 50
-25-20-15-10
-505
10152025
0 5 10 15 20 25
Reference (mmol/l)
Ro
uti
ne
(%
dif
fere
nc
e)
n = 50
Statistics & graphics for the laboratory 4
The quality of the comparison method
Cave: Total variance of a method comparison!CVTot = SQRT[CV2
Ref + CV2Rout]
Specifications are for CVRef = 0! Otherwise: Expand specifications!
Errors in the comparison methodExpand specifications!
Start specification: S (10.4%)
The comparison method has a bias B (2%)• The new specification becomes:
Snew = S + B (10.4 + 2 = 12.4%)
The comparison method has an imprecision (2%) that cannot be neglected in comparison to the specification (understood as "2 SD"-limit)
• The new specification becomes:Snew = 2 • [(S/2)2 + CVComp
2]0.5
(2•[5.2•5.2+2•2] = 11.1%)
In case of combination of both error types• The new specification becomes:
Snew = B + 2 • [(S/2)2 + CVComp2]0.5 (2+11.1 = 13.1%)
Interactive part
Statistics & graphics for the laboratory 5
The quality of the comparison ("Referral") method
2nd observation
May depend on the quality of the comparison method!
Glucose "biological criteria": Maximum deviation: 6.3%
3rd observation
May depend on the distribution of the outlying differences!
Interactive part
Potassium specification:10.4% ("biology")
Without error in the comparison method
New specification: 13.1%Because of error in the
comparison method, i.e. SE & RE 2%, REtest = 2%)
G (Potassium)
2,5
3,5
4,5
5,5
6,5
2,5 3,5 4,5 5,5 6,5
SPLIT 2: Referral
SP
LIT
1
n = 109
FAIL!FAIL!
G (Potassium)
2,5
3,5
4,5
5,5
6,5
2,5 3,5 4,5 5,5 6,5
SPLIT 2: ReferralS
PL
IT 1
PASSED!PASSED!
PASSED!PASSED!
FAIL!FAIL!
PASSED!PASSED!
FAIL!FAIL!
Decision………………………………………………………………………
-15
-10
-5
0
5
10
15
0 5 10 15 20 25
Reference (mmol/l)
Ro
uti
ne
(%
dif
fere
nc
e) n = 80
-15
-10
-5
0
5
10
15
0 5 10 15 20 25
Reference (mmol/l)
Ro
uti
ne
(%
dif
fere
nc
e) n = 80
Statistics & graphics for the laboratory 6
Glucose "biological criteria": Maximum deviation: 6.3%
Glucose "biological criteria": Maximum deviation: 6.3%
Glucose "biological criteria": Maximum deviation: 6.3%
Interactive part
Decision………………………………………………………………………
Decision………………………………………………………………………
Decision………………………………………………………………………
Factors that influence the interpretation of a method comparison
-15
-10
-5
0
5
10
15
0 5 10 15 20 25
Reference (mmol/l)
Ro
uti
ne
(%
dif
fere
nc
e) n = 40
-15
-10
-5
0
5
10
15
0 5 10 15 20 25
Reference (mmol/l)
Ro
uti
ne
(%
dif
fere
nc
e) n = 40
-15
-10
-5
0
5
10
15
0 5 10 15 20 25
Reference (mmol/l)
Ro
uti
ne
(%
dif
fere
nc
e) n = 40
-15
-10
-5
0
5
10
15
0 5 10 15 20 25
Reference (mmol/l)
Ro
uti
ne
(%
dif
fere
nc
e) n = 40
-15
-10
-5
0
5
10
15
0 5 10 15 20 25
Reference (mmol/l)
Ro
uti
ne
(%
dif
fere
nc
e) n = 40
-15
-10
-5
0
5
10
15
0 5 10 15 20 25
Reference (mmol/l)
Ro
uti
ne
(%
dif
fere
nc
e) n = 40
Statistics & graphics for the laboratory 7
Statistics behind2-sided 95% confidence limits of SD and sample size n
4th observation
May depend on the sample size
Summary – Interactive part• Decisions are influenced by
– "Quality" of specification– Quality of comparison method– Distribution at "critical" analyte values– Sample size (95% CL of, e.g., "2s")
Interactive part
PASSED!PASSED!
FAIL!FAIL!
Factors that influence the interpretation of a method comparison
PASSED!PASSED!
-15
-10
-5
0
5
10
15
0 5 10 15 20 25
Reference (mmol/l)
Ro
uti
ne
(%
diff
ere
nc
e)
-15
-10
-5
0
5
10
15
0 5 10 15 20 25
Reference (mmol/l)
Ro
uti
ne
(%
diff
ere
nc
e)
PASSED!PASSED!
DOUBT!DOUBT!
-15
-10
-5
0
5
10
15
0 5 10 15 20 25
Reference (mmol/l)
Ro
uti
ne
(%
diff
ere
nc
e)
FAIL!FAIL!
They are all from the same population!
(simulations with CVdiff 3.2%)
It's statistics!
The less samples you use,
the more uncertain your outcome is!
0,0
0,5
1,0
1,5
2,0
2,5
3,0
3,5
4,0
0 20 40 60 80 100
n (from n = 4)
2-s
ide
d 9
5% C
L (
SD
un
its)
Upper limit
Lower limit0,0
0,5
1,0
1,5
2,0
2,5
3,0
3,5
4,0
0 20 40 60 80 100
n (from n = 4)
2-s
ide
d 9
5% C
L (
SD
un
its)
0,0
0,5
1,0
1,5
2,0
2,5
3,0
3,5
4,0
0 20 40 60 80 100
n (from n = 4)
2-s
ide
d 9
5% C
L (
SD
un
its)
Upper limit
Lower limit
"True" CVdiff,true 3.2% Point estimate n = 40: the
CVdiff,exp can be
= 1.28 • 3.2 = 4.1%
= 0.82 • 3.2 = 2.6%0,0
0,5
1,0
1,5
2,0
2,5
3,0
3,5
4,0
0 20 40 60 80 100
n (from n = 4)
2-s
ide
d 9
5% C
L (
SD
un
its)
Upper limit
Lower limit0,0
0,5
1,0
1,5
2,0
2,5
3,0
3,5
4,0
0 20 40 60 80 100
n (from n = 4)
2-s
ide
d 9
5% C
L (
SD
un
its)
0,0
0,5
1,0
1,5
2,0
2,5
3,0
3,5
4,0
0 20 40 60 80 100
n (from n = 4)
2-s
ide
d 9
5% C
L (
SD
un
its)
Upper limit
Lower limit
"True" CVdiff,true 3.2% Point estimate n = 40: the
CVdiff,exp can be
= 1.28 • 3.2 = 4.1%
= 0.82 • 3.2 = 2.6%
Statistics & graphics for the laboratory 8
Method comparison S-cholesterol
Comparison method• Isotope dilution - gas chromatography/mass spectrometry (ID-GC/MS)• Negligible measurement error construct the x-axis of the difference plot by use of the ID-GC/MS, only.
Test method• Routine• Bias = 2.3%, CV = 3%.
Sample sizes• 80, 40, and 20
Plot• %-differences
Specifications• SEspec = 3% (NCEP) and TEspec = 10% (CLIA).
n = 80 n = 40
n = 20
The Bland & Altman approach
-12
-10
-8
-6
-4
-2
0
2
4
6
8
10
12
3 4 5 6 7 8 9
ID-GC/MS (mmol/l)
Ro
uti
ne
- ID
-GC
/MS
(%
)
+TE: 10%
-TE: -10%
+SE: 3%
-SE: -3%
+1.96s
-1.96s
Mean
-12
-10
-8
-6
-4
-2
0
2
4
6
8
10
12
3 4 5 6 7 8 9
ID-GC/MS (mmol/l)
Ro
uti
ne
- ID
-GC
/MS
(%
)+TE: 10%
-TE: -10%
+SE: 3%
-SE: -3%
+1.96s
-1.96s
Mean
-12
-10
-8
-6
-4
-2
0
2
4
6
8
10
12
3 4 5 6 7 8 9
ID-GC/MS (mmol/l)
Ro
uti
ne
- ID
-GC
/MS
(%
)
+TE: 10%
-TE: -10%
+SE: 3%
-SE: -3%
+1.96s
-1.96s
Mean
Observation…………………………………….…………………………………….
Observation…………………………………….…………………………………….
Observation…………………………………….…………………………………….
Conclusion• Easy visual inspections from the extended Bland&Altman plot.• Importance of the sample size• Account for the sample size via the statistical estimates
Statistics & graphics for the laboratory 9
Regression-based interpretation …
Thienpont LM, Van Nuwenborg JE, Bland JM, Altman DG. Stöckl D. Clin Chem 1998;44:849-57 Stat Meth Med Res 1999;8:135-60
… when systematic errors are related to the concentration!
Checklist
Consider• Sort of plot (absolute, %, "extended") • "Quality" of specification• Quality of comparison method• Distribution at "critical" analyte values• Sample size (95% CL of estimates, e.g., "2s")• Regression-based interpretation (systematic errors related to the concentration)
• Sample quality (clinical relevance)
The Bland & Altman approach
-6
-4
-2
0
2
4
3,5 4,0 4,5 5,0 5,5
IC (mmol/L)
(
%)
Ro
uti
ne
- IC
-4
-2
0
2
4
0 1 2 3 4 5 6 7Average fat content (g/100 ml)
Dif
fere
nc
e (
g/1
00 m
l)
-6
-4
-2
0
2
4
3,5 4,0 4,5 5,0 5,5
IC (mmol/L)
(
%)
Ro
uti
ne
- IC
-4
-2
0
2
4
0 1 2 3 4 5 6 7Average fat content (g/100 ml)
Dif
fere
nc
e (
g/1
00 m
l)
Statistics & graphics for the laboratory 10
References
Altman DG, Bland JM. Measurement in medicine: the analysis of method comparison studies. Statistician 1983;32:307-17.
Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet 1986;i:307-10.
Bland JM, Altman DG. Measuring agreement in method comparison studies. Stat Methods Med Res 1999;8:135-60.
Dewitte K, Fierens C, Stöckl D, Thienpont LM. Application of the Bland-Altman plot for the interpretation of method-comparison studies: a critical investigation of its practice. Clin Chem 2002;48:799-801.
Stöckl D, Rodríguez Cabaleiro D, Van Uytfanghe K, Thienpont LM. Interpreting method comparison studies by use of the bland-altman plot: reflecting the importance of sample size by incorporating confidence limits and predefined error limits in the graphic. Clin Chem 2004;50:2216-8.
Mantha S, Roizen MF, Fleisher LA, Thisted R, Foss J. Comparing methods of clinical measurement: reporting standards for Bland and Altman analysis. Anesth Analg 2000;90:593-602.
Pollock MA, Jefferson SG, Kane JW, Lomax K, MacKinnon G, Winnard CB. Method comparison - a different approach. Ann Clin Biochem 1992;29:556-60.Stöckl D. Beyond the myths of difference plots [letter]. Ann Clin Biochem 1996;33:575-7.
Stöckl D. Difference versus mean plots [reply]. Ann Clin Biochem 1997;34:571.Hyltoft Petersen P, Stöckl D, Blaabjerg O, Pedersen B, Birkemose E, Thienpont L, Flensted Lassen J, Kjeldsen J. Graphical interpretation of analytical data from comparison of a field method with a reference method by use of difference plots [opinion]. Clin Chem 1997;43:2039-46.
National Cholesterol Education Program. Recommendations for improving cholesterol measurements. US Department of Health and Human Services publication NIH 90-2964. Washington, DC: National Institutes of Health, 1990.Clinical Laboratory Improvement Amendments of 1988; Final Rule. Fed Reg February 28 1992;57: 7001-288.
The Bland & Altman approach
Statistics & graphics for the laboratory 11
Graphical interpretation of a method comparison
By use of integrated specifications
Data not within or at the limit of the specifications Interpret with graphical and statistical techniques
Method comparison exercises
InterpretationThe comparison shows visually that the quality of the test method is well within the specification. There is no need for further investigation (for, e.g., linearity, bias, Sy/x, etc.)
ConclusionInterpret the method comparison in first instance visually against the selected specification(s).
0
5
10
15
20
25
0 5 10 15 20 25Reference (mmol/l)
Glu
co
me
ter
XY
Z (
mm
ol/l
) AA
BB
C
C
D D
E
E
-20
-15
-10
-5
0
5
10
15
20
0 5 10 15 20 25
Reference (mmol/l)
Ro
uti
ne
(%
dif
fere
nc
e) Specification (6.3%)!
not Bland & Altman 2s
Glucose “biological criteria”:Max. deviation: 6.3%
Sort of specification:Proportional error
Choice: % bias plot
Statistics & graphics for the laboratory 12
Interpretation strategy
“When the data are not within or at the limit of the specifications”
General• Reconsider the quality of the samples (number, matrix) and the measurement protocol (incl. the comparison method) • Reconsider the internal quality control• Judge the comparison visually (concentration range, outliers) (consider all possible graphics)
Judge the data for linearity• Direct (if more appropriate, in the logarithmic plot) or• Use a “residuals plot” and check the sequence of the signs (+/-) of the residuals• If x and y indeed are linearly related, perform correlation analysis• If x and y are not linearly related, perform non-linear regression (only for the purpose of calibration)
Correlation analysis• If r <0.99 (broad range) or <0.975 (small to medium range), check again for outliers• When, thereafter, r sufficiently increases, perform linear regression without the outliers• Perform Deming regression (DR) (or Passing Bablok regression (PBR))• When r does not sufficiently increase, calculate SDy/x of OLR (note: use
alternatively SD of the differences yi - xi (SDdiff)) and compare with the total
imprecision of the method comparison due to both SDax and SDay
• a) If SDy/x or SDdiff ~ total imprecision, reduce the latter (e.g., by performing
multiple measurements)• b) If SDy/x or SDdiff >>> total imprecision
there is a substantial analytical difference between the methods due to sample-related effects
Deming (PB) regression• Use the regression parameters to find the cause of the poor quality:
–SE proportional (slope sign. 1)–SE constant (intercept sign. 0)–Matrix-related RE (SDy/x or SDdiff >>> total SDa)
• Improve the method (perform, if more appropriate, an in-depth evaluation of the "elements" of the method: calibration, recovery, interference studies, ...).
Adapted from: Stöckl D, Dewitte K, Thienpont LM. Clin Chem 1998;44:2340-6
Method comparison exercises
Statistics & graphics for the laboratory 13
Case study 1
Analyte: Estradiol in serum
Samples• 24 (range: 15 - 3000 pmol/l)• 4 series of 6 samples mixed from 4 native pools and 1 "stripped" serum ("6"/0
ml; 5/1; 4/2; 3/3; 2/4; 1/5)IQC: 3Design: 1 series, duplicatesMethod x: GC/MS, CV 6%: confirmed from IQC and duplicatesMethod y: Immunoassay, CV 6%: confirmed from IQC and duplicatesCVtot,expected: 8.5%
Graphical judgement: no irregularitiesCorrelation/regression data:
• r = 0.998; y = 1.03 (± 0.05) x - 10 (± 15) pmol/l, • Sy/x (%): 9%
Conclusion: Is the immunoassay equivalent to GC/MS?
We check all elements of the method comparison studyIQC: 3Design: 1 series, duplicates Method x: GC/MS, CV 6%: confirmed from IQC and duplicates Method y: Immunoassay, CV 6%: confirmed from IQC and duplicates CVtot,expected: 8.5% Graphical judgement: no irregularities Correlation/regression data:
• r = 0.998 ; y = 1.03 (± 0.05) x - 10 (± 15) pmol/l, • SDy/x (%): 9%
Samples• 24 (range: 15 - 3000 pmol/l)• 4 series of 6 samples mixed from 4 native pools and 1 "stripped" serum ("6"/0
ml; 5/1; 4/2; 3/3; 2/4; 1/5)
Conclusion: Decision not possible, wrong design! No native samples!
Strategy, General• [Re]consider the quality of the samples (matrix)
Method comparison exercises
Statistics & graphics for the laboratory 14
Case study 2
Case analogous to an example in literatureAnalyte: HDL-cholesterol in serum
Samples: 100 native serum samplesDesign: multiple series, singlicates
Method x: "reference method" (validated with the official reference method of CDC): indirect method (phosphotungstic acid/MgCl2)
Between-day CV: 3%
Method y: direct method (detergent + enzymatic)Between-day CV: 3.8%
CVtot,expected: 4.8%
Specifications (NCEP)SD < 0.044 mmol/l, or CV < 4%
Interpretation of combined absolute & %-specifications (select the highest)Example: 0.044 mmol/l and 4%
• Consider the measurement range and calculate at which concentration 4% = 0.044 mmol/l• Here: at 1.1 mmol/l is 4% = 0.044 mmol/l > 1.1 mmol/l, use the 4% specification, 1.1 mmol/l, use the 0.044 mmol/l specification
Specifications case 2SD < 0.044 mmol/l (up to 1.1 mmol/l), or CV < 4% (> 1.1 mmol/l) But, specifications are expressed in terms of RE
Transformation of specifications for RE to TEwith the formula: TE = SE + k • RETE = RE • k (k = 2, Westgard “classical”; k = 3, Ehrmeyer & Laessig; k = 4, Westgard, with IQC)We select: k = 2 for a method comparison TE = 2 • RE (specification) = 8%
Note: Only valid when the comparison method (x) is error free.Here: CV(x) = 3%, which cannot be neglected in comparison tothe CV of the test method (3.8%) Expand the specification
Method comparison exercises
Basic
introduction-participant
Datasets-MethComp
Statistics & graphics for the laboratory 15
0
1
2
3
0 1 2 3HDL-indirect (mmol/l)
HD
L-d
ire
ct
(mm
ol/l
)
Original specification (TE) = 8%Suppose SE = 0The CV(x) = 3%, which cannot be neglected in comparison to the CV of the test method (3.8%) and the specification.
The new specification becomes:Snew = 2 • [(S/2)2 + CVComp
2]0.5
= 2•[4•4+3•3] = 10.0% or= TE(abs) = 0.11 mmol/L
Interpretation of case study 2Graphical presentation&Specifications• Absolute: <0.11 mmol/l (up to 1.1 mmol/l)• Proportional: <10% (>1.1 mmol/l)
“When the data are not within or at the limit of the specifications”General
• Reconsider the quality of the samples (number, matrix) and the measurement protocol (incl. the comparison method)
• Reconsider the internal quality control • Judge the comparison visually (concentration range, outliers) (consider all
possible graphics)
Judge the data for linearity!-Direct (if more appropriate, in the logarithmic plot) or-Use a “residuals plot” and check the sequence of the signs (+/-) of the residuals
Something unusual?
0
1
2
3
0 1 2 3HDL-indirect (mmol/l)
HD
L-d
ire
ct
(mm
ol/l
)
Expansion of specifications
Method comparison exercises
Statistics & graphics for the laboratory 16
Judge the data for linearity• If the data are not linearly related
–Perform non-linear regression analysis for the purpose of calibration
Calibration-Linearisation and correction for SE: via the reverse plot and the trend line Note: The success of recalibration has to be checked with a new set of data!
0
1
2
3
0 1 2 3HDL-indirect (mmol/l)
HD
L-d
ire
ct
(mm
ol/l
)
Case study 2 (ctd.)
Method comparison exercises
Something unusual?
y = 1,1729x - 0,1518r = 0,9764
0
1
2
3
0 1 2 3
HDL-indirect (mmol/l)
HD
L-d
ire
ct
(mm
ol/l
)
0
0,5
1
1 ,5
2
2 ,5
3
3 ,5
0 0,5 1 1 ,5 2 2 ,5 3 3 ,5
HDL-in d ire c t (mmo l/l)
0
0,5
1
1 ,5
2
2 ,5
3
3 ,5
0 0,5 1 1 ,5 2 2 ,5 3 3 ,5
HDL-in d ire c t (mmo l/l)
-1
0
1
0 1 2 3
HDL-indirect (mmol/l)
H
DL
-dir
ec
t (m
mo
l/l) Judgement of linearity:
Residual-plot!
y = 0,1137x2
+ 0,4731x + 0,3907
0
1
2
3
0 1 2 3
HDL-direct (mmol/l)
HD
L-in
dir
ec
t (m
mo
l/l)
0
1
2
3
0 1 2 3HDL-indirect (mmol/l)
HD
L-d
ire
ct
(mm
ol/l
) Transformed data
y' = 0,1137*y2 + 0,4731*y + 0,3907
Statistics & graphics for the laboratory 17
Case study 3
Case analogous to an example in literatureAnalyte: HDL-cholesterol in serum
Samples: 100 native serum samplesDesign: multiple series, singlicatesMethod x: indirect method (phosphotungstic acid/MgCl2)
Between-day CV: 3%Method y: direct method (detergent + enzymatic)Between-day CV: 3%CVtot,expected: 4.2%
Specifications (NCEP)SD < 0.044 mmol/l ( 1.1 mmol/l), or CV < 4% (>1.1 mmol/l)
Note: The new specification for TE = 2•[4•4+3•3] = 10.0% orTE(abs) = 0.11 mmol/L
Visual interpretation -Too many points outside specifications-Otherwise no irregularities Further statistical, analytical investigations needed
General• Reconsider the quality of the samples (number, matrix) and the measurement protocol (incl. the comparison method) • Reconsider the internal quality control • Judge the comparison visually (concentration range, outliers) (consider all possible graphics)
Method comparison exercises
Basic
introduction-participant
Datasets-MethComp
0
1
2
3
0 1 2 3HDL-indirect (mmol/l)
HD
L-d
ire
ct
(mm
ol/l
)
-1
0
1
0 1 2 3
[HDL-ind. + HDL-direct]/2 (mmol/l)
H
DL
-dir
ec
t (m
mo
l/l)
Statistics & graphics for the laboratory 18
Judge the data for linearity• Direct (if more appropriate, in the logarithmic plot) or • Use a “residuals plot” and check the sequence of the signs (+/-) of the residualsIf x and y indeed are linearly related, perform correlation analysis• If x and y are not linearly related, perform non-linear regression (for the purpose of calibration)
Correlation analysisr = 0.9803 (small range!) • If r <0.99 (broad range) or <0.975 (small to medium range), check again for outliers• When, thereafter, r sufficiently increases, perform linear regression without the outliers• When r does not sufficiently increase, calculate SDy/x of OLR or SDdiff and
compare with the total SDa of the method comparison
• a) If SDy/x or SDdiff ~ total SDa, reduce the SDa of the method comparison (e.g.,
by performing multiple measurements)• b) If SDy/x or SDdiff >>> total SDa, there is a substantial analytical difference
between the methods due to sample-related effects Perform Deming regression (DR) (or Passing Bablok regression (PBR))
Use the regression parameters to find the cause of the poor quality• SE proportional: slope (1.010 ± 0.045) • SE constant: intercept (-0.03 ± 0.058)
No problems indicated by regression.
> Look at the random differences
y = 1,010x - 0,03r = 0,9803
0
1
2
3
0 1 2 3
HDL-indirect (mmol/l)
HD
L-d
ire
ct
(mm
ol/l
)
Case study 3 (ctd.)
Method comparison exercises
Linear regression95% CLs of intercept: –0.061 to 0.055of slope 0.965 to 1.055 Note: SDy/x (OLR) = 0.116 mmol/l
Statistics & graphics for the laboratory 19
Compare SDy/x from OLR (0.1164 mmol/l) with the total SDa of the method
comparison:SDy/x
2 = SDay2 + b2 SDax
2 (CVy/x = 2 CVay)
0.1164 >>> 0.0528 mmol/l (7.8% >> 4.2%)There is a substantial analytical difference between the methods due to sample-related effects
Alternatively: SD of the differences yi – xi can be used, here SDdiff = 0.1159 mmol/l
Note: "RE" in method comparison. Not only consider the value, but compare the spread of all points in a graphic
Observation:Too many points outside specifications due to "random" spread.
ConclusionImprove the method.If appropriate, perform an in-depth evaluation of the “elements” of the method: interference studies, specificity, method principle, ... WHICH ONE? X or Y?
Experimental investigations, done after the methodcomparison, according to literature
Interference studies• Hemolysis• Lipemia (caused most of the problems)• Bilirubinemia
-30
-20
-10
0
10
20
30
0 1 2 3
[HDL-ind. + HDL-direct]/2 (mmol/l)
H
DL
-dir
ec
t (%
) +2CV
-2CV
Case study 3 (ctd.)
Method comparison exercises
Note:Bland-Altman "2s" (2 CVdiff) = 15.6%
CVtot = 2 * 4.2% = 8.4%
Statistics & graphics for the laboratory 20
Case analogous to an example in literatureAnalyte: Troponine-I in serum
Samples: >200 native serum samplesDesign: multiple series, singlicatesMethod x: immunoassay 1, CV ~ 12%Method y: immunoassay 2, CV ~ 10%CVtot,expected: ~ 16%
Specifications None: Select 2 • CVtot,expected = 32%
The big slope and the high number of outliers• Have both methods specificity problems?• Is the slope only caused by a difference in calibration?
Can we be sure that both methods measure the same analyte?
-40-30-20-10
010203040
0 5 10 15 20 25 30 35
Troponine-I [1+2]/2 (µg/l)
Tro
po
nin
e-I
[2
-1]
(µg
/l)
0
10
20
30
40
50
0 10 20 30 40 50Troponine-I 1 (µg/l)
Tro
po
nin
e-I
2 (
µg
/l) Without outliers:y = 3,3 x + 1,2r = 0,9597With outliers:y = 2,1 x + 5,8r = 0,7134
Case study 4
Method comparison exercises
Graphical interpretation(incl. regression)
Two problems apparent:- the outliers- the big slope
Bias plotNote that the concentration range (x-values)(in particular of the "1" method) cannot longer be recognized>Problem of the large slope
Statistics & graphics for the laboratory 21
Case study 5
Case analogous to an example in literatureAnalyte: Potassium in serumAim: eventual recalibration
Samples: 60 native serum samplesMethod x: IC-reference methodBetween-day CV: 1.5% (for a design of 4 measurements per sample = 0.75%)Method y: ISEWithin-day CV: 1.1% (singlicates)
Specification (CLIA)TE = 0.48 mmol/l (= 10% at 4.8 mmol/l)
Regression in a method comparison
Judgement of the 95% CLs of the regression parameters in comparison to specifications
Uncertainty of the slope alone ca. 17%A recalibration via the method comparison exceeds the error budget!
Method comparison exercises
y = 1.16 x – 0.86 95% CLs of slope = ±0.1795% CLs of intercept = ±0.74 mmol/l
SpecificationCLIA limit: 10% at 4.8 mmol/l
y = 1.16x - 0.86r = 0.871
3
4
5
6
3 4 5 6Reference (mmol/L)
Ro
uti
ne
(mm
ol/
L) ..
.
Statistics & graphics for the laboratory 22
Notes
Notes
Statistics & graphics for the laboratory 23
Notes
Notes