Statistics for InterlaboratoryStatistics for InterlaboratoryComparisonsComparisons

IAAC PT WorkshopIAAC PT WorkshopSSããoo Paulo, Brazil Paulo, Brazil

20 September, 200420 September, 2004

Dan Tholen, MSDan Tholen, MS

Documents for LectureDocuments for Lecture

ISO FDIS 13528 ISO FDIS 13528 ““Statistical Methods forStatistical Methods foruse in Proficiency Testing byuse in Proficiency Testing byInterlaboratory ComparisonsInterlaboratory ComparisonsAPLAC Statistical proceduresAPLAC Statistical proceduresISO 5725 1-6, by reference in ISO 13528ISO 5725 1-6, by reference in ISO 13528

DIS 13528DIS 13528

ISO FDIS 13528ISO FDIS 13528

Originally approved as work item in 1997Originally approved as work item in 1997Cancelled as work item in 2001Cancelled as work item in 2001Re-approved in 2002Re-approved in 2002Final comments resolved, 2002 & 2003Final comments resolved, 2002 & 2003Approved as FDIS, June 2003 afterApproved as FDIS, June 2003 afterresolving 149 comments, to be publishedresolving 149 comments, to be publishedlate 2004late 2004

ISO 13528ISO 13528

Companion to ISO Guide 43-1Companion to ISO Guide 43-1Authored by TC-69, SC6, WG1Authored by TC-69, SC6, WG1Written as a StandardWritten as a StandardHigh interest / many commentsHigh interest / many commentsGoal is to describe optimal procedures,Goal is to describe optimal procedures,but to allow other procedures as long asbut to allow other procedures as long asthey are:they are:–– Statistically validStatistically valid–– Fully described to usersFully described to users

ISO 13528ISO 13528

Follows Template of Guide 43-1 Annex AFollows Template of Guide 43-1 Annex A–– Determine Assigned ValueDetermine Assigned Value

Robust StatisticsRobust StatisticsReference ValuesReference Values

–– Calculate Performance StatisticCalculate Performance StatisticZ scoresZ scoresOther performance scoresOther performance scores

–– Evaluate PerformanceEvaluate PerformanceRelativeRelativeFixedFixed

SD for proficiency testingSD for proficiency testing

Discussed in detail in section 6Discussed in detail in section 6SD as used in z scoresSD as used in z scoresCan also be thought of as 1/3 ofCan also be thought of as 1/3 ofevaluation intervalevaluation interval(when z>3 is action signal)(when z>3 is action signal)

For example if fixed interval is E = For example if fixed interval is E = ±±10%...10%...Then E = 3Then E = 3σσσ σ = E/3 = 10%/3 = 3.3%= E/3 = 10%/3 = 3.3%

Limiting the uncertainty of the AssignedLimiting the uncertainty of the AssignedValue (X): Section 4.2Value (X): Section 4.2

Establish limits for uncertainty of AVEstablish limits for uncertainty of AVuu(X) < 0.3(X) < 0.3σσ

When using fixed limitsWhen using fixed limits……

u(X) < 0.3(E/3)u(X) < 0.3(E/3)u(X) < E/10u(X) < E/10

Limiting the uncertainty of the AssignedLimiting the uncertainty of the AssignedValue (X): Section 4.2Value (X): Section 4.2

If this cannot be met thenIf this cannot be met then–– Look for a better way to determine AVLook for a better way to determine AV–– Incorporate uncertainty in scoreIncorporate uncertainty in score–– Warn participantsWarn participants

Limiting the uncertainty of the AssignedLimiting the uncertainty of the AssignedValue (X): Value (X): ExampleExample

When consensus mean and SD are used toWhen consensus mean and SD are used todetermine performance:determine performance:

Then u(X) = SD/Then u(X) = SD/√√nnSo one can have very small uncertainty withSo one can have very small uncertainty with

large number of labs.large number of labs.What n is needed to assure criterion is met?What n is needed to assure criterion is met?

Need 1/Need 1/√√n < 0.3, or n > (1/0.3)n < 0.3, or n > (1/0.3)2 2 or n>11or n>11and nand n≤≤11, then cannot meet criterion.11, then cannot meet criterion.

Determine the appropriate number ofDetermine the appropriate number ofReplicates: Section 4.3Replicates: Section 4.3

When a methodWhen a method’’s repeatability is large, its repeatability is large, itcan confuse interpretation of scorescan confuse interpretation of scoresDetermine n replicates so that:Determine n replicates so that:

σσrr //√√nn < 0.3 < 0.3σσ

•• All labs must perform the same numberAll labs must perform the same numberof replicates, even if of replicates, even if σσrr is different.is different.

Limiting the effect of repeatabilityLimiting the effect of repeatabilityExampleExample

Say E=10% and Say E=10% and σσrr = 2%: = 2%:Then Then σσrr //√√nn < 0.1E < 0.1E

So 2/So 2/√√nn < 0.1(10%) or < 0.1(10%) or

2/2/1.01.0 < < √√nnOr n>4Or n>4

This criterion can lead to large n replicates.This criterion can lead to large n replicates.

Reporting considerations: 4.6Reporting considerations: 4.6

Reporting:Reporting:–– determine typical repeatability determine typical repeatability σσrr

–– Do not round digits by more thanDo not round digits by more than σσrr/2/2NO TRUNCATED RESULTSNO TRUNCATED RESULTS–– ““Less thanLess than”” values not allowed values not allowed–– Results to be reported as determinedResults to be reported as determined–– Conflict with need to report same as to clientsConflict with need to report same as to clients–– Can have check-off box to note Can have check-off box to note ““<MDL<MDL””..

ISO 13528 ProceduresISO 13528 ProceduresCalculate Summary Statistics:Calculate Summary Statistics:

–– Outlier detection and removal are allowed ifOutlier detection and removal are allowed ifdone in a statistically valid waydone in a statistically valid way

–– Robust measures are preferredRobust measures are preferredMeanMeanSDSD

–– Preferred robust method is given, Preferred robust method is given, others areothers areallowedallowed if if

Statistically validStatistically validFully described to participantsFully described to participants

ISO 13528 ProceduresISO 13528 Procedures

Determine Assigned Value (Target)*Determine Assigned Value (Target)*–– Determined before PT shipmentDetermined before PT shipment

Result from formulationResult from formulationCertified reference valueCertified reference valueOther reference valuesOther reference values

–– Determined from PT dataDetermined from PT dataConsensus of expertsConsensus of expertsConsensus of participantsConsensus of participants

* Control the uncertainty of the assigned value* Control the uncertainty of the assigned value

Determine the Standard Uncertainty ofDetermine the Standard Uncertainty ofthe Assigned Valuethe Assigned Value

Determined Determined before PT shipmentbefore PT shipment–– Result from formulationResult from formulation

Uncertainty per manufacture process, usually veryUncertainty per manufacture process, usually verysmall relative to measurement uncertaintysmall relative to measurement uncertainty

–– Certified reference valueCertified reference valueUncertainty provided with certificateUncertainty provided with certificate

–– Other reference valuesOther reference valuesUncertainty calculated per GUM or otherUncertainty calculated per GUM or otherprocedureprocedure

Determine the Standard Uncertainty ofDetermine the Standard Uncertainty ofthe Assigned Valuethe Assigned Value

Determined Determined from data in PT shipmentfrom data in PT shipment–– Consensus of expert laboratories (p of them)Consensus of expert laboratories (p of them)

Each lab should know their MU, and report itEach lab should know their MU, and report ituuxx = 1.23( = 1.23(√√((ΣΣuuii

22))/p for robust mean (median)))/p for robust mean (median)Caution about bias in expertsCaution about bias in experts

–– Consensus of participants (p of them)Consensus of participants (p of them)Calculate robust mean and SD (s*)Calculate robust mean and SD (s*)uuxx = 1.23( = 1.23(s*)/s*)/√√ppCaution about bias due to method mixCaution about bias due to method mixCaution about lack of consensusCaution about lack of consensus

Robust AnalysisRobust Analysis

Algorithm A for mean and SDAlgorithm A for mean and SDStarts with x*=medianStarts with x*=median

s*=1.483 s*=1.483xxmedian|xmedian|xii-x*|-x*|

Limit data at x*+1.5s* and x*-1.5s*Limit data at x*+1.5s* and x*-1.5s*Extreme values trimmed to 1.5s*Extreme values trimmed to 1.5s*

New option added to use initial x* and s*New option added to use initial x* and s*

Robust AnalysisRobust Analysis

Algorithm A for mean and SDAlgorithm A for mean and SDCalculate new: Calculate new: x*=(x*=(ΣΣxxii)/p)/p

s*=1.134 s*=1.134√√ΣΣ(x(xii*-x*)*-x*)22/(p-1)/(p-1)

Trim data again, at 1.5s*Trim data again, at 1.5s*Recalculate new x* and s*Recalculate new x* and s*Repeat until convergenceRepeat until convergence

Quality check: Section 5.7Quality check: Section 5.7

When AV is determined prior to PT:When AV is determined prior to PT:–– Compare AV with robust mean or resultsCompare AV with robust mean or results–– Determine uncertainty of comparison Determine uncertainty of comparison uudd

–– If difference exceeds 2uIf difference exceeds 2udd then investigate then investigateWhen AV is determined from consensus:When AV is determined from consensus:–– Compare AV with a reference value from aCompare AV with a reference value from a

competent laboratory (could come fromcompetent laboratory (could come fromhomogeneity data)homogeneity data)

Compare robust SD with experienceCompare robust SD with experience

Determine Performance IntervalDetermine Performance Interval

5 ways to get 5 ways to get ““SD for ProficiencySD for Proficiency”” (for z (for zscores)scores)–– By prescription (set by AA)By prescription (set by AA)–– By experience (perception) of expertsBy experience (perception) of experts–– From a general model (From a general model (HorwitzHorwitz))–– By a precision experiment (ISO 5725-2)By a precision experiment (ISO 5725-2)–– From participant data (robust SD)From participant data (robust SD)

Determine Performance IntervalDetermine Performance Interval

Fixed Limits (or Fixed Limits (or ““Fitness for PurposeFitness for Purpose””))Can come from methods for SDCan come from methods for SDNot widely usedNot widely usedPreferred for interpretationPreferred for interpretation

Fixed percentage across rangeFixed percentage across rangeFixed value across rangeFixed value across rangeMixed or segmented.Mixed or segmented.

Calculate Performance StatisticCalculate Performance Statistic

Estimates of bias:Estimates of bias:Difference: Difference: D=(x-X)D=(x-X)Percentage Difference: D%=100(x-X)/XPercentage Difference: D%=100(x-X)/XD and D% can be evaluated with Fixed LimitsD and D% can be evaluated with Fixed Limits

Estimates of Relative PerformanceEstimates of Relative Performance–– rank or percentage rank (not recommended)rank or percentage rank (not recommended)–– z score (recommended) z=(x-X)/z score (recommended) z=(x-X)/σσ

EEn n z z’’ zeta zeta EEzz scoresscores

EEnn and zeta consider uncertainty of and zeta consider uncertainty ofparticipant result and assigned valueparticipant result and assigned value–– Requires consistent determination ofRequires consistent determination of

uncertainty by all laboratoriesuncertainty by all laboratorieszz’’ and and EEzz use uncertainty of assigned use uncertainty of assignedvalue onlyvalue onlyEEnn in use in calibration, starting to be in use in calibration, starting to beimplementedimplemented

Calculate Performance StatisticCalculate Performance Statistic

Scores that consider uncertainty:Scores that consider uncertainty:–– EEnn score ( score (““Error, normalizedError, normalized””))

EEnn = (x-X)/ = (x-X)/√√(U(U22lablab+U+U22

refref))

–– zz’’ scores (like z, includes scores (like z, includes uuxx))zz’’ = (x-X)/ = (x-X)/√√((σσ22+u+u22

XX))

–– zeta scores (like En, but with std. zeta scores (like En, but with std. UncertUncert))Zeta = (x-X)/Zeta = (x-X)/√√(u(u22

lablab+u+u22refref))

–– EEzz scores (puts U scores (puts UXX in numerator and denominator) in numerator and denominator) EEzz = (x-(X = (x-(X±±UUXX)/U)/UXX))

Evaluate performanceEvaluate performance

Compare performance statistic againstCompare performance statistic againstcriteria, determine acceptability; i.e.,criteria, determine acceptability; i.e.,

For fixed limits:For fixed limits:Bias < Limit Bias < Limit ““acceptableacceptable““Bias Bias ≥≥ Limit Limit ““unacceptableunacceptable””For z zFor z z’’ zeta: zeta:-2< z <+2 -2< z <+2 ””acceptableacceptable””-3< z -3< z ≤≤-2 or 2-2 or 2≤≤ z <3 z <3 ““warning signalwarning signal””z z ≤≤-3 or z -3 or z ≥≥3 3 ””unacceptableunacceptable””

Evaluate performanceEvaluate performance

EEnn <1 <1 ““acceptableacceptable””EEnn ≥≥1 1 ““unacceptableunacceptable””

Combined performance scoresCombined performance scores

Analyze data for each item independentlyAnalyze data for each item independentlySpecial process for Youden pairs (&reps)Special process for Youden pairs (&reps)Can be other reasons to combine resultsCan be other reasons to combine results–– PrecisionPrecision–– LinearityLinearity

Can count number of satisfactory scoresCan count number of satisfactory scoresNot recommended to combineNot recommended to combineperformance scores (such as average z)performance scores (such as average z)

Graphic Reports for PT roundGraphic Reports for PT round

Rank vs. Result (with or without MU)Rank vs. Result (with or without MU)

Graphic Reports for PT roundGraphic Reports for PT round

Histograms Histograms –– of results or scores of results or scores

Graphic Reports for PT roundGraphic Reports for PT round

Bar plot of standardized performanceBar plot of standardized performancestatistics (z h k)statistics (z h k)–– z, or other standardized scores (% error)z, or other standardized scores (% error)

h and k plots from 5725h and k plots from 5725–– h same as z, except always from sample SDh same as z, except always from sample SD–– k for repeatability (nk for repeatability (n≥≥2 replicates)2 replicates)

Graphic Reports for PT roundGraphic Reports for PT round

Youden plot (usually w/median lines)Youden plot (usually w/median lines)In this document uses only z scores.In this document uses only z scores.Should use sample results, for clarityShould use sample results, for clarity

Provides evidence of related results, whichProvides evidence of related results, whichcan suggest consistent biascan suggest consistent biasConsistent bias can suggest lack of clearlyConsistent bias can suggest lack of clearlydefined method.defined method.Confirm with rank correlation test.Confirm with rank correlation test.

Rank Correlation TestRank Correlation Test

Common statistical procedureCommon statistical procedureUsed when Youden Plot suggestsUsed when Youden Plot suggestsrelationshiprelationshipUse ranks of results from two samplesUse ranks of results from two samplesDocument shows critical values forDocument shows critical values forcorrelationcorrelation

Graphic Reports for PT roundGraphic Reports for PT round

Repeatability SDRepeatability SD

Graphic Reports for PT roundGraphic Reports for PT round

Split sampleSplit sample–– Check agreement in 2+labsCheck agreement in 2+labs–– Check difference between resultsCheck difference between results

Graphic Reports forGraphic Reports forMore than One PT RoundMore than One PT Round

Line plot (Line plot (ShewhartShewhart plot) for scores on plot) for scores onprevious roundsprevious rounds–– Use any standardized scoreUse any standardized score–– Show evaluation intervalsShow evaluation intervals–– Show test datesShow test dates

Graphic Reports forGraphic Reports forMore than One PT RoundMore than One PT Round

CUSUM control chartCUSUM control chart–– Can show trends affecting biasCan show trends affecting bias–– Choose some number to use (rolling sum)Choose some number to use (rolling sum)–– Sums should trend to zeroSums should trend to zero–– Not sensitive to current problemsNot sensitive to current problems

Graphic Reports forGraphic Reports forMore than One PT RoundMore than One PT Round

Plot of Standardized Laboratory BiasesPlot of Standardized Laboratory Biasesagainst assigned valueagainst assigned value–– Shows relationship between score and levelShows relationship between score and level–– Can mask time effectCan mask time effect……do bothdo both

Graphic Reports for more thanGraphic Reports for more thanOne PT RoundOne PT Round

Dot PlotDot Plot–– Show all samples on same chartShow all samples on same chart–– Show evaluation intervalsShow evaluation intervals–– Show dates or scheme codesShow dates or scheme codes

Annexes in 13528Annexes in 13528

A. SymbolsA. SymbolsB. Homogeneity and stability proceduresB. Homogeneity and stability procedures

–– No statistical testNo statistical test–– Test relative to evaluation intervalTest relative to evaluation interval

C. Robust proceduresC. Robust procedures–– Procedure A for mean and SDProcedure A for mean and SD–– Procedure S for SDProcedure S for SD

HomogeneityHomogeneity

HomogeneityHomogeneity–– 10 or more samples, 2 replicates10 or more samples, 2 replicates–– SDSDSS for samples (ANOVA calculation) for samples (ANOVA calculation)–– SDSDSS < 0.3 < 0.3σσ–– No F testNo F test

Can use experience to reduce testingCan use experience to reduce testingWhen evidence and theory prove homogeneousWhen evidence and theory prove homogeneous

HomogeneityHomogeneity

Homogeneity F test (not recommended)Homogeneity F test (not recommended)

F = (SDF = (SDSS22/s/srr

22))SSrr = repeatability SD = repeatability SDSS= Between samples= Between samples

FFcritcrit = F = F(.05,k-1, s(n-1))(.05,k-1, s(n-1)) k=# samples n=# replicatesk=# samples n=# replicates

High High SSrr insensitive test (large SDinsensitive test (large SDSS passes) passes)Low Low SSrr too sensitive test (small SDtoo sensitive test (small SDSS fails) fails)

StabilityStability

StabilityStability–– 3 or more samples, 2 replicates3 or more samples, 2 replicates–– Calculate overall meanCalculate overall mean–– Mean(H) Mean(H) –– Mean(S) < 0.3 Mean(S) < 0.3σσ–– No t testNo t test

High High SSrr insensitive test (big difference passes)insensitive test (big difference passes)Low Low SSrr too sensitive test (small difference fails)too sensitive test (small difference fails)

Robust AnalysisRobust Analysis

Algorithm S for pooled SDAlgorithm S for pooled SDApplied when there are SD estimates fromApplied when there are SD estimates fromall participants (replicates) (called all participants (replicates) (called ““ww””))When 2 replicates, use range (difference)When 2 replicates, use range (difference)

w*=median w*=median wwii..update with update with ηη from Table 19 from Table 19wwii* = trimmed * = trimmed wwii at at ηηw* = w* = ξξ√√((ΣΣ((wwii*)*)22/p/p))

Iterate if necessaryIterate if necessary

APLAC (NATA) Robust procedureAPLAC (NATA) Robust procedure

Calculate Quartiles Q1, median, Q3Calculate Quartiles Q1, median, Q3

IQR = Q3-Q1IQR = Q3-Q1MedianMedian is an is an estimate of meanestimate of meanNormalized IQRNormalized IQR is an is an estimate of SDestimate of SDIQRIQRNN = 0.7413 = 0.7413 x x IQRIQR

APLAC performance statisticsAPLAC performance statistics

Calculate relative performance measuresCalculate relative performance measuresBetween lab agreementBetween lab agreement

SSii = (A = (Aii+B+Bii)/)/√√22Within lab agreementWithin lab agreement

Di = (ADi = (Aii-B-Bii)/)/√√22 if median (Aif median (Aii)>median(B)>median(Bii)) (B (Bii-A-Aii)/)/√√22 if median(Aif median(Aii)<median(B)<median(Bii))

Calculate z-scores for these measuresCalculate z-scores for these measures

Possibilities for APLACPossibilities for APLAC

Estimate BiasEstimate BiasPercentage Bias(LabPercentage Bias(Labii) =) =

[((A[((Aii+B+Bii)-(M)-(MAA+M+MBB))/(M))/(MAA+M+MBB)] x 100%)] x 100%

Estimate repeatability:Estimate repeatability:DDAA= (A= (Aii-M-MAA) D) DBB= (B= (Bii-M-MBB) D) Davgavg=(D=(DAA+D+DBB)/2)/2rri1i1 = = √√[((D[((DAA - D - Davgavg))22+(D+(DBB - D - Davgavg))22)/2])/2] (no reps)(no reps)rri2i2 = = √√[((A[((Ai1i1-A-Ai2i2))22 + (B + (Bi1i1-B-Bi2i2))22)/2] (2 replicates))/2] (2 replicates)

The EndThe End

Thank you!Thank you!