8/12/2019 3B-Using CLSI Guidelines to
1/53
USING CLSI GUIDELINES TO
PERFORM METHOD EVALUATIONSTUDIES IN YOUR LABORATORY
James Blackwood , MS, CLSI
David D. Koch, PhD, FACB, DABCC, Pathology & LaboratoryMedicine, Emory University School of Medicine
Breakout Session 3BTuesday, May 1
8:30 10 am
8/12/2019 3B-Using CLSI Guidelines to
2/53
Outline
Learning Objectives Identify the seven performance characteristics that should
be evaluated prior to reporting results from a new test
method.
Verify the claims of manufacturers regarding analyticalperformance by following CLSI guidelines.
Demonstrate ongoing compliance with the method
evaluation criteria contained in accreditation guidelines.
Who is CLSI and what are these guidelines?
Method evaluation basic definitions and experiments
Use of the CLSI Evaluation Protocols
Use of StatisPro software in method evaluation
8/12/2019 3B-Using CLSI Guidelines to
3/53
Who is CLSI? Clinical and Laboratory Standards Institute
ANSI-accredited, global, nonprofit standards development organization
CLSI has over 2,000 members organizations such as IVDmanufacturers, hospital laboratories, reference laboratories, universities,
professional associations, and government agencies
We promote the development and use of voluntary consensus standardsand guidelines within the health care community.
Our documents help health care organizations meet their responsibilitieswith efficiency, effectiveness, and global acceptance.
8/12/2019 3B-Using CLSI Guidelines to
4/53
We Make the Blue Books
8/12/2019 3B-Using CLSI Guidelines to
5/53
Standards in the Clinical Laboratory
Goal of standardization in thelaboratory:
The right laboratory testat the right time with
the right result leads toquality diagnostics andimproved patient care, andimproved public health
around the world.
Standardized test
Standardized
procedure
Standardizedreporting
Improved outcomes
8/12/2019 3B-Using CLSI Guidelines to
6/53
CLSI documents are developed by volunteer experts from threedistinct constituencies: professions, government, and industry.
Under the supervision of a consensus committee, these volunteerswork on:
Document Development Committees
or
Standing subcommittees and Working groups
How are CLSI Documents developed?
8/12/2019 3B-Using CLSI Guidelines to
7/53
CLSI Consensus Process
Balance
Government
Industry
Professions
8/12/2019 3B-Using CLSI Guidelines to
8/53
Why are CLSI GuidelinesImportant?
The US Food and Drug Administration(FDA) recognizes over 100 CLSIdocuments.
The College of American Pathologists (CAP) recognizes 80 CLSIdocuments.
The Joint Commission recognizes over 144 CLSI documents.
All Evaluation Protocol guidelines in this presentation are recognized
by all three groups.
8/12/2019 3B-Using CLSI Guidelines to
9/53
Why are Evaluation ProtocolsImportant?
Provide clear and thorough guidance.
Evaluation protocols are guidelines for clinical laboratories andmanufacturers to characterize the performance of their analyticalsystems.
Ensure consistency with good laboratory practice.
Good laboratory practice requires clinical laboratories to verifyperformance claims before reporting results used for decisions aboutpatient care.
Help you to comply with the law!
Evaluation of performance characteristics is required by regulatory andaccreditation bodies in the United States.
See http://www.cms.hhs.gov/clia (493.1255).
8/12/2019 3B-Using CLSI Guidelines to
10/53
CLSI and Evaluation ProtocolsCLSI has over 25 Evaluation Protocol Guidelines.
These include:
EP05 Evaluation of Precision
EP06 Evaluation of Linearity
EP09 Evaluation of Bias and Comparability Using Patient SamplesEP10 Preliminary Evaluation (Bias, Carryover, Drift, Linearity)
EP15 Verification of Precision and Trueness
EP17 Limits of Detection and Limits of Quantitation
C28 Defining, Establishing, and Verifying Reference Intervals
8/12/2019 3B-Using CLSI Guidelines to
11/53
Performance CharacteristicsThe seven performance characteristics that should be evaluated before
reporting results of a new test method include:
1. Precision
2. Accuracy (measured bias) or comparability (measured differences)
3. Linearity over the measuring interval or analytical measurementrange (AMR)
4. Limit of detection (LoD) and limit of quantitation (LoQ or analyticalsensitivity)
5. Specificity or interference
6. Reagent or sample (analyte) carryover
7. Reference interval or decision value (interpretive information)
8/12/2019 3B-Using CLSI Guidelines to
12/53
Precision & Accuracy
8/12/2019 3B-Using CLSI Guidelines to
13/53
Apply a clinical perspective; set a target, an analyticalgoal, before you begin
Perform experiments that gather representative data
about a methods analytical performance
Convert data into statistical estimates of errors
Compare error estimates to specifications for medically
allowable error for an objective assessment of theerrors
8/12/2019 3B-Using CLSI Guidelines to
14/53
Introducing a New MethodEstablisha Need
MethodSelection
MethodEvaluation
MethodDevelopment
Define the
Quality Goal
Implementation
RoutineAnalysis
SubmitSpecimen
Quality ControlPractices
PreventiveMaintenance
ReportResult
8/12/2019 3B-Using CLSI Guidelines to
15/53
APPROACH IN METHOD EVALUATION:Evaluate imprecision and inaccuracy
IMPRECISION Refers to Random Analytic Error(Lack of repeatability, reproducibility)
INACCURACY Refers to Systematic Analytic Error(Lack of trueness)
1. Constant2. Proportional
TOTAL ERROR Combined error for a single result
8/12/2019 3B-Using CLSI Guidelines to
16/53
RELIABLE DECISIONS ABOUT PERFORMANCEREQUIRE:
1. Standards for acceptable performance
2. Experimental protocols to estimate
performance reliably
3. Criteria for comparing estimatedperformance with performance standards
8/12/2019 3B-Using CLSI Guidelines to
17/53
PERFORMANCE STANDARD (PS)PERFORMANCE STANDARD (PS)
Specify:
EA . . . Allowable error
XC . . . Decision level
Format:
PS = EA at XC
8/12/2019 3B-Using CLSI Guidelines to
18/53
ALLOWABLE ERROR (EA)ALLOWABLE ERROR (EA)
The amount of error that can be tolerated without
invalidating the medical usefulness of the result
or
causing the test to fail a proficiency testing event
8/12/2019 3B-Using CLSI Guidelines to
19/53
DECISION LEVEL (XC)DECISION LEVEL (XC)
Any concentration of the analyte that is critical formedical interpretation whether for
diagnosis,
monitoring, or
therapeutic decisions.
Laboratory data are most carefully interpreted at
these decision level concentrations.
8/12/2019 3B-Using CLSI Guidelines to
20/53
plasma glucose, mg/dL
1 2 3
0 20 50 80 126 160 200 260 300 340
DECISION LEVELS FOR GLUCOSEDECISION LEVELS FOR GLUCOSE
8/12/2019 3B-Using CLSI Guidelines to
21/53
Performance standards for Glucose Medical Decision
PS1 = 6.0 mg/dL @ 50 mg/dLHypoglycemia
PS2 = 10% = 12.6 mg/dL @ 126 mg/dL Impaired glucose control
PS3 = 30 mg/dL @ 300 mg/dL Poorly controlled diabetes
DECISION LEVELS FOR GLUCOSEDECISION LEVELS FOR GLUCOSE
8/12/2019 3B-Using CLSI Guidelines to
22/53
Sources of Allowable ErrorsSources of Allowable Errors
1. Proficiency testing requirements for acceptable performance
2. Literature guidelinesa. based on physician surveys
e.g.: Karon, Boyd & Klee, Glucose Meter Performance Criteria for TightGlycemic Control Estimated by Simulation Modeling, Clin Chem, 2010; 56:
1091-97
b. based on intra-individual biological variation of analyte Ricos C et al., Scand J Clin Lab Invest,1999; 59: 491-500
Fraser C, Biological Variation: From Principles to Practice, AACC, 2001
Internet at http://www.westgard.com/biodatabase1.htm
3. Input from clinicians and/or professional judgment
8/12/2019 3B-Using CLSI Guidelines to
23/53
8/12/2019 3B-Using CLSI Guidelines to
24/53
Performance Characteristics: Precision
CLSI Guidelines for Precision
EP15: a five-day procedure to verify that imprecisionmeets the claims of a measurement procedure
(EP15 is most frequently used by clinical laboratories for
method evaluation.)
EP05: a 20-day procedure to establish the imprecision fora measurement procedure
8/12/2019 3B-Using CLSI Guidelines to
25/53
Replication Experiment
1. Time period: within-run
within-dayday-to-day
2. Number of samples: minimum of 20
3. Sample matrix: simulate patient sample
4. Analyte concentration: medical decision limit
5. Calculations: mean, standard deviation (SD),coefficient of variation (CV)
8/12/2019 3B-Using CLSI Guidelines to
26/53
Performance Characteristics: Accuracy
Accuracy [Trueness] (Measured as Bias)
Bias: Estimate of a systematic measurement error; a quantitativemeasure of the average difference between results from ameasurement procedure and results from an accepted referencemeasurement procedure.
When a reference measurement procedure is not available for ananalyte, a best-available comparative method may be used tomeasure bias.
Frequently, clinical laboratories perform a comparison of patientsample results between a new and an existing measurementprocedure.
(correlation studies)
8/12/2019 3B-Using CLSI Guidelines to
27/53
CLSI Guidelines for Trueness (Measured as Bias)
EP15: a method comparison to verify that a new methodconforms to a manufacturers claim for comparability toanother procedure.
(minimum of 20 patient samples) EP09: a method comparison to establish a claim for
method comparability.
(minimum of 40 patient samples)
Performance Characteristics: Accuracy
8/12/2019 3B-Using CLSI Guidelines to
28/53
Comparison of Methods Experiment
CLSI EP9-A:
User Comparison of Quantitative Clinical Laboratory Methods
Using Patient Samples
1. Choice of comparative method:
critical for the conclusions which can be made
2. Number of test samples:
minimum N = 40
uniform distribution (EP9-A includes a table) a bin-box approach
8/12/2019 3B-Using CLSI Guidelines to
29/53
8/12/2019 3B-Using CLSI Guidelines to
30/53
8/12/2019 3B-Using CLSI Guidelines to
31/53
Three Approaches to AnalyzingComparison of Methods Data
1. correlation coefficient
2. t-test statistics
3. regression statistics
8/12/2019 3B-Using CLSI Guidelines to
32/53
Sensitivity of Statistical Parametersto Errors
Parameter Random Constant Proportional
LEAST SQUARES
SLOPE no no yes
Y-INTERCEPT no yes noSTD. ERROR yes no no
T-TEST
BIAS no yes yes
sd yes no yes
CORRELATION COEFFICIENTr yes no no
8/12/2019 3B-Using CLSI Guidelines to
33/53
Effect of range on the correlation coefficient
Range 0 to 300 70 to 110
Random Error 10 units 10 units
Corr. Coef., r 0.986 0.764
H
H
8/12/2019 3B-Using CLSI Guidelines to
34/53
8/12/2019 3B-Using CLSI Guidelines to
35/53
Linear regression statistics
Subject to certain limitations:
Data must be linear
Outliers must be carefully examined
Range of data must be wide:
a. r > 0.99 (Waakers et al.)
b. r > 0.975 (CLSI EP9-A)
8/12/2019 3B-Using CLSI Guidelines to
36/53
Recommendations for MethodComparison
Summary
Present graph of data
Present slope, y-intercept, and Sy/x
Present mean and standard deviation of X data Present correlation coefficient ONLY to show that least
squares regression is applicable; if not, use Deming or
Passing-Bablock regression statistics
8/12/2019 3B-Using CLSI Guidelines to
37/53
Performance Characteristics:LinearityLinearity Measuring Interval or Analytical Measurement Range (AMR)
A linearity study is used to establish or verify the measuring interval for ameasurement method.
Measuring Interval: the interval between lower and upper numerical
values for which a method can produce quantitative results suitable forthe intended clinical use.
The measuring interval is verified by demonstrating a linear relationship
between the measured and expected concentration relationships.
CLSI Guideline for Linearity Measuring Interval
EP06: procedures to verify or establish the linear measuring interval of ameasurement procedure.
8/12/2019 3B-Using CLSI Guidelines to
38/53
Performance Characteristics:
LoD/LoQLimit of Detection (LoD) & Limit of Quantitation (LoQ)(sometimes referred to as Analytical Sensitivity)
LoD: the lowest amount of analyte (measurand) in a sample that can be
detected with a stated probability.
LoQ: the lowest amount of analyte (measurand) in a sample that can be
quantified with acceptable precision and bias under stated experimentalconditions.
Usually, laboratories review and accept the manufacturers claims for LoDand LoQ.
But these characteristics can be tested by laboratories using:
CLSI Guideline for LoD and LoQ
EP17: procedures for verifying or establishing the LoD and the LoQ
8/12/2019 3B-Using CLSI Guidelines to
39/53
Performance Characteristics:
Interference Interference: an artifactual increase or decrease in the apparent quantity
of an analyte due to the presence of a substance that reactsnonspecifically with the measuring system.
Most manufacturers evaluate a large number of substances known orsuspected to be potential interferents. They report this information in the
Instructions For Use (IFU). It is not practical for most clinical laboratories to repeat such an
investigation and inspection of the manufacturers information isfrequently sufficient.
But these characteristics can be tested by laboratories using:
CLSI Guideline for Interference
EP7: procedures for testing constant error due to interference
8/12/2019 3B-Using CLSI Guidelines to
40/53
1. See CLSI EP7-A2
2. What to test:
Literature review
Always test hemolysis, lipemia, bilirubin
Tube additives
3. Concentrations to test: Interferent: highest compatible with life
Analyte: at medical decision levels
4. Volume of interferent
8/12/2019 3B-Using CLSI Guidelines to
41/53
Interference Experiment: N=?Number of Measurements / Replicates:
at least several samples per interferent
at least duplicates per sample
EP7 lists a table of N as a function of bias/stm (EA,I/Stm ), with whichone can determine how many replicates are necessary to reach95% probability of observing a certain magnitude of error:
EA,I/Stm No. Replicates EA,I/Stm No. Replicates
0.8 41 1.5 121.0 26 1.6 10
1.1 22 1.8 81.2 18 2.0 71.3 16 2.5 61.4 14 3.0 3
8/12/2019 3B-Using CLSI Guidelines to
42/53
Performance Characteristics:
Carryover Carryover: the discrete amount of reagent or analyte carried by the
measuring system from one test into subsequent test(s), thereby
erroneously affecting test results.
Reagent carryover among different measurement procedures on
multichannel automated analyzers is an evaluation that is usuallyconducted by measuring system manufacturers.
But this characteristic can be tested by laboratories using:
CLSI Guideline for Carryover
EP10: includes an assessment of sample carryover along with other
parameters.
NOTE: EP10 is intended to determine if a device has unacceptableperformance. It is recognized in the CAP Chemistry Checklist as anacceptable way to measure carryover.
8/12/2019 3B-Using CLSI Guidelines to
43/53
Performance Characteristics:Reference Intervals
Reference Interval: interpretive information for laboratory test results thatis frequently provided as the central 95% interval of results for a group ofwell-defined reference individuals.
Laboratories can produce reference intervals in a variety of ways,
including testing procedures found in
CLSI Guideline for Reference Intervals or Decision Value
C28: procedures for establishing a reference interval or verifying thesuitability of a manufacturer-proposed reference interval
8/12/2019 3B-Using CLSI Guidelines to
44/53
Transference of established reference intervals to an individual laboratory ora new method may be accomplished in a variety of ways:
1. Subjective assessment by a responsible individual; the Medical Director (sometimes called by divine judgment)
2. Donor testing
a. Verify with ~ 20 donor samples
b. Validate/Estimate using ~ 60 donor samplesc. Establish using ~ 120 donor samples
3. Calculation
use regression statistics from a comparison of methods study to calculate
reference limits for the new method (Y) that correspond to the referenceinterval limits of the former method (X).
Reference Interval Determination
Y = a + b X
8/12/2019 3B-Using CLSI Guidelines to
45/53
CLSI Makes Life Easier with
StatisPro
In October 2010, CLSI released
StatisPro software:
Direct, faithful implementation ofCLSI Evaluation Protocol Guidelines
Study Advisor step-by-step help foreach study
Four steps to complete a study:
Definition, Data Input, Analysis, and
Signoff
8/12/2019 3B-Using CLSI Guidelines to
46/53
8/12/2019 3B-Using CLSI Guidelines to
47/53
StatisPro Study Design
Performance Claimto be Verified
Study Goal
Identifying
Information
Details of
the Study
Description of
Materials Used
8/12/2019 3B-Using CLSI Guidelines to
48/53
StatisPro Data Entry
Copying and Pasting from any spreadsheet application or Windowsapplication with clipboard support is easy.
8/12/2019 3B-Using CLSI Guidelines to
49/53
8/12/2019 3B-Using CLSI Guidelines to
50/53
StatisPro Study Advisor
8/12/2019 3B-Using CLSI Guidelines to
51/53
StatisPro Demonstration Demonstrate EP15 (method comparison) and EP06 (linearity).
8/12/2019 3B-Using CLSI Guidelines to
52/53
User Experience with StatisPro StatisPro is useful when introducing new methods into your laboratory.
StatisPro is useful when performing six-month linearity or calibration verificationstudies.
By using StatisPro:
You are demonstrating compliance with regulatory and accreditation bodies.
You are ensuring that your laboratory delivers accurate results.
8/12/2019 3B-Using CLSI Guidelines to
53/53
Thank YouQuestions?