62
Method Validation Piet Meijer
Method Validation
Piet Meijer
Method Validation
Outline:
• Introduction
• Method Validation / Verification
• Conclusion
Method Validation
Introduction
Method Validation
Laboratory testing and diagnosis
Qualitative
Semi-quantitative
Quantitative
Correct Test Result ?
Method Validation
Environment
Sample
Reagents Consumables
CalibratorEquipment
Analytical principle
Personnel
Laboratory Test
Test Result
Method Validation
IQC EQC
Control of Analytical Process
Does the analytical test
system provides reliable
result?
Does the analytical test
system provides accurate
result?
Method Validation
Quality Control
Quality Control, either internal or external, is performed at the moment
a laboratory test is in use.
How do we know at the time we start to use a test if it is fit-for-purpose?
Method Validation
Method Validation
Evaluation of the characteristics of a method before this method is
taken into routine use.
Method Validation
• The lab shall validate examination procedures derived from the following
sources:
a) Non-standard methods
b) Laboratory designed or developed methods
c) Standard methods used outside their intended scope
d) Validated methods subsequently modified
Method Validation
ISO 15189
Method Validation
The validation shall be as extensive as is necessary and confirm, through
the provision of objective evidence (in the form of performance
characteristics), that the specific requirements for the intended use of the
examination have been fulfilled.
Method Validation
ISO 15189
Method Validation
New or modified
Method
Validation
or
Verification
Quality
Control
Implementation
Description of
characteristics
Method Validation
Method Validation
Validation / Verification
Validation is a procedure that is used for checking that a test and/or instrument meets the
requirements and specifications and that it fulfills its intended purpose.
Verification is the procedure in which already validated methods are verified for their
performance characteristics before the test is being introduced in routine use.
• No modification of the test
• The lab shall obtain information from the manufacturer/method developer for confirming the
performance characteristics of the procedure.
• The performance claims for the examination procedure confirmed during the verification
process shall be those relevant to the intended use of the examination results.
Method Validation
Validation METHOD Verification
Validation / Verification
Full assessment of the
characteristics of a
laboratory test
Verification of those
characteristics that are
relevant for the
intended use of the test
Method Validation
Validation / Verification
Verification
• Validated examination
procedures used without
modification
Validation
• Examination procedures
derived from the following
sources:
a) Non-standard methods
b) Laboratory designed or
developed methods
c) Standard methods used
outside their intended
scope
d) Validated methods
subsequently modified
Method Validation
• Creatinine measurement on an new chemistry-analyser
• Enzymatic assay: based on conversion of creatinine
• Method developed by firm, in manual full specifications
Validation / Verification
Validation or verification required?
Method Validation
Validation / Verification
• New POCT monitor for blood glucose
• Limited information about performance characteristics available
• No information on reference values for your population
Validation or verification required?
Method Validation
Method Validation / Verification
Method Validation
Performance characteristics should include consideration of:
• Trueness and Accuracy
• Precision including repeatability and intermediate precision
• Measurement uncertainty
• Analytical specificity including interfering substances
• Analytical sensitivity
• Detection limit and quantitation limit
• Measuring interval
• Diagnostic specificity and diagnostic sensitivity
Method Validation
ISO 15189
Method Validation
Action Plan for Validation / Verification
• Define criteria
• Make a schedule
• Perform the validation tests
• Reporting
• Evaluate
• Archive
Method Validation
Method Validation
https://pdfs.semanticscholar.org/6871/8039ed36445c5e02c1100f21632f6858eff1.pdf
This paper suggests one technique of validating methods. There are numerous other ways to validate
methods, all equally acceptable when scientifically justified.
Method Validation
CLSI evaluationprotocol
Function
EP05 evaluation of precision of quantitative measurement procedures
EP06evaluation of the linearity of quantitative measurement procedures: a statistical approach
EP07 interference testing in clinical chemistry
EP09 measurement procedure comparison and bias estimation using patient samples
EP10 preliminary evaluation of quantitative clinical laboratory measurement procedures
EP12 user protocol for evaluation of qualitative test performance
EP14 evaluation of commutability of processed samples
EP15 user verification of precision and estimation of bias
EP17 evaluation of detection capability for clinical laboratory measurement procedures
EP18 risk management techniques to identify and control laboratory error sources
Method Validation
CLSI evaluationprotocol
Function
EP19a framework for using CLSI documents to evaluate clinical laboratory measurement procedures
EP21evaluation of total analytical error for quantitative medical laboratory measurement procedures
EP23 laboratory quality control based on risk management
EP24assessment of the diagnostic accuracy of laboratory tests using receiver operating characteristic curves
EP25 evaluation of stability of in vitro diagnostic reagents
EP26 user evaluation of between-reagent lot variation
EP27 how to construct and interpret an error grid for quantitative diagnostic assays
EP28 defining, establishing, and verifying reference intervals in the clinical laboratory
EP29 expression of measurement uncertainty in laboratory medicine
Method Validation
Example Verification
• Creatinine measurement on an new chemistry-analyser
• Enzymatic assay: based on conversion of creatinine
• Method developed by firm, in manual full specifications
Method Validation
Determine
Accuracy/Trueness
Precision
Analytical specificity
Analytical sensitivity/Detection- & Quantitation limit
Measuring interval
Diagnostic specificity
Diagnostic sensitivity
Verification Plan
Method Validation
Accuracy / Trueness
Accuracy = Qualitative term for the degree by which the measurement result approaches the
reference value. Accuracy is a term including both precision and trueness
Trueness = concordance between the mean value of repeat measurements and the true
value. Degree of trueness is usually expressed as ‘bias’
How?
Use material with known value, e.g. reference material, WHO-material, method comparison
with patient material.
Method Validation
Reference material
Single measurement during 5 days
Criterium: difference between reference material and measured mean value should not exceed 6%
Reference material Creatinine
Mean value (µmol/L)
SD CV (%)
Difference to reference material (%)
1 436 439 3,5 0,8 0,7
2 244 248 1,4 0,6 1,6
3 52 55 0,9 1,6 5,8
Accuracy / Trueness
Method Validation
Determine
Accuracy/Trueness yes
Precision
Analytical specificity
Analytical sensitivity/Detection- & Quantitation limit
Measuring interval
Diagnostic specificity
Diagnostic sensitivity
Verification Plan
Method Validation
Repeatability (within-run precision):
Indicator for the concordance between measurement results if the method is repeatedly applied in
the same test run
Intermediate precision (between-run precision):
Indicator for the concordance between measurement results if the determinations are conducted
using the same material in the same laboratory in consecutive test runs.
Precision
Method Validation
Repeatability
Use 3 stable samples with 3 concentration levels (low, middle, high).
Analysis of 12 repeated measurements per level;
xxxxxxxxxxxx / mmmmmmmmmmmm / hhhhhhhhhhhh
Calculate from the last 10 measurements mean value, standard deviation and coefficient of
variation (CV) per sample
Method Validation
MeanResults for parameter X:
205
185
195
205
195
205
200
215
210
185
Sum: 2000
N = 10
Mean: 2000/10 = 200
Biorad: Basic Lessons in Laboratory Quality Control
Method Validation
Standard Deviation
Biorad: Basic Lessons in Laboratory Quality Control
Method Validation
Standard Deviation
Result Result - Mean (Result – Mean)2
205 5 25
185 -15 225
195 -5 25
205 5 25
195 -5 25
205 5 25
200 0 0
215 15 225
210 10 100
185 -15 225
900 SUM
SD = 900
9
= 10.0
Method Validation
Coeffcient of Variation
Mean = 200
SD = 10.0
CV = (10/200) * 100% = 5%
Method Validation
Intermediate Precision
Use 6 stable samples of which the first 3 are identical to the samples used for within-run
precision. Selection of samples should take reference interval into account.
Perform on at least 10 days the analysis for each sample
Calculate mean value, standard deviation and coefficient of variation (CV) per sample
Method Validation
Documentation firm regarding precision creatinine
Method Validation
Intermediate Precision
During 5 days single measurement of 3 levels.
Criterium: total CV % < 2%
Reference materiaal Creatinine
Gem (µmol/L) SD VC %
1 436 439 3,5 0,8
2 244 248 1,4 0,6
3 52 55 0,9 1,6
Method Validation
Precision and Trueness
Precison Good Bad Good Bad
Trueness Good Good Bad Bad
Method Validation
Determine
Accuracy/Trueness yes
Precision yes
Analytical specificity
Analytical sensitivity/Detection- & Quantitation
limit
Measuring interval
Diagnostic specificity
Diagnostic sensitivity
Verification Plan
Method Validation
Analytical Specificity
The ability of a method to detect only the analyte under consideration. Other components of
the sample should not influence the analytical result
Determine effect of lipemia, icteria and hemolysis, and also effect of potential interfering
factors like medication and M-protein
Interference study
Icteria/hemolysis/lipemia: adding interferent in increasing concentrations to a sample
Method Validation
Analytical Specificity
Method Validation
Interference study:
• Hemolysis, lipemia, icteria
Criterium:
• Difference between sample without interferent (=bilirubin) <4,6%
Analytical Specificity
Method Validation
Bilirubin (µmol/L) index Creatinine (µmol/L) % difference
1 7 16 114,9 0,0
2 9 18 114,9 0,0
3 8 16 114,3 -0,5
4 15 27 114,6 -0,3
5 25 41 114,3 -0,5
6 93 130 111 -3,4
7 205 270 108 -6,0
8 410 530 99,5 -13,4
9 573 784 91,6 -20,3
10 795 1015 81,1 -29,4
11 961 1235 71,9 -37,4
Analytical Specificity
Method Validation
Determine
Accuracy/Trueness yes
Precision yes
Analytical specificity partly
Analytical sensitivity/Detection- & Quantitation limit
Measuring interval
Diagnostic specificity
Diagnostic sensitivity
Verification Plan
Method Validation
The sensitivity of a test to measure a low concentration of a specific analyte.
Detection limit (Limit of Detection or LoD):
the lowest measurable analyte level that can be distinguished from zero
How: LoD can be calculated as the value lying two or three standard deviations above that of
the lowest standard
Analytical Sensitivity
Method Validation
Analytical Sensitivity
Method Validation
Detection limit:
• Blanco sample measured 10 times
• Blanco mean value + 3SD
Result:
• -0,78 µmol/L, SD 1,16, calculated detection limit: 2,71 µmol/L
(-0,78+(3x1,16))
• Detection limit according to firm specifications: 5 µmol/L
Analytical Sensitivity
Method Validation
Quantitation limit (Limit of quantitation or LoQ):
Lowest concentration that can be quantitatively measured with an established acceptable
precision and trueness, under established conditions
How: via a dilution experiment, determine concentration at which CV is ≤20%
Quantitation Limit
Method Validation
Specifications firm: 5 µmol/L
dilution Creatinine (µmol/L) CV %
1 60 0.7
2 30 0.9
4 15 1.6
6 10 11.6
10 6 22.5
12 5 25.8
15 4 27.9
Quantitation Limit
Method Validation
Determine
Accuracy/Trueness yes
Precision yes
Analytical specificity partly
Analytical sensitivity/Detection- & Quantitation limit partly
Measuring interval
Diagnostic specificity
Diagnostic sensitivity
Verification Plan
Method Validation
What?
The measuring range of the analyte concentration that a method can measure directly without
dilution, concentrating or other pretreatment procedures of the sample
How?
Spiking or dilution of a high level sample to obtain at least 5 different concentration levels
Measuring Range
Method Validation
Measuring Range
The laboratory decided not to verify the measuring range
Method Validation
Verification Plan
Determine
Accuracy/Trueness yes
Precision yes
Analytical specificity partly
Analytical sensitivity/Detection- & Quantification
limit
partly
Measuring interval no
Diagnostic specificity
Diagnostic sensitivity
Method Validation
How many patients do I detect with the test? (true positive)
How many patients do I miss? (false negative)
► Sensitivity and specificity of a test
Diagnostic Specificity and Sensitivity
Method Validation
http://notesmedicalstudent.blogspot.com
Diagnostic Specificity and Sensitivity
How sensitive your
test is to detect real
patients
How specific your
test is to exclude
false positve resultsTRUE NEGATIVETRUE POSITIVE
Method Validation
How?
Related to reference value study. Minimal 20-50 samples required monsters from suitable
patient population.
Diagnostic Specificity and Sensitivity
Method Validation
Reference interval
Interval between which 95% of values of a reference population fall into. Reference interval
depends on type of sample and measuring method.
Medical decision value
Analyte value at which medical professionals can determine whether a patient may be
suffering from a certain condition.
Reference Interval / Decision Value
Method Validation
Reference value check with minimal 40 samples of healthy volunteers (maximal 10% of the
measured samples are allowed to exceed the reference values).
Determine by measuring at least 120 samples of healthy volunteers or otherwise ‘normal’
patientpopulation for this analyte
Reference Interval / Decision Value
Method Validation
Determine
Accuracy/Trueness yes
Precision yes
Analytical specificity partly
Analytical sensitivity/Detection- & Quantification
limit
partly
Measuring interval no
Diagnostic specificity no
Diagnostic sensitivity no
Verification Plan
Method Validation
Conclusion
Method Validation
Laboratory testing and diagnosis
Qualitative
Semi-quantitative
Quantitative
Correct Test Result ?
Method Validation
New or modified
Method
Validation
or
Verification
Description of
characteristicsImplementation
Proper method validation / verification supports proper patient care!
