Copyright © 2009 AOAC Research Institute
THE AOAC RESEARCH INSTITUTE PERFORMANCE TESTED METHODSSM
PROGRAM
Validation of Rapid Microbiological Methods
Copyright © 2009 AOAC Research Institute
Outline of Workshop
About AOAC
Validation Overview
PTM Validation Components
Qualitative Study Design
Quantitative Study Design
Environmental Surface Testing
Report and Review
Certification and Publication
Renewal and Modification
About AOAC
Copyright © 2009 AOAC Research Institute
AOAC Vision and Mission
That there be worldwide confidence in analytical results
To serve the communities of analytical science by providing fit-for-purpose methods and services for assuring quality measurements
Copyright © 2009 AOAC Research Institute
AOAC
Solves problems by providing science based solutions
Globally recognized
Independent third party
Establishes standards
Voluntary consensus through stakeholder panels and working groups
Official and regulatory methods
Conducts conformity assessment
Official Methods of AnalysisSM
Performance Tested MethodsSM
Copyright © 2009 AOAC Research Institute
AOAC
AOAC does not:
Regulate products
Buy or sell food, beverage products, or proprietary technologies
Promote specific food and beverage products
Do risk assessments
Set tolerance levels
Own a laboratory or provide laboratory services
Copyright © 2009 AOAC Research Institute
AOAC INTERNATIONAL
Founded in 1884 in Philadelphia
Founded the Official Methods of AnalysisSM
Program
Originally Association of Official Agricultural Chemists
In 1965 became Association of Official Analytical Chemists
In 1991 became AOAC INTERNATIONAL
AOAC represents all of its method communities
Chemistry and microbiology methods
~ 3500 members worldwide
1/3 of members overseas (mostly Europe)
~2700 Official Methods
Copyright © 2009 AOAC Research Institute
AOAC Research Institute
Incorporated in 1991
Administers the AOAC Performance Tested MethodsSM
program
Commercial and proprietary methods
Chemistry and microbiology methods
Harmonized with AOAC Official MethodsSM
program
>100 Current certificates
International membership
PTM Certification Mark has international recognition
Copyright © 2009 AOAC Research Institute
AOAC RI Performance Tested MethodsSM
(PTM) Program
Experience validating a wide variety of method techniques
Cultural methods
Immunoassay
Lateral flow
Polymerase chain reaction (PCR), RT-PCR
Enzyme methods
Bioluminescence
Chemiluminescence
Electrical resistance
ID methods
Copyright © 2009 AOAC Research Institute
AOAC RI Performance Tested MethodsSM
(PTM) Program
Experience validating methods for a wide variety of microorganisms
Listeria
monocytogenes
& Listeria
spp.
Salmonella
E. coli, E. coli O157:H7, &
STEC
Coliforms
Total plate counts
Campylobacter
Vibrio
Staphlyococcus
aureus
& Staph
enterotoxin
Yeasts and molds
Bacillus anthracis
Copyright © 2009 AOAC Research Institute
Up to Date List of PTM Approved Methods
Validation Overview
Copyright © 2009 AOAC Research Institute
Programs for Rapid Methods
Performance Tested MethodsSM
Administered by AOAC Research Institute
Official Methods of AnalysisSM
Administered by AOAC INTERNATIONAL
Copyright © 2009 AOAC Research Institute
Method Validation Guidelines
AOAC Guidelines For Validation Of Qualitative And Quantitative Food Microbiological Official Methods Of Analysis
J. Assoc. Off. Anal. Chem.
85, 1187-1203 (2002)
Available at www.aoac.org
Copyright © 2009 AOAC Research Institute
PTM Program Technical Requirements
Method Developer Responsibilities:
Inclusivity
Exclusivity
Matrix Study
Robustness
Stability
Lot-to-Lot Variation
Independent Laboratory Responsibilities:
Matrix Study
1 Laboratory
1-4 Foods or surfaces (claim dependent)
Copyright © 2009 AOAC Research Institute
OMA Program Technical Requirements
Pre-Collaborative Study
Inclusivity
Exclusivity
Matrix Study
Collaborative Study
Matrix Study
At least 10 labs for qualitative methods
At least 8 labs for quantitative methods
1-6 Foods
AOAC, USDA, FDA, ISO, Health Canada reference methods
Reproducibility
Copyright © 2009 AOAC Research Institute
PTM Program Validation Procedure
Consulting
Required
Waiver can be granted upon approval of managing director
Application
New methods and modifications
Data Collection
Method developer and independent studies
Review
Performance Tested Methods status granted or declined
Certification and Publication
Copyright © 2009 AOAC Research Institute
Consulting
Method Developer Submits:
Consulting application and fee
Claim
Target analyte(s)
Matrices
Method instructions (package insert, user guide, SOP)
Copyright © 2009 AOAC Research Institute
Consulting
AOAC RI Provides:
Overview of current technical requirements
AOAC Guidelines
Appropriate reference methods
Specific validation protocols
Reviewed and approved by General Referee
Validation flowcharts
Copyright © 2009 AOAC Research Institute
Consulting
Benefits
Understanding for both sides
AOAC has better understanding of the method
Method developer has better understanding of AOAC requirements
Can significantly reduce method developer time and cost
Most current technical updates and requirements
Copyright © 2009 AOAC Research Institute
PTM Application
Method Developer Submits:
Application
Package insert, user manual, directions for use
Product labels
QA/QC synopsis
Application fee
Copyright © 2009 AOAC Research Institute
PTM Study
Method Developer Study
Inclusivity and exclusivity
Matrix Study
Lot-to-lot
Stability
Robustness
Copyright © 2009 AOAC Research Institute
PTM Study
Independent Laboratory Study
Contracted by AOAC RI
Method developer provides method materials and training
Performs matrix study
Report
Copyright © 2009 AOAC Research Institute
Review
Method developer submits complete study report and package insert (PI)
Reviewed by general referee and 2 expert reviewers
Method developer addresses reviewers’ comments/questions as necessary
Report and PI revisions submitted as necessary
Decision to grant or decline approval, based on reviewer consensus
Copyright © 2009 AOAC Research Institute
Certification
License Agreement and Use of the Mark
List on PTM Approved Methods website
Annual Review & Certification Renewal
Copyright © 2009 AOAC Research Institute
Publication
Certificate issued
Certification Mark issued
Certification Report published in Inside Laboratory Management
Study report published in Journal of AOAC INTERNATIONAL
Copyright © 2009 AOAC Research Institute
Harmonization with Official Methods of AnalysisSM
Method developer can use PTM study to satisfy
the OMA pre-collaborative requirement
Both
PTM and OMA requirements must be met
Allows for company to perform one large study
rather than two separate ones
Copyright © 2009 AOAC Research Institute
Harmonization Collaborative Study
Collaborative Study
Qualitative
10 labs minimum with valid data
1-
6 matrices, depending on claim
3 levels (uninoculated, low, and high)/matrix
6 test portions/level/matrix/method
Quantitative
8 labs minimum with valid data
1-
6 foods, depending on claim
4 levels (uninoculated, low, medium, and high)/matrix
2 test portions/level/matrix/method
Copyright © 2009 AOAC Research Institute
Harmonization Flowchart
Application for
harmonized study
RI prepares PTM & OMA protocols
Sponsor collects
internal data, IL performs
study
Sponsor writes report
Report reviewed by GR, Stat, 2
ER & 1 Comm. H
Approved study =
PTM certification
Collaborative study (OMA)
& review
Published in AOAC OMA and JAOACI
PTM manuscript published in
JAOAC
Certification report
published in ILM
Approved study = OMA First Action
OMA Final Action
Copyright © 2009 AOAC Research Institute
Harmonization with MicroVal
MicroVal
is a European organization for the validation and certification of alternative methods for the microbiological analysis of food and beverages.
http://www.microval.org/
EN-ISO 16140 –
“Protocol for the validation of the alternative methods”
forms the basis for the European
certification of alternative methods, thus fulfilling European legislation.
http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=30158
Copyright © 2009 AOAC Research Institute
Harmonization with MicroVal
Method developer can use MicroVal
Expert Laboratory study to satisfy the PTM inclusivity, exclusivity and matrix study requirements
Both
PTM and MicroVal
requirements must be met
Since all matrix study work is done by the Expert Lab, an additional independent lab study is not required
Additional PTM studies (lot-to-lot, stability, robustness) are performed by the method developer
Copyright © 2009 AOAC Research Institute
Harmonization with MicroVal
Allows for company to perform one large
study rather than two separate ones
Can use common expert reviewers, but still
include GR for AOAC
Each validating organization retains its own
acceptance criteria
Copyright © 2009 AOAC Research Institute
Validation Complete
Questions?
For additional information, go to http://www.aoac.org/
PTM Validation Components
Copyright © 2009 AOAC Research Institute
PTM Validation Components
Inclusivity
Exclusivity
RobustnessStabilityMatrix Study
Lot-to-lot
Copyright © 2009 AOAC Research Institute
Responsibilities
Method Developer
Inclusivity
Exclusivity
Robustness
Lot-to-lot/stability
Matrix Study
AOAC-RI
Coordinates independent laboratory testing
Matrix Study
Copyright © 2009 AOAC Research Institute
Inclusivity
To ensure the test kit can detect a variety of target organisms
Serovar, species, genus or class
At least 50% should be food or environmental isolates
Source: ATCC, NTCC, etc.
Origin: chicken, cheese, etc.
Cultures grown in test kit media
Test at approximately ten fold
higher than the LOD
Inclusivity
Copyright © 2009 AOAC Research Institute
Exclusivity
To determine the test kit’s ability to discriminate target organisms from non-target organisms
Report source and origin of all isolates
Cultures grown in non-selective media that supports growth
Does not need to be the same for all isolates
If an isolate generates a positive result
Re-test isolate in test method
selective enrichment
Report both results
Exclusivity
Copyright © 2009 AOAC Research Institute
Robustness
Introduce minor variations to the method and observe effects
Critical steps in the method
Steps that an end user could likely vary
Incubation temperature
Incubation time
Reagent volume
Sample volume
Robustness
Copyright © 2009 AOAC Research Institute
Lot-to-lot and Stability
To ensure the manufacturing is consistent between lots and to support the shelf life of the test kit
Can combine lot-to-lot and stability
Test 3 lots
Newly manufactured
Middle of term
Near expiration date
Stability
Lot-to-lot
Copyright © 2009 AOAC Research Institute
Matrix Study
Candidate method
Reference method
Claim
Allocation of matrix testing
Choosing Foods
Sample preparation
Analysis
Paired or unpaired samples
Matrix Study
Copyright © 2009 AOAC Research Institute
Matrix Study
Candidate Method
Method of analysis submitted for validation. The method can be proprietary or noncommercial.
Reference Method
The appropriate official method that is applicable to the analyte
and sample type that the
candidate method is intended to detect. Candidate methods are compared to reference methods when available.
Copyright © 2009 AOAC Research Institute
Matrix Study
AOAC Official Methods of Analysis
USDA FSIS Microbiological Laboratory Guidebook
US FDA Bacteriological Analytical Manual
ISO Methods
Health Canada
Other (Compendium, SMEDP, etc.)
Copyright © 2009 AOAC Research Institute
Claim Number of Matrices
Number of Categories
Variety At least 10 At least 5
Selected At least 5 At least 2
Food Category At least 5 1
Matrix Study Claim
Copyright © 2009 AOAC Research Institute
Matrix Study Allocation of Matrix Testing
Option 1
Method developers required to validate all claimed matrices
Independent lab evaluates 20% of total claimed matrices
Same strain/serovar
for each matrix as was tested in MD lab (if artificially contaminated)
Option 2
AOAC RI coordinates all
matrix study testing
Performed in an independent lab
Copyright © 2009 AOAC Research Institute
Matrix Study Allocation of Matrix Testing
Number of Claimed Matrices
Number of Matrices Evaluated by Independent
Lab
1-5 1
6-10 2
11-15 3
16-20 4
Copyright © 2009 AOAC Research Institute
Matrix Study Choosing Foods
Likely to contain target analyte
Associated with outbreaks
Where analyte
can survive
pH, water activity, spices, etc.
Target market for test method
Trial run of all foods before beginning validation study
Copyright © 2009 AOAC Research Institute
Matrix Study Choosing Foods
Use naturally contaminated samples whenever possible
Salmonella
in poultry products
Campylobacter
in carcass rinses
Artificially contaminated samples can be prepared if necessary
A different isolate should be used for each matrix tested
Details in Annex A of the Micro Guidelines
Copyright © 2009 AOAC Research Institute
Matrix Study Competitive Flora
At least one food type must be tested with competing flora (natural or artificial)
Naturally present in raw matrices
Always done in environmental surface studies
Helps simulate conditions found in nature
If artificially contaminating, competing organism should be at least ten times higher than target
Copyright © 2009 AOAC Research Institute
Matrix Study Cultures for artificial contamination
Matrix study should use foodborne
isolates
Single strain per matrix, not a pool of strains
Internationally recognized source
ATCC or equivalent
Laboratory isolate
Confirmation documentation required
Biochemical
Serological
Copyright © 2009 AOAC Research Institute
Matrix Study Cultures for artificial contamination
Lyophilized cultures
Unless known, recommended to do aging study of organism in matrix
Preparation
Historical counts
Turbidity
Known counts
Copyright © 2009 AOAC Research Institute
Matrix Study Food Preparation
Purchase
Sufficient amount to cover a paired or unpaired study, plus MPN analysis and aerobic plate counts
Screening
Recommended to use both the test method and reference method
If negative for target, prepare food for inoculation
Chop, cut, grind, melt, etc.
Use same lot of material for inoculated samples and uninoculated
control samples
Copyright © 2009 AOAC Research Institute
Matrix Study Artificial Contamination
Liquid
Add diluted organism to large
quantity and mix, stir, etc.
Dry
Add lyophilized organism to
food and mix: roll, shake, etc.
Copyright © 2009 AOAC Research Institute
Matrix Study Artificial Contamination
Solid/moist
Add diluted organism drop-wise over bulk quantity of food, homogenize as appropriate
Spray
Copyright © 2009 AOAC Research Institute
Matrix Study Artificial Contamination
Bulk inoculation required
Make sure to maintain integrity of matrix when inoculating with prepared culture
Be careful not to dilute matrix with the inoculum
Copyright © 2009 AOAC Research Institute
Matrix Study Food Stabilization
Perishable foods
Store 2-4 days at 2-6°C
Quantitative Listeria
store for 1 day at 2-6°C
Room temperature, shelf stable foods
Store minimum of 2 weeks at room temp
Test periodically to ensure level is stable
Frozen foods
Store minimum of 2 weeks at -20 to -30°C
Copyright © 2009 AOAC Research Institute
Matrix Study Analysis
Perform test method according to method instructions
Perform reference method according to most current published method
Perform MPNs
according to reference method
Determine aerobic plate count (APC)
All test portions, MPNs, and APC are to be initiated the same day
Copyright © 2009 AOAC Research Institute
Matrix Study Paired –
common enrichment
25g test portionenrichment
Candidate MethodPresumptive Result
Reference MethodIsolate and confirm
CONFIRMED RESULT
Copyright © 2009 AOAC Research Institute
Matrix Study Unpaired –
separate test portions
Candidate Method25g test portion enrichment
Transfer step,If necessary
Run AssayPresumptive Result
Isolate and confirm
Candidate MethodCONFIRMED RESULT
Reference Method25g test portion enrichment
Isolate and confirm
Reference MethodCONFIRMED RESULT
Copyright © 2009 AOAC Research Institute
Matrix Study Confirmations
All samples, regardless of presumptive result, must be subjected to isolation procedures for confirmation of target colonies
Confirm according to the reference method protocol, or as directed by the test method instructions if included
Confirm the number of colonies as outlined in reference method
Qualitative Method Study Design
Copyright © 2009 AOAC Research Institute
Qualitative Method
“Method of analysis whose response is either presence or absence of the analyte
detected
either directly or indirectly in a certain amount of sample.”
–
AOAC Guidelines
Copyright © 2009 AOAC Research Institute
Validation Study Components
Inclusivity
Exclusivity
Stability RobustnessMatrix Study
Lot-to-lot
Copyright © 2009 AOAC Research Institute
Inclusivity Qualitative
To ensure the test method can detect a variety of target organisms
Serovar, species, genus or class
At least 50% of food origin
Minimum of 50 isolates
100 for Salmonella
Report source and origin of all isolates
Inclusivity
Copyright © 2009 AOAC Research Institute
Exclusivity Qualitative
To determine the test method’s ability to discriminate target organisms from non-target organisms
Minimum of 30 isolates
Report source and origin
of all isolatesExclusivity
Copyright © 2009 AOAC Research Institute
Robustness Qualitative
Introduce minor variations to the method and observe effects
Critical steps in the method
Steps that an end user could likely vary
Incubation temperature
Incubation time
Reagent volume
Sample volume
Robustness
Copyright © 2009 AOAC Research Institute
Robustness Qualitative
Qualitative
Use 2 target isolates and 1 non-target
5 replicates of each for a total of 15 results
Culture organisms
Target -
in test method enrichment media
Non-target -
in non-selective enrichment media
Analyze with test method
Copyright © 2009 AOAC Research Institute
Lot-to-lot and Stability Qualitative
To ensure the manufacturing is consistent between lots and to support the shelf life of the test method components
Can combine lot-to-lot and stability
Test 3 lots
Newly manufactured
Middle of term
Near expiration date
2 target isolates and 1 non-target
5 replicates of each for a total of 15 results
Stability
Lot-to-lot
Copyright © 2009 AOAC Research Institute
Matrix Study Qualitative
Target Contamination Levels
Natural or artificial contaminationto achieve appropriate levels
Important to test within the range of the limit of detection of the test method
Goal is to achieve fractional results
for at least one level for at least one method (test or reference or both)
5-15 positive out of 20 replicates
Matrix Study
Copyright © 2009 AOAC Research Institute
Matrix Study Qualitative
Target contamination level
Low: 0.2-2 cfu/25g
High: 2-5 cfu/25g
Uninoculated: 0 cfu/25g
Adjust levels as necessary to achieve fractional results
Data from one level
is acceptable, as long as results are fractional
Often do 2 levels just in case
20 replicate test portions each for high and low levels, 5 for uninoculated
Per level per method if unpaired samples
Copyright © 2009 AOAC Research Institute
Matrix Study Qualitative
If natural contamination is found:
2 different lots, 20 replicate test portions per lot
20 test portions per lot per method if unpaired
Fractional recovery required for at least one lot
Food may be temperature abused or diluted with uncontaminated product to achieve appropriate levels
Copyright © 2009 AOAC Research Institute
Matrix Study Qualitative
Use a large scale Most Probable Number (MPN) to confirm contamination levels
3 x 100g, 3 x 10g, 3 x 1g, 3 x 0.1g
MPN = statistical estimate of contamination level
Determine using reference method
Use same lot as test portions
Initiate on same day as test portions
Copyright © 2009 AOAC Research Institute
MPN
10
0 g
10
0 g
10
0 g
10
g
10
g
10
g
1 g
1 g
1 g
0.1
g
0.1
g
0.1
g
100g sample + 900ml media
10g sample + 90ml media
1g sample + 9ml media
0.1g sample + 9.9ml media
Copyright © 2009 AOAC Research Institute
MPN
Copyright © 2009 AOAC Research Institute
Matrix Study Confirmation of qualitative levels -
MPN
Record number of positive flasks/tubes
Calculate MPN Index using 3 values
Level 100g 10g 1g 0.1g
High 3/3 3/3 0/3 0/3Low 2/3 1/3 1/3 0/3
Level 100g 10g 1g 0.1gHigh 3/3 3/3 0/3 0/3Low 2/3 1/3 1/3 0/3
Copyright © 2009 AOAC Research Institute
MPN Calculations Number of positive tubes
100g 10g 1g MPN/25g
3 2 2 5.253 2 3 7.253 3 0 6.03 3 1 11.53 3 2 27.5
Using MPN table, determine MPN Index
Copyright © 2009 AOAC Research Institute
Statistical Analysis Qualitative: Paired and Unpaired
Test for significant difference -
Chi-square (χ2)
χ2 results greater than
3.84 indicate a significant
difference between two methods at a 95%
confidence level
Sensitivity
Specificity
Copyright © 2009 AOAC Research Institute
Qualitative Statistics -
Paired Contingency Table
McNemar’s
χ2
= (|b
– c| – 1)2/ b
+ c
Sensitivity rate = a
/ a + b
Specificity rate = d
/ c
+ d
Test Method Result
Positive Negative Total
Reference Method Result
Positive a b a + b
Negative c d c + d
Total a + c b + d
Copyright © 2009 AOAC Research Institute
Qualitative Statistics-Paired Raw Data
SampleTest
MethodPresumptive
TestMethod
Confirmed
ReferenceMethodResult
1 - - -
2 + - -
3 + + +
4 + + +
5 - - -
6 + + +
7 + + +
8 - - -
9 - - -
10 - - -
11 - + +
12 + + +
13 + + +
14 + + +
15 + - -
16 + + +
17 - - -
18 + + +
19 + + +
20 + + +
Total pos 13 12 12
Uninoc.
1 - - -
2 - - -
3 - - -
4 - - -
5 - - -
Copyright © 2009 AOAC Research Institute
Qualitative Statistics-Paired Contingency Table
McNemar’s
χ2
= (|1 – 0| – 1)2/ 1 = 0
Sensitivity = 11 / 12 = 0.92 or 92%
Specificity = 8 / 8 = 1 or 100%
Test Method Result
Positive Negative Total
Reference Method Result
Positive 11 1 12
Negative 0 8 8
Total 11 9
Copyright © 2009 AOAC Research Institute
Qualitative Statistics-Unpaired Contingency Table
Conf. pos Conf. neg
Alternative Method
Pres. pos A B
Pres. neg C D
Reference Method E F
Mantel-Haenszel
χ2
= (n-1)(AF – (B+C+D)E)2
(A+B+C+D)(A+E)(B+C+D+F)(E+F)
where n = A
+
B
+ C
+ D
+ E
+ F
Relative sensitivity = A/E
Copyright © 2009 AOAC Research Institute
Qualitative Statistics-Unpaired Performance Parameters
Relative sensitivity = A/E
Proportion of presumptive positive results that
confirmed positive for the alternative method
relative to the proportion positive results for the
reference method
Copyright © 2009 AOAC Research Institute
Qualitative Statistics-Unpaired Raw Data
SampleTest
MethodPresumptive
TestMethod
Confirmed
ReferenceMethodResult
1 - - +
2 + - -
3 + + +
4 + + -
5 - - +
6 + + +
7 + + -
8 - - +
9 - - +
10 - - -
11 - + +
12 + + -
13 + + -
14 + + +
15 + - +
16 + + +
17 - - -
18 + + +
19 + + +
20 + + -
Total pos 13 12 12
Uninoc.
1 - - -
2 - - -
3 - - -
4 - - -
5 - - -
Copyright © 2009 AOAC Research Institute
Qualitative Statistics-Unpaired Contingency Table
Conf. pos Conf. neg
Alternative Method
Pres. pos 11 2
Pres. neg 1 6
Reference Method 12 8
Mantel-Haenszel
χ2
= (40 -
1)(88 -
108)2 = 0.1
(20)(23)(17)(20)
Relative sensitivity = 11/12 = 0.92 or 92%
Copyright © 2009 AOAC Research Institute
Data Presentation
Matrix
Inoculating strain
Level of contamination
MPN (cfu/25g)
Number of test portions
Reference method results
Alternative method results
Presumptive
Confirmed
Chi square
Sensitivity rate
Specificity rate
Quantitative Method Study Design
Copyright © 2009 AOAC Research Institute
Quantitative Method
“Method of analysis whose response is the amount of the analyte
measured either directly
(e.g. enumeration in a mass or volume) or indirectly (e.g. color absorbance, impedance, etc.) in a certain amount of sample.”
–
AOAC Guidelines
Copyright © 2009 AOAC Research Institute
Validation Study Components
Inclusivity
Exclusivity
Stability RobustnessMatrix Study
Lot-to-lot
Copyright © 2009 AOAC Research Institute
Inclusivity Quantitative
To ensure the test method can detect a variety of target analytes
Serovar, species, genus or class
Minimum of 30 isolates
Report source and origin of all isolates
Test cultures in non-selective broth
No inclusivity for total count methods
Inclusivity
Copyright © 2009 AOAC Research Institute
Exclusivity Quantitative
To determine the test method’s ability to discriminate target organisms from non-target organisms
Minimum of 20 isolates
Report source and originof all isolates
No exclusivity for total count methods
Exclusivity
Copyright © 2009 AOAC Research Institute
Robustness
Introduce minor variations to the test method and observe effects
Critical steps in the method
Steps that an end user could likely vary
Incubation temperature
Incubation time
Reagent volume
Sample volume
Robustness
Copyright © 2009 AOAC Research Institute
Robustness Quantitative
Use 1 target isolate and 1 non-target
1 target isolate at high and low level
5 replicates of each for a total of 15 results
Culture organisms in non-selective broth
Analyze using test method
Copyright © 2009 AOAC Research Institute
Lot-to-lot and Stability Quantitative
To ensure the manufacturing is consistent between lots and to support the shelf life of the test kit
Can combine lot-to-lot and stability
Test 3 lots
Newly manufactured
Middle of term
Near expiration date
Use 1 target isolate and 1 non-target
5 replicates of target at a high level and 5 replicates at a low level
Stability
Lot-to-lot
Copyright © 2009 AOAC Research Institute
Matrix Study Quantitative
Target inoculum
levels
Natural or artificial contamination at appropriate levels
If inoculating, important to have 10 fold between levels
Uninoculated: 0 cfu/g
Low: 10-100 cfu/g
Medium: 100-1000 cfu/g
High: 1000-10,000 cfu/g
5 replicate test portions per level
5 per level per method if unpaired samples
Matrix Study
Copyright © 2009 AOAC Research Institute
Matrix Study Quantitative
If natural contamination is found:
Quantitative method
3 different lots –
5 replicate test portions per lot
3 different contamination levels
Food may be temperature abused or diluted with uncontaminated product to achieve appropriate levels
Copyright © 2009 AOAC Research Institute
Statistical Analysis Quantitative
Perform log10
transformation of microbial counts
Perform outlier tests
Cochran and Grubbs or equivalent
Plot data
Test method on y-axis and reference method on x-axis
Calculate repeatability, sr
Calculate relative standard deviation for repeatability, RSDr
Perform significance tests
ANOVA, paired t-test or independent t-test
Copyright © 2009 AOAC Research Institute
SampleCFU/g
RMLog10
RMCFU/g
TMLog10
TM1 1500 3.18 1000 3.00
2 750 2.88 830 2.92
3 750 2.88 900 2.95
4 2400 3.38 920 2.96
5 1500 3.18 1000 3.00
Quantitative Statistics Medium Inoculation Level –
Raw Data
Copyright © 2009 AOAC Research Institute
Quantitative Statistics Method Agreement
Plot data
Test method on y-axis and reference method on x-axis
Use log10
values
Plot all levels on one graph
Provide regression line, line formula and correlation coefficient
Do not plot data if there is a missing value, i.e. >1100 or TNTC or <10
Missing data and corresponding value are removed from data analysis
Copyright © 2009 AOAC Research Institute
Quantitative Data 3 levels, ten fold apart
Copyright © 2009 AOAC Research Institute
Quantitative Statistics Repeatability (Standard deviation)
SampleLog10RM
Log10TM
1 3.18 3.00
2 2.88 2.92
3 2.88 2.95
4 3.38 2.96
5 3.18 3.00
Mean 3.10 2.97
Calculate sr
and RSDr
(RSDr
= Sr
/mean) for each method using the log10
values
Ref Methodsr
= 0.22RSDr
= 0.71
Test Methodsr
= 0.03RSDr
= 0.010
Copyright © 2009 AOAC Research Institute
Quantitative Statistics
t-Test or Analysis of Variance (ANOVA)
To determine if a significant difference between the alternative method mean and the reference method mean can be detected
Performed for each contamination level per matrix
Typically a 95% confidence interval
If p > 0.05, no significant difference detected
Copyright © 2009 AOAC Research Institute
Quantitative Statistics
Resulting data from t-test in Excel
Copyright © 2009 AOAC Research Institute
Data Presentation
Matrix
Inoculating organism
Contamination level
Organisms/g
Log10
organisms/g
Mean
Repeatability, sr
Relative standard deviation, RSDr
P-values from t-test or ANOVA comparison of means
Environmental Surface Testing Study Design
Copyright © 2009 AOAC Research Institute
Environmental Surface Testing
Test kits used in the analysis of environmental surfaces
Typically qualitative methods
Same parameters used for food testing are required for surface testing:
Inclusivity
Exclusivity
Robustness
Lot to lot/stability
Matrix study
Inclusivity
Exclusivity
RobustnessStabilityMatrix Study
Lot-to-lot
Copyright © 2009 AOAC Research Institute
Environmental Surface Testing
Reference methods
Listeria
–
USDA FSIS MLG
Salmonella
–
FDA BAM
Guideline
“Study Protocol for Validation of Environmental Surfaces Claims for Target Organisms.”
Draft
document
Copyright © 2009 AOAC Research Institute
Matrices
Stainless steel
Rubber
Wood
Food grade painted surfaces
Plastic
polyethylene, polypropylene, polycarbonate
Ceramic, glazed earthen material or glass
Sealed concrete
Cast iron (coated to prevent rusting)
Air filter material
Copyright © 2009 AOAC Research Institute
Claim
Environmental Surfaces – All 9 surfaces required
5 using sponge – 4” x 4” surface area
4 using swab – 1” x 1” surface area
Selected Surfaces – 2-8 Surfaces
Each surface listed
Individual Surface type
Copyright © 2009 AOAC Research Institute
Can add stabilizer to inoculum
Dilute in non-nutritive buffer
Inoculate surface
At least one surface must be contaminated with a competitor at a level 10 times greater than the target
Dry overnight
Recovery levels can vary
Strain/surface combinations
Laboratory conditions (humidity, temperature)
Preliminary testing is critical
Surface Inoculation
Copyright © 2009 AOAC Research Institute
Sample surface
Swab or sponge
Sponge/swab entire sampling area in both a horizontal and vertical motion
Swab/sponge held in neutralizing broth (DE broth) at room temperature for 2 hrs prior to testing
Add enrichment media to collection device
Surface Sampling
Copyright © 2009 AOAC Research Institute
Performing Study
If unpaired samples, side by side surface areas are tested, one for each method
Perform remainder of test method and reference method
Estimate initial level by determining plate count of inoculum
(MPNs
not applicable)
At least one level by at least one of the methods must achieve fractional recovery 5-15 positive test portions out of 20
Copyright © 2009 AOAC Research Institute
Performing Study
All samples, regardless of test method results must continue through confirmation process
Chi-square statistics and performance parameters are determined as per food studies based on whether test portions are paired or unpaired
Present data as per food studies omitting MPN results
Report and Review
Copyright © 2009 AOAC Research Institute
Report
Data collection is complete
Independent lab has submitted report
Internal lab testing is complete
Analyze data
See qualitative and quantitative sections for specific
statistical information
Compile into single report
Copyright © 2009 AOAC Research Institute
Report Format
Template is provided with the validation
outline
Format is in place to facilitate subsequent
publication in JAOAC
Indicate if method is PTM only or if harmonized
with OMA
Copyright © 2009 AOAC Research Institute
Report Format
Method / Kit Title:AOAC Performance Tested MethodSM
XXXXXX
AbstractMethod AuthorsSubmitting CompanyIndependent LaboratoryReviewers
1 Scope of method1.1Target analyte1.2 Matrices 1.3 Summary of validated performance claims
2 Definitions2.1 Limit of Detection2.2 Accuracy2.3 Precision
3 PrincipleGeneral description of the scientific principle of the test method.
4 General InformationGeneral information about the target analyte, matrix, occurrence of the analyte
in nature and its effect.
Copyright © 2009 AOAC Research Institute
5 Test Kit Information5.1 Kit Name5.2 Catalog Number 5.3 Ordering Information
5.3.1 USA.—
address with telephone, fax , e-mail, and website addresses.5.3.2 Europe.—
other offices if applicable.5.4 Test Kit Reagents
5.4.1 Reagent one.—
brief description.5.4.2 Reagent two 5.4.3 Etc. as needed.
6 Additional supplies and reagentsCite supplies and reagents as needed.—
brief description, specifications, and source.7 Apparatus
7.1 Heating block.—
brief description, specifications, source.7.2 Temperature-controlled waterbath.—
brief description, specifications, source.8 Standard Reference MaterialsExample: ECRC 96-1555 (available from the E.coli
Reference Center, Pennsylvania State University, University Park, PA., USA)9 Standard SolutionsExample: Tris-borate EDTA electrophoresis buffer (TBE).—
Dissolve 108 g Tris, 9.3 g NA2
EDTA, 55 g boric acid in ca 800 mL
H20. Adjust pH to 8.2 with concentrated HCl
and bring final volume to 1 L with H20.
Report Format
Copyright © 2009 AOAC Research Institute
Report Format
10 Safety Precautions
Example: Electrophoresis gels contain ethidium
bromide which is a potential mutagen. Observe all of the manufacturer’s precautions for handling and disposal of these gels.
11 General Preparation 11.1 Assemble the gel electrophoresis apparatus according to the manufacturer’s instructions. 11.2 Turn on the heating blocks and set temperatures to 37o and 95oC.
12 Sample Preparation
Describe how samples are processed.13 Analysis
13.1Title of first major step in analysis such as ExtractionGeneral description of process if needed.
13.1.1 Step-by-step description of process 13.1.2 Etc as needed.
13.2 Title of second major step in analysis such as Sample IncubationGeneral description of process if needed.
13.2.1 Step-by-step description of process 13.2.2 Etc as needed.
14 Interpretation and Test Result Report
Description of interpretation procedure and how results are reported.
Copyright © 2009 AOAC Research Institute
Report Format
15 Method Developer Validation Studies
Validation studies in the conducted in the Method Developer laboratory15.1 Title of first study such as “Robustness Testing”General description of the study such as: The effects of perturbations in four method parameters were investigated: 1)
temperature [69.5°C / 70.5°C]; 2) reduced sample volume (45 µL / 50 µL); 3) time delay (0 min / 15 min) in cooling rack; and 4) sample volume (3 µL / 5 µL).
15.1.1 Methodology15.1.2 Results
15.2 Title of the second study such as “Matrix Study”15.2.1 Methodology15.2.2 Results
16 Independent Validation Study
Validation study conducted by an independent laboratory under the direction of the AOAC Research Institute. 16.1 Title of study such as “Matrix Study”
16.1.1 Methodology
16.1.2 Results
17 Discussion
General discussion of the all of the studies with particular emphasis on unexpected results, new knowledge about the analyte, matrix, or analytical technique.
18 Conclusion
19 References
Copyright © 2009 AOAC Research Institute
Report Format
Abstract
Method Authors
Submitting Company
Independent Lab
Reviewers
Introduction
Materials and methods
Summary of results
Discussion
Conclusion
References
Acknowledgements
Copyright © 2009 AOAC Research Institute
Review
Method Developer submits
Study report
Package insert (directions for use)
QA/QC description
Package labels
Copyright © 2009 AOAC Research Institute
Review
Report, Package Insert, and review form sent to:
PTM: GR and 2 expert reviewers
Harmonized PTM/OMA: GR, 2 expert reviewers, 1 Committee H member, and statistical advisor
2 weeks allowed for initial review
Reviewer comments are sent to author
Technical consultant available to answer questions and assist with responses
Copyright © 2009 AOAC Research Institute
Review
Report and/or package insert revised as necessary and resubmitted to reviewers
One week allowed for review
Decision or additional comments sent to author
Same time schedule as above for any additional reviews, if needed
Review complete once consensus (for or against) is achieved
Copyright © 2009 AOAC Research Institute
Potential Problems
Fractional recovery requirement not met
Methods not followed exactly
Did not use correct or current reference method
Did not follow current validation policies
Data not acceptable
Test method did not perform as well or better than the reference method
Copyright © 2009 AOAC Research Institute
Potential Resolutions
Can retest if:
Fractional recovery is not achieved
Method(s) not followed exactly
Out of date or incorrect methods or policies were followed
Matrices can be removed from claim if test method is unsuccessful
To be determined by expert reviewers
Certification and Publication
Copyright © 2009 AOAC Research Institute
Approval Criteria
Applicant’s performance data supports and confirms all claims made in the method’s descriptive insert
Independent laboratory’s performance data corroborates the applicant’s performance data within the statistical limits specified in the testing protocol
Test method must perform as well or better than the reference method
Copyright © 2009 AOAC Research Institute
Approved
Upon approval, the method author will receive the following
Approval letter
Certification mark
Certificate
Web entry
License agreement
Request for ILM article/certification report
Copyright © 2009 AOAC Research Institute
Use of Certification Mark
License agreement is executed covering the Institute- owned PTM certification mark
Authorizes the method developer to use the mark on the approved test method, packaging, insert, informational and promotional materials
Denotes that the method was evaluated by the AOAC RI and confirms claims
Mark must be discontinued if certificate is not renewed
Copyright © 2009 AOAC Research Institute
Certificate Information
Certificate of Performance TestedSM
Status
Certificate No. (number)
The AOAC Research Institute hereby certifies that the performance of the test method designated as:
Full Test Method Namemanufactured byCompany Name
Company AddressCountry of Company
has been reviewed under the AOAC Research Institute's Performance Tested MethodsSM Program, and found to perform as stated by the manufacturer. This certificate authorizes the manufacturer to display the AOAC Performance TestedSM certification mark along with the statement - "THIS TEST KIT'S PERFORMANCE WAS REVIEWED BY AOAC RESEARCH INSTITUTE AND WAS FOUND TO PERFORM TO THE MANUFACTURER'S SPECIFICATIONS" - on the above mentioned test kit until December 31, (Current Calendar Year).
Signed for AOAC Research Institute: Date:________________
Scott G. Coates Managing Director
Copyright © 2009 AOAC Research Institute
Sample Web Entry
Copyright © 2009 AOAC Research Institute
Publications
Inside Laboratory Management
Summary of study
Serves as certification report
Journal of AOAC INTERNATIONAL
Complete study
Web site
http://www.aoac.org/testkits/testedmethods.html
Renewal and Modification
Copyright © 2009 AOAC Research Institute
Renewal
Applicant certifies that no changes have been made to the test method since the original PTM approval and the method performs as originally evaluated
Submit Method Modification Review Form if changes have been made
Granted for 1 year
Copyright © 2009 AOAC Research Institute
Renewal
Changes in test method
Notification -
responsibility of the licensee to notify
AOAC-RI of any test method changes
Failure to appropriately notify the AOAC RI of
changes may result in the cancellation of PTM
certificate
Copyright © 2009 AOAC Research Institute
Modification
Changes made in test method which effect:
Test method instructions
Labeling changes
Deletion or restatement of validated claims or procedures
Increase/decrease stability claims
Test method performance
Changes to method components
Changes to enrichments/procedure
Changes to equipment
Changes to claim
Changes to manufacturing process
Copyright © 2009 AOAC Research Institute
Modification
Review Levels
Level 1
Internal AOAC RI review
Change does not alter the validated performance of the test method
Level 2
Licensee data required
Reviewed by GR
Level 3
Licensee and independent data required
Reviewed by GR and 2 Expert Reviewers
Copyright © 2009 AOAC Research Institute
Modification
Testing requirements
Level 1
Document review, no additional testing required
Level 2
Matrix study required, at least 3 matrices
Modified method compared to reference method
Data collected by licensee
Copyright © 2009 AOAC Research Institute
Modification
Testing requirements
Level 3
Licensee and independent lab data required
Same as for new PTM study
Matrix extension
Matrix study required
Independent lab data required if total
claimed matrices for
the method exceeds 4
Copyright © 2009 AOAC Research Institute
Emergency Response Validation (ERV) Program
Purpose is to respond immediately to emerging food contamination crisis
Salmonella
contamination in peanut butter
Melamine in infant formula
Provides a rapid one-matrix, one-analyte
validation of
candidate method
Candidate methods evaluated using blind-coded samples prepared by independent lab
Special “Certificate of Validation”
granted upon approval
PTM certified and OMA validated methods eligible (where applicable)
Copyright © 2009 AOAC Research Institute
Cancellation of PTM Certificate
If a licensee has:
Not complied with original agreement relative to the use of the mark
Not responded adequately to complaints of poor performance reported by users
Modified the test method without notifying the RI
Failed to submit a renewal application
Requested PTM status to be discontinued
Copyright © 2009 AOAC Research Institute
Complaints
Formal complaints from applicant to be
directed to the AOAC RI Managing Director in
writing
Formal complaints from test method user
discussed with the method licensee
Failure to address user complaints will result in the
potential cancellation of PTM certificate
Copyright © 2009 AOAC Research Institute
Potential Problems
Incomplete method instructions
Too vague
Important parameters not described
Method not fully adapted to new matrices
Clinical method often applied to foods and other matrices that vary greatly in protein, fat, and other components
Manufacturing and quality control problems
Copyright © 2009 AOAC Research Institute
Summary
Background
Validation Overview
PTM Validation Components
Qualitative Study Design
Quantitative Study Design
Environmental Surface Testing
Report and Review
Certification and Publication
Renewal and Modification
Proposed Revisions to the AOAC Guidelines for Microbiological Method Validation
Copyright © 2009 AOAC Research Institute
AOAC RI Disclaimer
The proposed revisions contained in this presentation are being reviewed but have not been approved by the AOAC Methods Committee on Microbiology nor the Committee on Statistics
This presentation is applicable to the Performance Tested Methods
Program and the
Official Methods
of Analysis Program
Copyright © 2009 AOAC Research Institute147
Proposed Revisions
New terms and concepts for Qualitative Methods
POD –
Probability of Detection
Example analysis of qualitative SLV data
LPOD –
Cross-laboratory Probability of Detection
Repeatability
Reproducibility
Example analysis of qualitative collab
data
Study design changes
MPN estimation
Copyright © 2009 AOAC Research Institute
POD: Probability of Detection
The proportion of positive analytical outcomes for a qualitative method for a given matrix at a given bacterial level or concentration.
POD = x/N
For example, if 17 out of 20 replicates at a given concentration are positive by the candidate method, then the candidate method POD is 17/20 = 0.85 or 85% at that concentration.
Report POD values with 95% confidence intervals
Copyright © 2009 AOAC Research Institute
POD: Probability of Detection
PODC
= POD of candidate method
PODR
= POD of reference method
PODCP
= POD of candidate method presumptive result
PODCC
= POD of candidate method confirmed result
Replaces sensitivity and specificity
No distinction between matched and unmatched analyses
Does not require a reference method
Copyright © 2009 AOAC Research Institute
Graph POD Values vs. Concentration
Listeria Detection
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0 2 4 6 8 10 12
Concentration (MPN/25g)
Prop
ortio
n D
etec
ted
(PO
D)
CP CC C R
Copyright © 2009 AOAC Research Institute
dPOD: Difference of Two POD Values
dPOD
values are differences between two POD values:
dPODC
= PODC - PODR
dPODCP
= PODCP - PODCC
Calculate 95% confidence intervals on dPOD
values
If the confidence interval on dPOD
does not contain zero, then the two POD values are statistically different.
Copyright © 2009 AOAC Research Institute
LPOD: Cross Laboratory POD
Calculated as the mean POD across laboratories in a collaborative study:
LPOD = ∑x / ∑N
LPODC , LPODR , LPODCP , LPODCC
Calculate 95% confidence intervals of LPOD values
Graph LPOD vs. Concentration
Copyright © 2009 AOAC Research Institute
dLPOD: Difference of Two LPOD Values
dLPOD
values are differences between two LPOD values:
dLPODC
= LPODC
– LPODR
dLPODCP
= LPODCP
– LPODCC
Calculate standard deviation of dLPOD, sdPOD
Calculate 95% confidence intervals of dLPOD
values
If the confidence interval on dLPOD
does not contain zero, then the two LPOD values are statistically different.
Copyright © 2009 AOAC Research Institute
Repeatability
Calculate repeatability standard deviation, sr
sr
= √LPOD(1-
LPOD)
Calculate repeatability standard error, SEr
SEr
= sr
/ √n
Copyright © 2009 AOAC Research Institute
Reproducibility
Calculate the reproducibility standard deviation, sR
sR
= √∑
(PODl
–
LPOD)2/ L-1
Calculate 95% confidence interval on sR
Copyright © 2009 AOAC Research Institute
Comparison of Repeatability and Reproducibility
Compare SEr
to sR
If SEr
falls within the 95% confidence interval of sR
, then the laboratory effect is not significant.
The contribution of the laboratory effect to the variation in POD values across laboratories can be estimated by
slab
= √(sR2
– (sr2/n))
If sR
< Ser , then report slab
as 0%.
Copyright © 2009 AOAC Research Institute
Study Design Changes
Matrix Study –
Qualitative Methods
Single Lab
At least 3 levels: POD = 0, POD = 0.25-0.75, POD = 1.0
20 replicates each level
Collab
Minimum of 8 laboratories
At least 3 levels: LPOD = 0, LPOD = 0.25-0.75, LPOD = 1.0
12 replicates each level
Copyright © 2009 AOAC Research Institute
MPN Estimation
SLV
5-tube 3-level MPN
2-
or 3-fold serial dilutions
Can use reference method replicates in
calculation of MPN –
formula provided
Collab
Use reference method replicates from all labs
Copyright © 2009 AOAC Research Institute
Reminder
The proposed revisions contained in this presentation are being reviewed and have not been approved by the AOAC Methods Committee on Microbiology nor the Committee on Statistics
Questions?
Special Thanks to our AOAC
Statistical Advisors:
Paul Wehling, Robert LaBudde, and
Daryl Paulson