Christopher N. Greene, PhD Newborn Screening and Molecular Biology Branch,

Division of Laboratory Sciences NCEH, CDC

Tuesday, 9th July 2013

Assay Quality Considerations

National Center for Environmental Health U.S. Centers for Disease Control and Prevention

Molecular Testing

Testing Parameters Examples Target Analyte Genetic Mutations, Gene Dosage Platform PCR, qPCR, bead arrays, Fluorescent Probe, DNA

Sequencers Complexity Single- or multi-plex of single nucleotides,

Laboratory Developed Test, ASRs, FDA-approved, Gene Sequencing, Next-Gen Sequencing

Multiple molecular assays have been developed for screening procedures. These assays vary on their complexity, the target analyte, as well as the platform used for analysis.

Molecular Testing

When Adopting a New Test Required Evidence Test Development - LDT Accuracy, precision, reproducibility,

sensitivity, specificity, robustness Test Validation - ASR Clinical and analytic performance and

test limitations Test Verification – FDA approved Identify samples or reference

materials of known quantitative or qualitative values to verify test performance

Molecular tests require verification or validation of assay results. Level of evidence required is related to the type of assay, i.e. laboratory developed test, analyte specific reagents or FDA approved assays.

Laboratory Regulatory and Accreditation Guidelines

US Food and Drug Administration (FDA): approves kits and reagents for use in clinical testing

Clinical Laboratory Improvement Amendments (CLIA): Regulations passed by Congress1988 to establish quality

standards for all laboratory testing to ensure the accuracy, reliability and timeliness of patient test results regardless of where the test was performed

College of American Pathologists (CAP): Molecular Pathology checklist

State Specific Regulations

Testing Parameters

Accuracy

Sensitivity

Specificity

Precision

Limit of Detection

Range

Reproducibility

Robustness

Definitions

• Accuracy: Agreement between test result and the “true” result – Comparison of results between new method and a

reference method – Results of new method on certified reference materials

• Sensitivity: Ability to obtain positive results in

concordance with a positive result of a reference method – can the test detect a positive result

Definitions

• Specificity: Ability to obtain a negative result for a qualitative test in concordance with reference method

• Precision: Measure of random error in quantitative tests, the closeness of agreement usually expressed as standard deviation of CV

Definitions

• Limit of Detection: Lowest amount of analyte that is distinguishable from background or a negative control

• Range: Range of test values expected for the test population. For quantitative, high and low range of target, for qualitative, possible detectable alleles of the assay

Definitions

• Reproducibility: Equivalent to precision for qualitative and semi-quantitative tests. Can be used to indicate with-in or between run agreement of test results

• Robustness: Test precision, given small, deliberate changes in test conditions (e.g. incubation times, temperature variation)

Pre-Validation

Choose and evaluate assay methodology

Determining analytic performance of an assay involves:

Reviewing professional guidelines and relevant literature

Stringent design/analysis of primers and probes

Quality and quantity of extracted nucleic acid

Appropriate platform for the test

Availability of controls or calibrators

Optimization of amplification and detection

Verifying Assay Specifications

• Demonstrate that laboratory can obtain specifications comparable to manufacturer – Accuracy – Precision – Reportable Results

• Applies to unmodified FDA-approved tests

• Verify appropriate test results for population

• May require matrix-appropriate materials during verification

Assay Validation

• Performance characteristics of laboratory developed tests

• Reagents that are the components of a test that is created without regulatory approval – ASR

• Use of FDA-approved tests that have been modified for the testing procedure

Establish that the assay fulfills its intended use with the appropriate sample type and population

Modified FDA Test

• Alteration of any assay step or analyte for the test that may affect performance

• Changes could include: – Type or concentration of specimen – Storage specifications – Extraction method – Amplification parameters or platforms – Use of test for other purposes

Assay Validation

Required for: New testing methodology Assay modification – includes cross-checks for different

makes/models of instrumentation

Applies to:

Modified FDA assays Laboratory Developed methods

Controls for Each Run

Appropriate positive and negative controls should be included for each run of specimens being tested

Molecular Assay Controls

Positive controls: Inhibitors Component failure Interpretation of results Sources:

Residual positive DBS PT samples QC materials through purchase or exchange

Negative controls: Nucleic acid contamination

Common Molecular Assay Problems and Trouble Shooting

Temperature errors

Template/Sequence

PCR inhibitors

Buffers

Bad dNTPs

Bad primers

Bad enzyme

Critical Molecular Assay Components

Nucleic Acids: Prepare aliquots appropriate to workflow to limit freeze-thaw cycles Primers and probes dNTPs Genomic DNA

4-8°C: Up to one year -20°C: Up to seven years

Enzymes Benchtop coolers recommended

Fluorescent reporters Limit exposure to light Amber storage tubes or wrap in shielding (foil)

Positive Controls

Ideally should represent each target allele used in each run

May not be feasible when:

Highly multiplex genotypes possible

Systematic rotation of different alleles as positives

Rare alleles

Heterozygous or compound heterozygous specimens

Positive Controls

Assays based on presence or absence of product Internal positive amplification controls to distinguish true

negative from false due to failure of DNA extraction or PCR amplification

PCR amplification product of varying length Specimens representing short and long amplification products to

control for differential amplification

Quantitative PCR Controls should represent more than one concentration

Control copy levels should be set to analytic cut-offs

Quantitative PCR Controls

• Contamination controls • Calibration control with reliable low limit of

quantification • Measures for consistency, especially for

DBS • Measure of DNA quality – internal control

False Negative: ADO

Allele drop-out (ADO): the failure of a molecular test to amplify or detect one or more alleles

Potential causes: DNA template concentration

• Incomplete cell lysis • DNA degradation

Non-optimized assay conditions Unknown polymorphisms in target sites Reagent component failure

Major concern for screening laboratories Confirmation of mutation inheritance in families is not an option

DNA Degradation

Lane 1 + 7: 1kb size standard ladder

Lane 2: 100ng control genomic DNA

Lanes 3-5: Crude cell lysates

Question?

How can you control for presence of

sufficient amount/quality of DNA for a

PCR based test in a NBS lab?

PCR Amplification Controls

• Allele-specific amplification • Are there problems with this assay? • What additional controls would be useful?

Allele 1 + 2 Allele 2 Allele 1 Reference Negative

PCR with Internal Controls

Tetra-primer ARMS-PCR Simultaneous amplification of:

Positive amplification control

Mutation allele

Reference allele

Alternative to tetra-primer ARMS is to

include an additional primer set to amplify a different control sequence

Allele Drop-out in PCR Testing

ggacatgatGcactacatgctccaaggtagtggag Cgtgatgtacgaggttccat

5’

ggacatgatGcactacatgctccaaggtagtggag cctgtactaCgtgatgtacgaggttccat

5’

Allele Drop-out in PCR Testing

SNP in primer site

ggacatgatGcGctacatgctccaaggtagtggag gatgtacgaggttccat

5’

ggacatgatGcGctacatgctccaaggtagtggag

Cgtgatgtacgaggttccat 5’

False Negatives: Deletions

A

G

Reverse Primer

Forward Primer

Reverse Primer

Forward Primer

False Negatives: Deletions

A

G

Reverse Primer

Forward Primer

Reverse Primer

Forward Primer Deletion

False Positives

Potential causes:

Non-optimized assay conditions

Unknown polymorphisms in target sites

Gene duplications

Oligonucleotide mis-priming at related sequences

Psuedogenes or gene families

Oligonucleotide concentrations too high

Nucleic acid cross-contamination

How Many Samples?

• New test or to replace an old test? • How is the test to be used? • What performance criteria are most important? • How available are appropriate samples?

• For screening: daily load vs. weekly load vs.

monthly load – it depends

It depends…

CLSI Evaluation Protocols recognized by FDA as Consensus Standards

• EP5-A2: Evaluation of Precision Performance of

Quantitative Measurement Methods • EP9-A2: Method Comparison and Bias Estimation Using

Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative Clinical

Methods • EP12-A User Protocol for Evaluation of Qualitative Test

Performance • EP15-A2: User Demonstration of Performance for

Precision and Accuracy

Molecular Assay Proficiency Testing Material Sources

CDC NSQAP

UKNEQS

EuroGentest

CAP

Maine Molecular

SeraCare

Corielle

ECACC

In-house samples

Round-robin with other NBS laboratories

Professional Guidelines

American College of Medical Genetics (ACMG) Standards and Guidelines for Clinical Genetics Laboratories

Clinical and Laboratory Standards Institute (CLSI) MM01-A2: Molecular Diagnostic Methods for Genetic Diseases MM13-A: Collection, Transport, Preparation, and Storage of

Specimens for Molecular Methods MM14-A: Proficiency Testing (External Quality Assessment) for

Molecular Methods MM17-A: Verification and Validation of Multiplex Nucleic Acid

Assays MM19-P: Establishing Molecular Testing in Clinical Laboratory

Environments

Additional Sources