Midwest Biopharmaceutical Statistics WorkshopMuncie IN, May 24-26, 2010
Statistical Considerations for Defining Cut Points and Titers in
Anti-Drug Antibody (ADA) Assays
Ken Goldberg, Non-Clinical Statistics
Johnson & Johnson Pharmaceutical Research & Development, LLC, Chesterbrook, PA
Outline
• Introduction– Why are ADA and IR assays important?
• Two case studies1. RIA: How to define %binding?
2.ECL: How to define titer cut point?
3.Both use a Huber 3-parameter nonlinear logistic regression
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 2
Immune Response (IR) Assay
• Primary question: ADA, Yes or No?
• Every biologic must be evaluated.
• Safety and Efficacy concerns.
• Too much IR can kill a compound.
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 3
Biological Drug Products are Different than Traditional
Small Molecule Drugs
• Made by cells not chemists
• Complicated manufacturing process
• Small & simple vs large & complex chemical structures
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 4
Reference: Genentech, Inc. http://www.gene.com/gene/about/views/followon-biologics.html
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 5
Adverse Clinical Sequelae
• Hypersensitivity & autoimmunity
• Altered PK– Drug neutralization– Abnormal biodistribution– Enhanced clearance rate
Regulatory bodies require ADA
evaluation for all biologics
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 6
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 7
Immune Response (IR) Assay Challenges
• Cut Point for confidence that screening bioassay response (eg, ECL, OD, RLU, CPM) reflects immunogenicity
• Statistical issues of variance components, distributions, outliers, …
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 8
Screening Cut Point Flags 5% of Naïve Samples as False
Positive• Use Mean + 1.645 x SD with caution
– Only for normally independently distributed data without outliers
– Usually requires at least a transformation like logs
• Nonparametric often easier– Simply use 95th percentile– Caution if unbalanced design
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 9
ELISA Activity
Positive Control
Negative Control
PatientA
PatientB
PatientC
AssayControl
1.689 0.153 0.055 0.412 1.999 0.123
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 10
-1/ OD̂ .75
Frequency
0.0-3.5-7.0-10.5-14.0-17.5-21.0-24.5
25
20
15
10
5
0
-1.61Mean -3.872StDev 1.381N 118
Histogram of -1/ OD^.75Normal Distribution Overlaid
Mean and Standard Deviation based on mixed effects analysis of 117 non-outliers.
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 11
ELISA Cut Point Example
Analysis of an RIA Cut Point Assay Validation Experiment
• 6 Assay controls
• 2 Analysts with 3 assays each
• 2 Populations (Normal and Diabetes)
• 75 Naïve Human Serum samples
• Nonnormal data
• Unequal variances
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 12
RIA Histogram of 450 Naïve Sample Results
Transformed %Binding = ln(35+%Binding). Parametric Cut Point = 10.757 ± 3.524.Transformed Cut Point = 3.823 based on adjusted mean = 3.402 and total standard deviation = 0.256.
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 13
RIA Normal Probability Plot of 450 Naïve Sample Results
Transformed %Binding = ln(35+%Binding). Parametric Cut Point = 10.757 ± 3.524.Transformed Cut Point = 3.823 based on adjusted mean = 3.402 and total standard deviation = 0.256.
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 14
SAS Code
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 15
proc mixed; * For Cut Point; class sample run analyst; model t35Pct0_100= / ddfm=sat; random sample; random sample / type=sp(exp)(tube) subject=analyst*run; repeated / group=analyst*run;
proc mixed; * For Example Hypothesis Test; class sample run analyst; model t35Pct0_100 = Analyst Tube / ddfm=sat; random sample; random intercept tube / type=fa0(2) subject=analyst*run; repeated / group=analyst*run;
My RIA Notation
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 16
MinCPM = Minimum of the 2 Sample CPMsMaxCPM = Maximum of the 2 Sample CPMsAvgCPM = Average of the 2 Sample CPMsCV = Coefficient of Variation of the 2 Sample CPMs
B0 = Average of all 6 “Validation sample 0 ng/mL” CPMsB100 = Average of all 6 “Validation sample 100 ng/mL” CPMsB250 = Average of all 6 “Validation sample 250 ng/mL” CPMsB1000 = Average of all 6 “Validation sample 1000 ng/mL” CPMsNSB = Average of all 2-6 “NSB” (Non-Specific Binding) CPMsTC = Average of all 2-6 “TC” (Total Count) CPMs
Some RIA %Binding Definitions
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 17
Response
%CV Limit
Sample N
Sample Mean
Sample SD
Addend1
Sample %CV1
(MinCPM-B0)/(B100-B0)*100 450 -3.490 7.968 65 13.0
(AvgCPM-B0)/(B100-B0)*100 25 420 -1.173 8.373 85 10.0
(MinCPM-NSB)/(TC-NSB)*100 450 1.249 0.841 4.4 14.9
MinCPM/NSB 450 1.321 0.218 -0.7 35.0
AvgCPM/(TC-NSB)*100 20 403 5.459 1.119 3 13.2
MinCPM-B0 450 -59.339 151.356 1000 16.1
MinCPM/sqrt(B100*B0) 450 0.549 0.084 0 15.3
1CV of (Response + Addend) = Standard Deviation / (Mean + Addend) x 100%.
Addend chosen so that CV is not related to control concentration.
How to Choose the RIA %Binding Definition?
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 18
New versus Old RIA%Binding Definitions
• New: (MinCPM – B0) / (B100 – B0) – Repeat if CV > 25% and (MaxCPM – B0) /
(B100 – B0) > 12.0% (the Cut Point)
• Old: (AvgCPM – NSB) / TC– Repeat if CV > 20%
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 19
Attributes of Selected RIA %Binding Definitions
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 20
Response %CV Limit
Cut Point
LOD
(ng/mL) 0 ng/mL %Pos.
N
(MinCPM-B0)/(B100-B0) .120 23.5 0.04 450
(AvgCPM-B0)/(B100-B0) 25 .149 25.5 1.29 420
(AvgCPM-B0)/(B100-B0) 20 .153 25.0 0.10 403
(AvgCPM-NSB)/TC 20 3.380 31.7 0.112 403
RIA Validation Control Curve with Lower 1-sided 95% Prediction Limit 65 + %Binding = A+B·ConcentrationC
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 21
A Logistic Curve with an Infinite Plateau is Linear wrt X
C + R XH / ( MH + XH) =
Substitute α = C, = H, and R/β = MH
α + R X / (R/β + X) =Multiply second term by β/β
α + β R X / ( R + βX)Apply L’Hopital’s rule
Lim[ α + R β X / (R + β X) ] = α + β X (R)
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 22
RIA Naïve Sample %Binding vs Test Tube Order by Population
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 23
180160140120100806040200
40
30
20
10
0
-10
-20
-30
Tubepair
Min
Pct
0_100
DiabetesNormal
Population
Scatterplot of MinPct0_100 vs Tubepair
RIA Naïve Sample %Binding vs Test Tube Order by Analyst
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 24
180160140120100806040200
40
30
20
10
0
-10
-20
-30
Tubepair
Min
Pct
0_100
12.05
12
Analyst
Scatterplot of MinPct0_100 vs Tubepair
RIA Naïve Sample %Binding vs Test Tube Order by Analyst and Run
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 25
15010050
400
300
200
15010050
400
300
20015010050
1, 1
Tubepair
ln(3
5+
Pct
0_100)*
100
385.1
1, 2 1, 3
2, 1 2, 2 2, 3
385.1
Scatterplot of ln(35+Pct0_100)*100 vs Tubepair
Panel variables: Analyst, Run
RIA Naïve Sample Means vs Test Tube Order by Population, Analyst and Run
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 26
15010050
450
400
350
300
250
15010050
450
400
350
300
250
15010050
1, 1
Tubepair
ln(M
eanPct
+35)*
100
1, 2 1, 3
2, 1 2, 2 2, 3
DiabetesNormal
Population
Scatterplot of ln(MeanPct+35)*100 vs Tubepair
Panel variables: Analyst, Run
RIA Naïve Sample Mean %Binding vs CV by Analyst and Run
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 27
40-4
450
400
350
300
250
40-4
450
400
350
300
250
40-4
1, 1
lnCV
ln(M
eanPct
+35)*
100
1, 2 1, 3
2, 1 2, 2 2, 3
Scatterplot of ln(MeanPct+35)*100 vs lnCV
Panel variables: Analyst, Run
RIA Naïve Sample Minimum %Binding vs CV by Analyst and Run
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 28
40-4
400
300
200
40-4
400
300
20040-4
1, 1
lnCV
ln(3
5+
Pct
0_100)*
100
385.1
1, 2 1, 3
2, 1 2, 2 2, 3
385.1
Scatterplot of ln(35+Pct0_100)*100 vs lnCV
Panel variables: Analyst, Run
RIA Naïve Sample CPM CV vs Mean by Analyst
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 29
RIA Naïve Sample CPM CV vs Mean by Population and Control
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 30
RIA Probability Plots of ln(35+%Binding)•100 by Analyst
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 31
425400375350325300275250
99.9
99
95
90
80706050403020
10
5
1
0.1
ln(35+Pct0_100)*100
Perc
ent
385.1
344.0 24.90 225 2.052 <0.005339.9 24.84 225 3.863 <0.005
Mean StDev N AD P
12
Analyst
Probability Plot of ln(35+Pct0_100)*100
RIA Probability Plots of ln(35+%Binding)•100 by Population
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 32
450400350300250
99.9
99
95
90
80706050403020
10
5
1
0.1
ln(35+Pct0_100)*100
Perc
ent
341.8 19.87 150 0.542 0.162342.0 27.14 300 1.573 <0.005
Mean StDev N AD P
DiabetesNormal
Population
Probability Plot of ln(35+Pct0_100)*100Normal
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 33
Electrochemiluminescence (ECL) BioVeris Assay
• New way to determine screening cut point (Data = naïve samples)
• New way to determine titer cut point (not equal to screening cut point) (Data = positive samples’ Titration series)
• Estimator of Titer within-assay CV
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 34
Screening Cut Point DeterminationECL of Naïve Sample vs Diluent Alone with Cutoffs by Diluent ECL
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 35
Titer Definition
• Smallest distinct dilution in a titration series with a negative response– Response is Sample ECL mean / Diluent
Control ECL mean in this case study
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 36
Plot where Sample/Diluent Control ECL Ratio < 4 for 1 Selected Plate out of 24
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 37
Potential Problems with a Common Screening and
Titer Cut Point
• Highly diluted samples tend to be positive!– The opposite would not be a problem
• Titration curve too flat at cut point– Makes the titer highly variable– Common
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 38
Titer Cut Point Defined
• The continuous titer inverse predicted from it has CV ≤ 30.0% with 95% confidence
– 30.0% makes best case CV = worst case CV in ideal assay
– Continuous titer is exact dilution giving cut point (only as a theoretical concept)
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 39
Asymptotic CV
• CV Standard deviation of natural log ratio or titer
• CV of dilution@ratio CV of ratio / slope of titration curve@ratio
• CV of dilution decreases as ratio and slope increase
• These CVs are within-plate CVs
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 40
Four Theoretical Titer Distributions
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 41
42
50
25
0
Discrete Titer
Per
cent
5050 5050 5050
CV = 34.7% = ln(F)/ 250% at X and 50% at X*F. CV=ln(F)/2
842
75
50
25
0
Discrete Titer
Per
cent
12.5
75
12.5
CV = 34.7% = ln(F)/ 275% at X, 12.5% each at X/F and X*F
8421
50
25
0
Discrete Titer
Per
cent
1
4949
1
30% CV of Continuous Titer37.5%=> Discrete Titer CV =
168421
75
50
25
0
Discrete Titer
Per
cent
0.0312.39
75.16
12.390.03
30% CV of Continuous Titer=> Discrete Titer CV = 34.7%
Titer Cut Point Defined• A continuous (interpolated) titer inverse
predicted from it has CV<30.0% with 95% confidence– Exact dilution giving cut point (eg, 1.357
ratio) is the continuous titer– Continuous titer used here only as a
theoretical concept– Our cut-point 5 SD above diluent mean so
false-positives of noncensored titers unlikely
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 42
Summary
• All biologics need ADA evaluation
• Use controls to adjust for plate-to-plate variance and minimize the LOD
• Define titer cut point so best case CV = worst case CV in ideal assay
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 43
Acknowledgements:• Sheng Dai• Allen Schantz
Reference: Shankar, G. et al, (2008). Recommendations for the
validation of immunoassays used for detection of host antibodies against biotechnology products. Journal of Pharmaceutical and Biomedical Analysis. 48:1267–1281.
• Pam Cawood• Gopi Shankar• Bill Pikounis
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 44