1
OTC-TFM Monograph: Statistical Issues of Study
Design and Analyses
Thamban Valappil, Ph.D.
Mathematical StatisticianOPSS/OB/DBIII
Nonprescription Drugs AC MeetingNonprescription Drugs AC MeetingMarch 23, 2005March 23, 2005
2
Outline
• Introduction
• Summary of Statistical Issues
• Current TFM trial Design and Analyses with surrogate endpoints
• Statistical Issues of Study Design and Analyses
• Options for Trial Design and Efficacy Criteria using surrogate endpoints
3
Introduction• Previous presentations on issues involved in
validating surrogate endpoints.
• In the absence of clinical trials data, FDA still needs to address current products under review.
• This talk discusses issues related to analysis of data obtained on surrogate endpoints.
• Does not address clinical relevance of statistical findings or differences in analysis of data based on surrogate endpoints.
4
Summary of Statistical Issues
• Primary Endpoint of Log reduction in Bacterial Counts from baseline.
• Data Analyses & Variability Issues– Binary Outcomes
– Log Reduction (Mean vs. Median) outcomes
• Variability in methodology– Study Design and Controls
• Active
• Vehicle
5
Indication Controls Endpoints Statistical
tests
Sample size /arm
Surgical HandScrub
Active,
Vehicle or Placebo
Log Reduction,
Binomial
t-tests 66
PreOperative Skin Prep Active
Control
Log Reduction,
Binomial
Paired t-test 96
Healthcare
Personnel Handwash
Active control
Log Reduction
t-tests 54
Current TFM Recommendations
6
Current TFM Recommendations: Issues
TFM Recommends:– Randomized– Blinded (to persons determining counts only)– Use of Active Control– Use of Vehicle or Placebo Control (role not clearly
specified)
However, in the current TFM• A non-comparative study design is used in which the
test product is not directly compared to the Active Control.
• Mean log reduction meeting the threshold log reduction has been used to demonstrate efficacy.
7
• Although vehicle and placebo controls are mentioned in the TFM , majority of the NDAs only have test product and active control arms.
• Active controls have only been used for internal validation of study methods.
• Efficacy assessment does not include a direct comparison of Test product performance to Active control, vehicle or placebo.
Current TFM Recommendations: Issues
8
Statistical Issues of Study Design and Analyses• Primary Endpoint: Log Reduction
– Mean Log Reduction can be influenced by few extreme observations.
( Suggestion: Median log reduction may be another option. Median Log Reduction is less sensitive to few subjects with extreme log reductions or outliers. )
9
• The efficacy criteria in the current TFM are based on point estimates and do not include confidence intervals to evaluate variability.
• Consequently, a few extreme observations can potentially drive the efficacy results.
Statistical Issues of Study Design and Analyses
10
Example:
Log Reduction: Mean = 2.0, Median = 1.7Below 2-log: 78%
11
Active Control Test
Vehicle
Example: Log Reduction in Bacterial Counts
Threshold
.
12
Active Control
Test
Vehicle
.
Threshold
Upper Limit Mean Lower Limit
13
Active Control
Test Vehicle.
Threshold
Upper Limit Mean Lower Limit
14
Subject will be classified as a ‘success’ or a ‘failure’ based on meeting the threshold reduction.
Advantages:
• Outcome centered on number of subjects and not on organisms may be more clinically relevant.
• Effect of Variability is reduced.
Disadvantage:
This method does not differentiate the magnitude of log reductions among those who meet the criteria for “success”.
Primary Endpoint Using Binary Response
15
Example:
16
Study #1 (Surgical Hand Scrub)
Treatment Day Mean Median Success (%)Test 1 (1 log) 2.63 2.95 20/21 (95%) 2 (2 log) 3.25 3.17 21/21 (100%) 5 (3 log) 3.51 3.88 13/20 (65%)
Active #1 1 (1 log) 1.51 1.69 9/12 (75%) 2 (2 log) 2.37 2.36 8/12 (67%) 5 (3 log) 3.47 3.59 9/12 (75%)
Active #2 1 (1 log) 1.17 1.40 14/22 (64%) 2 (2 log) 2.02 1.80 10/22 (45%) 5 (3 log) 1.66 1.58 2/21 (10%)
17
Sample Size Issues
• In current TFM, sample size is estimated based on allowing a test product to be as much as 20% worse than active control in the mean log reductions. However, the basis for 20% margin is not clearly stated.
• Majority of the current submissions do not follow the recommended sample size as specified in the TFM and only use a sample size of ~30 subjects per treatment arm.
18
Options for Trial Design and Efficacy criteria
• Issue 1– How to analyze the data obtained on the surrogate
endpoint of log reductions in bacteria?
• Issue 2– How to take into account the variability in the data
collected, when measuring effect of the product?
• Issue 3– How to take into account the variability in the test
methodology?
19
Options for Trial Design and Efficacy criteria
• Issue 1: How to analyze the data obtained on the surrogate endpoint of log reductions in bacteria?
– Mean log reduction• It can be influenced by few extreme observations.
– Median log reduction• Less sensitive to outliers or extreme observations.
– Percentage of subjects who meet log reduction criteria• Outcome centered on number of subjects who meet the
threshold and may provide incentive to study “conditions of use” that provides highest success rates.
20
Options for Trial Design and Efficacy criteria• Issue 2: How to take into account the variability in the data
collected?
– Examine confidence intervals around the “outcomes” as defined on previous slide with a threshold for lower bound of confidence interval
• PRO: Improvement over examination of point estimates alone.• CON: Does not take into account the variability in test method.
– Examine confidence intervals around treatment difference between test product and some control
• PRO: Allows for examination of variability in methodology across treatment arms.
• CON: May require larger sample size for products with lower success rates.
21
Options for Trial Design and Efficacy criteria• Issue 3: How to take into account variability in the test
methodology?
– “Equivalence”/non-inferiority showing test product is no worse than active control by some margin
• PRO: Allows comparison with an active treatment to rule out loss of effect relative to active control
• CON: Lack of constancy of effect of active control in previous studies, possible overlap of effect of active and test product with vehicle, hence no basis to select a non-inferiority margin
– Superiority of test product to vehicle AND superiority of active control to vehicle
• PRO: Given lack of constancy of effect with both active and vehicle controls, allows internal validity of comparisons
• CON: May require larger sample size than current TFM standards (how much larger depends on product efficacy over vehicle)
22
Test Product
Active Control
Vehicle
S
S
S = Superiority
Controlling Variability in Test Methodology
?
23
Sample Size: Superiority Test Sample Size: Superiority Test 388
93
39
388
97
42
356
93
42
294
82
39
199
62
32
0
50
100
150
200
250
300
350
400
Sam
ple
Size
/arm
50 60 70 80 90
Success Rate for Test Product
Trt. Diff 10%Trt. Diff 20%Trt. Diff 30%
388
93
39
388
97
42
356
93
42
294
82
39
199
62
32
0
50
100
150
200
250
300
350
400
Sam
ple
Size
/arm
50 60 70 80 90
Success Rate for Test Product
Trt. Diff 10%Trt. Diff 20%Trt. Diff 30%
24
Thank you!
