Brian L. Strom, M.D., M.P.H.Chair and Professor, Department of

Biostatistics and Epidemiology

Director, Center for Clinical Epidemiologyand Biostatistics

George S. Pepper Professor of Public Health and Preventive Medicine

Professor of Biostatistics and Epidemiology, Medicine, and Pharmacology

University of PennsylvaniaSchool of Medicine

CCEB

CCEB

“Sampling”

1. What is an appropriate sample size of respondents to best determine risk of sound and look alike proprietary names in the prescription drug study group? In a focus group? In a survey document?

2. Should sample be randomly selected? Is it important to have a statistical significance for this type of evaluation?

CCEB

“Sampling”

• Introduction

• Very general principles of sample size calculations and sampling

• Application of general principles to this situation

• Recommendations for research, to guide the future

CCEB

Options in Research Design

• Analytic Studies– Experimental Study– Prospective Cohort Study– Retrospective Cohort

Study– Case-Control Study

• Descriptive Studies– Analyses of Secular

Trends– Case Series– Case Reports

Case-Control StudiesCase-Control Studies

DiseaseDisease

Coh

ort

Stu

die

sC

oh

ort

Stu

die

s

Fact

or

Fact

or

PresentPresent(cases)(cases)

AbsentAbsent(not exposed)(not exposed)

PresentPresent(exposed)(exposed)

AbsentAbsent(controls)(controls)

AA

DDCC

BB

CCEB

Determinants of Needed

Sample Size• Alpha

• Beta

• Variability (SD)

• Delta--how small a difference do you want to be able to detect?

CCEB

Determinants of NeededSample Size: Cohort

Study• Alpha

• Beta

• Incidence in the unexposed control group

• Delta--how small a RR do you want to be able to detect?

CCEB

Determinants of Needed Sample

Size: Case-Control Study

• Alpha

• Beta

• Prevalence in the undiseased control group

• Delta--how small a RR do you want to be able to detect?

CCEB

Sampling

Study Sample

Conclusion About a Population(Association)

Conclusion About Scientific Theory(Causation)

Statistical Inference

Biological Inference

CCEB

Application of GeneralPrinciples to This

Situation

• THE Central Principleof Research Design: The question is, what is the question??

CCEB

Application of GeneralPrinciples to This

Situation• In this situation, there are no

a priori hypotheses being tested to be able to consider sample size calculations or questions of sampling• What is being performed is

essentially qualitative research

CCEB

Recommendations for Research,

to Guide the Future

**Evaluate the current process in a quantitative fashion**

CCEB

General Strategy for Evaluation

• Standardize procedure• Test for reliability/

reproducibility• Test for validity• Make changes in the

procedure accordingly

CCEB

Standardize Procedure

• Choose among the current possible approaches a “standard” to be evaluated more rigorously

CCEB

Test for Reliability/Reproducibility

• Evaluate the same drug names in the same process with multiple different groups of survey prescribers and experts, to look for whether there is adequate agreement• If no reliability, validity is

impossible and procedure should be abandoned

CCEB

Test for Validity

• Gold standard needed:–Drug names rejected in initial FDA review

–Drug names withdrawn due to problems

–Direct measurement of error rate

CCEB

Retrospective: Drug NamesRejected in Initial FDA

Review

• Problem: was the initial review decision correct?

CCEB

Retrospective: Drug NamesWithdrawn Due to

Problems

• Knowledge of reviewers could be problematic–Other countries–Years ago, and using new

pharmacists

• Was the withdrawal decision a correct one?

CCEB

Prospective: DirectMeasurement of Error Rate

• Simulate real life situation in a study setting

• Choose good and bad options for new names

• Enter possible new names into prescription entry computer program

• Ask large numbers of docs to write orders

• Ask large numbers of pharmacists to “fill” each rx, entering into prescription entry computer program

• Directly measure resulting error rate

CCEB

Make Changes in theProcedure

Accordingly• Determine appropriate cutpoint

for expert ratings, ROC curve vs. gold standard• Determine appropriate sample

sizes through simulation, ie how many are needed to achieve results consistent with the gold standard• Modify processes accordingly

CCEB

Potential Sources of Support

• FDA extramural funds

• AHRQ patient safety funds

• NIA pharmacology program

• NIGMS pharmacology program

CCEB

Conclusions

• Applying a quantitative approach to evaluating what has so far been a qualitative one, could lead to major changes in the procedure, and major improvements in the net results

Statitsically incorrect