V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Core Clinical Research Concepts and Issues

Al M Best, PhDVirginia Commonwealth University

Task Force on Design and Analysis in Oral Health Research

Satellite Symposium, AADR

Boston MA: March 9, 2015

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Outline

Core Concepts– Defining the research question– Causation vs. association– Sources of error in clinical research– Multiple outcomes: Primary & secondary outcomes

Prominent Design Issues in Clinical Research– Randomization– Masking/blinding– Analysis: Intention-to-treat or Per Protocol?– Biomarkers and surrogate measures– Superiority, equivalence , or non-inferiority

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Outline Intro to Multivariate Statistical Modeling

– Regression Continuous outcomes: Linear … Dichotomous outcomes: Logistic … Counts: Poisson … Time to event: Proportional hazards

survival– Unified management of confounding and

effect modification in statistical models– Propensity methods

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Part 1

Core Concepts

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Defining the research question

Conceptual progression from general to specific

General question– Can GCF constituent levels distinguish

active chronic adult perio sites from inactive and simple gingivitis sites?

Specific hypothesis– Site GCF AST levels with active perio

exceed those with inactive perio and simple gingivitis.

Testable consequence– …

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Defining the research question: Testable consequence

Context– precise population: eligibility criteria* – sampling: patient sources, recruitment,

consent– measurement definitions:

detailed operational definitions distinguishing active sites, inactive sites,

gingivitis sites GCF sampling, storage, and AST

measurement procedures Prediction: a relationship between exposure and

outcome Formalization: refutable hypotheses* AKA: inclusion/exclusion criteria

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Defining the research question: Prediction

Prediction: a statistical relationship between exposure and outcome

–GCF AST will be tend to be higher in active than in inactive or simple gingivitis sites within the same or different patients

Formalization: a refutable hypothesis

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Defining the research question: Formalization

A refutable hypothesis Statistical formalization: Ho: Mean(Active) = Mean(Inactive) =

Mean(Gingivitis)– Which may be disproved beyond a

reasonable doubt through falsification by data via statistical hypothesis testing, in favor of:

Ha: Mean(Active) > Mean(Inactive) and Mean(Active) > Mean(Gingivitis)

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Defining the research question: Formalization

Estimating clinically relevant quantities Clinical formalization:

– Mean(Active) – Mean(Inactive) > clinically important delta and

– Mean(Active) – Mean(Gingivitis) > clinically important delta

– Or, ROC AUC for distinguishing active perio > 0.75

These quantities may be estimated, and the clinically relevant hypotheses proven or disproved, using confidence intervals and associated statistical tests

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Characteristics of a well-framed clinical research

question FINER–Feasible– Interesting–Novel–Ethical–Relevant

Source: Hulley, et al. (2007) Designing Clinical Research, Lippincot, Table 2.1

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

… a well-framed clinical research question

Feasible– Adequate subjects– Adequate technical expertise– Affordable in time and money– Manageable in scope: logistics

Interesting– Generalizable to more than 1 application– An answer would be intriguing to the

investigator and at least some other folks– Potentially fruitful, suggesting other

hypotheses

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

… a well-framed clinical research question

Novel – New information– Or in need of replication

Ethical– Avoids unacceptable risks

Relevant– How might the various outcomes

advance knowledge? Influence practice? Establish policy? Or guide future research?

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Causation vs. Association

What do we mean by “Exposure A causes Disease B”?

One approach: Logical consideration of evidence favoring causality– Consistent association across studies

and designs– Strong association (big differences,

strong trends)– Dose response relationship– Biologic plausibility

Note: investigators can come up with a plausible mechanism for virtually any association

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Causation vs Association What do we mean by “Exposure A causes

Disease B”? Empirical approach:

http://xkcd.com/943/

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Causation vs Association

What do we mean by “Exposure A causes Disease B”?

Empirical approach: Ruling out spurious associations

Statistical association– Tendency of characteristics to vary

together, so information on one allows guessing or predicting the other more accurately

– BUT: there are lots of ways we may be fooled

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Associations are common, and

most are worthless

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

How we measure statistical associations

Associations are what we observe, as– Differences or ratios of:

means or medians proportions odds rates

– Slopes of trends in statistical models – Correlation, regression coefficients

Causation → association, but not the other way No measure of association, in itself, implies

causation

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Ywww.phdcomics.com/comics/archive.php?comicid=1156

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Example of causal relationships

We think: influenza deaths less common in

immunized groups MI less common in hyperlipidemics on

statins than in untreated hyperlipidemics

Stroke probability ↑ with ↑ SBP, ↑ LDL ↑ exercise, ↓ DBP ↑ sugar consumption, ↑ DMFS ↑ cigarette smoking, ↑ alveolar bone loss

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Example of non-causal or unproven relationships

We are not sure: Antioxidant supplementation can lower US

cancer risk Calcium supplements reduce

demineralization after 65 Treating chronic periodontitis reduces CVD

risk Scaling and root debridement in pregnant

women will reduce preterm births

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Yhttp://www.phdcomics.com/comics/archive.php?comicid=1452

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Clinical research is a search for causes and predictors

Comparison (control) is critical!– Comparisons are needed to distinguish

real from bogus causes– Comparisons between groups are

needed to escape the particular and possibly idiosyncratic

– Statistical associations are how one summarizes comparisons among groups

– Causal pathways imply statistical associations

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Clinical research is a search for causes and predictors

Clinical research is about finding and distinguishing causal (or consistently prognostic) associations from those due to– Reverse causation– Measurement bias– Selection bias– Confounding– Chance

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Sources of error in clinical research

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Reverse causation? Does the outcome proceed from the clinical variable?

We observe: Prevalence of a disease is higher in those from whom a microbe is successfully cultured than in those from whom it is not

Is the microbe statistically associated with the disease– because the microbe causes the

disease?– or because the disease makes the site

more hospitable to colonization or infection by the microbe?

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Measurement (information) bias

Definition, Bias:– Systematic distortion of the estimated

intervention effect away from the “truth,” caused by inadequacies in the design, conduct, or analysis of a trial

Definition, Measurement bias:– Systematic or non-uniform failure of a

measurement process to accurately represent the measurement target

– Examples …

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Measurement (information) bias

Examples, Measurement bias:– different approaches to questioning,

when determining past exposures in a case-control study

– more complete medical history and physical examination of subjects who have been exposed to an agent suspected of causing a disease than of those who have not been exposed to the agent

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Measurement bias? NHANES III: “We estimate that at least

35% of the dentate US adults aged 30 to 90 have periodontitis, with 21.8% having a mild form and 12.6% having a moderate form”1

– Mesial and buccal surfaces– Two randomly selected quadrants– CAL≥3mm

Or: Full mouth prevalence= 64.7% vs partial = 39.1%2

1 JM Albandar, JA Brunelle, A Kingman (1999) "Destructive periodontal disease in adults 30 years of age and older in the United States, 1988-1994". Journal of Periodontology 70 (1): 13–29.

2 A Kingman & JM Albandar (2002) “Methodological aspects of epidemiological studies of periodontal diseases.” Periodontology 2000 29, 11–30.

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Conclusion “overall, the benefits of frequent brushing of teeth did not outweigh the damaging effect of frequent sugar consumption”

Hinds & Gregory (1995) National diet and nutrition survey: Children ages 1.5 to 4.5 years. Report of the Dental Survey, vol. 2, London: HMSO.

Conclusion: “for children who brushed their teeth twice a day or more, consumption of sugars and sugary foods did not appear to be associated with caries”

Gibson & Williams (1999) Dental caries in pre-school children: associations with social class, tooth brushing habit and consumption of sugars and sugar-containing foods. Caries Research 33,101–113.

Recall bias

Harris, et al. (2004) Risk factors for dental caries in young children: a systematic review, Community Dental Health, 21(S); 71-85

Is frequent sugar consumption related to childhood caries? Results using the British National Diet and Nutrition Survey (1995):

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Lead-time bias

Bias in evaluating screening programs– If time to outcome (e.g., death) is

measured from point of diagnosis, then early diagnosis will increase the estimated survival time by the interval between diagnosis-by-screening and when diagnosis would have occurred by ordinary means

– This can make early diagnosis appear to increase survival time, even when it has no effect … or even a damaging effect

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Lead-time bias

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Selection bias Definition: Bias from the use of a non-

representative group as the basis of generalization to a broader population

Can distort: disease frequency measures,exposure-disease associations

Example: Estimate prognosis from patients newly diagnosed and infer to patients hospitalized with the disease– Newly diagnosed patients have a much

broader spectrum of outcomes

“Selection bias” is also used to describe the systematic error that can occur when allocating patients to intervention groups.

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Selection (=sampling) bias

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Admission rate bias

Definition: When admission rates vary between exposed cases and unexposed controls

Respiratorydisease Yes No Totals %Yes Yes No Totals %YesYes 17 207 224 7.6 5 15 20 25.0No 184 2376 2560 7.2 18 219 237 7.6Totals 201 2583 2784 23 234 257

Relative risk= 1.06 Relative risk= 3.29Odds ratio= 1.06 Odds ratio= 4.06

In the general populationSubset: those hospitalized in

the prior 6 months.

Diseases of bones and organs of movement

Adapted from: Roberts, Spitzer, Delmore, Sackett (1978) J Chron Dis (31) 119-128

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Admission rate bias

AKA: Berkson’s bias

Respiratorydisease Yes No Totals %Yes Yes No Totals %YesYes 17 207 224 7.6 5 15 20 25.0No 184 2376 2560 7.2 18 219 237 7.6Totals 201 2583 2784 23 234 257

Relative risk= 1.06 Relative risk= 3.29Odds ratio= 1.06 Odds ratio= 4.06

In the general populationSubset: those hospitalized in

the prior 6 months.

Diseases of bones and organs of movement

Adapted from: Roberts, Spitzer, Delmore, Sackett (1978) J Chron Dis (31) 119-128

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Length-time bias Bias in evaluating screening programs: two

cases– a slowing progressing disease vs. – a rapidly progressing disease

In standard clinical practice: The observed proportion of those with the disease will be higher for the rapidly progressing disease because they have a shorter asymptomatic phase

In a screening program: The observed proportion of those with the disease will be higher for the slowly progressing disease because they are detectable over a wider range of time

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Length-time bias

Patients diagnosed by screening will, as a group, progress more slowly than those diagnosed by conventional means. Even if early treatment has no impact.

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Sources of bias

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Confounding

Informal definition: distortion of the true biologic relation between an exposure and a disease outcome of interest

Usually due to a research design and analysis that fail to properly account for additional variables associated with both– Such variables are referred to as

confounders or, less formally, as lurking variables

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Confounding examples “Honey, it’s not my fault!

It’s the one-night stand gene”

“The D4 mutation of the human dopamine receptor gene is linked with: gambling, alcoholism, drug use, overeating, political liberalism, passion for horror films, ADHD, extreme extraversion, impulsiveness”Garcia et al. (2010) Associations between Dopamine D4 Receptor Gene Variation with Both Infidelity and Sexual Promiscuity. PLoS ONE 5(11): e14162

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Confounding examples

Misidentified carcinogen Prior to the discovery of HPV, HSV-2 was

associated with the cervical cancer It is now well established that HPV is

central to the pathogenesis of invasive cervical cancer.And HSV-2 appears to increase the risk

Hawes & Kiviat (2002) Are Genital Infections and Inflammation Cofactors in the Pathogenesis of Invasive Cervical Cancer? JNCI 94(21): 1592-159

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Perio and CVD

Cigarette smoking is associated with adult perio and CVD

This produces an association between perio and CVD

Control for smoking to see the perio-CVD relationship clearly

Scannapieco et al. (2003) Associations Between Periodontal Disease and Risk for Atherosclerosis, Cardiovascular Disease, and Stroke. A Systematic Review. Annals of Periodontology (8)38-53

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Confounding

Amount of Annual %dental care Yes No Total loosing teethLess 20 373 393 5.1More 6 316 322 1.9Total 26 689 715

Relative risk= 2.73Odds ratio= 2.82

SurvivalOne year loss of any teeth

0

1

2

3

4

5

6

7

8

9

Less (n=393) More (n=322)

An

y te

eth

lo

st

Amount of dental care

Annual %

Note: artificial data

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Confounding

Amount of Annual %dental care Yes No Total loosing teethLess 3 176 179 1.7More 4 293 297 1.3Total 7 469 476

Relative risk= 1.24Odds ratio= 1.25

Amount of Annual %dental care Yes No Total loosing teethLess 17 197 214 7.9More 2 23 25 8.0Total 19 220 239

Relative risk= 0.99Odds ratio= 0.99

One year loss of any teeth: Clinic ASurvival

One year loss of any teeth: Clinic BSurvival

0

1

2

3

4

5

6

7

8

9

A, Less (n=179)

A, More (n=297)

B, Less (n=214)

B, More (n=25)

Any

teet

h lo

st

Amount of dental care

Annual %

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Confounding: Simpson’s paradox

Statistical associations in entire groups can differ from statistical associations in their subgroups

Worse:statistical associations in whole groups can be in the opposite direction from statistical associations in every subgroup!!!

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Multiplicity effects

http://xkcd.com/882/

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Multiplicity effects

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Multiplicity effects

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Multiplicity effects

N=47 perio, N=20 healthy

Analyzed for the presence of 300 species

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Controlling multiplicity effects Multiplicity: The proliferation of possible

comparisons in a trial. Common sources of multiplicity are:– multiple outcome measures,

assessment at several time points, subgroup analyses, or multiple intervention groups

Multiple comparisons: Performance of multiple analyses on the same data. Multiple statistical comparisons increase the probability of a type I error: “finding” an association when there is none

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Controlling multiplicity effects Analysis of multiple outcome variables

that reflect different disease manifestations, any of which might signal a spurious association due purely to chance

Analysis of the same variable at multiple time points after treatment initiation

Periodic analysis of accumulating partial results to see how a study is progressing, and whether there’s reason to stop collecting data earlier than initially planned

Post hoc subgroup comparisons are especially likely not to be confirmed in following studies

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Multiple outcome examples Caries

– visual exam– x-ray interpretation – Fiberoptic Transillumination (FOTI)– Electrical Caries Meter (ECM)– DiagnoDent (DD)

Perio– alveolar bone loss– clinical attachment level– pocket depth

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Multiple comparisons If we control the chance of a false positive

answer in evaluating a single outcome by a conventional statistical method, which permits a 5% chance of claiming a statistical association when none exists,– then by the time we’ve compared

treatments on 4 outcomes our chance of an error is MUCH HIGHER

We can’t stop asking questions, so three strategies are used to control this– Require stronger evidence to claim an

effect– Hierarchy of outcomes– Composite outcome

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Require stronger evidence to claim an effect

Example: When asking 5 questions Require stronger evidence for each

question so that the chance of claiming statistical association when none exists is only 1% instead of p < 5%

This insures that, overall, no more than 5% false positive errors– Various methods guide how much

stronger one should require evidence to be in different circumstances

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Composite outcome

Combine multiple outcomes into one index, and use that as a single outcome

Multivariate → Univariate– Meld several outcomes into one

question, by asking whether any function of any of them is associated with the exposure, e.g., treatment, and seeking a single yes or no

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Outcome hierarchy

Order outcomes from most to least important

Crucial decisions in planning and analyzing the study are based on the primary outcome atop the hierarchy

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Multiple outcome variables

Requiring stronger evidence to claim an effect is often overly conservative– Also, it’s hard to get folks to implement

it sincerely– Somehow lots of ‘buts’ get into the

interpretation Composite outcomes and outcome

hierarchies require choices clinical researchers are not always comfortable making– Results may be hard to interpret– Composite outcome methods are being

gradually adopted in some fields, e.g., stroke trials

Outcome hierarchy is most commonly used

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Disclose “data snooping”

Only pre-specified comparisons are to be believed

BUT, in a frankly “pilot study” the purpose is to find the subgroup, “dose”, time point, and outcome where the difference is largest

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Effect of Systemic Matrix Metalloproteinase Inhibition on Periodontal Wound Repair: A Proof of Concept Trial

Background: The adjunctive use of matrix metalloproteinase (MMP) inhibitors with scaling and root planing (SRP) promotes new attachment in patients with periodontal disease. This pilot study was designed to examine aspects of the biological response brought about by the MMP inhibitor low dose doxycycline (LDD) combined with access flap surgery (AFS) on the modulation of periodontal wound repair in patients with severe chronic periodontitis.

Methods: Twenty-four subjects were enrolled into a 12-month, randomized, placebo-controlled, double-masked trial to evaluate clinical, biochemical, and microbial measures of disease in response to 6 months therapy of either placebo capsules + AFS or LDD (20 mg b.i.d.) + AFS. Clinical measures including probing depth (PD), clinical attachment levels (CAL), and bleeding on probing (BOP) as well as gingival crevicular fluid bone marker assessment (ICTP) and microbial DNA analysis (levels and proportions of 40 bacterial species) were performed at baseline and 3, 6, 9, and 12 months.

Results: Patients treated with LDD + AFS showed more potent reductions in PD in surgically treated sites of >6 mm (P <0.05, 12 months). Furthermore, LDD + AFS resulted in greater reductions in ICTP levels compared to placebo + AFS. Rebounds in ICTP levels were noted when the drug was withdrawn. No statistical differences between the groups in mean counts were found for any pathogen tested.

Conclusions: This pilot study suggests that LDD in combination with AFS may improve the response of surgical therapy in reducing probing depth in severe chronic periodontal disease. LDD administration also tends to reduce local periodontal bone resorption during drug administration. The use of LDD did not appear to contribute to any significant shifts in the microbiota beyond that of surgery alone. Gapski, Barr, Sarment, Layher, Socransky, Giannobile J Periodontol 2004;75:441-452.

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Concepts

You need a testable, FINER research question

Try to rule out bias, confounding, chance Consider multiple outcome measures and

multiple predictors Disclose what you did [and why] with

enough detail so others may replicate

Mark Twain: There is something fascinating about science. One gets such wholesale returns of conjecture out of such a trifling investment of fact.

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Thanks

… for your participation

Acknowledgement: I’m grateful to Peter B Imrey, PhD for sharing his previous version of this talk.