80
Assessment of methodological quality and outcomes of clinical trials Christian Gluud Copenhagen Trial Unit Centre for Clinical Intervention Research
Assessment of methodological quality and outcomes of clinical trials
Christian Gluud
Copenhagen Trial Unit Centre for Clinical Intervention Research
RigshospitaletCopenhagen University Hospital
Evidence-basedclinical practice
• The patient’s values (concerns, expectations, preferences)
• The best clinical research evidence
Levels of evidence
Important aspects of clinical research
- all levels of the hierachy
• Systematic errors (domains) • Systematic errors (design) • Random errors (play of chance)
Levels of evidence
• Risk of systematic errors (domain)• Risk of systematic errors (design)• Risk of random errors
Important aspects of randomised clinical trials
and meta-analyses
• Systematic errors (domains)
• Systematic errors (design)
• Random errors (play of chance)
Domains associated with bias risk
• Generation of the allocation sequence• Allocation concealment• Blinding• Incomplete outcome data (intention-to-treat)• Outcome reporting bias• Industry bias• Other components associated with bias
Meta-analysis of several trials
Low risk of bias
High risk of bias
Overall
Ratio of odds ratios (ROR)Ratio of odds ratios (ROR)
Odds ratio of trials with unclear or inadequateOdds ratio of trials with unclear or inadequate
component (high risk of bias)component (high risk of bias)
divided bydivided by
odds ratio of trials with adequate odds ratio of trials with adequate
component (low risk of bias)component (low risk of bias)
Control of selection biasGeneration of the allocation sequence
• Low risk of bias Computer system, table of random numbers, or similar
• High risk of bias Not described or quasi-randomised (excluded)
BRANDO Sequence generation
• 112 meta-analyses with• 944 randomised clinical trials • 696 (73.7%) with high risk of bias• ROR 0.89 (95% CI 0.82 to 0.96)
Control of selection biasAllocation concealment
• Low risk of bias Central independent unit, sealed envelopes (only if perfect placebo)
• High risk of bias Not described or open table of random numbers
BRANDO Allocation concealment
• 146 meta-analyses with• 1292 randomised clinical trials • 916 (70.9%) with high risk of bias• ROR 0.93 (95% CI 0.87 to 0.99)
Control of detection bias(reporting bias and observer bias)
Blinding
• Low risk of bias Identical placebo or comparator
• High risk of bias Not described or not blinded
BRANDO Blinding
• 104 meta-analyses with• 1057 randomised clinical trials • 467 (44.2%) with high risk of bias• ROR 0.87 (95% CI 0.79 to 0.96)
Industry sponsorship and research outcome
• Lundh et al, The Cochrane Library 2012
• Cross-sectional studies, cohort studies, systematic reviews and meta-analyses that quantitatively compared primary research studies of drugs or medical devices sponsored by industry with studies with other sources of sponsorship
Favourable beneficial results
Favourable harms results
Favourable conclusions
Important aspects of randomised clinical trials
and meta-analyses
• Systematic errors (domains)
• Systematic errors (design)
• Random errors (play of chance)
Design errors affecting external validity of randomised trials include
‘wrong’: • Centres• Participants• Experimental intervention• Control intervention• Goal - explanatory or pragmatic• Trial structure - parallel group, crossover, etc.• Objective - superiority, equivalence, non-inferiority• Outcome• Unit of analysis
Important aspects of randomised clinical trials
and meta-analyses
• Systematic errors (domains)
• Systematic errors (design)
• Random errors (play of chance)
Mean number of patients per intervention arm (SEM) in 383 randomised trials
published in Gastroenterology from 1964-2000 (Kjærgard et al. 2002)
TA(C)E for hepatocellular carcinoma – trial sequential
analysis
The pernicious yin-yang interplay between random errors and systematic error
PUBLICATION BIASPUBLICATION BIAS
Number of patients with serum HBV DNA at end of treatment
Trial sequential analysis of serum HBV DNA
DARIS 4620 pts. Pc 54%, RRR 20%, alpha=5%, beta=20%, diversity=85%
Trial sequential analysis of serum HBV DNA at end of treatment
Number of pts without seroconversion from HBeAg to anti-HBe
DARIS 891 pts. Pc 86%, RRR 20%, alpha=5%, beta=20%, diversity 79%
Trial sequential analysis of seroconversion from HBeAg to anti-HBe
Trial variable Global Chinese
Trial registration Often insufficient Often insufficient
Sample size estimation 50% Less than 25%
Bias risks 95% More than 95%
Comparators Often ill chosen Often ill chosen
Outcomes Putative sourrogates Putative sourrogates
Results Mostly unreliable Mostly unreliable
Depesonolised data in register
Always insufficient Always insufficient
Experimental intervention- systematic errors (domains)- systematic errors (design)- PUBLICATION bias
Control intervention
Randomised clinical trials – most often false due to the people behind!
“…why most research findings are false!” JP Ioannidis
THANK YOU !THANK YOU !
The Cochrane
CollaborationAn international network of professionals,
preparing , maintaining, and disseminating systematic reviews of the effects of health care
Archie Cochrane (1979)
”It is surely a great criticism of our profession that we have not organised a critical summary, by speciality and subspeciality, adapted periodically, of all relevant randomised trials”
The Cochrane Collaboration
• www.cochrane.org• 670 000 randomised trials• Now 5000 systematic reviews• 500 new reviews per year• 500 updated reviews per year• JIF 6.2
Answer to the question:
When is it acceptable to make therapeutic decisions based on patient series, patient-control studies, and cohort studies?
Answer to the question:
When is it acceptable to make therapeutic decisions based on patient series, patient-control studies, and cohort studies?
Only if you have exophtalmus producing intervention effects!
Answer to the question:
When is it acceptable to make therapeutic decisions based on patient
series, patient-control studies, and cohort studies?
As we do not know the intervention effect a priory – and medical research is a
forward moving process – one should always test interventions in randomised clinical trials!
Thomas C Chalmers
• More than 500 publications• More than 25 000 citations• H index above 77
• Always randomize the first patient! The New England Journal of Medicine
1977;296(2):107
Rare diseases and randomised trials
• With 7,000,000,000 (7 billion!) individuals in the world the disease has to be extremily rare – say less than 1/10,000,000 – before it becomes difficult to conduct randomised clinical trials!
• International patient registries• Infrastructures like ECRIN
James Lind 1747
James Lind 1716-1794
James Lind allocated
Two oranges and lemons
Two cider
Two vinegar
Two elixir vitriol
Two spices and garlic
Two sea water
75 498 979 1647 31188823
16155
26322
45020
67522
99373
121584
131223
0
20000
40000
60000
80000
100000
120000
140000
1900 to 1946
1947 to 1951
1952 to 1956
1957 to 1961
1962 to 1966
1967 to 1971
1972 to 1976
1977 to 1981
1982 to 1986
1987 to 1991
1992 to 1996
1997 to 2001
2002 to 2007
Year period
Periodic growth of publications on randomised and controlled clinical trials
1127
1014945
881
713666 632
594 587 559
438 430 426355 349
254 250199 188 175
0
200
400
600
800
1000
1200
Number of publications on randomised and controlled clinical trials published per million inhabitants
James Lind observed
The two on oranges and lemons
regained health in a few days
and could care for the remainder,
that stayed ill
James Lind 1753
James Lind 1747
Yearly ‘International Clinical Trials’ Day’ !!!!!Yearly ‘International Clinical Trials’ Day’ !!!!!
George Löhner and…... 1835
George Löhner and his team of truth loving men
• Clear protocol, published before launch
• Large number of participants (n = 50)
• Perfect randomisation
• Placebo controlled (melted ice)
• Blinded for all parties
• Account for drop outs
• Statistical comparison
Downgrade the evidence in case of risks of errors
I More evidence needed!!!!!!!!!!!!!IIa Systematic review of randomised clinical trials with risk of systematic errors (bias) or of random errors (play of chance) IIb Single randomised clinical trial with risk of systematic errors (bias) or of random errors (play of chance) III Cohort studyIV Case-control studyV Case series
Upgrade the evidence in case of ‘exophthalmus’ producing effects
I Case-control studiesII Case series
Ex: - parachute for plane passengers - insulin for diabetic coma- blood transfusion for severe haemorrhage- defibrillation for ventricular fibrillation- ether to induce anaesthesia
Povl Heiberg 1897
Heiberg knew the solutions
- large numbers - large numbers !!
- randomisation - randomisation !!
- blinding - blinding !!
- independent research - independent research !!
The hierarchy of evidenceIa Systematic review of randomised clinical trials with low risk of systematic errors (bias) and of random errors (play of chance) Ib Single randomised clinical trial with low risk of systematic errors (bias) and of random errors (play of chance) II Cohort studyIII Case-control studyIV Case series
Low risk of systematic errors
High risk of systematic errors
Low risk of random errors
High risk of random errors
Random errors in small trials• False positive results
(type I error)
• False negative results
(type II error)
Components associated with bias risk
• Generation of the allocation sequence• Allocation concealment• Blinding• Incomplete outcome data (intention-to-treat)• Outcome measure reporting bias• Other components associated with bias (vested
interest bias)
TA(C)E for hepatocellular carcinoma – systematic review
Important aspects of randomised clinical trials
and meta-analyses
• Systematic errors (bias) • Random errors (play of chance)• Design errors
The hierarchy of evidenceIa Systematic review of randomised clinical trials with low risk of systematic errors and low risk of random errors Ib Single randomised clinical trial with low risk of systematic errors and low risk of random errors !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!II Cohort studyIII Case-control studyIV Case series
Dangers in non-randomised studies
• Confounding by indication and unmeasured confounding • Biological mechanisms - Limited time of diseases - Cyclical progression of diseases - When do we see patients?
• Psychological mechanism - We see what we want to see (BIAS)! - We believe what we want to believe (astrology)!
Deeks et al (2003) and the International Stroke Trial
• 19,435 patients with ischaemic stroke
• 467 clinical sites
• Randomised to aspirin versus placebo
• Dead or dependent at 6 months
• OR 0.95; 95% CI 0.89 to 1.01
Deeks et al (2003) and the International Stroke Trial
Deeks et al. resample 100 pts/group making
- Small randomised trials (n=14,000) from
randomised groups within centres
- Small controlled cohort studies (n=14,000)
from two centres
Deeks et al. 2003
Randomised clinical trials
Deeks et al. 2003
Controlled cohort studies
Deeks et al. 2003
Controlled cohort studies
Logistic regression
Deeks et al. 2003
Controlled cohort studies
Propensityscores
Meta-analysis of several trials
Low risk of bias
High risk of bias
Overall
Ratio of odds ratios (ROR)Ratio of odds ratios (ROR)
Odds ratio of trials with unclear or inadequateOdds ratio of trials with unclear or inadequate
component (high risk of bias)component (high risk of bias)
divided bydivided by
odds ratio of trials with adequate odds ratio of trials with adequate
component (low risk of bias)component (low risk of bias)
Control of selection biasGeneration of the allocation sequence
• Low risk of bias Computer system, table of random numbers, or similar
• High risk of bias Not described or quasi-randomised (excluded)
BRANDO Sequence generation
• 112 meta-analyses with• 944 randomised clinical trials • 696 (73.7%) with high risk of bias• ROR 0.89 (95% CI 0.82 to 0.96)
Control of selection biasAllocation concealment
• Low risk of bias Central independent unit, sealed envelopes (only if perfect placebo)
• High risk of bias Not described or open table of random numbers
BRANDO Allocation concealment
• 146 meta-analyses with• 1292 randomised clinical trials • 916 (70.9%) with high risk of bias• ROR 0.93 (95% CI 0.87 to 0.99)
Control of detection bias(reporting bias and observer bias)
Blinding
• Low risk of bias Identical placebo or comparator
• High risk of bias Not described or not blinded
BRANDO Blinding
• 104 meta-analyses with• 1057 randomised clinical trials • 467 (44.2%) with high risk of bias• ROR 0.87 (95% CI 0.79 to 0.96)
