Beyond evidence based medicine

Fiona GodleeEditor, BMJ

3rd European Conference on Scientific Publishing in Biomedicine and Medicine

LeydenMay 27 2010

Robert Merton

The normative structure of science

1942

Communalism

Universalism

Disinterestedness

Organised Skepticism

A comment on the state of the medical literature from Drummond Rennie, deputy editor, JAMA

There seems to be no study too fragmented, no hypothesis too trivial, no literature citation too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print.

McGauran et al. Trials 2010

GI outcomes, VIGOR, NEJM 2000

17/1000

Placebo vs Rofecoxib or Celecoxib: comparison of MI ratesJAMA 2001

Thrombotic events, Approve, NEJM 2004

BMJ 2005

Publication bias

Selective publication of results, so that positive results are over represented and negative ones under represented

Publication bias in medical research

Positive studies are more likely to be:• Published Stern and Simes, BMJ 1997

• Published faster Ioannides, JAMA 1998

• Published in higher impact journals Easterbrook et al, Lancet 1991: Tierney and Stewart, 1997

• Cited old style narrative review articles

Publication bias - Who’s to blame?

• Editors/peer reviewers– Mahoney 1977– Epstein 1990

• Researchers– Dickersin et al, Controlled Clinical Trials 1987– Olson et al, JAMA 2002

Publication bias linked to industry funding

• Rochon et al. Arch Intern Med 1994

• Lexchin, Bero et al. BMJ 2003

Duplicate publication and salami slicing

Trials of odansetron

• 84 trials that included information on 11 980 patients

• In reality only 70 trials and 8645 patients (17% of the studies had been published more than once and the number of patients had been inflated by 28%)

• Impossible to tell from published studies

• Four pairs of identical trials were published by completely different authors without any common authorship (Misconduct)

(Tramer et al, BMJ 1997: 315: 635-640)

Effectiveness of odansetron (Tramer et al, BMJ 1997: 315: 635-640)

Some things that bother editors

• Bias• Data manipulation/suppression• Duplicate publication • Fabrication• Falsification • Gift/ghost authorship • Plagiarism • Self delusion• Undeclared conflict of interest• Wrong observations/analysis/references

Errors in good faith

Trimming and cooking Fraud

Stephen Lock, BMJ

BiasData manipulation/suppression

Duplicate publication Fabrication

Falsification Gift/ghost authorship

Plagiarism Self delusion

Undeclared conflict of interestWrong observations/analysis/references

Errors in good faith

Trimming and cooking Fraud

Wrong observations

Wrong analysis

Wrong references

Bias

Self delusion

Manipulating data

Suppressing inconvenient facts

Gift authorship

Duplicate publication

Salami publication

Fabrication

Falsification

Plagiarism

Undeclared conflicts of interest

Stephen Lock, BMJ

Undeclared competing interest/ghosts and guests

• RCT of Vioxx versus Naproxen Lisse et al. Ann Intern Med 2003; 139: 539-46

• New York Times,24 April 2005“Merck designed the trial, paid for the trial, ran the trial…Merck came to me after the study was completed and said, ‘We want your help to work on the paper.’ The initial paper was written at Merck, and then sent to me for editing.”

Guests and ghosts are not “a thing of the past”

• Surveyed authors of 900 articles published in high impact general medical journals in 2008

• 1 in 5 respondents admitted to at least one guest author (no change since 1996)

• Nearly 8% admitted to at least one ghost author (slightly lower than 1996)

(Flanagin A, et al. Presented at Peer Review Congress, 2009)

Influence of financial links with industry on authors of reviews and letters

• Stelfox et al, NEJM 1998• Lexchin, Bero et al. BMJ 2003• Barnes DE, Bero LA. JAMA 1998• Yank, Rennie, Bero, BMJ 2007• Wang et al, BMJ 2010

Effect of conflict of interest statements on readers’ perceptions

• Two RCTs

Chaudry et al, BMJ 2002Schroter et al, BMJ 2004

• A research article with a financial conflict of interest was rated less highly by readers than the same paper with no COI or non-financial COI

What is it about conflicts of interest that bother us?

• Having them at all?• Failing to declare them?

• Is disclosure enough?

How do we act on disclosed information?

Three options

• Disclosure is enough in itself. After that, anything goes, caveat emptor

• Disclosure is followed by a judgement about the degree of potential conflict

• Disclosure of a conflict of interest is a bar to taking part (membership of a panel, authorship of an editorial, involvement in a piece of research)

Science depends on…

• Open critique and debate• Correction/retraction• Honesty and transparency• Identification and minimisation of bias • Repeatablity• Refutation or confirmation• Viewed as a public good• Ethics

Why does integrity matter?

Damage to the integrity of the biomedical literature brings:

• Pursuit of wrong avenues of scientific enquiry• Participants and patients put at risk• Waste of time and resources• Loss of public trust in science• Less willingness to fund research• Less willingness among patients and the public to take

part in research

Sources of pressure on integrity

• Science is a human enterprise• Human beings are fallible• Careers and huge sums of money are at • Libel laws increasingly being invoked, protecting those with

money• Science is based on a system of trust• Existing checks and balances are only partially effective • Training and supervision are inadequate• The biomedical literature is fragmented, making it hard to get

to the “truth”

With the result that

a lot of published research is of poor quality, and some of it (how much?) is fraudulent

“The community presents a natural work setting for an editor, not the laboratories or surgeries. The editor is a community worker and a teacher.”

Andrija Stampar, one of the founders of WHO

Problems with peer review

• Slow• Expensive• Biased• Unaccountable• Stifles innovation• Bad at detecting error• Hopeless at detecting fraud

• And journals have conflicts of interest too

How do journals make their money?

BMJ ?NEJM

Ads

Subs

Reprints

“Radical transparency”

• Nearly all decision making is carried publicly

• Not only the ordinary information of interest to the community is made freely available, but all (or nearly all) meta-levels of organizing and decision making are also published

• Includes draft documents, all arguments for and against a proposal, the decisions about the decision making process itself, and all final decisions

• Excludes data relating to personal security, passwords etc

(from Wikipedia)

Moving towards greater transparency

• Trial registration• Protocols• Data sharing• Mandate data availability• Unlimited space online• Uniform COI form• Open access• Open peer review/pre-publication history• Article level metrics

But will any of this really help?

A more radical future• Stop using opinion leaders with any financial

conflicts of interest?• Stop pharmaceutical companies directly

evaluating their own products?• An end to pre-publication peer review?• Publication of entire data sets

Why we need raw data

Godlee F, Clarke M

Why don’t we have all the evidence on oseltamivir?

BMJ 2009

Conclusions

• The integrity of the biomedical literature is essential if we are to practice evidence based medicine

• Loss of integrity distorts the scientific record and threatens public trust in science, funding and participation in research, and patient safety

• The threats to integrity are many and various• Some threats are caused by honest error, but a

significant proportion are intentional, for commercial or career gain (misconduct/fraud)

• Additional threats come from unnecessary and poorly done research

Conclusions

• Current efforts to protect the integrity of medical knowledge are inadequate

• Some hope lies in better education, greater transparency, and ever more rigorous peer review

• Tackling the influence of the drug industry remains a major challenge for clinicians, researchers, journals, journalists, and the public

• More radical alternative forms of research funding and data publication are needed

Thank you

fgodlee@bmj.com