Scientific commonsense and statistics in medicine: getting back to the Hill top John Worrall...

Scientific commonsense and statistics

in medicine: getting back to the Hill top

John Worrall Philosophy, Logic and Scientific Method

London School of Economics

Statistical Science & Philosophy of Science, LSE June 2010

EBM

• 1. Sketch evolution of EBM• 2. Seemed initially to endorse a very rigid view on what

counts as evidence• 3. Show how it has retreated/been clarified to bring it

more into line with scientific commonsense (aka some basic principles from philosophy of science)

• 4. Add as a historical footnote that EBM seems to have been gradually rediscovering the much more nuanced and commonsensical views of Austin Bradford Hill, whom it venerates as one of its founders.

EBM

EBM

• Not critical of RCTs• But only of inflated claims about their epistemic power• Indeed …

Evidence based everything!

• “A wise man proportions his belief to theevidence.” David Hume Enquiry Concerning Human Understanding

The evolution of EBM

• Word association: EBM → RCT → ‘Gold standard’• Initial impression: EBM says only evidence from RCTs

counts scientifically• Blood letting, e.g., may not be just an historical

phenomenon• Should trust neither ‘patho-physiologic’ rationale nor so-

called clinical expertise• Only trial evidence counts, and only RCTs are free from

bias


• Misinterpretation??• Certainly but:

• ‘[if] the study was not randomized we’d suggest that you stop reading it and go on to the next article in your search’ (Sackett et al Evidence Based Medicine, 3rd edition, p.108)


• Some pro-EBM cases (grommets for glue ear)• But more measured voices immediately pointed out:• Lots of contrary cases:


• thyroxine for myxoedema• insulin for diabetic ketoacidosis• vitamin B12 for pernicious anaemia, etc,etc

• appendicectomy for acute appendicitis etc.


• Retreat/clarification• Clinical expertise/ ‘patho-physiologic rationale’ to be

incorporated not ignored..• (How?)• Other types of evidence have some weight• But RCT retains a very special role• Evidence Hierarchy

An Evidence Hiearchy


• But only one of many• 2002 study found 40 different hierarchies• 2006 study added 20 more• All agree that the RCT remains king (trump, no overall

evaluation)• But also differences• a. some put meta-analyses top, others omit them• b.cohort/case control


• Also• 1. Explicit concession that RCTs are not needed for

‘dramatic’ effects (Glaziou et al)• 2. Fleeting recognition that it seems odd to hold that

some of the most clearly efficacious treatments do not have ‘best’ evidence

• 3. And odd that various unfortunate a priori judgments are endorsed


• Also also• Swing in the frequentist/Bayesian balance• Finally• One very influential voice:

Sir Michael Rawlins


• Rawlins certainly pro evidence in general sense• But scathing about ‘anorak EBM’


• 1. Evidence hiearchies internally unjustified since they overrate RCTs:

• “The notion that evidence can be reliably placed in hierarchies is illusory. Hierarchies place RCTs on an undeserved pedestal for .. although the technique has advantages it also has significant disadvantages. Observational studies too have defects but they also

have merit.”


• 2. Whole idea a mistake:• “Hierarchies attempt to replace judgement with an

oversimplistic, pseudo-quantitative, assessment of the quality of the available evidence. [In fact d]ecision makers have to incorporate judgements, as part of their appraisal of the evidence, in reaching their conclusions.”

Philosophy of Science to the rescue?

• Groundhog day?• Whole rationale for EBM distrust of judgment…• Flux →


• Need to start from a more fundamental perspective• RCT is not the embodiment of the scientific method• But is resorting to philosophy of science likely to resolve

controversy?• Many differences in logic of evidence but only need one

and a half fundamental and agreed principles


• Everyone agrees that “real” evidence for T not only• (i) accords with T; but also• (ii) is ‘unlikely otherwise’• So in particular evidence is the stronger the more

plausible alternatives it rules out• Popper, Bayes, Mill, Mayo .. all agree that support

depends on p(e,b)• Popper (and Mayo) – ‘severe test’• Bayes factor = p(e,h & b)/p(e,b)


• Principle 1 entails:• (i) When confronted with an apparently positive trial

result, the question always is: ‘is there a plausible alternative explanation for this outcome other than the (superior) effectiveness of the treatment?’

• (ii) You can’t be in a better epistemic position in a trial than if background knowledge supplies no reason to think that the experimental and control group are significantly different.


• These in turn entail:• 1. Randomizing is neither sufficient nor necessary for

genuine support from a trial outcome• 2. These plausibility judgments are going (often) to

depend on effect size• 3. The only solid reason to randomize is that it controls –

not for all possible confounders – but instead for the specific possible confounder ‘selection bias’

• (BUT if the selection bias can be eliminated by other means – or at any rate reduced – and the effect size is large …. )

Randomizing not sufficient

• Despite claims like these:• “In a randomised trial, the only difference between the

two groups being compared is that of most interest: the intervention under investigation.” (Mike Clarke)

• “As their name suggests, RCTs involve the random allocation of different interventions (treatments or conditions) to subjects. As long as numbers of subjects are sufficient, this ensures that both known and unknown confounding factors are evenly distributed between treatment groups.” (Wikipedia)

http://en.wikipedia.org/wiki/Random

http://en.wikipedia.org/wiki/Research_subject

http://en.wikipedia.org/wiki/Statistical_power

http://en.wikipedia.org/wiki/Confounding


• Everyone knows the sufficiency claim is false• Amusing example• Leibovici et al “Effects of remote, retroactive,

intercessory prayer on outcomes in patients with bloodstream infection: randomised controlled trial” BMJ 2001


• 3393 patients having a bloodstream infection while being an inpatient at the Rabin Medical Centre during 1990-6 were identified.

• In July 2000 a random number generator was used to divide these patients into two groups and which of these two became the treatment group was decided by a coin toss.

• 1691 were randomized to the intervention group and 1702 to the control.

• Checked for ‘baseline imbalances’ with regard to main risk factors for death and severity of illness.


• The names of those in the intervention group were given to a person ‘who said a short prayer for the well being and full recovery of the group as a whole.’

• Results: both length of stay in hospital and duration of fever were significantly shorter in the intervention group (p = 0.01 and p = 0.04)!

• Conclusion: ‘Remote, retroactive intercessory prayer said for a group is associated with a shorter stay in hospital and shorter duration of fever in patients with bloodstream infection and should be considered for use in clinical practice.’

• Somewhat tongue-in-cheek of course (‘No patients were lost to follow up’!!)


• Natural reaction shows we are all Bayesian:• “If the pre-trial probability is infinitesimally low, the results

of the trial will not really change it, and the trial should not be performed. This, to my mind, turns the article into a non-study, though the details provided (randomization done only once, statement of a prayer, analysis, etc) are correct.”

Randomizing not necessary

• As now acknowledged by Glaziou et al:• If you have an effect size so ‘dramatic’, and no reason to

think that the patients you are treating now are greatly different from those treated earlier, then there is no plausible alternative to the theory that the effect is produced by the treatment.


• Other ‘half’ principle:• Make sure you’re testing the theory you want to be

tested.

11/2. What is evidence, evidence for?

• Typical research reports:• ‘Efficacy and safety of ustekinumab .. in patients with

psoriasis..’ • ‘Active symptom control with or without chemotherapy in

the treatment of patients with malignant pleural mesothelioma ..’


• Report (usually randomized) trials on some selected group of patients

• involving a number of exclusion criteria • generally using some very precise treatment regimen;

and where• the treatment is given for some relatively brief period • Rawlins ‘ Most RCTs, even for interventions that are

likely to be used by patients for many years, are of only six to 24 months duration.’

• (N.B. However this is also true of non-RCTs)


• Result may be: D is ‘more effective’ than the comparator for condition C

• What exact theory has been tested?• Not really this (vague) claim but rather:• D when administered in a very particular way to a very

particular set of patients for a particular length of time is more effective than some comparator treatment (perhaps placebo).

• RCT provides – let’s say impeccable- evidence for this


• But this is not the claim that the practitioner is interested in.

• She wants evidence for:• D is effective (in a wide sense) when prescribed to the

sorts of patients she would like to prescribe it to.• ‘Target population’ will include the excluded• Dosage may be adjusted• For chronic conditions prescribed for a long time


• Usually run in terms of ‘external validity’• But better as: wrong theory being tested• NB not a Humean ‘purely philosophical’ issue• Specific grounds for thinking the target and study

populations different• Bartlett et al looked at RCTs on NSAIDs and Statins and

found older people, women and ethnic minorities all consistently underrepresented


• When looked at in my phil of sci way clear that • RCT gives strongest evidence for wrong theory (…)• By no means entails• RCT gives strongest evidence for right theory


• Truog argued this in case of ECMO• (largely on grounds of changing technologies)• Bluhm extended argument to chronic diseases• (largely on grounds of short term nature of trials

compared to long term nature of treatment)• NB not a case of ‘no RCTs so go down the hierarchy to

the “next best”’• Rather observational studies arguably give best

evidence - once the right theory has been identified

How much of this was anticipated by Hill


• “Criteria”• But actually ‘commonsense’ is more important• A 1950 paper ‘On the controlled trial’• 2 1965 papers in particular• “The Environment and Disease: Association or

Causation?”• *“Reflections on the Controlled Trial”*• (Continuing discussion in epidemiology of the “criteria”

and some periodic ‘rediscovery’.. But ..)


• Widely regarded as the prime mover re the randomized clinical trial

• BUT:• “The history of science … shows that frequently with a new

discovery … the pendulum at first swings too far. Has this been so with the clinical trial? Is it true, as Cromie (1963) has suggested, that ‘little or no credence is now given to clinical observations even by experienced investigators’ while there is ‘a blind acceptance of double-blind trails without a clinical evaluation of their shortcomings and their ability to mislead as well as to lead.”

• And he concludes in fact that• “Any belief that the controlled trial is the only way would mean not

that the pendulum had swung too far but that it had come right off its hook.”


• 1. We have to use ‘background knowledge’ in order to interpret any result:

• Need to look at any putatively positive result in whatever trial with ‘the fundamental question’ in mind

• “That fundamental question [is] – is there any other way of [plausibly] explaining the set of facts before us, is there any other answer equally, or, more likely, than cause and effect?”

• The Leibovici example shows that sometimes the answer is ‘ there must be but we can’t specify what it is’


• Same fundamental question whether the trial was randomized or not and even if there was no formal trial at all!

• Enthusiastically endorses Claude Bernard:• “.. it is imperative that we draw no precise line between observation

and experiment. It is just 100 years since the great experimentalist Claude Bernard (1865) wrote: ‘a physician observing a disease in different circumstances reasoning about the influence of these circumstances and deducing consequences which are controlled by other observations – this physician reasons experimentally even though he makes no experiments.”


• RCTs do have one - but only one – advantage:• “Faithfully adhered to [randomizing] offers three great advantages:

(1) it ensures that our personal feelings or judgments, applied consciously or unconsciously, have not played any part in building up the various treatment groups; from that aspect, therefore, the groups are unbiased; (2) it removes the very real danger, inherent in any allocation which is based upon personal judgments, that believing our judgments may be biased, we endeavour to allow for that bias in so doing may ‘lean over backwards’ and thus introduce a lack of balance from the other direction; (3) having used such a random allocation we cannot be accused by critics of having set up personally biased groups for comparison.”


• And this is plausible only if apparent effect is small• Indeed where it is large we not only do not need

randomization we don’t need any formal statistical tests.• Relates his investigation into sickness patterns in card

rooms in cotton mills in Lancashire.• The illnesses the card room workers suffered from were

so much worse and so different from workers in other parts of the mill, that Hill argued, the evidence established a causal link with the environmental conditions.


• He reports:• ‘My results were set out for men and women separately and for half

a dozen age groups in 36 tables. So there were plenty of sums. Yet I cannot find anywhere I thought it necessary to use a test of significance. The evidence was so clear cut, the differences between the groups were mainly so large, the contrast between

respiratory and non-respiratory causes of illness so specific, that no formal tests could really contribute anything of value to the argument. So why use them?’


• Statistics has uses, of course, but again a pendulum swinging too far:

• “To decline to draw conclusions without standard errors can surely … be silly? Fortunately I believe we have not yet gone so far as our friends from the USA where, I am told, some editors of journals will return an article because tests of significance have not been applied [!!]. Yet there are innumerable situations in which they are totally unnecessary – because the difference is grotesquely obvious, because it is negligible, or because, whether it be formally significant or not, it is too small to be of any practical importance.”


• So always same fundamental question – essentially “what else can it be?” – whatever results you are looking at.

• Basic philosophy underlying famous “criteria” of causality• strength, consistency, specificity, temporality, biological

gradient, plausibility, coherence, experiment, and analogy

• Want here just to articulate and endorse underlying view of evidence


• 1. Have ‘the facts before us’• 2.Is the relationship causal or ‘merely’ an association?• 3. Both answers are deductively consistent with the facts• 4. Need some further evidential input if we are to be on

as safe ground as possible• 5 Of course if we had rock solid evidence of a

deterministic mechanism wouldn’t even need stats.


• 6. But such clear cut ‘background’ evidence not available in cases of interest

• 7 Hill’s main point is we should not give up:• Background knowledge may at least assist us by

providing certain general constraints• 8. E.g. background knowledge tells us that many causal

mechanisms are linear• Hence more heavy smokers suffer more cancers

supplies further evidence for the causal link (dose-response)


• 9. Similarly background knowledge makes certain causal links plausible and others implausible without supplying detailed knowledge of the links

• (combination of plausibility and coherence)• Of course everything defeasible – but no excuse for

failing to make decisions in the light of current evidence


• Exercising judgment (Rawlins)?• Certainly not unanalysable judgment• We know a lot about the world ahead of any statistical

trial• Fisher’s understandable but egregious error• Bayesians don’t help by talking about ‘subjective priors’• Nothing subjective, e.g., about the view that remote

intercessory prayer can have no effect!


• Very much aware of 2nd issue too.• Indeed his very first criticism of many RCTs is that they

address the wrong question:• “many controlled trials that are published fall lamentably short of

what is really required .. The authors do not appear to have asked themselves at the outset the deceptively simple but dominating question ‘what precisely am I trying to find out?’”

• Always need to have an eye to generalising and generalising to the target population

• Makes a number of challenging points


• End by highlighting two:• 1. Double blinding may be a problem rather than a virtue• 2. Data mining is often good.• “There is one feature of the modern controlled trial that frequently

hampers the clinician in making acute and discriminating observations of his patient – and that is the double-blind procedure.”

• Of course he recognises that not blinding introduces the possibility of bias: especially in estimating subjective outcomes.


• But even where you can eliminate that by a division of labour

• In some situations double blinding is ‘injurious to the trial’• “Such situations arise when it is important, for the sake of a realistic

trial, that the doctor in charge of the patient be able to adjust the dose of a drug according to the patient’s reactions and according to his judgment of the patient’s requirements….It may well be asked therefore in the planning of a trial, which is the more important – for the doctor to be ignorant of the treatment and unbiased in his judgment or for him to know what he is doing and to be able to adjust what he is doing so as to observe closely the results and then make unbiassed judgments to the best of his ability and conscious mind?”


• NO reason why a controlled trial cannot be aimed at the question:

• “ If competent clinicians in charge of defined types of patients use drug X in such varying amounts and for such varying durations of time, and so forth, as they think advisable for each patient, what happens?”


• 2. Data mining may not be an epistemic sin but is often essential.

• Essential even within trials that show an overall marginal effect at best to try to “identify some sub-group of patients who do tend to respond favourably to the treatment.”

• Of course not just any sub-group – that would be epistemically sinful – but any sub-group that background knowledge tells you might plausibly react differently.


• Disagrees with Sir John McMichael who wrote• “the aim of a statistical trial is to include all the unpredictable

multitude of factors which can influence the outcome by a comprehensive sample. Unless the treatment shows a convincing difference in outcome in the whole group it is not permissible to separate out afterwards a sub-division of better results. Any sub-divisions should be done on other criteria before the trial begins.”

• BUT, Hill:• “McMichael is .. criticizing the analysis of the MRC report on long-

term treatment with anti-coagulants in terms of age and sex; two features in prognosis which invariably and so often call for divided attention that there could never be any question of before and afterwards.”


• But in any event• “.. I can see no argument in favour of his view, either statistical or

logical. If there is an ‘unpredictable multitude of factors which can influence the outcome’,then it is surely our job, and duty, to see whether in the analysis we can identify them and thus make them predictable.”

Conclusion

• Hill had a much more sophisticated (and yet commonsensical) view of evidence than that initially held by EBM-ers who thought of themselves as his followers

• His insights are only gradually (and partially) being rediscovered.

• EBM needs to keep on trying to get back to the Hill top.

Date post:	21-Dec-2015
Category:	Documents
View:	218 times
Download:	4 times

Scientific commonsense and statistics in medicine: getting back to the Hill top John Worrall...

Documents