How to Read a Paper Methodological Quality

8/12/2019 How to Read a Paper Methodological Quality

1/5

BMJ Publishing Group

How to Read a Paper: Assessing the Methodological Quality of Published PapersAuthor(s): Trisha GreenhalghReviewed work(s):Source: BMJ: British Medical Journal, Vol. 315, No. 7103 (Aug. 2, 1997), pp. 305-308Published by: BMJ Publishing GroupStable URL: http://www.jstor.org/stable/25175335.

Accessed: 19/12/2012 04:33

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at.http://www.jstor.org/page/info/about/policies/terms.jsp

.JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of

content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms

of scholarship. For more information about JSTOR, please contact [email protected].

.

Digitization of the British Medical Journal and its forerunners (1840-1996) was completed by the U.S. NationalLibrary of Medicine (NLM) in partnership with The Wellcome Trust and the Joint Information SystemsCommittee (JISC) in the UK. This content is also freely available on PubMed Central.

BMJ Publishing Groupis collaborating with JSTOR to digitize, preserve and extend access toBMJ: British

Medical Journal.

http://www.jstor.org

This content downloaded on Wed, 19 Dec 2012 04:33:56 AMAll use subject to JSTOR Terms and Conditions
http://www.jstor.org/action/showPublisher?publisherCode=bmjhttp://www.jstor.org/stable/25175335?origin=JSTOR-pdfhttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/stable/25175335?origin=JSTOR-pdfhttp://www.jstor.org/action/showPublisher?publisherCode=bmj


2/5

Education and debate

How to read a paperAssessing the methodological quality of published papers

Trisha Greenhalgh

Before changing your practice in the light of apublished research paper, you should decide whetherthe methods used were valid. This article considers fiveessential questions that should form the basis of yourdecision.

Question 1 :Was the study original?Only a tiny proportion of medical research breaksentirely new ground, and an equally tiny proportionrepeats exactly the steps of previous workers. The vast

majorityof research studies will tell us, at best, that a

particular hypothesis is slightlymore or less likely to becorrect than it was before we added our piece to thewider jigsaw. Hence, itmay be perfectly valid to do astudy which is, on the face of it, unoriginal. Indeed,the whole science of meta-analysis depends on the literature containing more than one study that hasaddressed a question in much the same way.

The practical question to ask, then, about a newpiece of research is not Has anyone ever done a similar study? but Does this new research add to theliterature in any way? For example:

Is this study bigger, continued for longer, or otherwise more substantial than the previous one(s)?

Is themethodology of this study anymore rigorous(in particular, does it address any specific method

ological criticisms of previous studies)?Will the numerical results of this study addsignificantly to a meta-analysis of previous studies?Is the population that was studied different in any

way (has the study looked at different ages, sex, orethnic groups than previous studies)?Is the clinical issue addressed of sufficientimportance, and is there sufficient doubt in the minds

of the public or key decision makers, to make new evidence politically desirable even when it is not strictlyscientifically necessary?

Question 2:Whom is the study about?Before assuming that the results of a paper areapplicable to your own practice, ask yourself thefollowing questions:

How were the subjects recruited? If you wanted to do aquestionnaire survey of the views of users of the hospital casualty department, you could recruit respondents

by advertising in the local newspaper. However, thismethod would be a good example of recruitment bias

since the sample you obtain would be skewed in favourof users who were highly motivated and liked to readnewspapers. You would, of course, be better to issue aquestionnaire to every user (or to a 1 in 10 sample ofusers) who turned up on a particular day.Who was included in the study?Many trials inBritainand North America routinely exclude patients withcoexisting illness, those who do not speak English,those taking certain other medication, and those who

Summary pointsThe first essential question to ask about themethods section of a published paper is:was thestudy original?

The second is:whom is the study about?Thirdly, was the design of the study sensible?

Fourthly, was systematic bias avoided orniinimised?Finally, was the study large enough, andcontinued for long enough, to make the resultscredible?

are illiterate. This approach may be scientificallyclean, but since clinical trial results will be used to

guide practice in relation to wider patient groups it isnot necessarily logical.1 The results of pharmacokineticstudies of new drugs in 23 year old healthy malevolunteers will clearly not be applicable to the averageelderly woman.

Who was excluded from the study? For example, a randomised controlled trialmay be restricted to patientswith moderate or severe forms of a disease such asheart failure?a policy which could lead to falseconclusions about the treatment of mild heart failure.

This has important practical implications when clinicaltrials performed on hospital outpatients are used to

dictate best practice in primary care, where the spectrum of disease is generally milder.

Were the subjects studied in real life*'circumstances? Forexample, were they admitted to hospital purely forobservation? Did they receive lengthy and detailedexplanations

of thepotential benefits of the

intervention? Were they given the telephone number of a keyresearch worker? Did the company that funded theresearch provide new equipment which would not beavailable to the ordinary clinician? These factors wouldnot necessarily invalidate the study itself, but theymaycast doubt on the applicability of its findings to yourown practice.

Question 3:Was the design of the studysensible?Although the terminology of research trialdesign canbe forbidding, much of what isgrandly termed criticalappraisal is plain common sense. I usually start withtwo fundamental questions:

What specific intervention or other manoeuvre was beingconsidered, and what was it being compared with? It istempting to take published statements at face value, butremember that authors frequently misrepresent (usu

This is the thirdin a series of 10articlesintroducingnon-experts tofindingmedicalarticles andassessing theirvalue

Unit forEvidence-BasedPractice and Policy,Department ofPrimary Care andPopulationSciences, UniversityCollege LondonMedical School/Royal Free HospitalSchool of Medicine,

WhittingtonHospital, LondonN19 5NFTrisha Greenhalgh,senior [email protected]

BMJ 1997;315:305-8

BMJ VOLUME 315 2AUGUST 1997 305

This content downloaded on Wed, 19 Dec 2012 04:33:56 AM

All use subject to JSTOR Terms and Conditions
http://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsp


3/5


Examples of problematic descriptions in the methods section of a paperWhat the authors said What they should have said (or should have done) An example of:

We measured how often GPs askpatients whether they smoke.

We measured how doctors treat lowback pain.

We compared anicotine-replacement patch withplacebo.

We asked 100 teenagers toparticipate in our survey of sexualattitudes.

We randomised patients to either'individual care plan' or 'usual care'.

'To assess the value of an educationalleaflet, we gave the intervention groupa leaflet and a telephone helplinenumber. Controls received neither.We measured the use of vitamin C in

the prevention of the common cold.

We looked in patients' medical records and countedhow many had had their smoking status recorded.We measured what doctors say they do when faced witha patient with low back pain.Subjects in the intervention group were asked to apply a

patch containing 15 mg nicotine twice daily; those in thecontrol group received identical-looking patches.We approached 147 white American teenagers aged12-18 (85 males) at a summer camp; 100 of them (31

males) agreed to participate.

The intervention group were offered an individual careplan consisting of...; control patients were offered...

If the study is purely to assess the value of the leaflet,both groups should have been given the helplinenumber.

A systematic literature search would have foundnumerous previous studies on this subject14

Assumption that medical records are100% accurate.Assumption that what doctors say

they do reflects what they actually do.Failure to state dose of drug ornature of placebo.

Failure to give sufficient informationabout subjects. (Note in this examplethe figures indicate a recruitmentbias towards females.)Failure to give sufficient informationabout intervention. (Enoughinformation should be given to allowthe study to be repeated by otherworkers.)Failure to treat groups equally apartform the specific intervention.

Unoriginal study.

ally subconsciously rather than deliberately) what theyactually did, and they overestimate its originality andpotential importance. The examples in the box usehypothetical statements, but they are all based on similar mistakes seen in print

What outcome was measured, and how? If you had anincurable disease for which a pharmaceutical companyclaimed to have produced a new wonder drug, you

would measure the eflBcacy of the drug in terms ofwhether itmade you live longer (and, perhaps, whetherlife was worth living given your condition and any sideeffects of the medication). You would not be too interested in the levels of some obscure enzyme in yourblood which the manufacturer assured you were a reliable indicator of your chances of survival. The use ofsuch surrogate endpoints is discussed in a later articlein this series.2

The measurement of symptomatic effects (such aspain), functional effects (mobility), psychological effects(anxiety), or social effects (inconvenience) of anintervention is fraught with even more problems. Youshould always look for evidence in the paper that the

outcome measure has been objectively validated?thatis, that someone has confirmed that the scale ofanxiety, pain, and so on used in this study measures

what it purports to measure, and that changes in thisoutcome measure adequately reflect changes in thestatus of the patient Remember that what is importantin the eyes of the doctor may not be valued so highly bythe patient, and vice versa.3

Question 4:Was systematic bias avoidedor mminiised?Systematic bias isdefined as anything that erroneouslyinfluences the conclusions about groups and distortscomparisons.4 Whether the design of a study is arandomised controlled trial, a non-randomised comparative trial, a cohort study, or a case-control study, theaim should be for the groups being compared to be assimilar as possible except for the particular difference

being examined. They should, as far as possible, receivethe same explanations, have the same contacts with

health professionals, and be assessed the same numberof times by using the same outcome measures.Different study designs call for different steps to reducesystematic bias:

Randomised controlled trialsIn a randomised controlled trial, systematic bias is (intheory) avoided by selecting a sample of participants

from a particular population and allocating them randomly to the different groups. Figure 1 summarisessources of bias to check for.

306 BMJ VOLUME 315 2AUGUST 1997




4/5


Target population (baseline state)

.*Allocation

Selection bias (systematicdifferences inthe comparisongroups attributable oincomplete randomisation)

Performance bras (systematicdifferences inthe careprovided, apart from theinterventionbeing evaluated)

Exclusionbias (systematicdifferences inwithdrawalsfrom the trial)Detection bias (systematicdifferences inoutcome

assessment)

Fig 1 Sources of bias to check for in a randomised controlled trial

Non-randomised controlled clinical trialsI recendy chaired a seminar in which a multidisciplinary group of students from the medical, nursing,pharmacy, and allied professions were presenting theresults of several in house research studies. All but oneof the studies presented were of comparative, but nonrandomised, design?that is, one group of patients (say,hospital outpatients with asthma) had received oneintervention (say, an educational leaflet) while anothergroup (say, patients attending GP surgeries withasthma) had received another intervention (say, groupeducational sessions). Iwas surprised how many of thepresenters believed that their study was, or was equivalent to, a randomised controlled trial. In other words,these commendably enthusiastic and committed youngresearchers were blind to the most obvious bias of all:they were comparing two groups which had inherent,self selected differences even before the intervention

was applied (aswell as having all the additional potential sources of bias of randomised controlled trials).As a general rule, if the paper you are looking at isa non-randomised controlled clinical trial, you mustuse your common sense to decide if the baseline differences between the intervention and control groups arelikely to have been so great as to invalidate any differences ascribed to the effects of the intervention. This is,in fact, almost always the case.56

Cohort studiesThe selection of a comparable control group is one ofthe most difficult decisions facing the authors of anobservational (cohort or case-control) study. Few, if any,cohort studies, for example, succeed in identifying twogroups of subjects who are equal in age, sex mix,socioeconomic status, presence of coexisting illness,and so on, with the single difference being their exposure to the agent being studied. In practice, much ofthe controlling in cohort studies occurs at the analysis stage, where complex statistical adjustment ismadefor baseline differences in key variables. Unless this is

done adequately, statistical tests of probability and confidence intervals will be dangerously misleading.7This problem is illustrated by the various cohortstudies on the risks and benefits of alcohol, which have

Intervention roup Controlgroup

Exposed to Not exposedintervention to intervention

Follow up Follow up

Outcomes V Outcomes

consistently found a J shaped relation betweenalcohol intake and mortality. The best outcome (interms of premature death) lies with the cohort who are

moderate drinkers.8 The question of whether teetotallers (a group that includes people who have beenordered to give up alcohol on health grounds, healthfaddists, religious fundamentalists, and liars, as well asthose who are in all other respects comparable with thegroup of moderate drinkers) have a genuinelyincreased risk of heart disease, or whether theJ shapecan be explained by confounding factors, has occupiedepidemiologists for years.8

Case-control studiesIn case-control studies (in which the experiences ofindividuals with and without a particular disease areanalysed retrospectively to identify putative causativeevents), the process that ismost open to bias is not theassessment of outcome, but the diagnosis of casenessand the decision as to when the individual became acase.

A good example of this occurred a few years agowhen a legal action was brought against the manufacturers of the whooping cough (pertussis) vaccine,which was alleged to have caused neurological damage

in a number of infants.9 In the court hearing, the judgeruled that misclassification of three brain damagedinfants as cases rather than controls led to theoverestimation of the harm attributable to whoopingcough vaccine by a factor of three.9

Question 5: Was assessment blind ?Even the most rigorous attempt to achieve a comparable control group will be wasted effort if the peoplewho assess outcome (for example, those who judgewhether someone is still clinically in heart failure, orwho say whether an x ray is improved from last time)know which group the patient they are assessing wasallocated to. If, for example, I knew that a patient hadbeen randomised to an active drug to lower bloodpressure rather than to a placebo, I might be morelikely to recheck a reading which was surprisingly high.

This is an example of performance bias, which, alongwith other pitfalls for the unblinded assessor, is listed infigure 1.

Question 6:Were prehminary statisticalquestions dealt with?Three important numbers can often be found in themethods section of a paper: the size of the sample; theduration of follow up; and the completeness of followup.

Sample sizeIn the words of statistician Douglas Altman, a trialshould be big enough to have a high chance of detecting, as statistically significant, a worthwhile effect if it

exists, and thus to be reasonably sure that no benefitexists if it isnot found in the trial.10 o calculate samplesize, the clinician must decide two things.The first iswhat level of difference between the twogroups would constitute a clinically significant effect

Note that this may not be the same as a statistically sig

BMJ VOLUME 315 2AUGUST 1997 307




5/5

Date post:	03-Jun-2018
Category:	Documents
Upload:	jose-alfonso-c
View:	226 times
Download:	0 times

How to Read a Paper Methodological Quality

Documents