Critical Appraisal

Evidence –Based Medicine

•Formulate problem

•Track down best evidence

•Appraise the evidence critically

•Implement results in clinical practice

•Evaluate performance

All published studies in reputable journals:

• Have the perfect design ?

• Use the most appropriate statistical analysis?

• Present the results in the best way?

Design Issues

Methodological issues I

• How were the subjects recruited?

• Who was included/excluded ?

• What justification is given for the sample size?

• Was bias avoided/minimised?

“A sample of patients were interviewed within the first week of their admission to hospital.

46 people were assessed, representing 15% of admissions over the study period”

Early trials of anti-histamine agents for seasickness during

World War 1

•All soldiers crossing the Atlantic on Ship A had active drug

•All soldiers crossing the Atlantic on Ship B had placebo

“The main contribution of statisticians in medical research is not to carry out statistical analyses but to inject a bit of logic into the

situation”

Methodological issues II

• Was random allocation to treatment used?

• Was potential degree of blindness used ?

• What was the duration and completeness of follow-up?

Example 1“Prevention of wound infection”

Randomisation techniques

In a randomised controlled trial:

“Patients were allocated to either a control or intervention group using date of birth”

Randomisation

Date of birth ‘randomisation’

= Systematic allocation (not randomisation)

Open to bias

RCT ‘programming error’ in allocation of treatmentAGE

60

50

40

30

20

10

0

control treatment

Presentation of Results

Descriptive Information

• Summary statistics (proportions, means, medians, ranges,

standard deviations)

• Characteristics of non-respondents

• Characteristics of withdrawals from clinical trials

“…mean (sd) of social class = 3.2 (1.2)”

Baseline characteristics

Treatment A

Treatment B

Male 48% 50% P=0.84

Age 58.6 43.2 P=0.19

Smokers 19% 13% P=0.35

Statistical Analysis

“We have chosen to use parametric statistics to stay in

line with the data analysis reported in the literature”

5-year survival rate after pancreatic resection

Treatment A (n= 8) 25%

Treatment B (n=19) 58%

(p=0.35)

Example 2“Patients with peripheral circulatory

disorders”

Report from Danish Newspaper

Impressive improvement in crime statistics 1983-1984

Number of reported crimes reduced by almost 50%

Report from Danish Newspaper

•In 1983, a man had made nuisance phone calls to the same woman (199 times)•Each had been reported and counted separately•The man was apprehended early in 1984

Data Torturing

“Opportunistic”[Fishing expeditions]

•Pore over data until a ‘significant’ association is found

•Devise a biologically plausible hypothesis to fit the association

Study of parents’ occupational exposures as a risk factor for birth defects in their

offspring

• Seven major categories of occupation identified

• No significant relationship

(for mothers or fathers)

Study of parents’ occupational exposures as a risk factor for birth defects in their

offspring

Categories split into:

• 64 separate occupations for mother

• 80 for fathers

• 5 significant relationships found

“Procrustean”

• Deciding on the hypothesis to be proved and matching the data to fit the hypothesis ie. selective reporting

[Procrustes, a robber in Greek mythology, made all his victims fit the length of his bed by stretching or cutting off their legs]

Exposure redefined to strengthen the association

In a study of adverse effects of oral contraceptives on the outcome of pregnancy

Exposure defined as OC used within 600 days before a delivery or miscarriage

Exposure redefined to strengthen the association

Why 600 days?

Why not 365 days (ie. 1 year)?

Or 2 years? Or 18 months? Or 500 days?........

Conclusion justified?

Example 3“Exposure to X-rays”

Hawthorne Effect

Hawthorne Effect

• Study by Western Electric Company to assess effects of illumination on production at the Hawthorne plant in Chicago

• Control group – worked under constant illumination

• Experimental group – worked under varying illumination

Hawthorne Effect

• Production increased in experimental and control groups at same rate

Hawthorne Effect

• Production increased in experimental and control groups at same rate

Increases in production were caused by the increased attention that workers received from management

Example 4“Bran study”

Regression to the Mean

Sir Francis Galton’s publication:

“Regression towards mediocrity in hereditary stature”

Very tall parents tend to have children who are not quite as tall, while short parents tend to have taller children

Conclusion:Never believe everything you read