The Importance of Systematic Reviews

The Campbell Collaboration www.campbellcollaboration.orgThe Campbell Collaboration www.campbellcollaboration.org

Howard WhiteCEO, Campbell Collaboration

@washeval @c2update @HowardNWhite

The importance of systematic reviews

The Campbell Collaboration www.campbellcollaboration.org


The seven piece study

But these are observational data, which don’t control for selection bias (people who eat more than five portions a day are wealthy, educated, health fanatics)

The five piece study

This is a systematic review, using data from 16 high-quality studies (observational data but analysis controls for confounders)




WHAT IS GOING ON HERE?


Errors in hypothesis testingH0 correct H0 false

Don’t reject H0 No error Type II error‘false negative’

Reject H0 Type I error‘false positive’

No error


Hypothesis testing: if H0 true

H0: ß = 0

Don’t reject H0

5% chance that when H0 true get a sample leads you to reject H0


Errors in hypothesis testing

H0 true H0 false

Don’t reject H0 No error Type II error

Reject H0 Type I error = 5%

No error


Hypothesis testing HA correct

0Incorrectly reject HA approx 40% of the time

Power = 1 – type II error

HA


Errors in hypothesis testing

H0 true H0 false

Don’t reject H0 No error Type II errorMaybe 20% but often

40-60%

Reject H0 Type I error = 5%

No error


The horrifying truth about hypothesis testing

• If the ‘null hypothesis’ is correct (null = no programme impact) then we will correctly agree with the null 95% of the time (we are wrong 5% of the time)

• But if the null hypothesis is wrong (the programme works) then we probably incorrectly conclude the programme doesn’t work 40-60% of the time!!!


Implications

• An under-powered RCT is no better than tossing a coin at determining if a successful programme is working so

• Power, power, power• We also need replicate ‘unsuccessful’ programmes• And we really REALLY need to do systematic reviews with

meta-analysis….


Pooling evidence

1 = no effect

Intervention worksIntervention is harmful

So pooling data allows us to overcome the high risk of Type II error

And goal scoring can be very misleading (DON’T do it)


A real life example

30-50% reduction in mortality

Corticosteriod for women about to deliver prematurely


So let’s look at the cancer graph again …


Eggs


So not much gives you cancer…


Source: Li-Xuan Sang et al. Consumption of coffee associated with reduced risk of liver cancer: a meta-analysis BMC Gastroenterol. 2013; 13: 34 doi: 10.1186/1471-230X-13-34

COFFEE AND LIVER CANCERBut coffee is good for you

http://dx.doi.org/10.1186/1471-230X-13-34


And this really matters… growing number of studies show most things don’t work.

• Education: 90 interventions evaluated in RCTs by IES - 90% had weak or no positive effects.

• Employment/training: Department of Labor-commissioned RCTs 75% weak or no positive effects

• Business: Over 13,000 RCTs of new products/strategies conducted by Google and Microsoft, 80- 90% no significant effects.

But are these sufficiently powered???Need to combine the evidence


There’s more to a systematic review than meta-analysis

- Systematic search- Systematic screening- Systematic coding- Systematic synthesis- Systematic presentation of resultsNot being systematic introduces bias – as we shall see


What the evidence synthesis process should look like

Source: Julia Littell: Campbell Systematic Reviews: Evidence for Implementation and Impact, GIC Dublin 2015


What the evidence synthesis process actually looks like


Over-emphasis on significant findings

Studies with significant findings are 2-3 times more likely to be published

Selective presentation of results


An example: the treatment of results from a single study of parent training (PT) versus multi-systemic training (MST) (a branded programme)

RCT assigning 43 abusive or neglectful families to either:• Parent training: group sessions discussing parenting

techniques• Multi-systemic therapy: individual family treatment tackling

multiple issues, e.g expectation re. child behaviour, child management, emotional support, parental behaviour change

Study looked at 30 outcomes on individual and child functioning, stress etc.


Outcome reporting and confirmation bias in action

9 out of 14 reviews report just one outcome from the paper, favouring MST



And how reviewers summarized the Brunk et al. paper…


No difference


What the systematic review says

Out of home placement: no differenceDelinquency: no difference

Family cohesion: no difference


Systematic reviews rebalance the evidence pyramid

Narrativereviews

More than 100 narrative reviews most stating MST is more effect than alternatives

MST is not consistently better or worse than alternatives


So drop narrative reviews in favour of systematic reviews to rebalance the evidence pyramid

More than 100 narrative reviews most stating MST is more effect than alternatives

MST is not consistently better or worse than alternatives


And in some areas, SRs have already made a difference

Crime and justice

1970s “Nothing works”Analysis of 231 studies:“With few exceptions, the rehabilitative efforts that have been reported so far have no appreciable effect on recidivism”Lipton, Martinson and Wilks ‘The Effectiveness of Correctional Treatment, 1975

Abandonment of rehabilitation in US and other countries (in prison at least get an incarceration effect)

But ….


The analysis was goal scoring:

and there is no mixed evidence, only poorly synthesized evidence. Meta-analysis gave clear results

though in fact 48% of studies found significant positive effects

showed ‘mixed evidence’ for all categories of intervention


Rehabilitation works

Review of 9 meta-analyses found ALL had positive average treatment effect

Source: Lipsey and Wilson (1993)

And prison no better than non- custodial sentences… or possibly worse

RCTs & 1 natural experiment

PSM

Source: Villletaz et al., 2013


Which means that…

• Prison is at best no better and possibly worse than non-custodial sentences

• Each additional year of prison increases recidivism by 3-4 per cent

Source: Petrosino et al. Scared Straight and Other Juvenile Awareness Programs for Preventing Juvenile Delinquency: A Systematic Review Campbell Systematic Reviews 2013:5

‘Shock approaches’ such as boot camps and scared straight are unsuccessful and even harmful


And this evidence is being used

“We must use sound, research-based, rehabilitation programmes for offenders so they do not re-offend.”


Similar story in policing

1997 Sherman et al. Preventing Crime: What works, what doesn’t, reviewing over 500 crime prevention programmes

UK £250 million ‘Crime reduction programme’

Source: Braga et al. Hot spots policing effects on crime. Campbell Systematic Reviews 2012:8

For example hotspot policing


Reviews can be used to answer both first generation questions (does it work) and second generation (design) questions.

Examples of looking at design questions


First generation: payment for environmental services

Tiny effect

0.3 % reduction in deforestationi.e. after 10 years 97% of land for which payments received would still have been forested in absence of the programme


What works by type of programme for teenage pregnancy

Source: Scher et al. Interventions Intended to Reduce Pregnancy-Related Outcomes Among Adolescents. Campbell Review 2006:12


Impact of welfare for work schemes by administration

• Source: Smedslund et al Work Programs for Welfare recipients. Campbell Review 2006:9

Employment

Earnings

Welfare payments

Welfare proportion

0.00 0.20 0.40 0.60 0.80 1.00 1.20

Bush Clinton Reagan 2nd Reagan 1st Carter Ford Johnson


Second generation: conditional cash transfers (CCTs)

• Mexico: Progressa launched 1996 (renamed Oportunidades)

• Brazil: 12 million families by 2010Cash payment on

conditions:• Education with 80%

attendance and maintaining certain grade

• Health: Ante-natal care, child immunization

Targeted both geographically and by means test

Design questions:• Do conditions

matter?• Timing, nature and

size of payment• Who to give it to


Meta-analysis allows us to get at design features, for example …CCTs have a larger effect on enrolment rates• Secondary than primary• The larger the transfer• The less frequent the transfer• If conditions include achievement not just attendance

And…


Conditionality works

Children 60% more likely to be in school with conditionality which is monitored and enforced compared to no conditions

But we need a lot of primary studies to exploit heterogeneity


Water, supply and sanitation interventions4 main types of intervention:

– Water supply improvement: source or point-of-use– Water quality: water treatment/protection at source or

point-of-use (households)– Sanitation: provision of facilities (improved latrines,

sewer connection)– Hygiene: soap, hygiene education

Usual outcome variable is child diarrhoea

The Campbell Collaboration www.campbellcollaboration.org www.3ieimpact.orgAuthor name44


45

Effectiveness results pooled (outcome = child diarrhoea)

NOTE: Weights are from random effects analysis

Water supply interventions

Subtotal

Water quality interventions

Subtotal

Sanitation interventions

Subtotal

Hygiene interventions

Subtotal

Multiple interventions

Subtotal

ID

Study

0.98 (0.89, 1.06)

0.58 (0.50, 0.67)

0.63 (0.43, 0.93)

0.69 (0.61, 0.77)

0.62 (0.46, 0.83)

ES (95% CI)

0.98 (0.89, 1.06)

0.58 (0.50, 0.67)

0.63 (0.43, 0.93)

0.69 (0.61, 0.77)

0.62 (0.46, 0.83)

ES (95% CI)

Ratio favours intervention 1.1 .5 .75 1 2

BUTEvidence largely from trials not actual projects

And hints of weak sustainability


Sustainability 1: less impact over longer periods

NOTE: Weights are from random effects analysis

Water supply (12 months or more)

Subtotal

Water quality (under 12 months)

Subtotal

Water quality (12 months or more)

Subtotal

Sanitation (12 months or more)

Subtotal

Hygiene (under 12 months)

Subtotal

Hygiene (12 months or more)

Subtotal

Multiple (under 12 months)

Subtotal

Multiple (12 months or more)

Subtotal

ID

Study

0.82 (0.71, 0.96)

0.56 (0.47, 0.66)

0.81 (0.67, 0.97)

0.64 (0.37, 1.10)

0.72 (0.60, 0.86)

0.67 (0.49, 0.91)

0.41 (0.23, 0.74)

0.77 (0.70, 0.85)

ES (95% CI)

0.82 (0.71, 0.96)

0.56 (0.47, 0.66)

0.81 (0.67, 0.97)

0.64 (0.37, 1.10)

0.72 (0.60, 0.86)

0.67 (0.49, 0.91)

0.41 (0.23, 0.74)

0.77 (0.70, 0.85)

ES (95% CI)

Ratio favours intervention 1.1 .5 .75 1 2


Sustainability 2: low compliance after a while

• Ceramic filter provision in Cambodia; 3 years later only 31% households were still using the filters (Brown et al, 2007)

• Pasteurisation in Kenya; 4 years later only 30% continued to pasteurise their water (Iijima et al, 2001)

• Programme promoting POU water disinfectant in Guatemala 1 year later; repeated use among only 5% of households from original trials (Luby et al, 2008).

• Water filters in Bolivia; compliance 67%; assessment made 4 months after trial ended (Clasen et al, 2006)

The Campbell Collaboration www.campbellcollaboration.org www.3ieimpact.orgAuthor name

Sustainability 3: lack of WTP

In Kenya, access to free chlorine increased uptake to over 60 percent, whereas coupons for even a 50 percent discount had a minimal effect


– Point of use water treatment has large health effect (community-level doesn’t)

– But challenge is to ensure sustained proper use

– Any water supply intervention not taking into account this demand element should be questioned

So the systematic review tells us that:


A quick word on the Campbell Collaboration• Coordinating groups (CGs) for

– Crime and Justice– Education– International development– Social welfare

• CGs manage editorial process– Three stage process: title, protocol, review– Any team can submit proposed title

• All published in Campbell Library, managed by Secretariat in Oslo


In summary

• Rigorous evidence matters• High quality systematic reviews sort out what is

rigorous and what is not• And synthesize the evidence in policy-relevant ways• - telling us what works and why

• Make them and use them!!

The Campbell Collaboration www.campbellcollaboration.orgThe Campbell Collaboration www.campbellcollaboration.org

Thank you

Visit our websitewww.campbellcollaboration.org

Follow us on Twitter @C2update & Facebook

Date post:	12-Apr-2017
Category:	Data & Analytics
Upload:	washington-evaluators
View:	439 times
Download:	1 times

The Importance of Systematic Reviews

Data & Analytics