Challenges in evaluating social interventions: would an RCT design have been the answer to all our problems?

Lyndal Bond, Kathryn Skivington, Gerry McCartney, Hilary Thomson

York September 2009

Overview

• Importance of evaluating effects on health of social interventions

• Welfare to work – health benefits – what is known• Pathways Advisory Service• Evaluation challenges

• Design• Comparison group• Data sharing• Ethics & recruitment

• Conclusions & where to from here?

Evaluating the health effects of social interventions programme

Programme’s aim:• To ensure policy decisions are based on the most

reliable research evidence available

By undertaking: • systematic reviews, bringing together existing

research on the effectiveness of social/policy interventions

• new studies evaluating the health impacts of social/policy interventions

What do we study?

The effects on health and health inequalities of:

Housing & community

regeneration

Transport policies

Tobacco control legislation

Work organisation & welfare to work policies

Welfare related policies & programmes

Social/policy interventions affecting:

• employment & income• welfare policies• means of improving

the health of the worst off & reducing health inequalities.

Welfare-related policy interventions: evidence for effectiveness

• Lack of evidence for effects on health

• House of Commons Health Committee (2009) inquiry into health inequalities concluded that the lack of evidence was:

“in large part due to inadequate evaluation

of the policies adopted”

Welfare to Work programmes

• Aim to reduce number of people claiming benefits by moving them into employment

• May be regarded as important ‘healthy public policy’ interventions • potential to impact on health and health

inequalities

• A positive relationship between employment & health (Waddell, 2006)

• ‘work is good for health’

But…

• Potential for health benefits for welfare recipients moving into employment is likely to be dependent on• type & suitability of the job • individual-level factors – age, health etc

• Evidence for health benefits of returning to work for groups such as those in receipt of illness-related benefits is limited

• There is a need to evaluate such interventions for their effect (positive or negative) on health

The Pathways Advisory Service

• PAS places employment & benefit advisors into GP surgeries to engage with IB recipients

• Provides advice about benefits & return to work including • guarantee of benefits in 12 months if return to

work not successful• GPs refer patients to the service• Uptake is voluntary• Patients can self-refer

DWP evaluation of pilot PAS

• Department for Work and Pensions (DWP) had commissioned an evaluation of the PAS pilot programme• process, & employment & benefit

outcomes • no health outcomes (Sainsbury, 2008)

• PAS rolled out before pilot evaluation completed & rolled out service different to that which was piloted

Our planned evaluation of PAS

• Robust evaluation of the effects on health of PAS• Presumed through a return to work?

• Experimental or quasi-experimental design• Self-reported health• Health services use

Challenges

• Evaluability of the intervention

• Who is eligible for and receives the intervention?

• Identifying and recruiting: issues of transfer of contact details to obtain informed consent to the study

• Defining a comparison group

Evaluability of the intervention

Cluster RCT - design of choice• appropriate & justified given uncertainty about harms & benefits

with respect employment, welfare, income & health

But• before an evaluation of health impacts could be planned PAS was

being rolled out across the UK• so not possible to make the case for this design

• use of well conducted non-randomised study designs has been advocated as an appropriate alternative • we proposed a quasi-experimental study; a non-randomised

controlled trial.

Who is eligible & who receives the intervention?

• Service is available to all IB recipients• GPs use their discretion to refer patients to PAS• Uptake is voluntary and patients can self-refer

• About 20% of IB recipients in participating practices are estimated to engage with the PAS intervention

• It is likely that those who are referred have:• relatively good health • are closer to returning to the labour market

Identifying & recruiting

1. IB recipients from DWP records • not possible due to implementation of a data transfer

ban following high profile loss of sensitive personal information by a number of Government departments

2. Target population of IB recipients through GP electronic records• indicating a GP has completed a medical reference for

an IB claim• Limited by GP practices who use particular software• Estimated about 70% of IB patients would be so

recorded

Defining a comparison group

• Need to be similar with respect to eligibility for intervention • in receipt of IB• age, health status, length of time on benefits, employment

history etc • Could identify IB recipients through GP records, registered with GP

practices not offering PAS. BUT • Referral and access to PAS was up to GPs’ discretion & patient

self-selection• proportion of IB recipients engaging with the service is small • those engaging with PAS likely to have relatively good health &

be closer to a return to the labour market than those not engaging with PAS.

• Making it difficult to identify an appropriate & available comparison group

IB Recipients in PAS GP Surgeries

Group 2:

80%

Group1:

20%

Exposed to PAS and engage withPASExposed to PAS and do notengage with PAS

IB Recipients in non PAS GP Surgeries

Group 3:

100%

Not exposed to PAS, and sodo not engage with PAS

Defining a comparison group

IB Recipients in PAS GP Surgeries

Group 2:

80%

Group1:

20%

Exposed to PAS and engage withPASExposed to PAS and do notengage with PAS

Defining a comparison group

Need to rely on capacity to collect appropriate data to adjust for baseline differences between Gp 1 & Gp3

Proposed recruitment

Local Research Ethics Committee required:• written opt-in consent for contact details to be given to researchers• GPs could send out only 1 reminder letter to potential participants.

Estimated <10% response

GP writes to potential participants identified through medical records

`

Potential participants given chance to stop contact details being given to

researchers

GPs pass contact details to the researchers

Researchers write to potential participants with more details about

the study

Participants who give informed written consent are recruited to study

Interpretation of the Data Protection Act

Data Protection Act – not clear • ongoing discussion regarding what the Act means for data sharing

for health research • it does not necessitate individual consent for all medical research • but its wording does not provide clear guidance on when consent

by individuals is necessary for medical researchMedical Research Council Guidance states:• there must be an acceptable balance of risk & benefit for

participants. • justification for disclosure of a limited amount of personal

information given in confidence without consent, • if patients should be given the opportunity to raise objections,

then limited disclosure is unlikely to give rise to any serious objections.

Summary of barriers

• No control over method of implementationcompromising study design

• Ban on DWP data sharing• limited identification• transfer of contact details of potential participants• the identification of a comparison group

• Data Protection Act – interpretation• limited capacity to recruit

RESULT:• unethical to undertake study due to bias and lack of power

Ways forward

• Closer & timely working between policymakers & researchers• Allowing input at early stage for evaluation

design• & develop policy-relevant evaluations

“All too often Governments rush in with insufficient

thought, do not collect adequate data at the beginning about the health of the population which will be affected by the policies, do not have clear objectives, make numerous changes to the policies and its objectives and do not maintain the policy long enough to know whether it has worked. As a result, in the words of one witness, ‘we have wasted huge opportunities to learn’. Simple changes to the design of policies and how they are introduced could make all the difference.” (2009)

House of Commons

Health Committee

Ways forward (2)

• Response to loss of data by Government departments by DWP has been to develop data sharing protocols:

Data sharing protocols• no data sharing protocols between the DWP

and research organisations unless a ‘commissioned’ evaluator for DWP

• DWP not researcher driven research/evaluation

• Way forward???

Ways forward (3)

Need clarity of legal and ethical interpretation of the Data Protection Act

• Is it guidance? Is it legislation? • Who decides the balance between individual & collective

good?• Are the evaluations of ‘non-medical interventions’

perceived to be less important & therefore not worth even ‘low risk’ research?

• Is that acceptable given:• Expensive• Unexpected consequences• Particularly unfair for disadvantaged groups

Back to control group

• This remains a fundamental issue

• Resolution? • Establish a cohort study to enable nested

evaluations of policies • Expensive, large numbers, but could be worth

it• Other suggestions are welcome

Conclusion

• Without resolution robust experimental or quasi-experimental evaluations of these types of interventions are not possible

“… it is nearly impossible to know what to do given the scarcity of good evidence and good evaluation of current policy. Policy cannot be evidence-based if there is no evidence and evidence cannot be obtained without proper evaluation. The most damning criticisms of Government policies we have heard in this inquiry have not been of the policies themselves, but rather of the Government’s approach to designing and introducing new policies which make meaningful evaluation impossible..”

House of Commons

Health Committee