Date post: | 03-Jan-2016 |
Category: |
Documents |
Upload: | christopher-norton |
View: | 216 times |
Download: | 1 times |
1
Government Social Research Unit
www.gsr.gov.uk
Randomised Controlled Trials ConferenceUniversity of York, 14-15 September 2006
Why Governments Need RCTs
Philip Davies PhDGovernment Social Research Unit
HM TreasuryLondon SW1A 2HQ
2
www.gsr.gov.uk
Answer
Because RCTs can help get policy initiatives
right more often than wrong
3
www.gsr.gov.uk
Caveats
• Governments have got to want to get things right
• ‘Getting things right’ is more than achieving an effect size
• Governments do things other than achieving impacts
• RCTs are not a panacea
• RCTs are only as good as their design and implementation
4
www.gsr.gov.uk
Governments Have Got to Want to Get Things Right
“Social science should be at the heart of policy making.
We need a revolution in the relationship between
government and the social research community - we
need social scientists to help determine what works and
why, and what type of policy initiatives are likely to be
most effective …”
(David Blunkett, 2000)
5
www.gsr.gov.uk
Governments Have Got to Want to Get Things Right
Government policy must be:
• Evidence-based
• Properly evaluated
• Based on best practice
6
Governments Have Got to Want to Get Things Right
• Policy making must be soundly based on evidence of what works
• Improve departments’ capacity to make best use of evidence
• Improve the accessibility of the evidence available to policy makers
7
Governments Have Got to Want to Get Things Right
Professional Skills for Government (2005)Analysis and Use of Evidence
• Anticipate and secure appropriate evidence• Test for deliverability of policy/practice – and evaluate• Use evidence to challenge decision making• Identify ways to improve policy/practice• Champion a variety of tools for collecting/using evidence• Ensure use of evidence is consistent with wider government
requirements• Work in partnership with a wide range of experts/analysts
8
What ‘Getting It Right’ Means
“…a society that would use social science methods and
evaluation techniques to vigorously try out possible
solutions to recurrent problems and would make hard-
headed, multidimensional evaluations of outcomes, and
when the evaluation of one reform showed it to have
been ineffective or harmful, would move on and try
other alternatives”
The ‘Experimenting Society’
(Donald T. Campbell)
9
What ‘Getting It Right’ Means
“A social experiment benefits society by providing better
information on which to base public policy. Such
information can improve policy in one of two ways: it
can lead policymakers to adopt a program or policy that
is found to have net social benefits, or it can lead to the
termination of an existing program that is found to have a
net social cost”
(Orr, L., 1999).
10
Why RCTs?(Recap)
Randomised Controlled Trials:
• Provide most precise estimates of the likely effects of policy interventions.
• Against a strong counterfactual – i.e. the net effect/impact• Reduce/account for various biases.• Establish cause and effect of policy interventions and
outcomes.• Raise the quality of evidence to support policy making.
11
So, Why Don’t Governments Use RCTS More Often?
• RCTs not on most policy makers’ radar
• RCTs not on many government researchers’ radar
• RCTS not on many academic researchers’ radar
• For some (many?) RCTs are anathema: Don’t like ‘random’ (too ‘chancey’)
Don’t like ‘control’ (no ‘wiggle room’)
Don’t like ‘trials’ (sounds inhumane / 1940s Germany)
• There are things other than ‘impact’
• There are things other than ‘evidence’
12
Things Other Than Impact Evidence
Research Evidence
Impact Evidence
Implementation Evidence
Descriptive Analytical Evidence
Economic and
Econometric Evidence
Ethical Evidence
ExperimentalQuasi-Experimental
SurveysAdmin DataComparativeQualitative
Cost-BenefitCost-Effectiveness
Cost-Utility
Experimental Qualitative
Theories of Change
Social EthicsPublic Consultation
Statistical Modelling
Multivariate Analysis
13
Things Other Than Evidence
“Research of any kind is seldom the
determining factor in shaping public
policy. Experimentation is no exception
to this rule” (Orr, L., 1999)
14
Things Other Than Evidence
Evidence
Experience & Expertise
Judgement
Resources
Values and Policy
Context
Habits & Tradition
Lobbyists & Pressure Groups
Pragmatics & Contingencies
15
Policy Makers’
Evidence
• Colloquial (Contextual)
• Anything that seems reasonable
• Policy relevant
• Timely
• Clear Message
Researchers’ Evidence
• ‘Scientific’ (Context free)
• Proven empirically
• Theoretically driven
• As long as it takes
• Caveats and qualifications
Different Notions of Evidence
16
Academics
Constituents, Consumers and Users
Media
Professional Associations
Lobbyists and Pressure Groups
Think Tanks/Opinion Formers
'Experts'
Special Advisers
Policy or Practice
Where Do Policy Makers Go For Evidence?
17
But Not All Is Lost
• Employment Demonstration Project (ERA)• Job Retraining and Rehabilitation Project (JRRP)• Restart evaluation (1990)• BA Visiting Officer Pilot• Evaluation of Employment Zones• New Deal 25+ Pilots• Evaluation of the New Deal for Young People Intensive
Gateway • Evaluation of the Restorative Justice programme
Examples of Recent UK Government RCTs
18
RCTs Are Only As Good As Their Design and Implementation
Common Design and Implementation Faults:
• Unclear policy objectives, populations, outcomes• Insufficient time (lack of strategic policy making)• Test condition often pre-determined• Sample sizes too small – no statistical power• Problems of randomisation, concealment, cross-over• High attrition (especially in control groups)• Intention to Treat analysis not used• Limited capacity (in the UK) to do high quality RCTs• Poor counterfactual too often the result
19
RCTs Are Only As Good As Their Design and Implementation
These faults can usually be overcome!
• Better strategic policy making• Better planning and protocol development• Involvement of researchers, policy makers, front-line staff• High quality piloting / pre-fieldwork testing• High quality research/project management• Careful real time data management• Use of experienced technical experts throughout• Policy Trials Service• Professional development of researchers and policy makers
20
Take Home Message
• RCTs have enormous potential for governments
• RCTs can help governments get things right more often
• RCTs are currently used considerably to UK Govnt.
• But there are still non-believers
• And those who remain to be convinced
• RCTs, like other types of research, must compete with other notions of evidence
• And with factors other than evidence
• There is a need to improve the capacity to do high quality RCTs in the UK
21
Contact
[email protected] Social Research UnitHM Treasury1 Horse Guards RoadLondon SW1A HQEngland
Tel: +44 (0)20 7270 5156
www.gsr.gov.uk