© Nuffield Trust22 June 2015

Matched Control Studies:Methods and case studies

Cono Ariti

cono.ariti@nuffieldtrust.org.uk

© Nuffield Trust

Predictive risk modelling

Resource allocation

Descriptive studies Evaluations

Integrated care pilots

nuffield trust

Nuffield Trust Research team – data linkage projects

Risk sharing for CCGs

nuffield trust

Combined predictive model

nuffield trust

Person based resource allocation

nuffield trust

Social care at end of life

nuffield trust

Cancer and social care

nuffield trust

Predicting social care costs

nuffield trust

Virtual Wards

nuffield trust

WSD

nuffield trust

British Red Cross

nuffield trust

© Nuffield Trust

Need for evaluation

Need to know what works• In a practical setting – “real world evaluation”• Clarify the debate• Likely impacts – unbiased results• Link to qualitative work

Refine programs• Obtain feedback and learnings – the pain of

implementation• Explore sub-groups – where did it work? Where

could it work?

© Nuffield Trust

Issues with evaluations

Randomised control trials • “Gold standard”• May not be feasible or ethical• Inclusion and exclusion rules can limit generalisation• Are still subject to poor implementation – can induce

bias• Potentially expensive!

Observational studies• Typically no “natural” experiment exists• Often no comparable control group to provide a fair

assessment

© Nuffield Trust© Nuffield Trust

Matched Control Studies - Methods

© Nuffield Trust

Matched Control Studies

The basic idea

• Match controls to those treated based on measured characteristics in existing datasets

• The control group and treated group should look similar “on balance”

• Mimics the idea of an RCT

• Based on propensity score theory (Rubin & Rosenbaum, 1983) and earlier work on matching (Cochran, 1965)

© Nuffield Trust

Matched Control Studies

Matching• Prognostic risk score• Demographics – age, gender, deprivation, ethnicity• Prior acute care service use – admissions, OP and

A&E attendances• Prior diagnoses, targeted chronic conditions

Balance• In this case all matching variables• Additional variables such as length of stay, additional

diagnoses and longer service use history• Assures comparability between the groups

© Nuffield Trust

Matching Algorithm

Algorithm• Exact match not possible• Computer intensive “genetic algorithm”• Uses a weighted Mahalanobis “distance” to

determine closest match• Automatically assesses balance and moves to an

improved solution

Assessing Balance• On overall group similarity• Compares means and distribution of variables in the

two groups

© Nuffield Trust

Analysis of matched control studies

• Standard statistical methods to estimate the difference in the two groups

• Regression models, difference in difference analysis

• By including matching variables in the statistical adjustment remaining imbalances can be reduced – “doubly robust”

• Methods exist for sensitivity analysis – impact of unobserved variables

• Some controversy around accommodating the matching in analysis

© Nuffield Trust© Nuffield Trust

Case Study 1: Telehealth Programme

© Nuffield Trust

Case Study 1: Telehealth program

Intervention:

• Remote monitoring for patients with long term conditions

Nuffield commissioned to evaluate impact:• Primary: Reduction in emergency hospital admissions?• Secondary: Reductions in Emergency attendances, outpatient

attendances, mortality

Methods:• Retrospective matched control study – use of already existing

administrative data

© Nuffield Trust

Description: Telehealth program

© Nuffield Trust

Matched control studies – broad aim

>30,000 individuals – resident in local area June 2010 to March 2013, did not receive telehealth and were eligible for matching

(local controls)

Aim to find 716 individuals who match almost exactly on a broad range of characteristics

Use this group as study control group

716 individuals – enrolled June 2010 to March 2013 & received Telehealth intervention & eligible for matching

© Nuffield Trust

Datasets available

Telehealth Nuffield trust

N = 716

• person details• dates of

service • type of service

Identifiers:Names, DOB,Addresses, etc

• dates & place of death for all people in England,

• associated hospital (HES) records

Identifiers:Nuffield Trust specific HESID

Administrative data ONS deaths Hospital inpatient, outpatient, AE

© Nuffield Trust

1

2

3

4

5

6

7

8

9

. .

1

2

3

4

5

6

7

8

9

. .

1

2

3

4

5

6

7

8

9

. .

Telehealth Data Linkage Service Nuffield TrustNew Identifier New Identifier New Identifier

(NHS no) (NHS no)

Names Names

Address Address

DOB DOB

HESID HESID

Telehealth person identifiers (File A)

© Nuffield Trust

Final datasets available for analysis

Nuffield trust

Identifiers:

HESID on all

ONS deaths Hospital inpatient, outpatient, AETelehealth data - desensitised

Use all this info to carry out matched control analysis

© Nuffield Trust

Control group – how well matched?

© Nuffield Trust

Control group – how well matched?

Telehealth Controls

© Nuffield Trust

Control group – how well matched?

Telehealth Controls

© Nuffield Trust

Key Result 1: Risk of admissions or death

© Nuffield Trust

Key Result 2: Changes in admissions or attendances (six months pre and post intervention)

© Nuffield Trust

Results

• Telehealth patients tended to be admitted for an emergency admission earlier than control patients

• There was no difference in mortality between the telehealth and control groups

• There were no statistically significant reductions in hospital admissions when comparing the period six months before and after the telehealth intervention

• In summary the Telehealth program did not have a significant impact on acute care outcomes

• Sensitivity analysis showed little evidence of an important unobserved variable

© Nuffield Trust© Nuffield Trust

Matched Control Studies: Summary

© Nuffield Trust

Matched Controls: Summary

Benefits• Makes full use existing data, with relative ease• Techniques applicable to many different types of

services and datasets• Decisions on what seems to work (and what may

not) based on more robust analyses leading to better informed decisions

Caveats• If important unobserved variables exist results may

be biased• The routine data sources must contain the relevant

data

© Nuffield Trust

Implementing locally – key enablers

Do you have … Can you …

• Access to data that contains the outcomes relevant to your evaluation?

• Access to data containing relevant matching characteristics?

• Do you have consent to access/link the data?

• Analysis tools to apply statistical methods to the data?

• Skilled analysts to analyse the data?

• Link multiple sources of data?• Handle large amounts of data

(millions of observations)?• Identify recipients of the

intervention?• Transform and augment that data

with bespoke variables?• Apply sophisticated matching

algorithms routinely to this data?• Analyse the data with a variety of

statistical methods and interpret the results appropriately?

© Nuffield Trust

www.nuffieldtrust.org.uk

Sign-up for our newsletterwww.nuffieldtrust.org.uk/newsletter

Follow us on Twitter:Twitter.com/NuffieldTrust

© Nuffield Trust

Thank you