Hilary Pinnock1 , Eleni Epiphaniou2 , and Stephanie J. C. Taylor2

1. Allergy and Respiratory Research Group, Centre for Population Health Sciences, The University of Edinburgh, Edinburgh, United Kingdom;

2. Centre for Primary Care and Public Health, Blizard Institute, Barts and the London School of Medicine and Dentistry, London, United Kingdom

Phase IV Implementation StudiesThe Forgotten Finale to the Complex Intervention Methodology Framework

The complex intervention methodology?• A framework that defines the iterative process for

developing and evaluating complex interventions in healthcare

• A systematic review of implementation studies identified significant problems with reporting standardso Inconsistent terminologyo Missing / unclear information

• There is a need for standardised checklists for the reporting of implementation studies.

Reporting standards: possible scope

• Key standards might include:

o An explicit evidence base from an RCT or

guideline recommendation;

o recruitment to the clinical service (not the

research)

o Inclusion of (at least some) population-level

outcomes using routinely collected data

o A description of the setting and the process of

implementing the service.

Phase IV “cycle”: possible scope

• Currently – the framework:o Illustrates a cycle of development and evaluationo Implementation is the final step

• Proposition: the research underpinning implementation should be visualized as a second inter-related cycle.

• The “phase III cycle” includes the iterative steps:o Development, and o Piloting

A similar process may be needed to translate the intervention into a practical service that can be tested in a phase IV implementation study

Evidence for healthcare

• Healthcare systems globally need evidence on which to base their clinical, management, and policy decisions.

• Many countries have organisations which review evidence and make objective recommendations about the comparative effectiveness of medical technologies.1 The focus may:o Be drugs (e.g., Australia2),o Be medical devices and procedures (e.g., Canada3),o Include public health interventions (e.g., the UK4).

1. https://www.academyhealth.org; 2. http://www.pbs.gov.au/info/industry/listing/participants/pbac;3. http://www.cadth.ca/en/products/healthtechnology-assessment/health-technology-assessments4. http://www.nice.org.uk/

Evidence for healthcare

• Complex interventions are those: made up of various interconnecting parts.5

• Complex interventions are beyond the scope of clinical effectiveness programs in most countries.1

HOWEVER

• Most healthcare interactions are “complex,” and the context in which a treatment is prescribed may have important consequences on adherence or appropriate use of medication.6

5. Campbell M, et al. BMJ 2000;321:694–696.1. https://www.academyhealth.org; 6. Moullec G, et al. Respir Med 2012;106:1211–1225.

Evidence for healthcare• Randomized controlled trials (RCTs) are considered the

gold standard research design7,8 and this is reflected in the process for grading the strength of guidelines and comparative effectiveness recommendations.9–12

YET• More pragmatically-designed trials may reveal

different results to the phase III RCTs,13 even in the context of the (relatively) simple administration of a drug treatment.

• There is an even greater need to implement and evaluate effectiveness in routine clinical practice in the context of complex interventions7. Eccles M, et al. Qual Saf Health Care 2003;12:47–52;

8 Higgins JPT, et al (the Cochrane Collaboration, 2011): http://www.cochrane-handbook.org; 9 GRADE: http://www.gradeworkinggroup.org/index.htm; 10 SIGN 50, 2011: http://www.sign.ac.uk/guidelines/fulltext/50; 11 The AGREE Research Trust, 2009: http://www.agreetrust.org; 12 NICE 2012: http://publications.nice.org.uk/theguidelines-manual-pmg6#nice-guidance13 Price D, et al. N Engl J Med 2011;364:1695–1707

Designing, Evaluating, and Implementing Complex Interventions

• In 2000, the UK Medical Research Council described “a framework for the development and evaluation of RCTs for complex interventions”14 o Forward citations suggest widespread global

adoption o Only methodological guidance for healthcare

researchers developing and evaluating complex interventions.

• The final step was to: “establish the long-term and real-life effectiveness of the intervention,” but the description of phase IV studies was limited to the statement: “this stage is likely to involve an observational study”14

• A 2008 update aimed to address this limited guidance

14. Medical Research Council, 2000: http://www.mrc.ac.uk/Utilities/Documentrecord/index.htm?d=MRC003372

Designing, Evaluating, and Implementing Complex Interventions

• The framework describes a cycle in which an intervention is:o Developedo Pilotedo Efficacy

tested in a Phase III RCT.

The complex intervention framework illustrates a cycle of development and evaluation that includes implementation. It recommends RCT results be disseminated widely with further research to assist the process of implementation.15

This implementation phase might be visualized as a second interrelated cycle also including iterative steps of development and piloting, to help translate the intervention into a practical service that can be tested in a phase IV implementation study.

Context is crucial, so that the intervention may be implemented in different ways in different settings.16

15. Craig P, et al. Medical Research Council guidance; 2008: http://www.mrc.ac.uk/complexinterventionsguidance16. Hawe P, et al. BMJ 2004;328:1561–1563.

Designing, Evaluating, and Implementing Complex Interventions

• Phase III trials are classically delivered:o Under tightly controlled conditions, involvingo Carefully selected patientso Highly motivated participants o Rigid research protocols to avoid the influence of confounding

variables, although pragmatic.17

• Healthcare professionals ALSO need information on:o Population-level effectivenesso the impact on time, resourceso Training requirements, o Workforce implications

of implementing interventions in routine care.18,19

17. Eldridge S, Kerry S. A practical guide to cluster randomised trials in health research. Chichester: Wiley; 2012.18. Meredith LS, et al. Psychiatr Serv 2006;57:48–55.19 Glasgow RE, et al. Am J Public Health 2003;93:1261–1267.

Designing, Evaluating, and Implementing Complex Interventions

• Phase IV studies aim to:o Evaluate the implementation by healthcare services of

research findingso Translate interventions of proven efficacy in research

settings into routine care. o Compare the new procedure or service with the existing

regimen, o Explore whether the new service/procedure improves:

– Patient outcomes– Quality of care– Service delivery, and/or– Health and social well being of the population7 – i.e. Determine the intervention’s true population effect20

7. Eccles M, et al. Qual Saf Health Care 2003;12:47–52;20. Kelly JA, et al. J Acquir Immune Defic Syndr 2008;47:S28–S33–1267.

Designing, Evaluating, and Implementing Complex Interventions

• Effectiveness of an intervention is assessed in a heterogeneous, unselected population and real-life clinical settings, and examines outcomes relevant to the patient, provider, social, and healthcare contexts (20, 22). The focus is on external validity and generalizability, and implementation studies are thus useful for informing policy.20

• Even pragmatic trials with broad entry criteria recruit selected populations, and additional professional time and enthusiasm may be provided to deliver an intervention.

• Effects are likely to be attenuated when an intervention competes with the demands of busy clinical practice, or may vary depending on the healthcare context20. Kelly JA, et al. J Acquir Immune Defic Syndr 2008;47:S28–S3321. Glasgow RE, et al. Am J Public Health 2003;93:1261–126722. Narayan KMV, et al. Diabetes Care 2000;23:1794–1798.

Designing, Evaluating, and Implementing Complex Interventions

• In an RCT, telephone consultations enabled 26% more people to have a routine review of their asthma.23 In our subsequent implementation study, the effect size was halved, reflecting the challenge of providing care as part of the normal practice workload for a population of about 33,000 patients, 1,800 of whom had active asthma.24

• In an implementation study, a patient is offered the intervention as part of routine clinical care with none of the selection imposed by requiring informed consent. Uptake of the intervention and attrition are important outcomes that will affect the effectiveness of the new service.23. Pinnock H, et al. BMJ 2003;326:477–479.24. Pinnock H, et al. Br J Gen Pract 2007;57:714–722.

(Poor) Quality of Reporting of ISs

• Recently conducted systematic review of implementation studies encountered significant challenges related to design and reporting. Difficulty ino Difficulty in identifying terms for the search strategy

(due to a plethora of non-specific descriptors in use for implementation studies) – “effectiveness trials,” “routine clinical care,” “implement*,” “real-world,” “phase IV,” and “pragmatic.”

o Difficulty in ascertaining (from the full text) recruitment route – clinical service (as per implementation studies) or via research (as per RCT) Although study aims and objectives,

o Many areas of unclear/missing information on: implementation strategies; professional training required; population descriptions (numbers and characteristics)

Category Phase III Trials Phase IV Implementation

Rationale

A trial tests a novel intervention,based on an explicit theory andsupported by developmental work.

An implementation study should be based on phase III evidence and/or guideline recommendations.

Population

Trial-specific inclusion and exclusion criteria limit the eligible population, sometimes to a small proportion of those with the condition of interest

Inclusion and exclusion criteria refer to eligibility for the clinical service with no additional study-specific criteria. The study population is thus the whole population clinically eligible for the new service.

Recruitment

Recruitment is to the research, with informed consent being required for participation. Access to the novel intervention thus requires consent to trial procedures.

– Patients may be recruited for study purposes, but they should be considered as a subgroup and their characteristicscompared with the whole eligible population.

Randomization

Often at individual level, though may be at service level in cluster RCTs with patients subsequently recruited within the centers.

Normally at service level, althoughit may be appropriate torandomly divide a populationand offer a new service initiallyto part of the eligible population.

Examples of Phase III and Phase IV implementation study descriptions

Standards for designing implementationresearch: examples• The study should be based explicitly on phase III

evidence e.g., an RCT and/or guideline recommendations.

• Implementation studies adopting novel interventions, should be considered phase III evaluations.

• Eligibility and recruitment must be for the clinical service, not the research.

• Patients should not have to participate in the research to receive the intervention, and this should be reported clearly.

• The population of interest is the whole eligible population, which should be defined and the characteristics described.

• This population may not be static: for example, there was a 20% turnover in patients on the asthma register during the course of the year in our implementation study (24).

24. Pinnock H, et al. Br J Gen Pract 2007;57:714–722.

Standards for designing implementationresearch: examples (II)• Outcomes and, ideally, the primary outcome should be at

population level. Uptake of the intervention is a crucial outcome.

• A subgroup may be recruited (e.g., to complete questionnaires]), but they should be considered as a subgroup, and their characteristics compared with the whole population.

• A comprehensive description of the intervention should include the setting and details of staff training, as well as the clinical intervention. In multicenter studies, the implementation process may vary in detail from site to site,16 but crucial components should be standardized and described.

• It is helpful if authors describe the process of implementing the service and reflect on the barriers and facilitators to inform readers wishing to replicate a service in their practice.

16. Hawe P, et al. BMJ 2004;328:1561–1563.

Standardization: a must• Existing quality standards for observational studies and

pragmatic trials (examples below) have some relevance, but none are wholly applicable to phase IV implementation studies: o Consolidated Standards of Reporting Trials,o Consolidated Criteria for Reporting Qualitative Researcho Preferred Reporting Items for Systematic Reviews and Meta-

Analyses checklists.25–27

o Strengthening the Reporting of Observational Studies in Epidemiology28

• WHAT’S REQUIRED?A checklist for quality standards when reporting on

phase IV implementation studies that researchers can use to improve completeness and transparency of their reporting, and editors can apply to assess the quality of publication.29,30

25. Schulz KF, et al. BMC Med 2010;8:18; 26 Altman DG, et al. Lancet 2008;371:1149–1150; 27 Moher D, et al. PLoS Med 2009;6:e1000097; 28 von Elm E et al; STROBE Initiative. J Clin Epidemiol 2008;61:344–349; 29 Newhouse R, et al. Med Care 2013;51:S32–S40; 30 Rycroft-Malone J, et al. Worldviews Evid Based Nurs 2011;8:189–190.

Conclusions• The updating of the complex intervention

framework in 2008 signaled the recognition that phase IV implementation research is a crucial component of the evaluation of complex interventions in healthcare.

• This message now needs to be adopted by researchers, funders, and guideline developers. No evidence base on a complex intervention can be considered complete until implementation in real-world practice has been evaluated in a phase IV study