Developing guidance on the Developing guidance on the conduct of narrative synthesis conduct of narrative synthesis in systematic reviews in systematic reviews An ESRC Research Methods Programme project An ESRC Research Methods Programme project Jennie Popay, Helen Roberts, Amanda Jennie Popay, Helen Roberts, Amanda Sowden Sowden , Mark , Mark Petticrew Petticrew , Lisa Arai, Mark Rogers, Nicky Britten , Lisa Arai, Mark Rogers, Nicky Britten With Katrina With Katrina Roen Roen , Sally Baldwin* and Steven Duffy. , Sally Baldwin* and Steven Duffy. * * Died 28/10/2003 Died 28/10/2003
Transcript
Developing guidance on the Developing guidance on the conduct of narrative synthesis conduct of narrative synthesis
in systematic reviewsin systematic reviews
An ESRC Research Methods Programme projectAn ESRC Research Methods Programme project
Jennie Popay, Helen Roberts, Amanda Jennie Popay, Helen Roberts, Amanda SowdenSowden, Mark , Mark PetticrewPetticrew, Lisa Arai, Mark Rogers, Nicky Britten, Lisa Arai, Mark Rogers, Nicky Britten
With Katrina With Katrina RoenRoen, Sally Baldwin* and Steven Duffy. , Sally Baldwin* and Steven Duffy.
** Died 28/10/2003Died 28/10/2003
Structure of the presentationStructure of the presentation•• What is narrative synthesis (NS)What is narrative synthesis (NS)•• Why do we need guidance?Why do we need guidance?•• The guidance; methods and structureThe guidance; methods and structure•• Applying the guidance: overviewApplying the guidance: overview•• Demonstration review of evidence on Demonstration review of evidence on
the implementation of smoke alarm the implementation of smoke alarm programmesprogrammes
Structure of the presentationStructure of the presentation•• What is narrative synthesis (NS)What is narrative synthesis (NS)•• Why do we need guidance?Why do we need guidance?•• The development of the guidanceThe development of the guidance•• Application of the guidance: overviewApplication of the guidance: overview•• Demonstration review of data on the Demonstration review of data on the
implementation of smoke alarm implementation of smoke alarm programmesprogrammes
What is Narrative Synthesis ?What is Narrative Synthesis ?•• Variants on NS widely used in evidence synthesis and Variants on NS widely used in evidence synthesis and
recommended in some guidelines on the conduct of recommended in some guidelines on the conduct of systematic reviews of effectiveness but:systematic reviews of effectiveness but:–– no consensus on what no consensus on what ‘‘itit’’ is or how trustworthiness is established.is or how trustworthiness is established.–– And a systematic and transparent approach to narrative synthesisAnd a systematic and transparent approach to narrative synthesis is is
frequently absent.frequently absent.
•• So how do we define it? So how do we define it?
““An approach to the synthesis of findings from multiple An approach to the synthesis of findings from multiple studies the defining characteristic of which is that it relies studies the defining characteristic of which is that it relies primarily on the use of words and text to summarise and primarily on the use of words and text to summarise and explain explain –– to to ‘‘tell the storytell the story’’ -- of the findings of multiple of the findings of multiple studies. NS can involve the manipulation of statistical data studies. NS can involve the manipulation of statistical data and it can be used to synthesis evidence relevant to a wide and it can be used to synthesis evidence relevant to a wide range of questions including but not restricted to those range of questions including but not restricted to those concerned with effectivenessconcerned with effectiveness””. .
Structure of the presentationStructure of the presentation•• What is narrative synthesis (NS)What is narrative synthesis (NS)•• Why do we need guidance?Why do we need guidance?•• The development of the guidanceThe development of the guidance•• Application of the guidance: overviewApplication of the guidance: overview•• Demonstration review of data on the Demonstration review of data on the
implementation of smoke alarm implementation of smoke alarm programmesprogrammes
•• Increasing demand for trustworthy Increasing demand for trustworthy evidence from research to inform decisions; evidence from research to inform decisions;
•• Many syntheses involve diverse types of Many syntheses involve diverse types of evidence evidence –– from research and potentially from research and potentially from other sources. from other sources.
•• Making sense of large diverse bodies of Making sense of large diverse bodies of research is challenging.research is challenging.
•• Ensuring a synthesis product can be trusted Ensuring a synthesis product can be trusted is important for people who use it. is important for people who use it.
•• Specialist synthesis methods for Specialist synthesis methods for methodologically uniform research are methodologically uniform research are relatively well developed.relatively well developed.
Why do we need guidance?Why do we need guidance?
Why do we need guidance?Why do we need guidance?
•• NS does not rest on authoritative body of knowledge or NS does not rest on authoritative body of knowledge or reliable and rigorous techniques tested over many years. reliable and rigorous techniques tested over many years.
•• As a result: As a result: –– There is great variability in the practice of NSThere is great variability in the practice of NS–– The methods used in reviews adopting a NS approach are not The methods used in reviews adopting a NS approach are not
transparent transparent –– Reproducibility of the products of these reviews is an Reproducibility of the products of these reviews is an
important issueimportant issue
•• Guidance would help to address some of these Guidance would help to address some of these problems but there are currently no examples of such problems but there are currently no examples of such guidance in the methodological literatureguidance in the methodological literature
What is the guidance about?What is the guidance about?•• Advice on the conduct of NS in the context of Advice on the conduct of NS in the context of
systematic reviews of research evidencesystematic reviews of research evidence
•• Descriptions of specific tools & techniques that Descriptions of specific tools & techniques that can be used in the synthesiscan be used in the synthesis
•• The product The product –– at a minimum at a minimum –– is a summary of is a summary of the current state of knowledge in relation to a the current state of knowledge in relation to a specific review question specific review question
•• It focuses in particular on systematic reviews of It focuses in particular on systematic reviews of evidence on effectiveness and implementationevidence on effectiveness and implementation
Who is it aimed at?Who is it aimed at?
•• Aims to be accessible to a wide range of peopleAims to be accessible to a wide range of people
•• Users wonUsers won’’t need to be expert reviewerst need to be expert reviewers
•• They will need to have a reasonable level of They will need to have a reasonable level of research experience research experience –– not novices!not novices!
WHY?WHY?
The nature of evidence synthesisThe nature of evidence synthesis
•• At its simplest At its simplest involves juxtaposition involves juxtaposition of findings from of findings from multiple studies multiple studies perhaps with some perhaps with some analysis of common analysis of common themes or findings themes or findings across studiesacross studies
•• More sophisticated More sophisticated approaches involve approaches involve integration or integration or interpretation of interpretation of results from multiple results from multiple studies with aim of studies with aim of producing new producing new knowledge. knowledge.
NS guidance will always involve juxtaposition and may involve integration/interpretation depending on studies
Therefore require research competency to be done well
Structure of the presentationStructure of the presentation•• What is narrative synthesis (NS)What is narrative synthesis (NS)•• Why do we need guidance?Why do we need guidance?•• The development of the guidanceThe development of the guidance•• Application of the guidance: overview Application of the guidance: overview •• Demonstration review of data on the Demonstration review of data on the
implementation of smoke alarm implementation of smoke alarm programmesprogrammes
Where does the NS process Where does the NS process ‘‘fitfit’’??
Main elements in a systematic review: Main elements in a systematic review: •• Focus of review, searching/mapping Focus of review, searching/mapping
evidenceevidence•• Specify the review questionSpecify the review question•• Select studies to include in reviewSelect studies to include in review•• Extract data and assess study qualityExtract data and assess study quality•• Synthesise findings from included studiesSynthesise findings from included studies•• Report the results of review and Report the results of review and
disseminatedisseminate
How we developed the guidanceHow we developed the guidance
•• Multidisciplinary team (health/social care, Multidisciplinary team (health/social care, quantitative/qualitative research)quantitative/qualitative research)
•• Use of expert review panelUse of expert review panel
•• Began with a review of methodological Began with a review of methodological literature literature
•• The search of methodological literature The search of methodological literature involved three elements: involved three elements: –– Traditional database searchTraditional database search–– Search of internet sitesSearch of internet sites–– Knowledge of team membersKnowledge of team members
•• Process generated 1,309 items after reading Process generated 1,309 items after reading and review of abstracts and full texts included and review of abstracts and full texts included 69 items judged to have some relevance. 69 items judged to have some relevance.
How we developed the guidanceHow we developed the guidance
Identified by team
N = 54
Database search
N = 1,145
Internet searchN =110
Titles & abstracts
Titles & abstracts
Titles & abstracts
IncludeN=54
IncludeN=121
IncludeN=89
ExcludeN = 0
ExcludeN = 1,024
ExcludeN = 21
Full articles
Full articles
Full articles
IncludeN=13
ExcludeN = 41
IncludeN=23
ExcludeN = 98
IncludeN=25
ExcludeN = 64
Total included texts, n = 69
HandsearchN=8
The literatureThe literature
–– None of the included text specifically related to None of the included text specifically related to narrative synthesis narrative synthesis
–– Some of the included texts described established Some of the included texts described established methodologies e.g. metamethodologies e.g. meta--analysis; metaanalysis; meta--ethnography, ethnography, ‘‘case surveycase survey’’ method etc.method etc.
–– Some aspects of these approaches are relevant to the Some aspects of these approaches are relevant to the conduct of a narrative synthesisconduct of a narrative synthesis
–– Other Other texts give general texts give general ‘‘tipstips’’ on structuring results, on structuring results, presenting data etc.presenting data etc.
•• Prescriptive guidance Prescriptive guidance –– a cook book a cook book -- on the on the conduct of NS is not feasibleconduct of NS is not feasible
•• The specific approach in a narrative synthesis The specific approach in a narrative synthesis will depend on: will depend on: –– Types of research included in the review (e.g. Types of research included in the review (e.g.
only only quanquan; only ; only qualqual; mixed); mixed)–– Study characteristics (design, participants, Study characteristics (design, participants,
outcomes etc.)outcomes etc.)
We concluded that:We concluded that:
•• Stage 1Stage 1: 2 parallel processes reading the literature to: : 2 parallel processes reading the literature to: •• Inform design of a framework for the conduct of NSInform design of a framework for the conduct of NS•• Identify tools and techniques relevant to the NS process Identify tools and techniques relevant to the NS process
•• Stage 2Stage 2: developed draft guidance including a proposed : developed draft guidance including a proposed framework for NS and examples of the use of tools and framework for NS and examples of the use of tools and techniques at different points in the synthesis processtechniques at different points in the synthesis process
•• Stage 3Stage 3: Applied the guidance to two : Applied the guidance to two ‘‘demonstrationdemonstration’’reviews reviews –– one on effectiveness the other one on effectiveness the other implementation implementation -- in order to provide practical examples in order to provide practical examples of the use of the guidance. of the use of the guidance.
•• Stage 4Stage 4: revised the guidance on the basis of comments : revised the guidance on the basis of comments from expert reviews and experience of from expert reviews and experience of ‘‘demonstration demonstration reviewsreviews’’. .
How we developed the guidanceHow we developed the guidance
The general framework for a NSThe general framework for a NS
•• Developing a theory of changeDeveloping a theory of change
•• Developing a preliminary synthesisDeveloping a preliminary synthesis
•• Exploring relationships in the dataExploring relationships in the data
•• Assessing the robustness of the synthesis productAssessing the robustness of the synthesis product
Not stages Not stages –– NS an iterative process!NS an iterative process!
The NS frameworkThe NS frameworkTheory of changeTheory of change: : •• PurposePurpose: to inform : to inform
decisions about the decisions about the review question and review question and what types of what types of studies to includestudies to include
•• To contribute to To contribute to interpretation of interpretation of findingsfindings
•• To assess To assess applicability of the applicability of the findingsfindings
•• Theory building and Theory building and theory testing is a theory testing is a neglected aspect of neglected aspect of systematic reviewssystematic reviews
•• Systematic reviews Systematic reviews have focused too have focused too much on description much on description and too little on and too little on explanationexplanation
•• SR could be more SR could be more powerful than single powerful than single studies re theorystudies re theory
The NS FrameworkThe NS FrameworkThe preliminary synthesisThe preliminary synthesis::•• PurposePurpose: To organise : To organise
findings from included findings from included studies to provide an studies to provide an initial description of initial description of patterns across the patterns across the studies.studies.
•• In effectiveness reviewsIn effectiveness reviewsthe preliminary synthesis the preliminary synthesis would focus on:would focus on:–– Direction of effectsDirection of effects–– Size of effectsSize of effects
•• In implementation In implementation reviewsreviews preliminary preliminary synthesis would focus synthesis would focus on: on: –– Identifying and listing Identifying and listing
facilitators and barriers to facilitators and barriers to implementationimplementation
–– Exploring the relationship Exploring the relationship between thesebetween these
The NS FrameworkThe NS FrameworkExploring relationships Exploring relationships
•• Purpose inPurpose ineffectiveness reviewseffectiveness reviewswould be to consider would be to consider the factors that might the factors that might explain any explain any differences in the differences in the direction and/or size direction and/or size of effects across of effects across included studiesincluded studies
•• Purpose in Purpose in implementation implementation reviewsreviews would be towould be to–– consider factors that consider factors that
might explain any might explain any differences in report differences in report facilitators and/or facilitators and/or barriers andbarriers and
–– To understand why To understand why and how interventions and how interventions have an effecthave an effect
The NS FrameworkThe NS FrameworkAssessing robustnessAssessing robustness•• PurposePurpose: to provide : to provide
an assessment of the an assessment of the strength of evidence strength of evidence forfor–– drawing conclusions drawing conclusions
about size and about size and direction of effect direction of effect
–– or factors shaping or factors shaping implementationimplementation
–– and and generalisabilitygeneralisability of of synthesis product synthesis product
quality of included quality of included studiesstudies
–– Judgements about Judgements about the relevance of the relevance of study findings for study findings for the reviewthe review
–– Trustworthiness of Trustworthiness of the product of the product of synthesissynthesis
Structure of the presentationStructure of the presentation•• What is narrative synthesis (NS)What is narrative synthesis (NS)•• Why do we need guidance?Why do we need guidance?•• The development of the guidanceThe development of the guidance•• Application of the guidance: Application of the guidance:
OverviewOverview•• Demonstration review of data on Demonstration review of data on
the implementation of smoke the implementation of smoke alarm programmesalarm programmes
Demonstration reviewsDemonstration reviews•• Why did we do this? Why did we do this?
–– To have practical experience of the guidance to To have practical experience of the guidance to improve itimprove it
–– To have practical examples of decisions about which To have practical examples of decisions about which tools/techniques to use and why to include in the tools/techniques to use and why to include in the guidanceguidance
•• What did we do?What did we do?–– A blinded replication of a Cochrane systematic review A blinded replication of a Cochrane systematic review
of the effectiveness of interventions aiming to of the effectiveness of interventions aiming to increase use of domestic smoke alarms increase use of domestic smoke alarms
–– A review of evidence on the implementation of A review of evidence on the implementation of interventions aiming to increase use of smoke interventions aiming to increase use of smoke alarms. alarms.
The NS of effectiveness dataThe NS of effectiveness dataIn brief: In brief: •• Reached broadly similar conclusions to the Reached broadly similar conclusions to the
Cochrane systematic review it was Cochrane systematic review it was ‘‘replicatingreplicating’’
•• SubSub--group analysis in Cochrane review offered group analysis in Cochrane review offered additional insights into additional insights into ‘‘moderator effectsmoderator effects’’ --aspects of interventions impacting on aspects of interventions impacting on effectiveness effectiveness -- not identified in the NSnot identified in the NS
•• Close scrutiny of studies in NS lead to more Close scrutiny of studies in NS lead to more extensive & detailed insight into implications extensive & detailed insight into implications for future research than Cochrane reviewfor future research than Cochrane review
The example of data on the The example of data on the implementation of smoke implementation of smoke
alarm programmesalarm programmes
•• In earlier work on the synthesis of In earlier work on the synthesis of implementation data, we identified implementation data, we identified reports of domestic smoke alarm reports of domestic smoke alarm promotion interventions. promotion interventions.
•• For that work, the primary studies were For that work, the primary studies were taken from the systematic review taken from the systematic review conducted by conducted by DiGuiseppiDiGuiseppi and Higginsand Higgins
Application of the guidance: Application of the guidance: implementation of smoke alarm implementation of smoke alarm programmesprogrammes
•• We updated this literature search, and We updated this literature search, and drew on data on implementation from drew on data on implementation from seven reports of interventionsseven reports of interventions–– Three of these were from same Three of these were from same
programme in Australiaprogramme in Australia–– Two from two programmes in USTwo from two programmes in US–– Two from London programmeTwo from London programme
•• These reports all contained relatively These reports all contained relatively ‘‘thickthick’’ data on implementation data on implementation
Application of the guidance: Application of the guidance: implementation of smoke alarm implementation of smoke alarm programmesprogrammes
•• Team members each worked Team members each worked through the guidance using these through the guidance using these data on implementationdata on implementation
•• This work is almost completeThis work is almost complete•• Final draft of the guidance will be Final draft of the guidance will be
available end of Octoberavailable end of October
Application of the guidance: Application of the guidance: implementation of smoke alarm implementation of smoke alarm programmesprogrammes
Preliminary synthesis: textual Preliminary synthesis: textual descriptiondescription•• The The target populationtarget population in this study was Arabicin this study was Arabic--, ,
ChineseChinese-- and Vietnameseand Vietnamese--speaking communities in speaking communities in Australia. The Australia. The interventionintervention consisted of the following consisted of the following components: selling cheap smoke alarms; information, components: selling cheap smoke alarms; information, demonstrations and installation instructions. The demonstrations and installation instructions. The outcomeoutcome measure was installation rates amongst those measure was installation rates amongst those who had bought smoke alarms, after 10who had bought smoke alarms, after 10--15 days. The 15 days. The methods of methods of evaluationevaluation were: questionnaires and focus were: questionnaires and focus groups to assess barriers to purchasing and installing groups to assess barriers to purchasing and installing smoke alarms. The smoke alarms. The resultsresults showed installation rates of showed installation rates of 65% (Vietnamese), 52% (Arabic) and 35% (Chinese) 65% (Vietnamese), 52% (Arabic) and 35% (Chinese) after 10after 10--15 days; and 50% (Chinese). The 15 days; and 50% (Chinese). The questionnaires identified 2 barriers (lack of awareness of questionnaires identified 2 barriers (lack of awareness of the need for smoke alarms and unsympathetic the need for smoke alarms and unsympathetic landlords). The focus groups identified 12 barriers which landlords). The focus groups identified 12 barriers which the authors grouped under 3 headings (lack of the authors grouped under 3 headings (lack of awareness; rented accommodation; and installation) awareness; rented accommodation; and installation) (Young et al 1999). (Young et al 1999).
Preliminary synthesis: Preliminary synthesis: tabulationtabulationAuthor & year Location/setting Target
population Method Main findings
Roberts et al (2004)
London, UK Urban
All residents on estate (n=x)
Interviews Problems with smoke alarms (sensitivity, false alarming) identified as major barriers to implementation
Camit et al, Young et al Camit
NSW, Australia Mixed (?)
Chinese, Vietnamese, Arabic-speaking (n=x)
Focus groups Implementation successful using multi-faceted, language appropriate approach.
Campbell De Long Resources
Oregon, USA Mixed (?)
All residents, but focus on Latino-speaking (n=x)
Interviews Successful implementation heavily dependent on landlords’ attitudes.
Smoke alarm Theme Evidence The design of smoke alarms has serious implications for their use.
• False alarms mentioned by number of respondents (UK, USA).
• Not all respondents understand how to disable alarm (UK).
Individual Theme Evidence Perceived invulnerability to fire • Some respondents mentioned that
people may think ‘It’s never going to happen to me’ (USA).
• Fire not seen as a serious risk (NSW). Lack of awareness of the importance of smoke alarms/general lack of fire safety awareness.
• Fire safety educational materials need to be tailored for targeted population (USA).
Cultural factors have effect on smoke alarm use/functioning
• Recent migrants do not understand need for smoke alarms (USA, NSW).
• Some groups (e.g. Chinese) fry oily food and this can set off alarms (UK).
Age/poverty can affect smoke alarm installation
• Some respondents considered themselves too ‘old’ to be worth spending money on (UK).
• Free/reduced price smoke alarms can be incentive (NSW).
Community Theme Evidence Landlords play a role in promoting/hindering good fire safety awareness.
• Landlords need to set an example of good fire safety awareness (USA).
• Landlords may not give permission for installation of smoke alarms (NSW).
1) Barriers/levers to acquisition of smoke alarms General
Barriers Levers Problems accessing communities/gatekeepers Gaining trust of key community ‘players’ and leaders
Suspicion of ‘authority’ or local government Emphasising separation from distrusted authority/alliance with trusted partners
Specific to smoke alarm campaigns Lack of awareness of benefits of smoke alarms Running well-coordinated, culturally appropriate awareness
campaign Perceived cost of smoke alarms Giveaway or availability of reduced price alarms
Perception that household is not at risk of fire (due to type of house or characteristics of household members)
Awareness campaign
2) Barriers/levers to installation of smoke alarms General
Barriers Levers Anxiety about damage to property Landlord approval/permission for installation, or landlord
example of installation Specific to smoke alarm campaigns Inability/unwillingness to install alarm Installation of alarm by project worker
3) Barriers/levers to continued use of smoke alarms Barriers Levers False alarms Education about triggers for false alarms/re-installation of alarm Problems with maintenance Project workers offer to maintain alarms/education about maintenance
Next stepsNext steps•• We need to revise guidance on We need to revise guidance on
basis of the testing exercise and basis of the testing exercise and expert consultationexpert consultation
•• Exploring idea of making the Exploring idea of making the guidance a webguidance a web--based resourcebased resource
•• Workshops, conferences etcWorkshops, conferences etc•• Encourage people to use itEncourage people to use it
Thanks for listening. For more information, go Thanks for listening. For more information, go to: to: http://www.city.ac.uk/chrpu/projects/narrativhttp://www.city.ac.uk/chrpu/projects/narrativesynthesis.htmlesynthesis.html
