Issues and Challenges with synthesis of qualitative evidence
Professor Jane NoyesDirector, Centre for Health-Related Research
Bangor UniversityConvenor Cochrane Qualitative Research Methods Group
Editor Journal of Advanced Nursing
1. Protocol - How and when to select a synthesis approach?
3. Use of conceptual/theoretical frameworks to inform synthesis
4. Organisation and management of data for synthesis
5. What challenges are people reporting with specific approaches?
6. What methodological work still needs to be done?
Some issues to consider
Protocol - How and when to select a synthesis approach?
Upfront in protocol?
Once pool of evidence is known?
Decision to conduct a qualitative evidence synthesis
Purpose of the additional qualitative synthesis
Primarily to integrate and interpret qualitative and quantitative evidence within a single approach or integrated model
Can be used to develop explanatory theory
To interpret synthesised qualitative evidence and develop explanatory theory or models
To aggregate/summarise/ integrate qualitative data to address specific questions in relation to a Cochrane intervention review
· Realist review· EPPI approach· Narrative synthesis· Bayesian synthesis· Critical interpretive synthesis
· Meta-ethnography· Thematic analysis with
theory generation· Grounded theory
· Thematic analysis without theory generation
· Meta-aggregation· Meta-summary
Product:· Explanatory theory, analytical or
conceptual framework or interpretative framework/mechanism
Product:· Aggregated findings from
source papers
How best to organise and manage qualitative data ?
Example of two WHO/Cochrane reviews
WHO Alliance commissioned reviews
Evidence Synthesis on Strategies for Implementing Interventions for Achieving
MDGs 4, 5 or 6
2010- 2011
Implementation Research Platform
Our reviews also linked with GREAT Initiative
•Guideline development
•Research priorities
•Evidence synthesis
•Applicability of evidence
•Transfer of knowledge
Primary objective of our workPrimary objective of our work
To Optimize the delivery of key interventions to attain MDGs 4 & 5 (Optimize4MNH) through task-shifting
To identify effective practices that can be delivered by a lower cadre of health worker that would increase access to effective interventions in a safe way
To identify and make recommendations on cadre(s) of workers who can increase access to effective practices in a safe way
WHO Alliance commissioned reviews - linked to existing Cochrane effectiveness reviews
1.Task-shifting to improve access to health care providers for MCH: Systematic review of qualitative
research on barriers and facilitators to the implementation of lay health worker programmes2. Doctor/Nurse substitution to improve access to
health care providers for MCH: Systematic review of qualitative research on barriers and facilitators to
the implementation of nurse-led care
Experienced multi-skilled team (plus research officers) Dr Claire Glenton- Knowledge Scientist (NKC) – Cochrane EPOC Dr Simon Lewin- medic and SR - NKC, EPOC and South African
Cochrane Centre – EPOC Dr Arash Rashidian – GP and mixed method HSR researcher (BU,
Tehran University, Iran) Dr Chris Colvin - PhD cultural anthropology MPH in epidemiology and
biostatistics. – South Africa Dr Metin Gülmezoglu
WHO/World Bank Special Programme of Research, Development and Research Training in Human Reproduction – Who Geneva/Cochrane.
Prof Jane Noyes – health services researcher – Cochrane qualitative research methods group BU and Cochrane.
Online virtual global community of practice (1500 members) Regular face to face meetings with WHO experts (x 2 days in May
2011)
Organising and managing evidence for synthesis Early decision in protocol development –
lump or split question at question development stage (PICO or SPICE)
Split usually equates to more focussed conceptually secure review question
We had no choice to revise scope: What to do if you cannot split/reduce review
and it remains a large unmanageable ‘lump’?
Developing a Typology to support synthesis of evidence
Doctor/Nurse substitution Developed/developing or developed country
context High medium or low complexity re
substitution/task Close or distant proximity of medical support High/low training/preparation of nurse Context of care delivery (team/alone/available
resources/rural/urban/in-hours/out of hours)
Model of implementation to aid interpretation of barriers and facilitators evidence Realist approach:
Develop an a priori theory/propositions of what works:
Context Mechanism Outcome
Or model of implementation such as the PARiHS Framework map data against context, evidence, facilitation concepts
Logic models
We are looking at the use of logic models in Cochrane
For examples and instruction on developing a Logic Model see: University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html
Management of evidence for synthesis
Issues and challenges:
1.Nurse doctor substitution – 50 studies
2.Lay health worker – 100s of studies
Do we sample – if so how?
What criteria do we use? High/low quality evidence?
High/low coverage of interventions?
High/low coverage of context?
Challenge is that low quality, low coverage of context and intervention usually occur together
G.R.E.A.T. Project April 20, 202316 |
The GRADE approachThe GRADE approachThe GRADE approachThe GRADE approach
Grades of Recommendation Assessment, Development and
Evaluation
www.GradeWorking-Group.org
G.R.E.A.T. Project April 20, 202317 |
Quality of evidence across studiesQuality of evidence across studies
17
I BIIVIIII AIV
Quality: High
Outcom
e # 1
Outcom
e # 2
Outcom
e # 3
Quality: M
oderate
Quality: Low
Reporting the synthesis
Depends largely on approach to synthesis.
Narrative – themes
New theory - model
Issues:
Reporting quotes form original studies
Themes derived from a single study
Challenges experienced by authors
1. Meta-ethnography – Lewin et al
2. Realist synthesis – Rycroft-Malone et al
3. Thematic analysis – Thomas and Harden
2. Complex interventions – Noyes et al
“I think you should be more explicit here in Step Two.”
Methodological work CQRMG
1. Testing Lewin, Oxman and Glenton’s tool to assess the complexity of interventions ,
2. Developing guidance on when it may be appropriate to use a conceptual model/framework/theory to inform understanding and interpretation of complex evidence and effects.
Determining:
3. When qualitative research should be included alongside a review of the effects of a complex intervention
4. How to structure qualitative reviews so as to facilitate the integration of their findings
5. How to structure effects reviews so as to facilitate the integration of their findings
6. How quality appraisal/assessment of bias should be addressed and reported in hybrid reviews
7. Methods for integrating and using qualitative data to understand complexity
8. How to present the results of a review that has integrated effects and qualitative data, for instance in a Summary of Findings table
Your thoughts?
Open discussion and questions
