Qualitative Evidence Synthesis (QES):
Karin HannesCentre for Methodology of
Educational Research
Qualitative Evidence Synthesis: AgendaPART 1• Its context (and how I became triggered by QES)• How to formulate questions and search for qualitative insightsPART 2• Its popularity• Its role in systematic reviews (practical examples)• A list of developed approaches• A comparison of characteristics of two commonly used approachesPART 3• New developments
– Context specific versus multi-context reviews– Mixed methods reviews
CONTEXT AND STEPWISE APPROACHPART 1:
CONTEXT
Emma,Born the 6th of October 2010
CONTEXT
• Women who exercised did not lose significantly more weight than women in the usual care group.
• Women who took part in a diet or diet plus exercise programme, lost significantly more weight than women in the usual care.
• There was no difference in the magnitude of weight loss between diet and diet plus exercise group.
• The interventions seemed not to affect breastfeeding performance adversely.
CONTEXT
A recent study in the Journal of the America College of Nutritian found that those who ate cereals where lower in weight compared to those who ate meat and eggs, bread or skipped breakfast.
CONTEXT
Logical, rational reasoning:
• IF a diet helps to loose weight after pregancy.• IF cereals have proven to work well as a diet.
• THEN the consumption of cereals will lead to weight loss after pregnancy!
• After having consumed cereals for several months……………..
CONTEXT
What is evidence?• Evidence of ‘effectiveness’: the extent to which an intervention,
when used appropriately, achieves the intended effect.• Evidence of ‘feasibility’: the extent to which an intervention is
practical and practicable, whether or not an intervention is physically, culturally or financially practical or possible within a given context.
• Evidence of ‘appropriateness’ the extent to which an intervention fits with a situation, how an intervention relates to the context in which it is given.
• Evidence of ‘meaningfulness’: the extent to which an intervention is positively experienced by the population and relates to the personal experience, opinions, values, beliefs and interpretations of the population.
CONTEXT
Systematic ReviewsIF I am not interested in evidence of effectiveness,
BUT in feasibility, appropriateness, meaningfulness,…
Qualitative Evidence Synthesis:A process of summarizing qualitative research findings, either aggregative or interpretive, by comparing and analysing texts derived from multiple accounts of an event or situation as reported in basic qualitative research studies.
Explore questions such as • how do people experience illness?• why does an intervention work (or not), for whom and in what circumstances…? • what are the barriers and facilitators to accessing health care?• what impact do specific barriers and facilitators have on people, their experiences and
behavior?
CONTEXT
Systematic Reviews• Could I still use the methodology outlined for
SR to answer these questions? Could I use it in the same way?
1. Question formulation PICO becomes SPI(C)E2. Searching Sensitive (all-inclusive) versus Specific
(Selective)3. Critical appraisal Proponents versus Opponents4. Synthesis Variety of Approaches5. Recommendations Not always the goal, could be
building theory as well
CONTEXT
QES: Stepwise approachPICO becomes SPICE
I Intervention/(topic of) Interest: diet(C) Comparison: (training, placebo)
E Evaluation: Attitude, view, opinion on...
•Elements that hinder the diet•Impact of the diet on the general welbeing of the mother•Opinions on how to best integrate the diet in daily family life
Targeted suggestions on how to make things work.
P Perspective: Mothers in a post-natal situation
S Setting: Western SPI(C)E !
For some qualitative questions there isn’t an intervention to be evaluated.
QES: Stepwise approachSearching qualitative evidence: problems
• Little result from searching the major databases– 30% database & handsearch– 50% ‘snowballing’– 24% personal contacts(Greenhalgh, 2005)
• Major problems:– Bad indexing– Less developed and tested methodological filters
– MeSH-term: qualitative research
• General rules:– Use methodological filters– Examine references– Use the related article features in major databases– Search for citations (backword and forward)
QES: Stepwise approachSearching qualitative evidence: INTERTASC
www.york.ac.uk/inst/crd/intertasc/index.htm
• nursing methodology research OR (qualitative OR ethnolog* OR ethnog* OR ethnomethodolog* OR emic OR etic OR phenomenolog*[Title/Abstract]) OR (hermeneutic* OR heidegger* OR husserl* OR colaizzi* OR giorgi* OR glaser OR strauss [title/abstract]) OR (kaam* OR manen OR participant observ* OR constant compar* [title/abstract]) OR (focus group* OR grounded theory OR "narrative analysis" OR lived experience* OR life experience* [title/abstract]) OR (theoretical sampl* OR purposive sampl* OR ricoeur OR spiegelberg* OR merleau [title/abstract]) OR (metasynthes* OR meta-synthes* OR metasummar* OR meta-summar* OR metastud* OR meta-stud* [title/abstract]) OR (maximum variation OR snowball [title/abstract]) OR (field stud* OR field note* OR fieldnote* OR field record* OR action research [title/abstract]) OR (thematic analys* OR content analy* OR unstructured categor* or structured categor* [title/abstract]) OR (participant observation* OR nonparticipant observation* OR non participant observation* [title/abstract]) OR (tape recording OR "tape record*" OR "video record*" OR "audio record*" OR taperecord* OR audiorecord* OR videotap* OR videorecord*)
http://www.library.ualberta.ca/subject/healthsciences/QualitativeFilters/index.cfm
Methodology
Citations/authors
Techniques
QES: Stepwise approachSearching qualitative evidence: debate
• We need a sensitive approach to searching that includes all potentially relevant studies.
• We need a search strategy that is specific and purposeful, including studies that are relevant to our synthesis. This might include working with a saturation point for inclusion.
• Related to the goal and/or approach of a particular study.
Searching: addressed in upcoming presentation
QES: Stepwise approachCritically appraising evidence: debate
• The more you appraise the more you obstruct the creative, interpretive nature of qualitative research. The most important criterion is relevance!
• The more you appraise the smaller the chance that you would end up with statements and conclusions that do not represent the truth. The most important criterion is (methodological) quality!
‘Critical appraisal’ addressed in upcoming presentation
PART 2: Qualitative Evidence Synthesis (QES) approaches
QES in the scientific literatureThe role of QES (in SR)
A list of developed approachesComparing characteristics of two commonly used approaches
QES in the public health literature
Timetrend based on the reviews from:Dixon-Woods & colleagues (2007)Hannes & Macaitis (2011)
Quantitative
Accumulating
Strictly comparable
More power
Through data
Nature
Aim
Studies
Result
Synthesis
Meta-analysis
Evidence-synthesis
Qualitative
Make sense of data
Basic comparability
Added value in content
Through interpretation
The role of QES in SR: different aims
The role of QES in SR• Inform: reviews by using evidence from qualitative research to help
define and refine the question, and to ensure the review includes appropriate studies and addresses important outcomes (scoping review)
• Enhance: reviews by synthesizing evidence from qualitative research identified whilst looking for evidence of effectiveness (process and implementation issues).
• Extend: reviews by undertaking a search to specifically seek out evidence from qualitative studies to address questions directly related to the effectiveness review (mixed method or multilevel synthesis).
• Supplement: reviews by synthesizing qualitative evidence within a stand-alone, but complementary review to address questions on other than effectiveness (stand-alone or parallel synthesis).
Note:These should be distinguished from a narrative report of a quantitative SR. When individual studies cannot be pooled quantitatively (reason: heterogeneity), they may still have useful qualitative information to be shared with the reader.
Narrative synthesis because pooling is not possible? Rather not labelled as QES…
Example:Narrative report
Adding a bit won’t work.
The need: full SR of QR !
‘Mixed Method Approach’Children & Healthy Eating - EPPI-centre: eppi.ioe.ac.uk
Children’s Views Trials
Recommendation for interventions Good quality Other
Do not promote fruit and vegetables in the same way
None None
Brand fruit and vegetables as an ‘exciting’ or child-relevant product, as well as a ‘tasty’ one
5 5
Reduce health emphasis in messages to promote fruit and vegetables particularly those which concern future health
5 6
Example: Extending ReviewWhich interventions match recommendations derived from children’s views and experiences?
Example: Supplementing review• Barroso J, Powell-Cope GM. Metasynthesis of Qualitative Research on Living
with HIV Infection. Qual Health Res vol 10, nr 3, 2000.:
– Understand experience of adults living with HIV infection– 21 articles– Method: constant comparative analysis– Themes:
• Finding meaning in HIV: death, loss, surviving• Shattered meaning: fear, irreparable• Human connectedness: meaningfull relationships alienation, isolation• Focusing on the self: actions to enhance fysical and emotional health• Negociating care: active role• Dealing with stigma: personal, family, society
The role of QES in SR• Provide evidence on the subjective experience of those involved in
developing, delivering and receiving an intervention OR/AND
• Provide a research-based context for interpreting and explaining trial results
• How to achieve change (more effectively)?• How to improve interventions?• How to ‘fit’ subjective needs?• What other type of interventions might be needed?
OR/AND
• Reveal the extent to which effective interventions are actually adopted in policies and practice (what are barriers and bridges?)
OR/AND
• Contribute to the understanding of heterogeneity in outcomes
A list of developed approachesQES-approach DevelopedMeta-ethnography Noblit & Hare, 1988
Meta-summary Sandelowski & Barosso (2008)
Meta-study Paterson et al (2001)
Realist synthesis Pawson et al (2004)
Meta-narrative mapping Greenhalgh (2005)
Critical Interpretive Synthesis Dixon-Woods et al (2006)
Narrative Synthesis Popey et al (2006)
Textual narrative synthesis Lucas (2007)
Ecological triangulation Banning (unknown)
Framework synthesis Brunton et al (2006), Oliver et al (2008)
Meta-interpretation Weed (2005)
Meta-aggregation Joanna Briggs Institute (2001), Hannes and Lockwood (2010)
Bayesian meta-analysis Roberts et al (2002), Voils et al (2009)
Content analysis Evans and Fitzgerald (2002), Suikkala & Leino-Kilpi (2000)
Case Survey Yin & Heald (1975), Jensen & Rodgers (2001)
Qualitative Comparative analysis Cress & Snow (2000)
Thematic synthesis Thomas & Harden (2008)
Cross-case analysis Miles & Huberman (1994)
Grounded theory Finfgeld (1999) Kearney (2001), Eaves (2001)
… …
A list of developed approaches• MAKING SENSE OF THE
MYRIAD OF QUALITATIVE EVIDENCE SYNTHESIS METHODS
• No clear guidance about how to determine which of the existing synthesis methods best fits a particular purpose.
• possible considerations…
A list of developed approachesConsider Questions to be asked
The nature of the research •Will the synthesis method result in the expected and desired outcomes?•Is the method congruent with the goals of the synthesis project?•Does the primary research support the method?
The nature of the researcher •How tolerant is the researcher to the amount of structure and ambiguity that is inherent in the method?•Is his/her epistemological stance congruent with that of the synthesis method?
The nature of the research team •Is there the necessary mix of disciplinary, methodological, and other perspectives among the research team to enact this method?• Is the expertise needed for this method (e.g., statistical analysis, theoretical) available?
Resource requirements •How much personnel, time and effort are required for this method?•Is there adequate funding to support expenses incurred in implementing this method?
Paterson, B. (2011). Introducing Qualitative Evidence Synthesis. In Hannes, K., & Lockwood, C. (eds.). Qualitative Ev. Synthesis: choosing the right approach. Wiley-Blackwell, UK.
Qualitative Evidence Synthesis approachesDecision to conduct a qualitative evidence
synthesis
To aggregate / summarise / integrate qualitative data to address specific questions in relation to a Cochrane
intervention review
To interpret synthesised qualitative evidence and
develop explanatory theory or models
Primarily to integrate and interpret qualitative and quantitative evidence
within a single approach or integrated model
Can be used to develop explanatory theory
Purpose of the additional qualitative synthesis
Thematic analysis without theory generationMeta-aggregation
Realist reviewEPPI (mixed) approach (thematic)Narrative synthesis
Meta-ethnographyThematic analysis with theory generationGrounded theory
ProductExplanatory theory, analytical or
conceptual framework or interpretative framework/ mechanism
Product Aggregated findings from source papers
Comparing characteristics of different approaches (developed for synthesis)
Meta-ethnography Meta-aggregation
Purpose Seek and reveal similarities and differences, achieving a degree of innovation
To aggregate findings of included studies and aid decision making
Epistemology Idealism Realism
Quality Assessment Not discussed, relevance argument
Required, using standardized critical appraisal instrument
Synthesis Refutational and reciprocal translation, line of argument synthesis
Aggregation of findings into categories and of categories into synthesized findings
Outcome Higher order interpretation of findings, theory
Standardised chart informing practice and policy
PART 3: Recent developments
Context-specific versus multi-context reviewsMixed methods reviews
Context-specific versus multi-context syntheses
Quantitative Review Qualitative Review
Context-specific versus multi-context reviews
Multi-context reviewsExhaustive search
Little access to or knowledge of local databases and experts
Targets a broad audience (but no-one in particular)Findings may be too general
Risk of downplaying important local characteristicsContext may get lost
Potential low level of acceptance in end-users
Wide ranging in scopeAbility to cross compare different settings
Works for topics were little heterogeneity between settings is expectedFindings are more likely transferable to a broad range of settings
Context-specific versus multi-context reviews
• Context-specific syntheses do well in responding to the needs and policies of a targeted setting.
• Multi-context syntheses assist in building a cumulative knowledge base and are an excellent choice when little heterogeneity is expected.
• Integrating the best of both– Umbrella reviews, in which insights in a particular phenomenon
generated from different settings could be summarized.– Transcontextual adaptation, which means modifying insights in such a
way that they become relevant and reply to the needs and policies of a targeted setting.
Mixed methods reviews: definition
•‘Mixed research synthesis’ (Sandelowski et al., 2006)•‘Mixed studies review’ (Pluye et al., 2009)•‘Mixed methods synthesis’ (Harden & Thomas, 2005)
• ‘Mixed methods research synthesis’ = A synthesis in which researchers combine primary qualitative, quantitative, and mixed methods studies, and apply a mixed methods approach in order to integrate those studies, for the broad purposes of breadth and depth of understanding and corroboration (Heyvaert, Onghena & Maes, 2011).
•Promising utility for research and practice: Combining strengths of qualitative and quantitative techniques and studies
Paired: Concurrent but separate review example – 1 team
• Lins S, Rücker G, Motschall E, Langer G, Antes G, Meyer G. Efficacy and experiences of telephone counselling for informal carers of people with dementia (Protocol). Cochrane Database of Systematic Reviews 2011, Issue 5. Art. No.: CD009126. DOI: 10.1002/14651858.CD009126
– Qualitative thematic synthesis related to the efficacy of telephone counselling• Leiknes KA, Berg RC, Smedslund G, Jarosch-von Schweder L, Øverland S, Hammerstrøm KT, Høie B. Electroconvulsive therapy
for depression (Protocol). Cochrane Database of Systematic Reviews 2011, Issue 5.– Meta-aggregative approach to qualitative synthesis
Paired: A/S -QUAL + QUAN–Equal status-Concurrent
CochraneIntervention
Review
QualitativeEvidenceSynthesis
Research Question(s)
Synthesis of Qualitative and Quantitative Studies
Paired: Concurrent but separate review example – 2 teams
• Embuldeniya, G. Et al. Perceived impacts and experiences of peer support in chronic disease– Qualitative meta-ethnography exploring issues related to an ongoing Cochrane effectiveness
review (Doull MJ, O'Connor AM, Robinson VA, Tugwell P, Wells G. Peer support strategies for people with chronic disease to enhance health and promote health equity. Cochrane Database of Systematic Reviews 2005, Issue 3.)
CochraneIntervention
Review
QualitativeEvidenceSynthesis
Research Question(s)
Synthesis of Qualitative and Quantitative Studies
Paired: A/S -QUAL + QUAN–Equal status-Concurrent
Retrospective, stand alone review example
Noyes, J., Popay, J (2007).Directly observed therapy and tuberculosis: how can a systematic review of qualitative research contribute to improving services? A qualitative meta-synthesis. Journal of Advanced Nursing, 57, 227-243. Narrative Summary approach extending/enhancing a Cochrane intervention review Volmink J. & Garner P (1997) (updated
2002, 2006) Directly observed therapy for treating tuberculosis. Cochrane Database Systematic Reiews 19(2), CD003343.
Jordan J, Rose L, Dainty KN, Noyes J, Clarke S, Blackwood B. Factors which impact on the use of weaning protocols for reducing the duration of mechanical ventilation in adults and children. Meta-ethnography extending/enhancing a recent Cochrane intervention review (Blackwood B, Alderdice F, Burns KEA, Cardwell CR,
Lavery G, O'Halloran P. (2010) Protocolized versus non-protocolized weaning for reducing the duration of mechanical ventilation in critically ill adult patients. Cochrane Database of Systematic Reviews (in press).
CochraneIntervention
Review
QualitativeEvidenceSynthesis
Published New Retrospective, stand alone
-Equal status-Sequential
QES: conclusion
• If you wish to embark in the field of qualitative evidence synthesis you will soon realise that
– There are no fixed standards– There is no general consensus (and will there ever be one?)
– You shall have to travel slowly,– By your own means (but the CQRM- and CP&I-groups can help)
– On small and potentially difficult roads…
• BE PREPARED FOR A POTENTIAL DELAY!