Date post: | 28-Mar-2015 |
Category: |
Documents |
Upload: | william-rogers |
View: | 225 times |
Download: | 0 times |
Systematic reviewing for Nursing, Therapy & Allied health- Qualitative Evidence Synthesis (SyNTAQuES)Andrew Booth, Chris Carroll & Janet Harris, School of Health and Related Research (ScHARR), University of Sheffield.
By the end of this workshop participants will be able to:
• Understand role of qualitative evidence syntheses (QES) /systematic reviews and types of questions they can address.
• Be able to formulate an appropriate question to be addressed by a qualitative systematic review/evidence synthesis.
• Be able to construct an efficient search strategy for retrieval of primary qualitative research and to select appropriate methodology filters for qualitative studies.
• Apply explicit criteria regarding collection/analysis of qualitative data to appraise the quality of a qualitative research report.
• Understand main features and techniques required to conduct QES, specifically thematic synthesis and framework synthesis.
• Understand primary considerations when selecting a synthesis method and how these impact upon the conduct and reporting of a qualitative evidence synthesis/systematic review.
Programme
MORNING
What is Qualitative Systematic Review/ Evidence Synthesis
What are your choices?Formulating a Question for Qualitative ReviewSearching & Sifting for Relevant Studies
BreakQuality Assessment [Plenary]Quality Assessment Workshop
Programme
AFTERNOON
Introducing Thematic Synthesis/Framework SynthesisThematic Synthesis/Framework Synthesis WorkshopFeedback
BreakProducing Your Review for PublicationWhere to Next?
Close
What is Qualitative Systematic Review/Evidence Synthesis?
Andrew BoothSchool of Health and Related Research,
University of Sheffield, UK
Umbrella terms:
• Qualitative Systematic Review (but also non-quantitative, e.g. not meta-analysis)
• Qualitative Metasynthesis (Sandelowski et al, 2001)
• Qualitative Research Synthesis (Howell Major & Savin-Baden, 2010)
• Qualitative Evidence Synthesis (Cochrane Collaboration: Hannes & Lockwood, 2012)
Topic=("qualitative systematic review" OR "qualitative evidence synthesis" OR "qualitative research synthesis") OR Topic=(metastudy OR metasynthesis OR "meta synthesis" OR "meta ethnography" OR "meta ethnographic" OR "metaethnography" OR "metaethnographic") OR Topic=("systematic review of qualitative") = 645 Currently about 10 per month
61% from 2008-2012; 37.5% from 2010-2012
So what is a QES?
• qualitative evidence synthesis – an umbrella term increasingly used to describe a group of review types that attempt to synthesise and analyse findings from primary qualitative research studies (Booth et al, 2011).
Qualitative Evidence Synthesis
• Quantitative meta-synthesis, or meta-analysis, aims to pool numerical results of individual quantitative studies, qualitative meta-synthesis looks for “themes” or “constructs” that lie in or across individual qualitative studies.
• Within broader category of “qualitative meta-synthesis,” (Cochrane preferred term = qualitative evidence synthesis) narrow term “meta-ethnography” refers to specific method of data synthesis most widely adopted to date.
Booth et al, 2006
Key Difference
• Goal not aggregative in sense of “adding studies together,” as with meta-analysis.
• Interpretative in broadening understanding of a particular phenomenon.
An Example• Paterson and colleagues
identified 38 studies examining first hand experience of living with diabetes.
• Prevailing metaphor = concept of “balance”. Specific sub-themes identified across multiple studies included “knowing one's body,” “learning how to manage diabetes,” and “fostering supportive, collaborative relationships with others.”
Stages of a Qualitative Evidence Synthesis?
• Formulating the review question
• Conducting a systematic literature search
• Screening and selecting appropriate research articles
• Analyzing and synthesizing qualitative findings
• Maintaining quality control
• Presenting findings(Sandelowski & Barroso, 2007)
Framework for systematic reviews of qualitative research (Garside, 2010)
Stage Typical activities
Developing research question Assemble team; Consult; Agree approach
Scoping exercise Identify relevant research; Refine methods
Identifying relevant literature Develop Inclusion/Exclusion criteria; Focused searches; Citation searches
Initial assessment of study reports Preliminary reading; Identify theories; Assess utility/relevance
Analysis and synthesis Reading & rereading study reports; Constant comparison; Assess validity
Preliminary synthesis Categorising; tabulating; mindmaps; Explore relationships
Full synthesis Thematic analysis; translation of findings; Theory development; rival explanations
Dissemination Target audiences; Limitations of review
Throughout Multiple viewpoints; Reflexivity; Audit trail; Ongoing consultation; revisit review purpose
Why do it?
• Perspectives of Patients, Relatives or Carers• Of Condition• Of Intervention
• Perspectives of Staff• Implementation• Barriers and Facilitators
• Patient Centred Outcomes
• Development and Testing of Theory
• Answering “What Works for Whom Under Which Circumstances?”
• Acceptability of Treatments
• Explaining Differential Effectiveness
• Designing and Evaluating Complex Interventions
Some Examples - 1• Perceptions of patients
and physiotherapists on patient participation
• Systematic search in six databases using a set of key words, extracted relevant data, performed quality assessment and synthesized findings from selected studies.
• Narrative synthesis of qualitative studies. Retained 11/160 studies.
• Two main themes: conceptualization of patient participation and patients' role preferences. Patient participation included goal setting, information exchange, decision-making and exercise training; often influenced power relation between patient and physiotherapist. Patients' willingness to participate varied; they often did not play their desired role.
SCHOEB & BÜRGE (2012)
Some Examples - 2• Incentives and barriers to
lifestyle interventions for people with severe mental illness.
• Eight electronic databases [1985–March 2009] plus Cochrane Library and Google Scholar. Electronic ‘hand’ searches of key journals and explosion of references also undertaken.
• Narrative synthesis of qualitative, quantitative and mixed methods studies
Reports possible incentives and barriers including:
•Illness symptoms, treatment effects, lack of support and negative staff attitudes as possible barriers;
•Symptom reduction, peer and staff support, knowledge, personal attributes and participation of staff as possible incentives.
ROBERTS & BAILEY (2011)
Centres and Networks: EPPI-CentreJoanna Briggs InstituteCochrane CollaborationCampbell Collaboration
SoftwareEppiReviewerQARI
BooksNoblit & Hare (1988) - Meta-EthnographyPatterson et al (2001) - Meta-Study of Qualitative Health ResearchSandelowski et al (2001) - Handbook for Synthesizing Qualitative Research Petticrew & Roberts (2003) - Systematic Reviews in Social SciencesMays, Pope, Popay (2007) - Synthesizing Qualitative and Quantitative Health ResearchHowell Major & Savin-Baden (2010) - Introduction to Qualitative Research SynthesisHannes & Lockwood (2011) - Synthesizing Qualitative Research: Choosing the Right ApproachSaini & Schlonsky (2012) - Systematic Synthesis of Qualitative ResearchBooth et al (2011) – Systematic Approaches to a Successful Literature ReviewGough, Oliver & Thomas (2012) - An Introduction to Systematic Reviews
What are your Choices?
Andrew BoothSchool of Health and Related Research, University of
Sheffield, UK
Specific Methods
Realist Synthesis
Meta-Interpretation
Critical Interpretive Synthesis
Thematic Synthesis
Framework Synthesis Narrative Synthesis
Meta-Ethnography
Best Fit Synthesis
Meta Study
Meta Narrative
EPPI-Centre Method
Matrix Method
Bayesian Meta-Synthesis
Meta- Aggregation
Decision to conduct a qualitative evidence synthesis
To interpret synthesised qualitative evidence and develop
explanatory theory or models
Purpose of the additional qualitative synthesis
or
Choice of Synthesis (Adapted from Noyes & Lewin, 2011)
To aggregate/ summarise/ integrate qualitative data to address questions in relation to a specific intervention review
Primarily to integrate and interpret qualitative and quantitative evidence within a single approach or integrated model. Can be used to develop explanatory theory.
Thematic analysis without theory generationMeta-aggregationMeta-summary
Product: Aggregated findings from source papers
Meta-ethnographyGrounded theoryThematic analysis with theory generation
Framework synthesis Realist Review
EPPI ApproachMatrix MethodNarrative SynthesisBayesian SynthesisCritical Interpretive synthesis
Product: Explanatory theory, analytical or conceptual framework or interpretative framework/mechanism
Best fit synthesis
Bad Reasons for Choosing Method
• Frequency of Use of Method (e.g. Meta-Ethnography)
• Popularity/”Sexiness” of Method (e.g. Realist Synthesis)
• What a friend/ colleague/ mentor has used (once!)
• Bad experiences of others (may have been inappropriate!)
Key Considerations
1. Role of Theory
2. Available Expertise
3. Extent of Description versus Interpretation
4. Intended ProductOther Considerations: Richness of Data and Available Time/ Resources
1. Will You Generate, Explore, Test Theory (Gough et al, 2012)?
• Generate – may require “suspension of disbelief” – quality assessment/ value judgement may come later (cp. Brainstorming)- Grounded Theory, Meta-ethnography
• Explore – looking for patterns - Narrative Synthesis, Thematic Synthesis
• Test – quality assessment differentiates well-supported and unsupported data - Framework Synthesis (incl. Best Fit Synthesis)
• NB. We (Carroll & Booth) are currently conducting empirical work on systematic identification of Theories
2. What Expertise Can You Access?
Epistemological versus Technical:•Expertise in Qualitative Research Methods (e.g. Grounded Theory; Framework Analysis, Thematic Analysis)•Expertise in Synthesis Methods (incl. Searching, Data Extraction, Quality Assessment, Interpretation) •Knowledge of Topic Area
http://www.jiscmail.ac.uk/lists/asqus.html
3. Will You Describe or Interpret?
All Reviews figure on a continuum between Description and Interpretation
•Description – What does the data say? – factual reporting of studies, themes etc…
•Reader does work of interpretation
• Interpretation – What does the data mean? –subjective interpretation of data and themes etc…
• Reviewer does work of interpretation – may be contested
4. What is Your Intended Output/Product?
• “the output of some methods of synthesis (Thematic Synthesis, textual Narrative Synthesis, Framework Synthesis, and ecological triangulation) is more directly relevant to policymakers and designers of interventions than the outputs of methods with a more constructivist orientation (Meta-Study, Meta-Narrative, Meta-Ethnography, Grounded Theory, CIS) which are generally more complex and conceptual” (Barnett-Page & Thomas, 2009)
• Thematic Synthesis (including Meta-Aggregation) and Framework Synthesis produce findings to directly inform practitioners (Thomas & Harden, 2009)
• Interpretive approaches (e.g. CIS, Meta-Ethnography) produce a model that requires practitioners to interpret relevance and applicability to their own context
• Narrative Synthesis or EPPI-Centre (matrix) methods may help to integrate and present quantitative/qualitative work
How Rich (“Thick”) is Your Data?
• Qualitative data from “thin” studies (or textual responses to surveys) will not sustain interpretive approaches
• Limited to Meta-Aggregation, Thematic Synthesis, Framework Synthesis, Narrative Synthesis –type approaches
• Rich/“Thick” reports will sustain Meta-Ethnography/Grounded Theory – may allow selective sampling/ theoretical saturation
• NB. Is “Unit of Analysis” Individual Study (Meta-Aggregation, Thematic Synthesis) or “Body of Evidence” (e.g. Meta-Narrative or Critical Interpretive Synthesis approaches) or even Theory (Framework Synthesis/Best Fit Synthesis)?
How Long?/How Much Have You Got?
• “Richer” approaches make fuller use of data – require fewer studies
• Meta-Aggregation, Thematic Synthesis can handle large numbers of studies
• Framework/best fit approach for speed
32 papers (775 patients and carers) reporting help-seeking experiences for at least 20 different types of cancer.
Synthesis Quiz
For each of the following scenarios identify the review characteristics and try to match to an appropriate type of synthesis
Scenario A: You are working in a general health technology assessment team conducting a synthesis of Group Therapy for Postnatal Depression. There are few studies and most reports are Nursing Standard/Times case reports. There is no prevalent theory. You will produce recommendations on what works for whom.
1. Role of Theory 2. Expertise3. Describe/Interpret 4. Output/Product
Scenario B: You are working as a group of topic experts and experienced qualitative researchers to examine the phenomenon of “Willingness to Hasten Death”. There are less than eight rich qualitative studies. No-one has yet conducted a synthesis to look at what is meant by the concept. Your review will help those who work in terminal care to gain a better understanding of the phenomenon. 1. Role of Theory 2. Expertise3. Describe/Interpret 4. Output/Product
Scenario C: You are a member of a government-funded Institute producing best practice reviews for nurses and nursing managers. You are asked to examine all evidence, quantitative and qualitative, for physical restraint in residential homes. You are expected to turn around a report in a very short time frame. Your final review is expected to include Recommendations for Practice. 1. Role of Theory 2. Expertise3. Describe/Interpret 4. Output/Product
Scenario D: Your review team, with access to clinical expertise, is commissioned to review the evidence on taking vitamins and supplements to prevent colorectal cancer. Because of the focus on long term outcomes there are few directly relevant qualitative studies. You decide to look for “related” studies and hypothesise attitudes to these specific agents from a wider evidence base. Attitudes to medicine are well-theorised but not within such a long-term prevention perspective. Your qualitative review will be a short chapter in a much longer effectiveness/cost effectiveness report.
1. Role of Theory 2. Expertise3. Describe/Interpret 4. Output/Product
Safest Options!
If…•There is a Pre-existing Theory or Framework….
Then•….Framework Synthesis (including Best Fit Synthesis)
If…•There is No Theory or Framework…
Then•…Thematic Synthesis (Can also act as first stage of Meta-Ethnography)
If…•There is a Proximate (Close-ish!) Theory or Framework….
Then•….Best Fit Synthesis
Formulating a Question for Qualitative Review
Janet HarrisSchool of Health and Related Research, University of
Sheffield, UK
Framework for systematic reviews of qualitative research (Garside, 2010)
Stage Typical activities
Developing research question Assemble team; Consult; Agree approach
Scoping exercise Identify relevant research; Refine methods
Identifying relevant literature Develop Inclusion/Exclusion criteria; Focused searches; Citation searches
Initial assessment of study reports Preliminary reading; Identify theories; Assess utility/relevance
Analysis and synthesis Reading & rereading study reports; Constant comparison; Assess validity
Preliminary synthesis Categorising; tabulating; mindmaps; Explore relationships
Full synthesis Thematic analysis; translation of findings; Theory development; rival explanations
Dissemination Target audiences; Limitations of review
Throughout Multiple viewpoints; Reflexivity; Audit trail; Ongoing consultation; revisit review purpose
A 15 minute guide
• What is the focus?• Brainstorm questions• Refine• Prioritize• Determine next steps• Reflect
(Adapted from Rothstein and Santana (2011) Teaching Students to Ask Their Own Questions: One small change can yield big results. Harvard Educational Letter)
Focus and brainstorm
Our topic of interest today is: •Taking a client-centred approach to rehabilitation after stroke
Our FOCUS – the problem that has initiated a review of qualitative research : •Health professionals have different ideas about client-centred approaches to rehabilitation
Our AIM:•To produce evidence from a synthesis of the literature that will guide effective client centred-approaches
BRAINSTORM•Ask as many questions as you can about this topic•Do not stop to discuss, judge, or answer the questions.•Write down every question exactly as it is stated.•Change any statement into a question
What did you come up with?
• Broad or narrow question?
• Open or closed?
• Advantages/disadvantages of the questions?
Prioritize• Which questions do you want to pursue
and why?• Consider
• Likelihood of finding qualitative studies• Relevance of studies to problem focus• How the research synthesis will be used
What people have
researched
What I think I am
looking for
Next steps would be…
• Conduct a ‘scoping search’ to see whether the published research seems to be a match for your problem focus
• Use the brief scoping of abstracts to refine the question further
What I think I am
looking for
What I can realistically find!!
The final question for the qualitative review was:
• What are stroke survivors’ experiences with rehabilitation? (Peoples et al, 2011)
Searching and Sifting for Relevant Studies
Andrew BoothSchool of Health and Related Research, University of
Sheffield, UK
Framework for systematic reviews of qualitative research (Garside, 2010)
Stage Typical activities
Developing research question Assemble team; Consult; Agree approach
Scoping exercise Identify relevant research; Refine methods
Identifying relevant literature Develop Inclusion/Exclusion criteria; Focused searches; Citation searches
Initial assessment of study reports Preliminary reading; Identify theories; Assess utility/relevance
Analysis and synthesis Reading & rereading study reports; Constant comparison; Assess validity
Preliminary synthesis Categorising; tabulating; mindmaps; Explore relationships
Full synthesis Thematic analysis; translation of findings; Theory development; rival explanations
Dissemination Target audiences; Limitations of review
Throughout Multiple viewpoints; Reflexivity; Audit trail; Ongoing consultation; revisit review purpose
Relevant Studies…
1. Will match the focused question for the review (e.g. The Experience of Rehabilitation for Post-Acute Stroke Survivors), AND
2. Will employ a qualitative research method (for data collection and analysis) [OR, if the review protocol permits, will include qualitative data e.g. from open-ended survey questions]
Other Considerations…
• How much literature is there? How is it characterised? e.g. Was it conceived as Qualitative Research?
• How far back will I go? Has the Intervention/Experience of Living with The Condition Changed?
• What Settings, Countries, Languages will I include? How important is Context?
Post acute stroke rehabilitation
http://www.nlm.nih.gov/nichsr/hedges/search.html
Relevant Studies…
1. Will match the focused question for the review (e.g. The Experience of Rehabilitation for Post-Acute Stroke Survivors),
Matching the Focused Question
• Combination of natural language (e.g. stroke) and index terms (e.g. CEREBROVASCULAR DISORDERS and BRAIN INFARCTION)
• May include subheadings e.g. /rehabilitation as well as index terms
• Need to optimise Sensitivity (Getting everything you want) and Specificity (Getting only what you want)
Sample Search Strategy (SIGN Guideline)
Relevant Studies…
, AND
2. Will employ a qualitative research method (for data collection and analysis) [OR, if the review protocol permits, will include qualitative data e.g. from open-ended survey questions]
Challenges
• Qualitative research is small part of the research literature
• Indexing in databases is inconsistent and variable
• Qualitative studies sometimes have creative titles or inadequate abstracts
• MEDLINE might not be your preferred resource.
• If searching in limited databases try MEDLINE and CINAHL
How would you find qualitative studies?
Sources of Evidence
10/04/23 © The University of Sheffield
Identifying Qualitative Research - Terminology
Generic terms: e.g. “qualitative” plus “ESCAPADE”Exploratory Methods: Focus group, Grounded theory, Action Research, Content analysis, Thematic analysis Software: Nudist or NVivoCitations: Glaser & StraussApplication: Ethnology, PsychologyPhenomenon: Perceptions, Attitudes, User Views, Standpoint, ViewpointApproaches: EthnographicData: Stories, Narratives, Descriptions, Themes, FindingsExperiences: Encounters, Experiences
What is a methodological filter?
• “A hedge or filter is a standardised search strategy that is designed to be used in conjunction with a subject search to retrieve valid studies from the (primary) medical literature”.
• Filters work in one of two ways:
• by identifying particular publication types or study designs most likely to answer a question
• by isolating subject or free-text terms most likely to be associated with high-quality studies
How do they work?
• Filters come from 3 different sources:-
• Subject heading
• Keyword
• Publication type
How do I use a methodological filter?Step One: Carry out a subject search as usual
using subject headings and/or free text
Step Two: Apply methodological filter appropriate to question you are asking
One-line filter
Maximum sensitivity filter
Maximum specificity filter
Mid-range filter
One-line filters
• MeSH Headings• e.g. Qualitative Research [Medline 2003-]
• Keyword • “Findings”
• Publication Type• Research [CINAHL only]
Methodological filters - 1
1. qualitative$
2. findings
3. interview$
4. interviews.DE.
5. 1 OR 2 OR 3 OR 4
Grant MJ. “How does your searching grow? A survey of search preferences and the use of optimal search
strategies in the identification of qualitative research.” Health Info Libr J. 2004 Mar; 21(1):21-32.
Methodological filters - 2
• Best sensitivity: exp interviews
• Best specificity: audiorecording.sh.
• Best optimization of sensitivity and specificity: exp study design
Wilczynski NL, Marks S, Haynes RB. Search strategies for identifying qualitative studies in CINAHL. Qual Health Res. 2007 May;17(5):705-10.
Methodological filters - 3
• Don’t forget MeSH heading Qualitative Research
• Introduced in 2003, so coverage is limited
• But you can combine it with one of the previous filters by using the “OR” operator
Sources of filters
• ISSG search filters resource http://www.york.ac.uk/inst/crd/intertasc/
• SIGNhttp://www.sign.ac.uk/methodology/filters.html
• McMaster Hedges Project http://hiru.mcmaster.ca/hiru/HIRU_Hedges_home.aspx
• PubMed Clinical Querieshttp://www.ncbi.nlm.nih.gov/corehtml/query/static/clinical.shtml
Each Question requires a different solution...
• Findings showed that a simple search strategy (broad-based terms - 3 search terms) was as effective as a complex one (free text - 48 search terms) in locating qualitative research on patients’ experiences of living with a leg ulcer.
• May be feasible to restrict searches with a clear nursing focus to CINAHL bibliographic database.
• Replication with other nursing topics is required.Flemming K, Briggs M. Electronic searching to locate qualitative
research: evaluation of three strategies. J Adv Nurs. 2007 Jan;57(1):95-100.
“Supplementary” Methods
• Citation searches
• Reference lists
• Hand searching
10/04/23 © The University of Sheffield
Citation searching- Google Scholar
In Conclusion• Identifying Qualitative Research is Challenging
• Simple strategy with qualitative terms (e.g. Interview*; qualitative; findings etc) may be more effective than exhaustive lists of terms
• May not be necessary to search as many databases as quantitative topics
• But supplementary methods (e.g for books and theses) may be very important!
• Cp. Cluster searching!
Reading the Reports
Janet HarrisSchool of Health and Related Research,
University of Sheffield, UK
Aims and objectives
• What do we mean by quality assessment?
• What does it involve?
• How do you do it?
• What are the strengths and weaknesses of different tools or checklists?
Framework for systematic reviews of qualitative research (Garside, 2010)
Stage Typical activities
Developing research question Assemble team; Consult; Agree approach
Scoping exercise Identify relevant research; Refine methods
Identifying relevant literature Develop Inclusion/Exclusion criteria; Focused searches; Citation searches
Initial assessment of study reports Preliminary reading; Identify theories; Assess utility/relevance
Analysis and synthesis Reading & rereading study reports; Constant comparison; Assess validity
Preliminary synthesis Categorising; tabulating; mindmaps; Explore relationships
Full synthesis Thematic analysis; translation of findings; Theory development; rival explanations
Dissemination Target audiences; Limitations of review
Throughout Multiple viewpoints; Reflexivity; Audit trail; Ongoing consultation; revisit review purpose
Quality assessment: Basic criteria• Carrying out ethical research• Importance of the research• Clarity and coherence of the report• Use of appropriate and rigorous methods• Importance of reflexivity or attending to
researcher bias• Importance of establishing validity or
credibility• Importance of verification or reliability
(Cohen et al, 2008)
Quality assessment: 3 stages
1. Filtering against minimum criteria, involving adequacy of reporting detail
2. Assessing technical quality of the study’s elements - methodological soundness, credibility, validity
3. Judging theoretical consistency
Filtering: Relevance and adequacy of reporting
• Relevance: generally judged against title and abstract in the first instance
• Reporting: Is the study process reported well enough to judge technical rigour? This depends on your area of interest…• Intervention
Core elements of the intervention as plannedNon-core elements e.g. components that are added to suit a
particular context or/or population
• Theory development: Relationship between Context, Mechanisms and Outcomes
• Implementation: barriers and facilitators to implementing the intervention
Quality appraisal: Basic criteria
Technical quality – Qualitative concepts TechniquesCredibility: the representation of data fits the views of the participants studied, the findings hold true
•outside auditors or participants validate findings (member checks)•peer debriefing, •attention to negative cases, •independent analysis of data by more than one researcher •verbatim quotes•persistent observation (stay in the field long enough)
Confirmability: findings are qualitatively confirmable through the analysis being grounded in the data, through examination of the audit trail
assessing the potential effects/impact of the researcher during all steps of the research processReflexivity toward personal influences, biasproviding background information on the researcher’s background, education, perspective, school of thought
Dependability: process of research is logical, traceable and clearly documented, particularly on the methods chosen and the decisions made by the researchers
peer review, debriefing, audit trailstriangulation, the use of different methodological approaches to look at the topic of researchreflexivity to keep a self-critical account of the research processcalculation of inter-rater agreements
Transferability: research findings are transferable to other specific settings
providing details of the study participants to enable readers to evaluate for which target groups the findings potentially hold trueproviding contextual background information, demographicsproviding thick description about both the sending and the receiving context
Validity: linking data collection
and analysis to conclusions• When evaluating
methodological soundness, we need to know • whether the set of
arguments or the conclusion derived from a study necessarily follows from the premises.
• whether it is well grounded in logic or truth.
• whether it accurately reflects the concepts, the ideas that it is intended to measure.
Theoretical quality
• Does your review question aim to develop an explanation for something – a theory?
• If it does, then you need to consider theoretical quality
• How was the theory developed?• Is there congruence between the data and the
interpretation of findings? Can you see a logical connection or translation of findings into theory?
Revisiting relevance?
• Relevance can be a multistage process• The first stage involves excluding obviously irrelevant
papers (filtering)
- The second stage involves weighing relevance to the review question with the quality of reporting and technical competence at establishing validity Data may be relevant and insightful but not rigorously analyzed...do you include it?
Criteria1) There is congruity between the stated philosophical perspective and the research methodology.2) There is congruity between the research methodology and the research question or objectives.3) There is congruity between the research methodology and the methods used to collect data.4) There is congruity between the research methodology and the representation and analysis of data.5) There is congruity between the research methodology and the interpretation of results.
6) There is a statement locating the researcher culturally or theoretically.
7) The influence of the researcher on the research, and vice-versa, is addressed.
8) Participants, and their voices, are adequately represented.9) The research is ethical according to current criteria or, for recent studies, there is evidence of ethical approval by an appropriate body.
10) Conclusions drawn in the research report do appear to flow from the analysis, or interpretation, of the data.TOTAL
QARI Critical Appraisal Instrument
Criteria categorization and definition
Criteria categorization and definition
Study design:Yes - if e.g., “a case study approach was used . . .”, “phenomenology was used . . .”No - If paper does not specify study design Unclear if unsure
Method of data collection:Yes - If details of the data collection method are given e.g., piloting; interviews, topic guides for interviews; number of items in a survey; use of open or closed items; validation, and so forth.No - If only states “focus group”, “interviews were used” or “questionnaire was used”Unclear if unsure
Selection of participants:Yes - If the sampling and recruitment of participants is described in full or explicitly as e.g., purposive, convenience, theoretical and so forth.No - If only details of participants are given, e.g. age, gender, numberUnclear if unsure
Methods of data analysis:Yes - If full details of analysis method are given, e.g., transcription and form of analysis (with reference or full description of method), validation tests, and so forth.No - If only states “content analysis” or that “data were analyzed”Unclear if unsure
Quality of Reporting Tool (Carroll et al, QHR 2012)
Appraising a paper for a qualitative review
• Group 1: Use the QARI appraisal tool
• Group 2: Use the Quality of Reporting Tool
• Make your decision about the quality of the paper
• Technical quality
• Relevance
• Theoretical quality
Integrating papers
• Would you use this paper in the qualitative review?• Poor methodological quality but
relevant data and valuable insights
• Good quality, but poor interpretation and limited insight
• Which gets privileged – the quality score or useful insight?
Learning outcomes
• An understanding of • What we mean by quality assessment
• What critical appraisal involves
• How you conduct an appraisal
• The strengths and weaknesses of different tools or checklists
• Which tool might be best for you
Introducing framework synthesis and thematic
synthesis
Chris CarrollSchool of Health and Related Research, University of
Sheffield, UK
Aims and objectives
• What do we mean by framework synthesis and thematic synthesis?
• How do you extract data for synthesis?
• How do you perform both types of synthesis?
• How do you “complete” the synthesis?
Time Content
13.15-13.35 Introduction to two types of synthesis
13.35-14.45 Extracting and coding data: IntroductionPractical (10 mins)Feedback (5 mins)How to do thematic analysisPractical (15 mins)Feedback (10 mins)Analysis and synthesis
14.45-15.00 Feedback
15.20-15.45 Producing your review for publication (AB)
Framework for systematic reviews of qualitative research (Garside, 2010)
Stage Typical activities
Developing research question Assemble team; Consult; Agree approach
Scoping exercise Identify relevant research; Refine methods
Identifying relevant literature Develop Inclusion/Exclusion criteria; Focused searches; Citation searches
Initial assessment of study reports Preliminary reading; Identify theories; Assess utility/relevance
Analysis and synthesis Reading & rereading study reports; Constant comparison; Assess validity
Preliminary synthesis Categorising; tabulating; mindmaps; Explore relationships
Full synthesis Thematic analysis; translation of findings; Theory development; rival explanations
Dissemination Target audiences; Limitations of review
Throughout Multiple viewpoints; Reflexivity; Audit trail; Ongoing consultation; revisit review purpose
A Framework
How do you create a framework?
Develop own framework or themes or concepts for coding Oliver S et al: A multidimensional conceptual framework for analysing
public involvement in health services research. Health Expectations 2008, 11:72-84.
Brunton G, Oliver S, Oliver K, Lorenc T. A Synthesis of Research Addressing Children’s, Young People’s and Parents’ Views of Walking and Cycling for Transport London. London, EPPI-Centre, Social Science Research Unit, Institute of Education, University of London; 2006.
Identify or create a thematic framework or conceptual model Carroll C, Booth A, Cooper K. A worked example of “best-fit”
framework synthesis: A systematic review of views concerning the taking of potential chemopreventive agents, BMC Medical Research Methodology 2011; 11: 29
An existing conceptual framework
Conrad K, et al (1996). The worksite environment as a cue to smoking reduction. Research in Nursing & Health, 19 21-31.
A Framework
Framework synthesis
Thematic synthesis
The case study
The question:
What are the barriers and facilitators of patient autonomy post stroke?
The studies:
1. Proot I et al, Patient autonomy during rehabilitation: the experiences of stroke patients in nursing homes, International Journal of Nursing Studies, 2000; 37: 267-76.
2. Mangset M et al, ‘We’re just sick people, nothing else’: ... Factors contributing to elderly stroke patients satisfaction with rehabilitation, Clinical Rehabilitation, 2008; 22: 825-35.
Data extraction
What is it?
An attempt to reduce a mass of material (your included papers) to a much smaller body of text and numbers, amenable to analysis and the interpretation of findings
Data extraction form
Location Setting Sample (n) Age, Gender, Ethnicity and other relevant variables (e.g. Marital
status, socio-economic status) Intervention (if any) Outcomes and results Study design Data collection method Quality assessment criteria Further citations
Format?????
Example form: Proot 2000
What results do you extract?
What is your question?
Question:What are the barriers and facilitators of patient autonomy post-stroke?
Keep the question in mind as you read:Are the data relevant to this question?Is the question answered by the data?
Extraction of results
Results sections of included studies: Authors’ statements clearly-supported by data
and/or Verbatim quotations
“... It was often difficult to distinguish [Schutz’s] first- from second-order constructs”
Atkins et al. Conducting a meta-ethnography of qualitative literature: Lessons learnt, BMC Med Res Methodol 2008; 8: 21
Schutz A. Collected Papers 1971; vol.1: 361.
From: Mangset p.829
Extracting data for thematic analysis
Extracting data for thematic analysis
Extracting or “coding” data using framework analysis
Framework developed from ...
Kirkevold M et al. Promoting psychosocial well-being following a stroke: developing a theoretically and empirically sound complex intervention, International Journal of Nursing Studies, 2012; 49: 386-97.
Extracting or “coding” data using framework analysis
Code number Theme Definition
1 Dignity A self-concept characterised by self-valuing, self-acceptance, usefulness and belief in oneself
2 Well-being A basic mood of joy, pleasure and wellbeing; the absence of feelings of emptiness or sadness
3 Coping Ability to adapt and cope within home and hospital settings
4 Family Support Relations with immediate family
5 Support for patient’s coping efforts
Patients take responsibility and are supported by others in this
6 Bodily changes and impairments
Difficulties generated by residual or post-stroke conditions
Handout 5
Extracting or “coding” data using framework analysis
Extracting or “coding” data using framework analysis
Extracting or “coding” data using framework analysis
Extracting or “coding” data using framework analysis
Extraction and coding: Practical
Consider the full framework in Handout 5 Consider extracts A, B and C on Handout 6 Code the extracted data according to the framework by
entering the number of the framework code in the right-hand column on Handout 6
Note: Data may be assigned more than 1 theme No code in the framework, in your opinion, might capture, “fit”
or reflect the data. If this is the case, enter “New theme” into the right-hand column
10 minutes: Individual and compare with neighbour; 5 minutes: Discussion of results and experiences
What now?
Secondary thematic analysis Practical: Thematic analysis Discussion of practical Synthesising your data
What now?
You have extracted your data
AND You have coded your data against your framework
(framework analysis) Or you may have coded some of your data but not
others (“best fit” framework synthesis)
OR You are conducting thematic synthesis anyway ...
You now need to assign themes to those data without themes, i.e. Using secondary thematic analysis (data reduction, display, conclusion)
How do you do it?
Generating new themes
“Evidence does not speak for itself. It requires interpretation .... on the basis of our personal experience and a range of ideas that we use to make sense of our observations”
Kelly M et al. Evidence-based public health: A review of the experience of NICE developing public health guidance in England, Social Science and Medicine, 2010; 71: 1056-62.
Developing new themes
C. “…“I will take that paper when P. is finished with it. I want to study it. Then I'll take my shoes off, and practise as I see it in the picture” (p04b, p. 11), Proot, p.271
Other / New theme? Interpret Create Revisit and revise Discuss Agree ...
= Learning self-help skills / guided self-determination
Thematic analysis: Practical
Consider extracts D, E and F on Handout 6 Develop and assign new themes to these extracts and
enter them in the right-hand column on Handout 6
Note: Data may be assigned more than 1 theme There may also be code in the framework which, in your
opinion, might capture, “fit” or reflect the data. If this is the case, also enter this code into the right-hand column
15 minutes: Individual and compare with neighbour; 10 minutes: Discussion of results and experiences
Generating new themes: Practical
Feedback Interpretation Bias?
Reviewers might have agendas too
What is the result of the secondary thematic analysis?
Synthesis
Report the results of the new thematic framework Narrative structured by theme
Report each theme with reference to studies and data in order to: Specify the evidence base for the theme Illustrate the theme with reference to actual data
McInerney P & Brysiewicz P. A systematic review of the experiences of caregivers in providing home-based care to persons with HIV/AIDS in Africa, JBI Library of Systematic Reviews, 2009 ;7(4):130-153
Synthesis
Synthesis
Reduce the findings into a smaller number of categories and then a single finding (metasynthesis, according to the Joanna Briggs Institute approach):
McInerney P & Brysiewicz P. A systematic review of the experiences of caregivers in providing home-based care to persons with HIV/AIDS in Africa, JBI Library of Systematic Reviews, 2009 ;7(4):130-153
Synthesis
A new framework
Code number Theme Definition
1 Dignity A self-concept characterised by self-valuing, self-acceptance, usefulness and belief in oneself
2 Well-being A basic mood of joy, pleasure and wellbeing; the absence of feelings of emptiness or sadness
3 Coping Ability to adapt and cope within home and hospital settings
4 Family Support Relations with immediate family
5 Support for patient’s coping efforts
Patients take responsibility and are supported by others in this
6 Bodily changes and impairments
Difficulties generated by residual or post-stroke conditions
7 Self-determination Define theme as a concept, based on data
8 Need for information Define theme as a concept, based on data
9 Lack of shared decision-making
Define theme as a concept, based on data
10 Acceptance of therapy Define theme as a concept, based on data
A new framework and synthesis
Theme Categories
Dignity
Psychological factors
Well-being
Self-determination
Coping
Bodily changes and impairments Physical factors
Need for information
Interaction with professionals and family
Lack of shared decision-making
Family Support
Support for patient’s coping efforts
Acceptance of therapy
Synthesis
Alternatively, go beyond the framework to create a new conceptual model or theory ...
Revisit data to explore the relationships between the themes or findings of your framework
The “richer” or “thicker” the data, the deeper the resulting model or theory
Narrative and diagrammatic representations
Synthesis
Do you stop there?
Test your synthesis? Disconfirming cases
Booth A, Carroll C, Illott I. Desperately Seeking Dissonance: Identifying the “Disconfirming Case”, Qualitative Evidence Synthesis, Qualitative Health Research 2012 (in press).
Morse, J. M. The significance of saturation. Qualitative Health Research 1995; 5, 147-149.
Sensitivity analysis By quality, population, location, setting etc. By frequency and “thickness”
Carroll C, Booth A, Lloyd-Jones M. Should we Exclude Inadequately-reported Studies from Qualitative Systematic Reviews? An Evaluation of Sensitivity Analyses in Two Case Study Reviews, Qualitative Health Research 2012; 22(10)
Boeije et al (2011) Making a difference: towards a method for weighing the evidence in a qualitative synthesis, Journal of Evaluation in Clinical Practice 2011 17(4):657-63
What is the purpose of your review and synthesis?
To answer questions and to solve problems that cannot be addressed by quantitative research
To illuminate patient and client experience and to develop theory
To be useful
Learning outcomes
• An understanding of • What we mean by framework synthesis
• What we mean by thematic analysis
• How you conduct framework and thematic synthesis, or “best-fit” framework synthesis
• The strengths and weaknesses of the different methods
• How you “complete” your synthesis
Summary
Extract and interpret data
Synthesis
Test your synthesis
Integrating and writing-up ...
Producing Your Review for Publication
Andrew BoothSchool of Health and Related Research, University of
Sheffield, UK
Framework for systematic reviews of qualitative research (Garside, 2010)
Stage Typical activities
Developing research question Assemble team; Consult; Agree approach
Scoping exercise Identify relevant research; Refine methods
Identifying relevant literature Develop Inclusion/Exclusion criteria; Focused searches; Citation searches
Initial assessment of study reports Preliminary reading; Identify theories; Assess utility/relevance
Analysis and synthesis Reading & rereading study reports; Constant comparison; Assess validity
Preliminary synthesis Categorising; tabulating; mindmaps; Explore relationships
Full synthesis Thematic analysis; translation of findings; Theory development; rival explanations
Dissemination Target audiences; Limitations of review
Throughout Multiple viewpoints; Reflexivity; Audit trail; Ongoing consultation; revisit review purpose
Between Two Traditions
• Systematic Review Publishing Guidelines (e.g. PRISMA – formerly QUOROM)
• Primary Qualitative Research Reporting Traditions
Qualitative Research
Systematic Reviews
QES
Requirements for a Published QES
• Systematic Review requires:• Explicit methods
• Transparency
• Audit Trail
• Review Question
• Search strategies & sources
• Quality Assessment
• Method of Synthesis
• Strategies to reduce bias
• Primary Qualitative Research requires:
• “Believability”
• Findings Grounded in the Data
• Themes/Constructs
• Selective Findings
• Reflexivity
Purpose of Study
Databases and Keywords
Search Strategy
Methods of Appraisal/ Extraction
Methods of Synthesis
PRISMA Flow Diagram
NB. This is non-standard but informed by QUOROM/ PRISMA
Themes
Findings
Studies
Narrative synthesis
Verbatim Extract
Unsubstantiated Comment
Relationship between Themes
Structuring Recommendations
Where to Next?
Andrew BoothSchool of Health and Related Research, University of
Sheffield, UK
Further Reading
Howell Major & Savin-Baden (2010)
Hannes & Lockwood (2011)
Further Resources
http://www.mendeley.com/groups/518691/cochrane-qes-register/
Further Support
http://www.jiscmail.ac.uk/asqus http://cqrmg.cochrane.org/
Further Training
• Look out for SyNTAQuES – II (the Intermediate Version) from the Higher Education Academy
http://www.heacademy.ac.uk/events
• Advanced Training is available from: ESQUIRE13, September 2nd-4th 2013, ScHARR, University of Sheffield
http://www.sheffield.ac.uk/scharr/shortcourseunit
Previous materials at:
http://esquiresheffield.pbworks.com
Some Issues to Think About!
Or it’s not all that simple really!
Qualitative Systematic Review
Systematic Review Qualitative Data
Explicit? (cp. Iterative)
Comprehensive? Qualitative Research?
Reproducible? (cp. Reflexivity/ Subjectivity/ Interpretation)
(cp. Theoretical Saturation; Purposive Sampling)
Particular Type of Study (e.g. all Grounded Theory Studies)?
Follows “System”: from Systematic Review or from Primary Qualitative Research?
With/Without Quantitative Research?
Qualitative Data?