Extensions to traditional statistical meta-analysis James Thomas Systematic Reviews for Complicated and Complex Questions, ESRC Methods Festival, St Catherine’s College, Oxford, 10 th July 2014. EPPI-Centre Social Science Research Unit Institute of Education University of London - PowerPoint PPT Presentation
(1) Extensions to traditional statistical meta-analysis James Thomas Systematic Reviews for Complicated and Complex Questions, ESRC Methods Festival, St Catherine’s College, Oxford, 10 th July 2014 EPPI-Centre Social Science Research Unit Institute of Education University of London 18 Woburn Square London WC1H 0NR Tel +44 (0)20 7612 6397 Fax +44 (0)20 7612 6400 Email [email protected]Web eppi.ioe.ac.uk/ The EPPI-Centre is part of the Social Science Research Unit at the Institute of Education, University of London
Transcript
(1)
Extensions to traditional statistical meta-analysis
James Thomas
Systematic Reviews for Complicated and Complex Questions, ESRC Methods Festival, St
Catherine’s College, Oxford, 10th July 2014
EPPI-CentreSocial Science Research UnitInstitute of EducationUniversity of London18 Woburn SquareLondon WC1H 0NR
The EPPI-Centre is part of the Social Science Research Unit at the Institute of Education, University of London
(2)
Outline
• What is meta-analysis, and why are ‘extensions’ needed?
• What is complexity? And why is it a challenge in systematic reviews?
• Extensions to traditional meta-analysis when complexity is encountered
• Acknowledgement: presentation draws on: O’Mara-Eves A, Thomas J (2013) Methods for configurational synthesis: extensions to traditional meta-analysis for addressing intervention complexity and contextual variation in reviews. 21st Cochrane Colloquium: Quebec 19-23 September
(3)
Form review team (involve ‘users’)
Formulate review question, conceptual framework and inclusion criteria (develop ‘protocol’)
Search for and identify relevant studiesDescribe studies
Assess study quality (and relevance)
Synthesise findings
Communicate and engage
Map
Synthesis
The common stages of a systematic review; focus here on synthesis
(4)
• Typically are used to address three key research questions:– What is the overall estimate of the size of the
effect and its precision?– Is there heterogeneity across the study effects?– What (if any) variables explain differences
across the study effects (if heterogeneity is present)?
Traditional meta-analytic models
(5)
Aggregative approaches in research
Aggregative reviews predominately add up (aggregate) findings of primary studies to answer a review question…
… to indicate the direction or size of effect
(6)
Newman M, Bird K, Tripney J, Kalra N, Kwan I, Bangpan M, Vigurs C (2010) Understanding the impact of engagement in culture and sport: A systematic review of the learning impacts for young people. London: Department for Culture, Media and Sport. http://culture.gov.uk/images/research/CASE-systematic-review-July10.pdf
RCT forest plot: Does children’s participation in structured arts activities improve their cognitive learning outcomes?
• Current popular meta-analytic methods are limited to:– Questions of differences between
two groups or correlations or variables
– One causal / relational proposition at a time
– One outcome at a time
Are we limiting ourselves?
(8)
• Policymakers and practitioners usually do not ask a single narrow aggregative question
• They begin with a particular problem and ask “what is the best way to achieve outcome X?”
• They also ask “does it vary according to…?” and “What does X mean to Y?”
• And they ask these questions relating to complex problems -
Whose questions are we addressing?
(9)
A complex intervention
• Defined in MRC guidance as: “interventions with several interacting components… Many of the extra problems relate to the difficulty of standardising the design and delivery of the interventions, their sensitivity to features of the local context, the organisational and logistical difficulty of applying experimental methods to service or policy change, and the length and complexity of the causal chains linking intervention with outcome.”
• Craig P et al (2008): Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ 337
Some would say the above is merely complicated…
(10)
Complicated and complex
• Truly complex interventions are best conceptualised as dynamic processes– Virtuous circles– Feedback loops– Non-linear step changes in responses /
outcomes– Multiple ‘routes’ to effectiveness
• Rogers PJ. Using Programme Theory to Evaluate Complicated and Complex Aspects of Interventions. Evaluation. 2008;14(1):29-48
(11)
Complex reviews of social research:
• Start from a given ‘problem’– often a known outcome and population
• Aim to identify a range of possible ‘solutions’• Rarely aim to come to a single answer• Acknowledge that there are rarely replications
of interventions in social research– Use heterogeneity to better understand factors that
influence the impact of interventions– Contain detailed and complex conceptual frameworks
(programme theories etc)• So to configure findings (as well as aggregate)
(12)
Configurative approaches in research
• Configurative reviews predominately arrange (configure) the findings of primary studies to answer the review question….
• … to offer a meaningful picture of what research is telling us
Meta-regression Catalá-López et al. BMC Psychiatry 2012, 12:168
(16)
Network meta-analysis
• Facilitates an estimate of the relative effectiveness of interventions – even when they have not been directly compared with one another in a trial
Thorlund and Mills Systematic Reviews 2012, 1:41
(17)
Problems with ‘traditional’ approaches
• The above approaches work well in some situations, BUT
• There are rarely replications of complex interventions (is it possible to have a genuine replication?); leads to a lack of data
• Even when an analysis has many studies, interventions, contexts etc. all differ– Lots of unexplained heterogeneity
• Symmetrical nature of correlational analysis
(18)
Complexity and correlational analysis
• Correlation is symmetric– When testing for a connection between cause
and effect, also tests equally for absence of cause and absence of effect
• Correlation therefore cannot detect multiple causal pathways*– E.g. Asserting that ‘interventions which are
delivered by peers tend to be effective’ should not require that those not delivered by peers are not effective
* There are usually too few studies to use interaction variables
(19)
Another approach: qualitative comparative analysis (QCA)
• Originally developed by Charles Ragin in political science and historical sociology
• Not a correlational approach• Useful for small numbers of
studies• Focus can be on
combinations of intervention components
• More inductive mode of analysis than above
Thomas J, O’Mara-Eves A, Brunton G (2014) Using Qualitative Comparative Analysis (QCA) in systematic reviews of complex interventions: a worked example. Systematic Reviews. 3: 67
Suggests that intensity & qualityare sufficient to gain effective outcome
(20)
Summary
• Meta-analysis can address a single focused question, requiring a straightforwardly aggregative answer
• “Real world” questions tend to require configuration AND aggregation
• Extensions to meta-analysis are able to configure and aggregate study findings successfully
• But few current methods can cope with genuinely complex situations
