Meta-analysis: Mega-silly or mega-useful?

- Hans Eysenck

Research question: Is there an abnormal cytokine profile in autism?

Yes - 2 studies No - 6 studies

Study 1 Study 3Study 2 Study 4

Study 5

Study 6

Study 7

Study 8

6 studies indicate ‘no’ so should we conclude there’s no abnormal cytokine profile in autism?

Yes No

Study 1 (n = 200; clinical diagnosis)

Study 3 (n = 10; self-report)

Study 2 (n = 100; clinical diagnosis

Study 4 (n = 8; self-report)

Study 5 (n = 13; self-report)

Study 6 (n = 5; self-report)Study 7 (n = 15; self-report)

Study 8 (n = 17; self-report)

Big differences in study quality but are the 2 ‘yes’ studies worth more than then 6 ‘no’ studies?

Meta-analysis is an objective and transparent technique to synthesise data from a number of related studies.

Doing a meta-analysis isn’t particularly hard, it’s just hard work.

How do you interpret a meta-analysis?

It’s very easy for others to ‘game’ a meta-analysis to get the outcome they want - watch out for this.

9 Circles of scientific hell

‘Sins’ that can influence the data in your

meta-analysis

Sins that are often

overlooked in meta-analysis

This letter from Eysenck put meta-analysis on the map - his concerns are still valid today.

1. How did they search for articles?

2. What was their inclusion criteria?

Garbage in, garbage out

3. Were the studies homogenous?

4. Did they account for publication bias?

4. Do the authors have an agenda to push? Conflicts of interest even more important here.

How do you do a meta-analysis?

If you want the theory read these two books

1. Have a good research question

•Is there a debate in the literature? •Perhaps a research question is settled but you want to look

at a moderator

2. Pilot your search terms•Too broad and you’ll be swamped, too narrow and you’ll

miss papers •Use relevant databases (Pubmed + Embase will have you

covered) •Also a good ‘feasibility’ check

3. Document everything!•Can someone reading your paper recreate your analysis? •This makes your analysis transparent

3. Extract the data•Can help having an ‘data extraction’ form where you enter

important study details •Gold standard is having 2 people do this and a third

adjudicating any disagreement

There’s a few software packages you can use;

• Comprehensive meta-analysis (recommended)• R packages (tricky but more flexibility with figures)• An excel spreadsheet that comes with Cumming (2014)

You can extract almost any data to create a common effect size

• P-values and sample size• Means and SDs• Correlation coefficients (‘easiest’ meta-analysis)• Still not enough info? Contact the author!

•Most authors oblige (it’s a citation!)•Not likely they’ll have data if older than 10 years

The software/package will calculate common effect sizes (even if you’re extracting different types of data) and then calculate a summary effect size

Forest plot

sub-summary effect size (i.e., what the

overall impact of one cytokine?

Overall effect size (i.e., what’s the summary of

ALL studies?)

Publication bias?•Are there ‘missing’ studies? •A scatterplot of standard error against individual effect size •Large studies tend to have small SE (near top) •There should be an even spread (especially near the bottom)

Should be about 4 more studies here

What happens if there’s bias?•You can impute the missing studies and re-analyse •If your overall conclusions don’t change with the inclusion of the

studies you’re in the clear

Imputed studies

Forest plot

Are these different?

Here you can get some clues as to which factors are driving a result (i.e., is this due to one cytokine?)

Other common moderator analyses

• Gender - is this only found in one gender?• Age - is this stronger/weaker in older people?• Study quality - what’s the effect of ‘bad’ studies?• Different types of measures • Clinical groups - e.g., Bipolar vs. schizophrenia

Meta-analysis is a better approach than a ‘traditional’ narrative review, in most cases.

It’s also possible to do meta-analysis with brain imaging data but this is for another time

Questions?

If your thinking of doing a meta-analysis I’d be happy to help!

http://xkcd.com/1447/