Meta-Analysis

CORRELATION VS. CAUSATION

Correlation and Causation

“The 2011 Chicago Pedestrian Crash Analysis identified a strong correlation between community areas with high numbers of pedestrian crashes and community areas with high crime rates.”

It’s a strange fact that when an area of Chicago has more street crime, it has more car crashes, and when it has less street crime, it has fewer car crashes.

What could explain this strange correlation? Let’s consider 3 explanations.

Street Crime

I had to look up “street crime” because I didn’t know exactly what it meant. Wikipedia quotes London’s police definition:

“Robbery, often called 'mugging', and also includes thefts from victims in the street where property is snatched and the victim is not assaulted.”

So, Like This…

Street Crime Causes Car Accidents

There are lots of ways street crime could be responsible for more car accidents. 1. If criminals are running away, they may be reckless crossing the street. 2. If they drive away in their cars, they may drive dangerously. 3. If they buy cars with stolen money, they may care less about their cars. 4. If there are lots of muggings, people may drive more often to avoid the crime that occurs on the streets.

Another Reason

One person actually had another theory:

5. Street crime results in more people getting shot (if you rob someone with a gun and they don’t give you their money, you shoot them). This means more ambulances on the roads, which tend to drive fast and dangerously.

Car Accidents Cause Streetcrime

What about the other causal direction? Can car accidents cause crime? This is less plausible, but there are certainly possibilities.

1. If you need money to fix your damaged car, maybe you turn to crime. 2. If someone you know is killed in a car crash, maybe you get angry at the world and become a criminal.

One Additional Possibility

I’ve at least heard of crimes that work like this. Someone runs into your car while you’re stopped at a red light or stop sign. You get out to talk with them and they rob you and drive away.

So maybe it’s not street crime causes car crashes or car crashes cause street crime, but car crashes = street crime!

Common Cause Explanations

There are also various common cause explanations, where some variable C causes both car crashes and street crime. This seems reasonable.

What could C be?

1. “Deviant Tendency”

According to this explanation, some people have a “deviant tendency,” which is just a fancy way of saying that they like to break the rules, and they aren’t motivated by the same moral concerns as the rest of us.This deviant tendency, so the explanation goes, causes them to break the rules of the road, which brings about accidents, and to break the rules that prohibit robbery.

2. Poverty

Not having money, or medical care, or food, or whatever can obviously lead people to become criminals who rob other people.

But can poverty cause car crashes? Here are two ways in which this might happen: by causing people to be lack empathy, and by causing people to be fatalists.

Lack of Empathy

Empathy is an emotional connection that we have to other people and animals. It is when other people’s happiness makes us happy and other people’s sadness makes us sad.

Some evidence suggests that poverty can lead people to lack empathy, and be unconcerned with the suffering of others.

Train Deaths in Mumbai (Bombay)

For example, every year in Mumbai, there are six thousand deaths caused by trains. That’s 16 deaths every day in just one city. People die by getting pushed off the platform in front of moving trains, by getting pushed off moving trains, and by getting trampled to death by people crowding onto trains. How can people be more concerned with pushing onto the train than with the people they’re pushing to death?

Empathy

The suggested explanation is that for so long, the people in Mumbai have been so poor, that they have lost their empathetic connection to others. They don’t feel any sadness or loss when someone else is trampled or pushed off a train. Who cares about that guy? I got what I wanted!

A Bad Explanation?

Maybe that’s true, but it might not explain the correlation between street crime and car accidents.

The reason is that rich people lack empathy too. If you have lots of money, you tend to think of other people as “undeserving peasants” and not care what happens to them.

Fatalism

Fatalism is the idea that whatever will happen will happen (“que sera sera”). The future is fixed, and there’s nothing that can change it.

If poverty causes fatalism, then the poor might think that it’s not worth driving safely or using their turn signals, or whatever, because those things can’t change what will happen.

Fatalism

This is a better explanation, because rich people tend not to be fatalists. They believe that their hard work and effort is what made them rich, not merely the unchanging hand of fate. The poor are more likely to view their situation as not due to their own laziness or incompetence, but rather due to outside forces beyond their control.

Multiple Factors

Finally, there’s no reason to believe that the correlation might not result from all of these factors. It might be that sometimes street crime causes car crashes, sometimes car crashes cause street crime, and sometimes poverty causes both car crashes and street crime. When you “add up” all these effects, the correlation between street crime and poverty becomes strong.

The Point

What is the point?

People very often argue from a correlation claim to a causation claim. This is typically an inference to the best explanation. What best explains the correlation between chocolate eating and Nobel prizes? Maybe chocolate causes you to win Nobel prizes.

The Point

As a critical thinker, your job is to evaluate their argument and decide whether to believe the conclusion.

To do this, you have to look to see whether there are better explanations than the one being offered. “It’s more likely,” you might say, “that wealth is the common cause here.”

META-ANALYSES

Meta-Analysis

A meta-analysis is an analysis of analyses.

In clearer terms, it is a study that looks at lots of different experiments that have been conducted on the same problem, and tries to “put together” all of the findings.

Motivation & Example

Sometimes babies are born prematurely (like me).

Unfortunately, premature babies are more likely to suffer and die.

Steroids

In New Zealand, doctors had the idea that giving steroids to premature babies might improve their chances of survival. They did seven separate studies over nine years. Two studies showed some benefit, but five of the studies were unable to reject the null hypothesis, that steroids did not help. As a result, doctors stopped using the treatment.

Blobbogram

Blobbogram

A blobbogram is a summary of a bunch of studies. Each study is represented by a line, so you can see the seven studies that were conducted on steroids in the previous slide. The line down the middle is the “no effect” line: if a study line crosses it, then that study can’t rule out the null hypothesis, that steroids are no better than placebo treatment.

Blobbogram

Lines to the left represent positive findings. Since two studies showed positive effects, we can see two lines that are completely to the left of the “no effect” line. The length of a study’s line represents its confidence: longer lines are more uncertain. Even though 5 studies touch the “no effect” line, there seems to be a trend here: the lines tend to be to the left, positive side.

Meta-Analysis

A meta-analysis is a way of “summing up” all the information contained in different studies of the same thing.

The blue diamond to the left of the “no effect” line represents the combined meaning of all the studies: steroids work to save the lives of premature babies.

Large Effect Size

In fact, the effect size is a reduction in the risk of death between 30% and 50%.Here’s what this means in human terms. In the US, 4 million children are born each year. 12% or about half a million of them are premature. There are about 5,000 neonatal deaths (in the first month of life) due to premature birth and complications associated with it.

Probabilities

So the likelihood that a premature baby (in the US) will die is around 5,000 in 500,000 or 1 in 100 or 1%.

This of course is right now, after we learned to use steroids. So that 1% figure is a 30% to 50% reduction of the earlier figure, which should be 1.4% to 2%.

The Human Cost

That’s up to twice as many deaths. Between 2,150 and 5,000 babies per year died– in the US alone– because they weren’t given life-saving treatment.This happened for eight years after we had all the information to know that the treatment worked. No one had put that information together in a meta-analysis.

Meta-Analysis

Now I hope it’s clear why these things are important. Ben Goldacre called the idea of meta-analyses an idea “that has saved the lives of more people than you will ever meet.” And it’s true!

But what exactly is a meta-analysis and how does it work?

Literature Review

The literature review is usually a search of databases containing abstracts of all the published literature or registered trials, like PubMed, Embase, or Web of Science. Researchers look for studies that have been conducted on certain topics like “breast cancer” and “wine”. This is the stage at which a meta-analysis can go really wrong for two reasons: cherry picking and the file drawer problem

Cherry Picking

If you see cherries in the store, you might notice that most or all of them were ripe and healthy.But you couldn’t conclude that most or all cherries are ripe and healthy. These ones have been selectively picked for good condition.

Cherry Picking

Cherry picking is another name for “the fallacy of incomplete evidence,” and it’s related to confirmation bias and selection bias.

If I want to prove that a treatment works and I pick only those studies that are positive and ignore lots and lots of negative studies, then I’m “cherry picking” my studies.

“Systematic Review”

In the “bad old days” (before the 1980s) review articles were unsystematic, meaning that people writing the reviews included some studies that were relevant, but not all. This resulted in cherry picking: later it was shown that systematic meta-analyses often had the opposite conclusions of unsystematic reviews. A systematic review looks at all the relevant studies, not just some.

The File Drawer Problem

A second and more difficult problem for literature reviews is that the published literature is a biased sample of all the studies that have been done. As we saw last time, there is publication bias against negative results: people tend to publish positive findings, but to leave negative findings unpublished, sitting in the “file drawers” of their offices (metaphorically).

Detecting Publication Bias

One clever way to detect whether publication bias is operative is to make a funnel plot.

A funnel plot is a graph where the x-axis is the effect we’re looking for (number of premature deaths, for example) and the y-axis is how good the study is: how many people are in it, what the “variance” is, that sort of thing.

Good Funnel

Bad Funnel

Inverted V

We always expect the best studies (at the top) to be closest to the actual effect, and the lower-quality studies to be further away from the actual effect– randomly to the left or right of it.

If there’s no publication bias, we get an “inverted V” shape: lower quality studies move further away from the actual effect in both directions.

Publication Bias

If there is publication bias, however, we get only half of an inverted V. The lower quality studies that had negative outcomes weren’t published, so there is no “left half” of the V, only a “right half.Not a lot can be done when there is serious publication bias. This is something we have to do before hand.

Trial Registration

One common idea is that we require every trial that is conducted to be registered before it is conducted. In the registration, it describes the methods, what the researchers are trying to find, and how they intend to measure those effects. If it isn’t registered, it can’t be published when it is completed, and if it’s registered but not published, we can always ask the researchers for their data afterward.

Quality

There is a balance that we want to maintain, however. While it’s good to include more studies in your meta-analysis, it’s bad to include low quality studies, especially since the effects of publication bias are stronger with low quality studies. In addition, low quality studies are subject to other biases as well (that’s why they’re “low quality”) so including them biases our results.

Selection Criteria

The way to maintain this balance is to state in advance the “selection criteria”– the features an experiment must have to be included in the analysis. For example, we might only want studies with control groups, where the control is a placebo and not “no treatment”, and the participants have randomly been assigned to the groups.

Cherry Picking?

Sometimes people with vested interests will argue that selection criteria amount to cherry picking. Studies of bad medicine often don’t have adequate randomization procedures or aren’t placebo controlled (or the placebo is not comparable to the “real” treatment– as when sugar pills are compared to acupuncture).

GMOs

For instance, one meta-analysis showed that GMOs (genetically modified organisms) had the same nutritional value as organic produce. People who are against GMOs argued that the analysis was cherry picked because it only included RCTs and not observational studies.That’s the point! Observational studies are full of biases that RCTs don’t have.

Measuring Quality

Even among the studies that get included, there will be differences in quality. We can measure the quality of the experiments, and report several figures:

What does the meta-analysis say if only the best studies are included? What does it say if all the studies that meet the criteria are included?

Jadad Scores

1. Do the researchers say that the study is randomized? Yes: 1 point, No: 0 points.If yes, was the method described and is it appropriate? Yes: 1 point, No: -1 point.2. Do they say it’s double-blind? Y: +1, N: 0.If yes, described and appropriate? Y: +1, N: -1.3. Did it describe the people who dropped out of the study? Y: +1, N: 0.

The Analysis

Finally, it’s time for the analysis. This involves complicated statistics that aren’t really important for us.

The basic idea is that you take all the results from all the studies and “weight” them according to some measure of the power of the study to detect effects (number of people, e.g.)

“Voting” on the Truth

The idea is that all of the experiments in the literature are going to “vote” on what the truth is. An experiment gets more “votes” if it’s more likely to “know” the truth, and less votes if it’s more likely to be “guessing”. We then use lots of complicated statistics to determine how close the “vote” is likely to get to the truth. If the “vote” is positive, and very likely to be true, then we conclude that there really is a positive effect.

Example Conclusion

This is taken from a meta-analysis of acupuncture literature for pain:“A small analgesic effect of acupuncture was found, which seems to lack clinical relevance and cannot be clearly distinguished from bias. Whether needling at acupuncture points, or at any site, reduces pain independently of the psychological impact of the treatment ritual is unclear.”

Cherry Picking

Sometimes there are lots of meta-analyses on the same topic. Different meta-analyses can make different choices: some might include unpublished data, they might have different selection criteria, and they might use different statistical methods. But we must remember not to cherry pick our meta-analyses! We can’t say “this one says acupuncture doesn’t work” when other ones say “it does” or vice versa.

SUMMARY

Meta-Analysis

Today we learned that lots of inconclusive information can be turned into conclusive knowledge by putting it together in a meta-analysis.

This can save lives– and it has saved lives, more people than you will ever meet in your life!

Methods

We looked at the methods of meta-analysis and saw some ways that they can go wrong. Sometimes this should lead us to reject the result (if the meta-analysis is not systematic, then it is biased) and sometimes it should lead us to be careful in interpreting the result (if there’s publication bias, the results of the meta-analysis are less certain).

Cherry Picking

Importantly, we talked A LOT about cherry picking. The whole goal of science is to remove bias. The reason why we randomize, why we control, why we double blind experiments– is to remove biases. If we then cherry pick our experiments in our meta-analysis or cherry pick which meta-analysis we believe, we have just undone all of our efforts and stuck the bias right back in!

HW6

HW5

In homework 5 I asked you to identify a treatment that was important to you.

Thanks! I learned a lot about your lives, and that was meaningful for me.

HW6

Now I want you to look into the scientific evidence for that treatment.

Don’t worry: you can do it!

Step 1

If necessary, do some quick background research on your illness/ medicine.

For example, if you take panadol for menstrual cramps, you might look up “panadol” and “menstrual cramps” on Wikipedia.

Step 1

Here we learn:

Panadol = paracetamol

Menstrual cramps = dysmenorrhea

Step 2

Go to Google Scholar:

scholar.google.com

Type in your illness and your treatment, maybe with “meta analysis” in the front.

Step 3

What did the researchers do? What studies did they look at? What question were they answering?

Step 4

What did the researchers find? Was your treatment effective for your illness?

Problem #1

“There’s no meta analyses!”

That’s fine. Sometimes science is incomplete. Look at any of the research articles on the topic. Ask the same questions: what did the researchers do? What did they find?

Problem #2

“There’s no research articles!”

That’s fine. Find a similar issue, or some other issue you’re interested in and look into that. If there’s no research on moxibustion, look at acupuncture in general.