Date post: | 28-Jul-2015 |
Category: |
Education |
Upload: | janine-benbouajili-ma |
View: | 40 times |
Download: | 0 times |
CARGO CULT SCIENCE
Feynman spoke of what he called cargo cult science
Cargo cults were observed by anthropologists - formed by South Sea Islanders living in pre-scientific cultures that came into contact with manufactured goods through small aircraft pilots landing their planes
The Islanders wanted the planes (and their cargo) to return, so they built “airplanes” and “runways”
The planes didn’t return as the Islanders hoped
TOOTH FAIRY SCIENCE
Coined by Harriet Hall, MD
“You could measure how much money the Tooth Fairy leaves under the pillow, whether she leaves more cash for the first or last tooth, whether the payoff is greater if you leave the tooth in a plastic baggie versus wrapped in Kleenex. You can get all kinds of good data that is reproducible and statistically significant. Yes, you have learned something. But you haven’t learned what you think you’ve learned, because you haven’t bothered to establish whether the Tooth Fairy really exists.”
ANSWER
The pattern was generated randomly!
“What this ingenious experiment shows is just how bad we are at correctly identifying random sequences. We are wrong about what they should look like: we expect too much alternation, and to us, even truly random sequences seem somehow too lumpy and ordered.” – Ben Goldacre
HUMANS ARE EXPERT PATTERN-FINDERS
Did our propensity for Type 1 errors help us survive as a species?
BUT WE ALSO HAVE A LIMITED VIEWPOINT
“You cannot sense whether a pill improves intelligence, or cures the common cold, or whether MMR causes autism. Your tiny, beautiful ingot of human experience in the world does not present you with sufficient information to spot patterns on that scale: it’s like looking at the ceiling of the Sistine chapel with one eye through a very long cardboard tube.”
-Ben Goldacre
WHAT CAN STATISTICS DO FOR US? When used correctly…
Separate the signal from the noise
Reveal the pattern in the chaos
Find the enduring among the transient
THE UNIVERSE IS A BIG PLACE
One of the fundamental issues of scientific inquiry is we can’t just measure things directly
There are usually far too many instances of what we would like to observe to examine each one
We can’t gather all people with a particular disease at our hospital to undergo an intervention or answer our questions
So we have to resort to taking samples of the total population
THAT DRESSWhat if we want to know how many people in the Cincinnati metro area see that dress as blue and black and how many see it as white and gold?
We can’t show that dress to and poll 2.2 million people (although some people attempted this through social media)
If we take a sample, however, it might or might not be representative of the true proportions of people who see the 2 different color combos
If we take repeated samples and average, we will get closer, but taking repeated samples is not usually practical due to limited time and resources
We have to factor in ERROR
We have to rely on PROBABILITIES
IS OUR MODEL ANY GOOD?
Question: How well does it explain the outcome data based on our hypothesis?
The hypothesis must be decided in advance of data collection
The statistical model must reflect the alternate hypothesis
H0 = There is no relationship between age and seeing that dress as blue and black
HA = There is a relationship between age and seeing that dress as blue and black
ASSESSING GOODNESS What is the probability that the alternate hypothesis is WRONG?
What is the probability the data can be explained equally well by just CHANCE?
Recall:
predicted outcome = model + error
Need a test statistic
test statistic = =
TEST STATISTIC Recall that the whole point of this exercise is to determine what is (most likely) happening in the real world
The test statistic is based on known distributions
If the probability of obtaining the test statistic is low (conventionally < .05), then we say that we fail to reject the null hypothesis
We have some confidence that the alternate hypothesis reflects something happening in the real world and our model is a good one
FACING REALITY
Type 1 error:
We think there is an effect in the real world population based on our test statistic but there is none
alpha = .05
Acceptable level of Type 1 error: 5 out of 100 collections of sample data will result in a test statistic that leads us to conclude there is a real effect when there is none
Type 2 error:
We think there is no effect in the real world population based on our model but there is one
beta = .20
Acceptable level of Type 2 error: 20 out of 100 collection of sample data will lead us to conclude there is no real effect when there is one