Lessons from reviewing 1,889 stories Gary Schwitzer Publisher HealthNewsReview.org
Transcript
Lessons from reviewing 1,889 stories Gary Schwitzer Publisher
HealthNewsReview.org
Our criteria: Does the story explain Whats the total cost? How
often do benefits occur? How often do harms occur? How strong is
the evidence? Are there alternative choices? Is the condition
exaggerated? Is this really a new approach? Is it available? Whos
promoting this? Do they have a financial conflict of interest? 69%
66% 65% 61% 57% Percent unsatisfactory after 1,889 story reviews 7
years
Ignore or Minimize Potential Harms
Most common flaws Conveying a certainty that doesnt exist
Exaggerating effect size Using causal language to describe
observational studies Failing to explain limitations of surrogate
markers/endpoints Single source stories with no independent
perspective Failing to independently analyze quality of
evidence
Bad science is no excuse for bad journalism.
Exaggerating effect size Absolute vs. Relative Risk Two ways of
saying the same thing One way relative risk reduction makes effect
size seem larger. Risk of something happening in one group relative
to risk in another. Other way absolute risk reduction makes effect
size seem smaller. The chance that something will happen to someone
a ratio of number of people who have an event divided by all of the
people who could have the event. We absolutely think you should use
the absolute figures.
Tamoxifen cut breast cancer rate in half
Chance of breast cancer over 6 yrs. Women on placebo 3.3% Women
on tamoxifen 1.7% 50% relative risk reduction 1.6% absolute risk
reduction OR if 100 women took tamoxifen instead of placebo for 6
years, there would be 2 fewer cases of breast cancer.
Oh, by the way. Tamoxifen users were >2 times more likely to
develop uterine cancer or blood clots in lungs/legs. In relative
risk terms, thats 210% greater risk! (Absolute risk increased from
0.5% up to 1.1%) Net effect for every 1,000 women on tamoxifen: 16
fewer breast cancers 5 more serious blood clots 6 more uterine
cancers
Risk reduction must be stated in absolute, not just relative,
terms NBC News reported, In women, aspirin reduces the rate of
stroke blood clots in the brain, the most common type, by 24% but
had little effect on heart attack risk. In men, the daily aspirin
cuts heart attack risk by 32 percent but had little effect on
stroke risk. HealthNewsReview.org comment: 24 percent of what? And
32 percent of what? The journal article upon which the story is
based did state the absolute values for both benefits and risks.
Over 6.5 years: Daily aspirin is predicted to result in: - 2 fewer
strokes and 2.5 more major bleeds per 1000 women - 8 fewer heart
attacks and 3 more major bleeds per 1000 men. To many viewers,
those numbers will mean more than 32% or 24%.
Last example: absolute vs. relative 3rd graf of NYT story:
About 8 percent of 2- to 5-year-olds were obese in 2012, down from
14 percent in 2004. From the published study: Overall, there have
been no significant changes in obesity prevalence in youth or
adults between 2003-2004 and 2011-2012.
Results suggest a systematic bias of major newspapers
preferentially covering medical research with weaker methodology.
More likely to cover observational studies & less likely to
cover randomized clinical trials When they cover observational
studies, they select those of inferior quality.
Observational study vs. randomized clinical trial Observational
study: observing what happens to people under exposure conditions
that have been self-selected or determined by influences outside
the control of the researcher. The researcher can choose what
exposures to study, but does not influence them. (Susceptible to
bias and confounding factors) Randomized clinical trial: A true
experiment. The researcher controls the exposure and assigns people
randomly to experimental or control group.
Using causal language to describe observational studies
Observational studies cannot prove cause and effect so it is
inaccurate to use terms like benefitsprotectsreduces risk These
studies can only show a statistical association, so all you can say
is that. We offer a primer, Does The Language Fit The Evidence?
Association Versus Causation.
Association Causation
Stories on: Alcohol can make men smarter CBS Eating greater
variety of fruit reduces risk of Type 2 diabetes NYT Small
reduction in sodium in American diet could save half a million
lives/yr. NYT Coffee may protect against heart failure ABC, CBS,
HealthDay, CNN The St. Louis Post Dispatch reminded readers:
Researchers caution, however, that they cant be sure whether these
associations mean that drinking coffee actually makes people live
longer. Coffee can kill you Atlanta magazine Coffee may cut risk
for some cancers CNN Reader comments: I am so fing sick of these
studies. Stop wasting our time. Correlation IS NOT causation!!!!
The statistics book in a class Im taking uses coffee as an example
of statistics run amok.
Surrogate markers/endpoints Measuring an outcome, such as a lab
test, that substitutes for measuring an important life event but
may not be an important finding in itself. A difference to be a
difference must make a difference Drug may reduce cholesterol but
not prevent MI/death Drug may increase bone density but not reduce
hip fractures Cardiac Arrhythima Suppression Trial (CAST) drugs to
reduce premature ventricular contractions. Study stopped early
because drugs led to higher death rate and nonfatal problems than
placebo. Avastin for breast cancer - drug may extend
progression-free survival but not impact overall
Failure to explain limits of surrogate markers USA Today: New
drug may turn back the clock on heart disease. HealthNewsReview.org
analysis: The story focuses excessively on the unprecedented
changes in cholesterol achieved with the drug. these are surrogate
markers that may not reflect any real benefit on the outcomes that
matter to patients. WebMD: Beet Juice Good for Brain: Drinking Beet
Juice Increases Blood Flow to Brain and May Fight Dementia.
HealthNewsReview.org analysis: Surrogate endpoints do not
necessarily reflect on overall health outcomes. And especially not
from a study of 14 people over four days. And from this we get the
headline that drinking beet juice may fight dementia ???
HealthNewsReview.org linked to 10 stories that framed a tiny short-
term study of a biochemical marker of bone turnover as if it were
definitive advice to women that, for example, Two glasses of wine a
day ends menopause misery.
Whom to believe? Framing matters. Dueling news stories on new
prostate cancer radiation therapy: Breakthrough? Or not a home run?
Medical News Today: breakthroughextends lives vs. NY Times Well
blog (quoting independent expert): I think this is a big deal. Its
not a home run, but its a nice advance.
Whom to believe? Framing matters. Dueling news stories on brain
wave scan for ADHD. Important tool? Or waste of money? NJ
Star-Ledger, Fox News, NY Daily News, Bloomberg all gushing praise
such as this should be an important tool quote from MD in the study
vs. NY Times quoted independent expert skeptical about the test LA
Times quoted independent expert The current scientific research
really doesnt support (this) as a diagnostic tool. I would caution
people. ABC News quoting independent expert: I dont know that this
going to help...its going to make people spend money needlessly.
They can charge for it and it gives you a pseudo-scientific basis
for diagnosis.
Whom to believe? Framing matters. Dueling news stories on
melanoma detection device. Helping doctors? Or causing
dissent/polarizing the field? Cleveland Plain Dealer: helps doctors
detect deadly cancer WTVF Nashville & KFSN Fresno among the
stations using a syndicated news services story that claimed the
Cleveland Clinic named the device one of its top medical
innovations of 2013. KDVR Denver allowed a doctor to call it the
biggest advance in melanoma in his 26-year career vs. NY Times
quoted 3 independent experts: 1. This should still be considered to
be in the developmental stage. 2. A dermatologist on the FDA panel
who voted against approval 3. Biostatistician: (The test) just says
everything is positive. I dont think it helps an aggressive doctor
and unaggressive doctors could do just as well if they were more
diligent without the device.
Whom to believe? Framing matters. Dueling news stories on meds
for mild-moderate depression. Give pause about prescribing? Or no
reason to avoid meds? HealthDay: Severely depressed gain most from
antidepressants: But thats no reason to avoid meds for moderate
cases. vs. LA Times: Study finds medication of little help to
patients with mild, moderate depression USA Today (quoting
independent expert): Such findings demonstrate a failure in the
system: These drugs were not thoroughly tested in mild to moderate
depression prior to their approval.
Whom to believe? Framing matters. Dueling news stories on
Alzheimers drug. BY THE SAME NEWS SOURCE! Reverses disease in mice?
Or a flop? MedicalDaily.com: Cancer drug may help reverse
Alzheimers. AND Targretin a flop: Potential Alzheimers drug fails
retrial.
Whom to believe? Framing matters. Dueling news stories on
limitations of colonoscopy. Miss many cancers? Or concerns
overblown? NY Times: Colonoscopies miss many cancers, study finds.
VS. ABC News: The Case for Keeping Colonoscopy: Cancer Experts Say
Concerns Over Colonoscopy Effectiveness May Be Overblown. CBS News
Katie Couric: Dont use this study as an excuse not to get
screened.
Whom to believe? Framing matters. Dueling news stories on
prostate cancer screening study. Reduces deaths? Or isnt saving
lives/value questioned? MSNBC: Regular prostate screening reduces
deaths HealthDay: PSA testing cuts cancer death risk Bloomberg: PSA
screening reduces deaths vs. Associated Press: prostate cancer
screening isnt saving lives, study finds CNN.com: Value of mass
prostate cancer screenings questioned.
The final line of the NEJM article: More information on the
balance of benefits and adverse effects, as well as the
cost-effectiveness, of prostate- cancer screening is needed before
general recommendations can be made. Maybe that should have
received a little more attention.
Framing Star Tribune: Boy, 12, dies after historic transplant
to treat HIV, leukemia. He was in line to become the second person
in the world to be cured of both deadly illnesses by the
extraordinary type of bone marrow transplant, doctors said. Maybe.
But in fact he was in line to die from a transplant that was not
historic because it did not treat his HIV & leukemia.
The tyranny of the anecdote Stories may use only positive,
glowing patient anecdotes but fail to capture trial dropouts,
compliance problems, patient dissatisfaction or the choice to
pursue less aggressive options. In one example, four major US news
sources all used the same breakthrough patient anecdote of one mans
reported improvement from a heart failure stem cell experiment.
This suggests another common problem: public relations people
spoon-feed journalists the patient anecdotes that put an
intervention in the most positive light.
Single source stories & journalism via news release are
unacceptable ABCs Good Morning America asked, Could a cure for
obesity be just a pill away? HealthNewsReview.org analyzed the
sourcing in this story: In a bizarre twist, the reporter used the
potential conflicts of interest of the doctor he interviewed in the
studio as evidence of his expertise. He noted that the doctor is a
consultant to the manufacturer of the pill and then said, Youre the
right man to talk to. Really? In a recent journal publication, this
researcher listed financial ties to 17 drug and device makers,
including the maker of the (pill in question) viewers were not
clearly told that all of the results reported in the story came
from a trial controlled by the company that has not been
independently reviewed. The Wall Street Journal reported,
Depression treatment Cymbalta effective as pain reliever, Lilly
says. HealthNewsReview.org wrote: This 189-word story failed on
almost every count: no independent perspective; a quote apparently
lifted from a company news release; The company news release
actually did a better job of explaining the study than did this
news story.
Stories about screening tests Sir Muir Gray: All screening
programmes do harm; some do good as well. All screening stories
should explain the tradeoffs of potential benefits and potential
harms
Rates of new diagnosis and death for five types of cancer in
the US, 1975-2005. Moynihan R et al. BMJ 2012;344:bmj.e3502 2012 by
British Medical Journal Publishing Group
Unquestioning news coverage of disease awareness campaigns
Movember misinformation in Chicago Sun-Times from star of In
the Bedroom with Dr. Laura Berman She wrote that men arent aware of
the warning signs of prostate cancer and went on to list some,
including frequent lower back pain and swelling of the legs and
feet and weight gain. But there are no warning signs that are
specific for prostate cancer. Her message could create unnecessary
anxiety and lead to unnecessary testing. She mentioned an
industry-funded survey suggesting that men greatly underestimate
their risk of prostate cancer. She did not provide a reference for
that survey. But peer-reviewed published work has shown that men
vastly overestimated such risk. She also wrote as a statement of
fact that more frequent ejaculation could help men to lower their
risk of prostate cancer. At best, the research on this point is
inconclusive, with other studies suggesting that men who have more
ejaculations are more likely to develop prostate cancer. She may
know something about ejaculation but shes shooting blanks on this
one.
Fawning coverage of new technologies (proton beam therapy &
robotic surgery) KERA public radio in Dallas reported on both
technologies in its Battlefield Breakthrough series. Headline:
Zapping Cancer With A Ray-Gun. We wrote: The story presents not one
word of caveats, concerns or limitations. There was no independent
source, no data about efficacy. Instead, there was fascination:
looks like something out of Star Wars. Headline: Robots Slice Time
Under The Knife. We wrote: Star Wars was a theme in that one as
well. In fact, the formula was the same as for part one with proton
more battlefield breakthroughs. Maybe we need to wait a bit to see
how this arms race plays out with evidence and data before
proclaiming battlefield breakthroughs just yet.
Going easy on health business stories Reuters Headline:
Medtronic tests stents for erectile dysfunction. Our comment: An
initial feasibility study in 30 men is worthy of news coverage?
With a company-sponsored investigator as the only source? Not in
our view. Chicago Tribune Headline: Area firms pneumonia treatment
passes test. Our comment: It features the company presidents
positive assessment of the study results but does not seek
independent comment. It fails to make clear that the results of the
trial have not been peer-reviewed or published and that results of
a related trial have not been released at all. 56 New York Times
Headline: Test of Eye Drug Is Said To Show Success in Elderly. Our
comment: The piece shouldve slapped a bigger warning label on this
cocktail of leaked summaries of secret results, speculation, and an
inadequate dash of independent evaluation. Wall Street Journal
Headline: AstraZeneca Says Brilinta Beats Plavix in Clinical Trial.
Our comment: lets a drug company get away with making superiority
claims without releasing data. All information in the story appears
to come from company sources. Los Angeles Times Headline: Drug for
menstrual cramps in the works. Our comment: This story reports that
data were presented at a scientific meeting. But it didnt tell
readers what the data showed. But it did allow a company VP the
only person quoted to say this could be a breakthrough. Wow.
Minneapolis Star Tribune HealthNewsReview.org found four stories
within six weeks in which the Star Tribune established a clear
pattern of writing nice things about local medical device
industries, but did not lead in reporting negative stories (e.g.,
on troubling Medtronic stories in their own back yard, often
scooped by Milwaukee Journal-Sentinel in another state, a smaller
paper.)
Blog roll: Dr. Richard Lehmans weekly journal review on a BMJ
blog. Funny. Hard-nosed. A treasure. Retraction Watch/Embargo
Watch. Knight Science Journalism Tracker Skeptical Scalpel blog.
Not Running A Hospital blog.
1. Not all studies are equal. All have flaws. You and your
audience should appreciate the limitations inherent in any study
design. 2. If you rely on medical journals as a main source of your
news, you are getting and giving an imbalanced view of new
treatments, tests, products and procedures. 3. Both industry
interests and journal policies tend to promote positive findings.
4. If you rely on presentations at scientific meetings as a main
source of your news, you may be promoting claims that have not been
validated by any independent expert sources. 5. There are a growing
number of resources that can help you evaluate claims and evaluate
the underlying science in journal articles, presentations at
scientific meetings, news releases and interviews with
experts.
Good journalism has a subtle feature of reticence. We dont
publish everything we hear. We filter. We curateand this requires
the willingness to not publish things that are unlikely to be true.
Joel Achenbach, Washington Post The Shroud of Turin, pseudoscience
and journalism
8th straight AHCJ workshop discussing HealthNewsReview.org
could be our last