The Sorry State of Science Writing
Seth Mnookin laments a series of embarrassing failures in science writing in recent months but rejoices in the rich dialog that followed.
Seth Mnookin laments a series of embarrassing failures in science writing in recent months but rejoices in the rich dialog that followed.
After a rehashing of the much-discussed Jonah Lehrer saga and pointing to some absurd pseudo-science that has made its way into the highest levels of mainstream American journalism, he points to what to me is a far more damning problem:
Wednesday, a team of researchers based in France published a paper in PLOS ONE titled “Why Most Biomedical Findings Echoed by Newspapers Turn Out to be False: The Case of Attention Deficit Hyperactivity Disorder.” (The paper’s authors were intentionally evoking the title of John P. A. Ioannidis’s groundbreaking 2005 piece, “Why most published research findings are false,” which built off of his earlier JAMA paper, “Contradicted and Initially Stronger Effects in Highly Cited Clinical Research.”) After examining every newspaper report about the ten most covered research papers on ADHD from the 1990s, the authors were able to provide empirical evidence for a troubling phenomenon that seems to be all but baked in to the way our scientific culture operates: We pay lots of attention to things that are almost assuredly not true.
That might sound crazy, but consider: Because it’s sexier to discover something than to show there’s nothing to be discovered, high-impact journals show a marked preference for “initial studies” as opposed to disconfirmations. Unfortunately, as anyone who has ever worked in a research lab knows, initial observations are almost inevitably refuted or heavily attenuated by future studies — and that data tends to get printed in less prestigious journals. Newspapers, meanwhile, give lots of attention to those first, eye-catching results while spilling very little (if any) ink on the ongoing research that shows why people shouldn’t have gotten all hot and bothered in the first place. (I have a high degree of confidence that the same phenomenon occurs regardless of the medium, but the PLOS ONE study only examined print newspapers.) The result? ”[A]n almost complete amnesia in the newspaper coverage of biomedical findings.”
So, to summarize: one of our biggest stars was revealed as a fraud; publications that should be exemplars of nuanced, high-quality reporting are allowing confused speculation to clutter their pages; researchers and PIOs are nudging reporters towards overblown interpretations; and everything we write about will probably end up being wrong anyway — not that we’ll bother to let you know when the time comes.
But he notes that the Internet generally and blogs and social media in particular make it easier than ever before for the good science to get out there:
And yes, the ENCODE coverage highlighted some of deep-rooted flaws in how we value and communicate about science - but the snarled, labyrinthine debate also highlighted the incredible opportunities available to anyone interesting in reading, or writing, about complex scientific issues. Genetics is a subject I know precious little about — and one I hope to write about in the future. Five years ago, it would have been difficult to know where to start. Today, I turned to Princeton genomics and evolutionary biology professor Leonid Kruglyak‘sTwitter stream. Among the many places that directed me was biochemist Mike White’s posts at the Finch and Pea and evolutionary biologist T. Ryan Gregory’s posts on ENCODE at his blog, Evolver Zone: Genomicron. Once I began pulling on those threads, they lead me to computational biologist Sean Eddy’s “ENCODE says what?” post at Cryptogenomicon, the 4,900-word, “My own thoughts” post that Ewan Birney, the lead scientist on the ENCODE project, put up simultaneous to the ENCODE papers’ publication, and Birney’s response to the reactions/backlash that ensued.
It wasn’t until I sat down to write this post that I realized that those are alldocuments written by people who are not only working scientists but also experts in the fields in question. When I began searching out work by science writers, I found subtle, sedulous pieces like Ed Yong’s “ENCODE: The rough guide to the human genome” and Brendan Maher’s “Fighting about ENCODE and junk.”
The end result of all of my reading was manifold: I now have a good grasp of the ENCODE project; I’m aware of some of the big issues facing genetics; I understand why the initial coverage proceeded the way it did, why that coverage was criticized, and how to avoid similar mistakes in my own work in the future; and I have learned of, and in some cases made contact with, a range of dynamic scientists dealing with these issues.
The same can be said for most any field of endeavor: A lot of crap gets out there but virtually all of it is thoroughly debunked by experts who simply had no access to the public debate a little over a decade before.
Alas, I would argue that the latter in no way makes up for the former. For every Seth Mnookin out there joyously seeking out the truth, there are 10,000* consumers who never go beyond the “Good Morning America” coverage. Once stupid factoids get out there, they become permanent truths for some significant portion** of society. So, while this is the greatest time in the history of mankind for wonks, we seem to be firmly into the post-truth era for the vast bulk of information consumers.
__________________
*I’m sure there’s a study out there with this very ratio, but I simply made it up for the purposes of this essay.
**I’m too lazy to make up a percentage here. Go with 27% if you require more specificity; it’s not implausible.
I think a distinction should be made between science writing and science reporting. There are people who attempt to translate science as it becomes established, and they do a pretty good job. On the other hand, science reporting may go for the exciting headline, and not set context and expectations in every piece.
When the Sci Am blog covers science news they do a good job, as science writers, to put it in context and say what it means and what it doesn’t mean. Though Sci Am thinks:
Scientists play a large role in bad medical reporting
The general reader should always be cautioned that a specific study is never an end-point. It may be interesting to hear that coffee prevents (some kind of) cancer, but chances are it will be followed by another study that coffee causes (some other kind of) cancer.
For all the years put in on diet and health there is a lot which is not settled. Particularly because we didn’t have the tools to tackle human microbial interaction. Reporting on what it all means will roll in over the next 20 years or so.
Unfortunately, publications like Scientific American have politicized themselves. Where they used to stick to objective reporting, their pieces are now full of unacknowledged political and social bias.
@John Burgess:
I think you do yourself a disservice, painting with that broad brush. Sci Am is a larger umbrella now that it is no longer just a print magazine and book publisher. Their website has a lot of direct connection blogs to working scientists. Etc.
So, in that broad group, who are you actually singling out for “unacknowledged political and social bias?”
If you didn’t have a link and a solid example, that would be pretty bad.
Some of the best and most popular books are in the field of science:
Brian Greene is a best seller.
Books about string theory and quantum physics are selling well.
@john personna:
I don’t know, but if I had to hazard a guess I would say he is talking about global warming.
@OzarkHillbilly:
I hope not, because anyone who faults Sci Am for that one also needs to fault NASA and the British Royal Society and on and on.
HA!
Not intending to single out one individual but just jumping to the first writer in the Washington Posts’s “Medicine and Science” section, Lena H. Sun. As best as I’m able to determine Ms. Sun is a staff writer.
A quick check of her bibliography shows she has written articles on science, medicine, transport, and politics. I’m guessing that she’s a J-school graduate and hasn’t taken a science course since high school.
If you’re looking for a reason for bad science reporting, that’s it. Most articles are written by staff writers and they’re generalists without specific background and that’s not enough today.
As a non-scientist (comic understatement) I use a Five Year rule. No initial finding is true unless it’s held up for five years. This saves me a lot of trouble at the grocery store.
My second rule is the Bring Out Your Dead rule. Whenever I read or hear that this or that is killing us I check to see whether Eric Idle is collecting dead people in the street.
If the scare headline is still true after 5 years, and if the dead are being hauled away in carts, I begin to think it may be true.
Of course there’s a third rule, the Vice Support Rule, which states that I will treat as true any report which confirms me in a vice.
The problem is the Science News Cycle:
http://www.phdcomics.com/comics/archive.php?comicid=1174
@Dave Schuler: Correct, when trying to summarize gene mapping or the ABC math theorum in eight column inches when all you know is Gregor Mendel’s pea plant studies or basic linear equations, you are bound to get it wrong.
@michael reynolds:
I was kind of hoping for a link and example too.
@john personna:
Those are three very good rules. I note that I tend to follow the third rigorously myself.
Oops, got the quotes in the wrong order above.
Signs and portents: enough to stand your hair on end:
Rare planetary alignment : http://www.csmonitor.com/…/2012/…/Venus-Jupiter-Mercury-and-moon-t.
Sun storm threatens earth: http://www.youtube.com/watch?v=PIg_DZurejI
Asteroid’s near miss ( was it really an asteroid?): http://www.tgdaily.com/space-features/62127-asteroid-near-miss-expected-next-year
Worst drought in US history: http://www.dailymail.co.uk/news/article-2172931/Drought-devastating-26-states-largest-natural-disaster-U-S-history.html
Mississippi River drying up: http://articles.latimes.com/2012/sep/01/nation/la-na-mississippi-river-20120902
Western US bakes: http://www.nytimes.com/2012/06/26/us/heat-driven-wildfires-continue-to-consume-the-west.html
@Delmar:
Good collection, but being science readers we already know what makes our hair stand on end 😉
@john personna: I completely agree. Most science reporting in newspapers and general interest magazines is terrible.
In fairness, I’m not even sure it is possible to have good science reporting, i.e. how can you convey a complex subject to an audience that is flipping through the magazine at the dentist? I had one college biology course that required, as an exercise, to write an abstract of a scientific article. It was an eye opener in that It took days to get it right. In this age, I’m not sure a science journalist is given the time and the editorial support to do a proper article.
Of course, there is some responsibility on the reader’s part. It takes time and attention to read a science article and get a good understanding. I don’t think that in our ADD society many make the effort.
I have it on good authority that we are actually all dead, having been frozen by the CO2 induced ice age predicted in the 1970s.
However, in a stunning reversal of fortune, scientists have been able to teleport us Into a parallel universe, where we actually are still alive. Unfortunately, we are all now doomed by CO2 driven global warming.
Damned if you do, damned if you don’t.
First off, bless you for the 27%, sir.
I wonder if the poor state of science writing is just another manifestation of the marketization of America. The traditional gatekeepers such as JAMA, Scientific American and so forth have to sell themselves much more than before – and the buyer/seller relationship means both sides influencing the other.
When considering the arts, sciences, philosophies, etc. quality and truth are less of a concern than whether or not it sells.
Come to think of it, Goldman Sachs, AIG, et al agree with my thesis! Where’s my jet?
@Drew:
This, of course, demonstrates why bad science reporting is so popular — because its easily animated for political purposes.
It’s pretty easily demonstrated that the entire “Ice Age” during the 70’s was the product of the work of a few researchers being picked up by the press (http://www.realclimate.org/index.php/archives/2005/01/the-global-cooling-myth/). A far greater number of climate researchers were already predicting *warming* at that time. But Time, Newsweek, and National Geo all ran with the cooling research.
The problem is that, rather than talking about the actual research, in particular in Scientific Journals at the time (and subsequently), people who want to deny climate change for personal or political reasons use what the media was poorly reporting to create a potent myth that allows them to “prove” that “climate science doesn’t know what it’s talking about.”
This also speaks to how important it is for actual scientists to be involved in the public sphere and talk about the interpretation of their research.
@Drew:
If it helps, almost certainly every theory we have today in science is as wrong as Newton’s mechanics was in about 1900, when the only two “clouds on the horizon”, to quote Lord Kelvin, was the troubling lack of ether drift, and the irritation of unco-operative black body radiation.
If you want 100% guaranteed predictions, science is a very bad place to look. The theories are adapted, or even discarded all the time. That doesn’t make it political, its just by its nature always going to be trial and error. Scientists predictions are always turning out to be wrong, even in (or perhaps especially in) very basic things.
Theology and formal systems such as math are where you want to look if you want guaranteed predictions. Neither in the past has proven useful in themselves (ie without science) in creating much technology, but there seems to be a real movement in parts of the states to discard anything which is just a theory (such as climate change, evolution, and presumably soon gravity, relativity, quantum mechanics, oxidation-reduction reactions, and everything else in science … because it never gets better than theory).
Blah, blah, blah
Even though our offices are in NY. I refuse to go. The rising seas have made it more like Venice than NY.
My god man, what about my new Allen and Edmonds….
@Drew:
Lol Drew, that’s what Watson said to Crick … “Blah, blah, blah”
@john personna:
Yeah, but Watson and Crick were scientists, and some of their predictions were actually wrong. Obviously not people to be taken seriously, their findings on things like the double helix were no doubt poltitically motivated, or they would have been 100% on all of it.
I really wonder what many of the people complaining about failed predictions in science (Drew comes to mind) think science is – they seem to have no concept (let alone experience) with the idea that even a successful theory is going to get some things wrong, or will be just unable to solve some problems analytically. Even things which sound simple in principle, like the 3-body problem.
Maybe it just comes from only learning about science from elementary and high school text books, where’s its all presented as a obvious givens, and not structures slowly put together by trial and error. I actually like the caveate they wanted in biology class that evolution is just a theory. They should have the same caveate on every physics and chemistry text book as well – everything from F=ma (which is actually wrong, it should be F=dp/dt) to chemical reactions to plant physiology is just theory. There is no proof, no 100% certain law in science. Its all just theory.
@Drew:
You’re behind the curve on this, dude. The cimate change skeptic hired by the Koch Brothers to disprove climate change switched sides, concluded it’s real, and it’s man-made.
But still, I mean, how could the earth be round? We’d fall off, right?
@michael reynolds:
Meh, 5 years for major findings might be a little short; for minor findings it’s not nearly enough. As an early postdoc I worked in a major lab whose most significant finding of the last 5 years (a the time of my arrival) was likely fabricated. I tried repeating the finding w/some minor tweaks and the person who made the findings/write the paper freaked out. I did nothing but made a mental note. I then went to a smaller lab, retested the finding, determined it was definitely false. When I went to write up the paper, my PI wouldn’t allow it, “I can’t afford to piss off your old boss, he’s too powerful”.
Unless you can change that dynamic, science is going to be slow, troubled, and shitty.
Science journalism, OTOH, is mostly just the problem of people who want to make their work seem more impactful running into journalists who can’t possibly understand the details of the work. The better writers enlist the help of anonymous scientists in the field to summarize the benefits/flaws.
Science today, since thats the period being discussed ,is about making money by monopolizing information that can be held hostage and sold for a price. Marketing of this proprietary information is based on fear mongering and exaggeration, two effective methods. Truth isn’t in the equation unless coincidently it is true. Science Journals have sold out and are owned, controlled and influenced by money. One only has to read about Dr Andrew Wakefield to see what happens to a researcher who doesn’t keep information quiet. Then media coverage exaggerated and misinformed the public and Wakefield became the target of MMR makers and government health officials. In the end, we are somehow supposed to believe, according to official sources, that Wakefield’s crucifiction and professional castration means all vaccines are safe. Calling that a stretch is an understatement.
@Ross Coe:
Oh please. The number and depth of epidemiological vaccine studies is incredibly vast. Only the profoundly uninformed (or scientifically ignorant) make claims about the mythic “unknowns” of vaccines.
@Mr. Prosser: “Correct, when trying to summarize gene mapping or the ABC math theorum in eight column inches when all you know is Gregor Mendel’s pea plant studies or basic linear equations, you are bound to get it wrong. ”
I’d bet that 90% of the journalists writing on science know neither of the above.
@Drew: “I have it on good authority that we are actually all dead, having been frozen by the CO2 induced ice age predicted in the 1970s. ”
Nobody, and I mean nobody, predicted a ‘CO2 induced ice age’ in the 1970’s.
@Crusty Dem:
That is a sad story, and seems to confirm the old pessimism that science can move only as fast as old scientists die off.
Well, we hope that old scientists aren’t sitting athwart every path forward.
Often, even that’s not enough. The bad findings usually fall away because A) people can’t replicate the findings do there’s no follow-up and B) when new techniques make obvious the old findings are bogus. In the 10-15 yrs that takes, the original result is generally forgotten and the negative consequences to the scientists doing the bogus work are nil.
Occasionally you get the satisfaction of finding out someone reused a figure (the number 1 way fraud is caught) or digitally manipulated an image, but more often, it’s never caught. Or worse, the culprit is caught and still manages to wriggle free.
Science is real. Our reporting of it isn’t, carried out as it is by lowly-paid journalistic hacks who are at the mercy of some marketing-driven editor determined to “punch up” the article and attract eyeballs.
Go read the original papers. Then follow the results. What the sceptics say, then the counter-sceptics, then the counter-counter-skeptics.
Be humble in your reporting. Half of the silliness is due to editors taking out the caveats and cautions expressed by the researchers. We’re just reporting on the data, folks, and it may be wrong.
Oh, and if they have shoved a graph at you that doesn’t have error bars on the data, ignore everything. It’s a hack press release, written by someone who doesn’t know what science is.
There’s a lot of stuff out there, but the Great Unwashed prefers to have its “science” fed to it in small, easily digestible bits that confirm whatever social theories it already held as gospel truth. Very few Americans have the guts to stand up and say: the data shows something different, my belief was wrong.
Oh, and for those of you who want to know what would disprove evolution? Easy, already answered by J.B.S. Haldane: “Rabbits in the Pre-Cambrian.”
@Crusty Dem: I’m going to waste a small amount of time respondind because you are obviously a lout who thinks he knows about vaccines. Maybe you are ignorant of the fact that most countries have vaccine injury compensation program. Thats all that needs saying.
@Ross Coe:
I’m a lout with a PhD who tried to set up experiments based on the now-discredited vaccine/autism link during my postdoctoral career. I’ve forgotten far more about vaccines than you’ll ever know. I wasted a lot of time because if Wakefield’s fraud (and it was definjtely fraud)
Despite vast attempts to find a substantive link, there’s no there there. Vaccine injury programs exist to refund people injuried from vaccine reactions, most commonly anaphylaxis in response to allergies (for example, some vaccines have chicken egg components).
@Crusty Dem: Tried to set up. Isn’t that the same as doing nothing? So your PHD hasn’t trasfered into itelligence by the sound of it and you come across as a B.S.er. You sound like many I’ve encountered over the lasy 18 years who spout the status quo and puff out their chests like heroes. The main feature about the vaccine/ autism connection is that the science doesn’t yet exist to prove it but we are told by louts it has. Vaccine reactions include seizure disorders, narcolepsy and death. Why its a stretch to think it could cause autism is for the small minded to figure out.
@Ross Coe: As an ivory tower PhD, I have obviously spent much time doing nothing, all paid for with your tax dollars. The time spent reading papers and writing grants? WIth the goal of setting up proper scientific experiments to elucidate the unknown? All wasted compared to a genius, such as yourself, who has clearly learned so much from reading established science by the likes of Wakefield and his genius protege, Jenny McCarthy. Clearly you have felled me with your vague knowledge and bravado. Kudos to you, good sir, for having the strength and integrity to ignorantly dismiss an expert with a mere flick of your wrist. Perhaps one day I can unlearn all that I have studied and become as confident as you. Good day.
@Crusty Dem: Let me guess, you wear clown suits as often as possible. Your PHD doesn’t impress there are hundreds of thousands of PHD’s.