Medical Study: Don’t Trust Medical Studies
A new study (“Scientific Evidence Underlying the ACC/AHA Clinical Practice Guidelines” by Pierluigi Tricoci; Joseph M. Allen; Judith M. Kramer; Robert M. Califf; Sidney C. Smith Jr) in the prestigious Journal of the American Medical Association, in Stuart Buck‘s words, “analyzes more than two decades of heart care guidelines (that is, the guidelines that your doctor might follow in deciding how to treat you) from the American College of Cardiology and the American Heart Association. The study found that the overwhelming majority of recommendations are not supported by good evidence.”
I don’t have access to the article but Stuart provides the following quote:
Level of evidence provides the link between recommendations and evidence base. Although there is significant variation among individual guidelines in available evidence supporting recommendations, the median of level of evidence A recommendations [i.e., those supported by more than one randomized trial] is only 11% across current guidelines, whereas the most common grade assigned is level of evidence C, indicating little to no objective empirical evidence for the recommended action. . . . Interestingly, our findings are reflective of a specialty — cardiology — that has a large pool of research to draw on for its care recommendations. Guidelines in other medical areas in which large clinical trials are performed less frequently may have an even weaker evidence-based foundation. [emphases original]]
This, from the abstract, is even more damning:
Recommendations issued in current ACC/AHA clinical practice guidelines are largely developed from lower levels of evidence or expert opinion. The proportion of recommendations for which there is no conclusive evidence is also growing. These findings highlight the need to improve the process of writing guidelines and to expand the evidence base from which clinical practice guidelines are derived.
I’ve noted for years that the standards for publication in medical journals are far less rigorous than for publication in social science journals. This, despite the former being far more important and the widespread perception that the latter isn’t really science at all.
bene caca et irruma medicos
(There is an older paper as well, Why Most Published Research Findings Are False. It was covered at Marginal Revolution and Overcoming Bias, IIRC.
Now, here’s something I’d like you to think about. Such studies often… heck… usually… end up influencing our healthcare by becoming a mandated treatment for insurnace purposes… or in the case of government healthcare, a law. You will follow their mandate, regardless of the whole thing being based on limited or non-existant scientific evidence. It’s one way government getting between you and your doctor can ruin your health… for no good reason.
Now, if you think it won’t happen that way, may I point to the increasingly draconian laws based on the supposedly scientific studies surrounding “global warming”?
This is nice to know since my husband had malignant melanoma last year and opted to do just 30 days of high dose Interferon infusions as opposed to the recommended 30 days of high doses followed by a year of low dose self injections because of a couple of studies that we read claiming the year of self injections were not necessary.
The people who do these studies should realize that sometimes we’re really counting on them to not pull information out of their asses and trust that no one in the medical/sciences field would be so irresponsible as to publish false information.
No, you may not.
But its science and science is always right.
Like I have said before, I a big believer in the scientific method. However when people treat it as a religion that is beyond questioning, I get irate, In addition it shows their ignorance.
(Chuckle) Too close to the truth?
Bit, do you understand the difference between statistical studies and physical models?
(We probably need to wrap our heads around Karl Popper, not an easy task, to understand the relative confidence we can have in such things.)
Of course.
But I also understand you can’t transfer massive amounts of wealth on the idea that what warmiong is corruing… or nowdays NOT occurring, is from the SUN. And it would appear without argument that both the WH and congress are all about transferring wealth to the government.
The model as suggested by James article gives us a clear picture of how, through the use of governmental/corporate force, ‘science’ that isn’t science at all, is accepted as unarguable fact. It works both in medicine, in which the government is getting increasingly involved, and in the myth of ‘global warming’, as well.
Doesn’t that argument apply to “truth,” Bit?
If you extend it that way, all of your positions fail the same test … they would be implemented by fallible agents.
(I didn’t think you were an absolute anarchist!)
So the most costly medical system in the world is based on inadequate guidelines. Are the guidelines in other countries any better – or just less expensive?
Actually, I think James missed the point of the article. It doesn’t say that the medical studies are passing along incorrect information, or that their conclusions are invented/bogus/untrustworthy, but that the guidelines which should be based on these studies are not following the empirical evidence. There’s a huge difference between those two assertions.
This has nothing to do with “standards for publication in medical journals”, but rather with the ability of the organizations who should be utilizing study results to do so consistently and correctly.
The headline is misleading. The study did not address (at least not in the freely available sections) the validity or lack thereof of medical studies. It addressed medical guidelines not having sufficient grounding in medical studies. That is the guidelines are not based on the available studies or there are insufficient studies in areas where these guidelines are being written. These are separate and distinct issues.
A potential issue in guidelines may be level of confidence. Are they issuing guidelines because there were some apparent trends in studies that were not statistically significant? (ex/ alpha 0.08)
How is that possible?
The standards used in medical testing are quite rigorous and have resulted in phenomenal outcomes. I seriously doubt that one social science paper in a hundred meets the level of rigor in the typical medical trial.