How often do flawed studies end up doing real harm?

November 20, 2026 5 answers
Asked by 5w!kelzavo

Research influences decisions about medicine, diet, crime, and all sorts of other things. What happens when a study is flawed, or when people treat its findings as stronger than they really are? How can we tell the difference?

Answers

3f!zaldqani

A study can be fine for the question it asked and still be misused. For example, observations about pain treatment during a hospital stay cannot tell you much about addiction or opioid use months after discharge.

That distinction matters with claims about opioid safety. Marketing and prescribing decisions can turn limited evidence into a much broader promise, but it would be too simple to blame a crisis on one paper. Drug promotion, prescribing practices, regulation, and other factors all matter too.

Reply to this answer
_e2+kelzaro

Start with the actual outcome and comparison, not just the headline. A treatment described as 90% effective might reduce a risk from 10% to 1%, or from 0.1% to 0.01%. Both are a 90% relative reduction, but the absolute differences are very different.

Also check who was studied, for how long, and whether the study measured a real-world outcome or a proxy. Small samples, missing follow-up, and lots of outcomes tested without adjustment can all make a result less convincing.

4k.dornzelo:

The same goes for claims about products killing bacteria. Check the conditions and which organisms were tested before assuming the claim applies everywhere.

Reply to this answer
8q+farlqeno

Sometimes the issue is what never gets published. If positive studies are easier to find than negative or inconclusive ones, a stack of published papers can give a misleading picture. Selective reporting within a study can do that too.

Look for a trial registration or protocol and compare the planned outcomes with the published results. Systematic reviews can help, especially when they look for unpublished results, although a review is only as useful as the evidence it can find.

Reply to this answer
_x3~torqavo

A result that finds no clear difference is not automatically proof that a treatment does nothing. The study might be too small, measure the wrong outcome, or follow people for too little time. On the other hand, those limitations are not proof that it works either.

That gets especially contentious when the subject is politically charged. It helps to separate what the data actually show from claims about why researchers did or did not publish them.

Reply to this answer
_z8~fendqari

There isn't a simple count of studies that made society worse. A bad paper might go nowhere, while a weak finding repeated in guidelines, advertising, or news coverage can have a much bigger reach.

Replication and correction are part of how research catches mistakes, but they can take time. For a practical check, I'd put less weight on one dramatic study and more on whether independent research using different methods points in the same direction.

Reply to this answer

Add your answer

Answers are reviewed before publication.