Common Mistakes in Empiric Reading: How to Avoid Them and Improve Your Research Skills

Reading evidence badly is easy to do without noticing, because the mistakes feel like reasonable interpretation in the moment. You draw a conclusion the data does not support, trust a number that means less than it seems, or accept a causal story that the study never earned. This article catalogs the most common errors in evidence-based reading and, more usefully, gives you concrete ways to catch each one before it distorts what you take away.

Want expert help putting this into practice? EmpireReading can guide you through it.

Mistaking correlation for causation

This is the error that launches a thousand bad conclusions. A study finds that people who do X tend to have outcome Y, and the reader concludes that X causes Y. But correlation alone cannot establish that, because a third factor may drive both, or the causation may run backward. People who take vitamins are healthier, but they also tend to exercise, eat better, and have more money, any of which could explain the health difference.

To avoid it, train yourself to pause at every causal verb, caused, boosted, reduced, and ask what kind of study produced it. If the design is observational, the honest reading is "these things are associated," not "one causes the other." Only randomized experiments, or careful causal methods, license causal language. When you catch yourself telling a cause-and-effect story about a mere association, stop and downgrade the claim. A useful test is to actively invent a third explanation: for any "X causes Y" claim built on a correlation, spend thirty seconds asking what kind of person does more of X and whether that same kind of person would have more Y anyway. If you can name a plausible hidden factor, the causal story is not yet earned.

Being fooled by statistical significance

Related: Empirereading - Expert Advice for Informed Decision-Making.

"Statistically significant" sounds decisive, but it only means the result is unlikely to be pure chance; it says nothing about whether the effect is large enough to matter. A huge study can find a statistically significant difference so tiny it is meaningless in practice. Readers routinely treat significance as a stamp of importance, when the two are entirely separate questions.

The fix is to always look for the effect size, the actual magnitude, in units you understand, and judge whether it is big enough to care about. A weight-loss method with a "significant" effect of half a pound over a year is significant and useless. Ask two questions, not one: is the effect real, and is it large? Both must be yes before a finding should change your behavior.

Ignoring the sample and its limits

Every study is run on specific people in specific conditions, and its conclusions travel only as far as that sample resembles the situation you care about. A common mistake is generalizing a finding from 40 college students to the entire population, or from a two-week trial to lifelong use. The result may be perfectly valid for who was studied and irrelevant to you.

Before applying any finding, check the sample size, the population, the duration, and the setting, and ask honestly whether you belong to it. Small samples produce noisy, unreliable estimates that often shrink or vanish in larger studies. If the people studied differ sharply from your context, treat the finding as a lead to investigate, not a conclusion to act on. "Shown for whom, over what period" should be a reflex question.

Falling for cherry-picked and one-off results

See also: Empirereading Best Practices: Enhancing Data-Driven Decision Making.

A single striking study is the least reliable form of evidence, yet it is the most likely to make headlines. Individual findings are noisy, and a substantial fraction fail to replicate when others try to repeat them. Building a belief on one dramatic result, especially one that contradicts a larger body of prior work, is a recipe for being wrong in an exciting direction.

Cherry-picking compounds this. A study that measures many outcomes will find some "significant" by chance, and authors or reporters may spotlight the one flattering result while ignoring the rest. Guard against both by preferring the weight of evidence, a systematic review or a consistent body of studies, over any lone finding, and by asking whether a celebrated result was the study's stated main outcome or a lucky secondary one.

A pre-flight checklist before you trust a finding

You can catch most of these errors with a short routine applied before you accept any evidence-based claim. Run through it whenever a finding is important enough to act on.

• Design: is it experimental or observational, and does that match the causal claim being made?
• Effect size: how big is the effect in plain units, not just whether it is significant?
• Precision: how wide is the uncertainty, the confidence interval or margin of error?
• Sample: who was studied, how many, for how long, and do I resemble them?
• Replication: is this one study or a consistent body of evidence?
• Comparison: what baseline or control is the result measured against?

If a claim cannot survive these six questions, it is not settled knowledge, however confidently it was presented. Most misreadings come from skipping this check because the conclusion felt intuitively right.

Watch for the errors that come from inside your own head

Not all mistakes are in the text; many are in the reader. Confirmation bias makes us accept evidence that fits our existing beliefs and scrutinize evidence that does not, so we end up "learning" mostly what we already thought. The antidote is to apply your harshest reading to the studies you most want to be true, and your fairest reading to those you would rather dismiss.

Two related traps round out the list. First, relative-versus-absolute framing: a "50 percent risk reduction" can mean a change from two in a thousand to one in a thousand, so always convert to absolute terms before reacting. Second, trusting summaries over sources: newsletters and headlines routinely drop the caveats, so when a claim matters, trace it upstream to the original. Practicing these checks on a steady diet of accurate, evidence-first material like that found at EmpireReading turns them from effortful into automatic. The reward is not cynicism but calibration: you end up believing true things more strongly and false things less, which is exactly what good reading is for.