Not all "studies" are equal

Here's the quiet secret that changes how you read every health claim forever: the word "study" covers a hundred-fold range of trustworthiness. A test-tube experiment is a study. A survey of what 500 people remember eating is a study. A double-blind trial where 400 people were randomly given a pill or a placebo is a study. A statistical pooling of 143 such trials is a study. They all get quoted with the same confident phrase — "studies show" — and they are absolutely not the same thing.

Scientists organise this range into a shape: the evidence pyramid. The higher up a claim's evidence sits, the more confident you're entitled to be. Learn the five floors once, and you'll be able to place almost any claim in about five seconds.

The evidence pyramid, from anecdote at the base to meta-analyses at the top Meta-analyses & reviews strongest — pools many trials Randomised trials (RCTs) the gold standard Observational studies cohort, case-control Animal & in-vitro mechanism only Anecdote & opinion weakest — where marketing lives
Higher = more trustworthy; wider = more plentiful. There's an ocean of anecdote and cell-culture data at the bottom and a scarce, hard-won layer of pooled trials at the top. Supplement marketing quotes almost entirely from the bottom two floors — while dressing the language up to sound like the top.

The five floors, bottom to top

  • Anecdote & expert opinion. "It worked for me", "my trainer swears by it", a doctor's hunch. Valuable for generating ideas, worthless as proof — no control, no blinding, endless bias. (Lesson 06.)
  • Animal & in-vitro studies. Cells in a dish, mice on a diet. Essential for understanding how something might work, but a molecule that kills cancer cells in a petri dish usually does nothing useful in a person. (Lesson 05.)
  • Observational studies. Watching what large groups of people already do and tracking outcomes. Good for spotting patterns, weak on cause, because the people who take a supplement differ from those who don't. (Lesson 04.)
  • Randomised controlled trials (RCTs). Randomly assign people to the supplement or a placebo, then compare. Randomisation is what finally lets you say "the pill caused the difference". (Lesson 02.)
  • Systematic reviews & meta-analyses. Gather every decent trial on a question and combine them. The closest thing to a final answer we have — when done well. (Lesson 03.)
⚠
The pyramid is a starting prior, not a verdict

Height is a strong hint, not a guarantee. A tiny, sloppy RCT can be less trustworthy than a huge, careful cohort study, and a meta-analysis of ten weak trials just gives you a precise-looking wrong answer. Use the floor to set your expectations — then check how well that particular study was actually done. The rest of this path teaches exactly that.

Worked example One ingredient, four floors of "proof"

Watch how the same turmeric/curcumin claim changes strength depending on the floor it's standing on:

"Curcumin killed tumour cells in the lab" — floor 2 (in-vitro). True, and nearly meaningless for a human swallowing a capsule, because almost none of it reaches your bloodstream.

"People who eat more turmeric have lower rates of X" — floor 3 (observational). Interesting, but those people differ in diet, income and lifestyle in a hundred ways.

"A randomised trial found curcumin reduced knee-pain scores vs placebo" — floor 4 (RCT). Now we're talking — provided it was big enough and well run.

"A meta-analysis of those RCTs found a modest, consistent benefit" — floor 5. The strongest statement available. Same molecule, four very different levels of confidence — and a label will always quote you the most impressive-sounding one.

You don't need to become a scientist. You need one reflex: whenever you meet "studies show", quietly ask which floor? That single question deflates most of the shelf, and it's the backbone of everything else in this path.

What to remember

When you hear “studies show”, ask one thing: which floor?

  • Five floors, weakest to strongest: anecdote → animal/in-vitro → observational → RCT → meta-analysis. Higher means more trustworthy.
  • Marketing quotes the basement. Cell-dish and "people who eat X" findings get dressed in top-floor language. Height tells you how much confidence you've actually earned.
  • Height is a prior, not a verdict. A sloppy RCT can lose to a careful cohort. Use the floor to set expectations, then judge the individual study — the rest of this path shows how.
Try it · ~1 min

Place three claims on the pyramid.

  1. Find any supplement's "the science" section (a product page or ad works).
  2. For each claim, ask what kind of study it rests on — a dish, mice, a survey, a trial, or a review of trials?
  3. Notice how often the boldest promises sit on the lowest floors. That mismatch is the whole point.

Next in this path

Lesson 02RCTs — why randomisation is the whole point Lesson 07How much evidence to demand, per supplement

References

1
Murad MH, Asi N, Alsawas M, Alahdab F. New evidence pyramid. BMJ Evidence-Based Medicine. 2016;21(4):125–127. doi:10.1136/ebmed-2016-110401
2
OCEBM Levels of Evidence Working Group. The Oxford Levels of Evidence 2. Oxford Centre for Evidence-Based Medicine. cebm.ox.ac.uk

Progress saves automatically in this browser · skip ahead anytime · no streak.