EXPLAINER
Design evidence vs. validity
Short answer: Design evidence says your study plan can recover an effect, assuming your inputs are right. Validity says your measure really reflects the real-world thing, in real people, and your conclusion holds up. They are different. A study can have strong design evidence and still produce an invalid conclusion — for example, if the measure is wrong or the sample is not like the people you care about.
Two different questions
Think of a bathroom scale.
- Design evidence is like checking that the scale’s mechanism can register a change — if you add a known weight, the dial moves the right amount. That is a question about the instrument’s plumbing.
- Validity is the harder question: does the scale read true pounds for real bodies, and would a doctor be right to act on the number?
You can pass the first and fail the second. The mechanism works, but the scale is mis-calibrated, or it was only ever tested on one kind of object.
Why people mix them up
A “PASS” feels like a stamp of truth. It is not. In design testing, a PASS means: given the effect size, noise, and sample we assumed, this design would recover the signal. Every word after “given” is an assumption. If the assumptions are wrong, the PASS still appears — but the real-world conclusion can be false.
Validity is built differently. It comes from many checks over time: does the measure agree with a trusted reference? Does the result repeat in a new, independent sample? Does it predict real outcomes? One study rarely settles it.
What this means for your claims
- A passed design check lets you say: “Our plan can recover this effect under stated assumptions.”
- It does not let you say: “This is true,” “This tool measures X in people,” or “This is safe to act on.”
- For anything that touches health, safety, or money, treat a model as unproven until it is validated on independent, outcome-linked data.
The honest boundary
Good design evidence is necessary but not sufficient. It earns you the right to collect data — not the right to make a validity or real-world claim. Keeping these separate is what protects your credibility.
Bring us one planned study.
We will show where it can recover, where it is at risk, and what has to change.
Request a design review