EXPLAINER
What is recoverability testing?
Short answer: Recoverability testing checks whether your planned study could find the effect you care about before you spend money collecting real data. You build a practice version of your study on the computer, hide a known answer inside it, run your real analysis plan, and see if the analysis gets that answer back. If it reliably recovers a signal you planted, the design can recover a real one. If it can't, the study would likely fail no matter how careful you are.
The plain version
Imagine you want to know if a new teaching method raises test scores. You plan to test 30 students. Before you run it, you ask a fair question: if the method really worked, would a study this size even notice?
Recoverability testing answers that. The steps are simple:
- You state the effect you expect and how big it is.
- You make fake data on the computer with that effect built in.
- You run the exact analysis you plan to use on the real data.
- You check: did the analysis find the planted effect? How often? How close was its estimate?
If it finds the planted effect most of the time and gets close to the right number, your design passes. If it misses, or its guesses are far off, the design is at risk or fails — and you fix it before collecting anything.
Why it matters
Most studies are judged after the data is in, when it is too late and too expensive to change the design. Recoverability testing moves the check to the start, when fixing things is cheap. A common result is a clear verdict — PASS, RISK, or FAIL — with suggested fixes like a larger sample, a cleaner measure, or a simpler comparison.
What this is — and what it is not
- It is evidence about your design: can this plan recover a signal, under the assumptions you stated?
- It is not proof that your real result will be true. A planted effect is not a real one.
- It is not a validity check on your measure. It does not prove your test, scale, or survey really captures the thing you care about in real people. (See design evidence vs. validity.)
The honest summary: a passing design test means “this plan can work if our assumptions hold,” not “our answer is correct.”
Bring us one planned study.
We will show where it can recover, where it is at risk, and what has to change.
Request a design review