Will my study recover the effect? A pre-data checklist

The checklist

1. Name the effect — and the smallest size worth caring about. Write down what you expect to find and the smallest version of it that would still matter to you. “Some difference” is not enough; pick a number.
2. Write down your assumptions. How much natural noise or variation is there? How many people or units can you collect? How many might drop out? How messy is your measure? Be honest — optimistic numbers give you a false pass.
3. Make practice data with the effect built in. Simulate a dataset on the computer where you know the true answer, using the size and noise from steps 1–2.
4. Run your real analysis plan on the practice data. Use the exact test or model you plan to use later — not a simpler one. The plan you test must be the plan you run.
5. Check recovery. Does the analysis find the planted effect often enough? (This is statistical power — the aim is usually 80–90% of the time.) Are its estimates close to the truth, or biased high or low?
6. Stress-test the assumptions. Change the numbers: smaller effect, more noise, more dropout. Does the design still recover the signal, or does it fall apart the moment reality is less kind than your best case?
7. Decide. Read the verdict like a traffic light:
   • PASS — the design recovers the effect across reasonable assumptions. Collect data.
   • RISK — it works only under your most hopeful numbers. Improve it first.
   • FAIL — it misses the effect even when planted. Do not collect yet.

Common fixes when you don’t pass

• Increase the sample size.
• Use a cleaner, more reliable measure.
• Simplify the comparison or the model.
• Reduce dropout, or plan for it.
• Narrow the question to one the design can actually answer.

What this checklist is not

It does not prove your future result will be true, and it does not prove your measure is valid in real people (see design evidence vs. validity). It tells you one thing well: whether your plan is strong enough to be worth running.