What is minimum detectable effect (MDE)?
What sets the MDE?
Lift is the difference in results between a group that saw the ads and a comparable control group. The MDE is the smallest lift your test can tell apart from noise. The vendor guide above calculates it for the actual baseline conversion rate, group sizes and test duration1. In practice, these set it:
- Group size. More people or regions in the test and control groups give a smaller MDE.
- Baseline. How often the outcome happens without ads. Rare outcomes are harder to measure.
- Power. The chance the test spots a real effect when there is one.
- Error threshold. How much risk of a false alarm you accept. See statistical significance.
- Test length. A longer test collects more outcomes, at the cost of more time with ads held back.
There is no universal MDE. It depends on the baseline and the group sizes, so we do not give a figure here. Group sizing is covered in sample size.
Why does MDE matter?
So compare two numbers before you start. One is the MDE of the design. The other is the effect you would need to see to change a decision. If the MDE is larger, the test cannot answer your question, however carefully it is run.
A paper on designing incrementality experiments on Facebook notes that lift studies differ from standard A/B tests, and gives its own power and sample size calculation for them2. A geo test works with regions rather than people, so it needs its own sizing too.
For ZFD: If a design cannot detect an effect big enough to matter, we say so before spending the budget. Our view: an honest "this test is too small" beats a confident "no effect".
Worked example
Easily confused with
Minimum detectable effect vs Sample size
How much data you need. MDE is what that data lets you see. They are two sides of one trade-off.
Minimum detectable effect vs Statistical significance
Whether a measured difference is unlikely to be chance. It does not say the effect is big enough to matter.
Minimum detectable effect vs The expected effect
What you hope the ads did. MDE is what the test can see. They are often not the same.
FAQ
Is a smaller MDE always better?
A smaller MDE is more sensitive. It also needs more data, a bigger control group or a longer test, and a bigger holdout withholds more marketing. The aim is an MDE below the effect you need, not the smallest possible.
Can a small brand run a useful test?
Sometimes. It depends on whether the MDE of the design fits the decision. If it does not, another method may serve better. For web tests, see A/B testing with low traffic.
Let’s find what’s actually working.
Next step. Check what your numbers look like on explicit assumptions. Use the Is your ROAS real? calculator. For the concept behind it, start with incrementality.