What is an inconclusive A/B test result?
What does inconclusive actually mean?
An A/B test splits an audience between two versions and compares them. Statistical significance checks whether the gap between them is unlikely to be chance alone. If the gap is not clear enough, the result is inconclusive.
That says the test could not separate a real effect from noise. It does not say there was no effect. The same vendor glossary stresses that an inconclusive result is not proof of no effect1.
Why do tests end up inconclusive?
Three reasons come up most.
- Too little data. The test did not have enough visitors or conversions. See sample size.
- The effect was smaller than the test could see. Every test has a minimum detectable effect (MDE): the smallest lift it can reliably detect. A real lift below that line looks like nothing.
- A faulty setup. For example, traffic did not split between versions as planned.
What should you do next?
Do not call it a loss yet. Check what size of effect the test could detect and whether the split worked. Then choose: run it longer, run a bolder version, or stop and spend the traffic on a better idea.
Also avoid peeking: stopping early because a number looked good or bad.
Low traffic makes inconclusive results more likely. Read A/B testing low traffic ecommerce for what is realistic at your traffic. For the wider picture, see how to improve ecommerce conversion rate.
For ZFD: An inconclusive test is a result. We write down what it could and could not detect.
Worked example
Easily confused with
Inconclusive test result vs A losing test
A loss means the new version did worse, with enough evidence to say so.
Inconclusive test result vs A winning test
A win needs significance and an effect big enough to matter.
Inconclusive test result vs Statistical significance
A threshold on chance. It says nothing about how big the effect is.
FAQ
Does inconclusive mean the change did nothing?
No. It means the test could not tell. The effect may be too small for the test, or the test may have been set up badly.
Should I rerun an inconclusive test?
Only after you know why it was inconclusive. Rerunning the same test on the same traffic may give the same answer.
Can I look at segments to find a winner?
Treat segment differences as ideas for the next test, not as findings. Slice the data enough ways and noise will look like a pattern.
Sources
Let’s find what’s actually working.
Next step. Check what your ad numbers look like on explicit assumptions. Use the Is your ROAS real? calculator.