What is statistical significance?
How does statistical significance work?
A test compares two groups, such as people who saw an ad and people who did not. The groups will never be exactly equal. Some gap is expected by luck alone.
Significance asks one question: if the ads did nothing, how likely is a gap at least this large? You set the threshold before the test starts. If the gap is rarer than that threshold, the result is called significant.
That is the second half of the question. A result can be statistically significant and commercially trivial. Check the size of the effect, not only the verdict.
Does "not significant" mean the ads do not work?
No. This is a common misreading. "Not significant" means the test could not separate the effect from chance. The effect may be absent. Or the test may have been too small to see it.
A research chapter on measuring the effects of advertising reports that measuring ad returns can need very large samples, and that results can come with wide intervals1. So a flat result can simply be an inconclusive one. See sample size for why.
Worked example
Easily confused with
Statistical significance vs Sample size
How much data you need. Too little makes significance hard to reach.
Statistical significance vs Minimum detectable effect
The smallest lift a test design can reliably find.
Statistical significance vs Importance
Significance is about chance. Importance is about money. Only you can judge the second.
Statistical significance vs A/B test winner
A significant result in one test is not a promise it will repeat.
FAQ
What is a p-value?
It is the number behind the check. It says how likely a gap at least this large would be if the ads had no effect. A small p-value means a gap like this would be rare if the ads did nothing.
If a result is significant, should we act on it?
Not automatically. Ask how big the effect is, how wide the uncertainty is and what it is worth after margin.
Let’s find what’s actually working.
Next step. Check what your ROAS looks like on explicit assumptions. Use the Is your ROAS real? calculator.