What is peeking in A/B testing?
Why is peeking a problem?
A standard test is designed with a planned end. You decide the sample size first, run to that point, then read the result. The statistical significance check assumes you look once, at the end.
Look every day and stop when the number crosses the line, and you give chance many chances to cross it. Early on, small samples swing wildly. Sooner or later one swing looks like a win. The paper "Always Valid Inference", by Johari and co-authors, describes standard p-values as wholly unreliable when testers stop based on continuous monitoring, and proposes p-values and intervals that stay valid whenever you decide1. The authors say the method runs in a large commercial A/B testing platform1.
How do you avoid peeking?
- Fix the plan first. Set the sample size and duration before the test starts. Write them down.
- Do not act on early reads. Look if you must, but do not stop or ship because of it.
- Use methods built for monitoring. Sequential methods are designed to allow continuous monitoring1. A vendor guide to the peeking problem also presents sequential testing as the fix2. It sells an A/B testing tool, so treat it as one view.
- Keep a test log. A record of what was planned and what was stopped shows whether anyone peeked.
Traffic limits make this harder, not easier. A low-traffic shop is tempted to stop early because the wait is long. How many tests make sense is covered in A/B testing low traffic ecommerce.
Worked example
Easily confused with
Peeking (A/B tests) vs A/B test
The experiment itself. Peeking is a mistake in how it is read.
Peeking (A/B tests) vs Statistical significance
The check that peeking inflates.
Peeking (A/B tests) vs Inconclusive test result
No significant difference found. The opposite risk from a false win. It can mean the test was too small for the effect, see minimum detectable effect.
Peeking (A/B tests) vs Early stopping
A near-synonym. Early stopping is fine in a sequential design. It is a problem in a fixed one.
For the bigger picture of testing to improve sales, read how to improve ecommerce conversion rate.
FAQ
Is it ever fine to look at a test early?
Looking is not the problem. Acting on it is. Sequential methods exist to allow monitoring, but they must be chosen before the test starts.
Does peeking only affect A/B tests?
The same logic applies to any test read against a planned sample, including ad tests.
Let’s find what’s actually working.
Next step. Check what your ROAS looks like on explicit assumptions. Use the Is your ROAS real? calculator.