A/B testing low traffic ecommerce: how many tests make sense?

An A/B test randomly splits visitors between two versions and compares them on a metric you chose before the test started. Three ideas decide whether it can work.

• Sample size: how many visitors or conversions each version needs for the test to detect a real effect.
• Minimum detectable effect (MDE): the smallest lift the test can reliably detect. One testing-software vendor describes it in its documentation as the smallest improvement you want to detect, and notes that a low MDE needs more traffic³.
• Duration: the time it takes to collect that sample. The same vendor's documentation describes the classic method as total sample divided by traffic, and recommends running for at least one business cycle of seven days⁴.

Say whether your MDE is relative (measured against your current rate) or absolute (in percentage points). Calculators differ, and mixing the two makes a test look shorter or longer than it is.

Here is the planning order:

1. Write down your current conversion rate for the page.
2. Decide the smallest lift worth shipping. Below that, you would not act on the result anyway.
3. Use a sample size calculator to find visitors needed per version.
4. Divide by weekly traffic per version to get the duration.

Example (illustrative). Say the calculator asks for 40,000 visitors per version, and each version gets 5,000 visitors a week. That is eight weeks for one test. Run tests on one page one after another, and that page supports six tests a year at most, if no promotion or stock problem spoils a run.

Notice what the sums do to ambition. If you want to detect a small lift, the test gets long. If the page cannot support a long test, the only effects you can see are big ones.

Statistical significance will not rescue you. It checks whether a difference is unlikely to be chance, at a threshold chosen in advance. It does not say whether the difference is big enough to matter.

The temptation is human. You launch on Monday, check on Wednesday, see a winner and want to bank it.

Peeking means checking a fixed-horizon test before it reaches its planned sample and acting on what you see. Researchers on continuous monitoring of A/B tests show that repeatedly looking and stopping when the result looks significant inflates the false-positive rate. Methods called sequential tests are designed to allow continuous monitoring⁵.

In practice:

• Fix the sample and the duration before launch, and do not call it early.
• Run the test for at least a full weekly cycle, so every day of the week is represented. People buy differently on a Tuesday and a Sunday.
• Avoid starting a test just before a sale or a major campaign. The sale changes who is buying.
• If you must monitor continuously, use a method built for it, not a fixed-horizon test you keep refreshing.

A false winner is worse than no result. You ship it, you stop looking, and the page quietly does no better than before.

With low traffic, you cannot detect small effects. So stop trying.

Test changes big enough to move behaviour: the offer, how delivery and returns are shown, the structure of the product page, what the first screen says. A button colour will never clear the bar.

Convert's argument that low-traffic testing is possible comes with conditions: rigorous pre-test analysis and higher risk margins². We would put that in plain terms. A lower bar for evidence means a higher chance of shipping a change that did nothing. If you accept that, record it, and treat the result as a lean, not proof.

One more trick, with a caveat. You can test against an earlier event, such as add to cart, because it happens more often than purchase. You get more events and a shorter test. But a change can lift add to cart and not orders. If you use the earlier metric, say so before the test and check orders afterwards.

Not testing is not the same as doing nothing.

• Fix. If the problem is clear, such as a payment error or a form that rejects valid input, fix it. No experiment needed.
• Research. Watch sessions, read support emails, run short usability sessions. These find problems. They do not size them.
• Monitor. Ship the change, then compare the metric with a baseline you wrote down beforehand. A before-and-after comparison is weaker than a test, because anything else that changed gets the credit. Say so when you report it.

The order of work matters. Our view: research and clear fixes first, tests second, and only where traffic can carry them.

An inconclusive test result means the test found no statistically significant difference. A testing vendor's glossary defines it that way, and adds that it is not the same as proving there is no effect⁶. The test may have been underpowered, or the effect too small for the set-up to see.

There are three common reasons, and each has a different next step:

• The test was too small. The effect may be real but below your MDE. Next step: test a bigger version of the idea, or stop.
• The effect is smaller than you hoped. The change works a little. Next step: decide whether a small effect is worth keeping.
• The set-up was faulty. Tracking broke, traffic was split unevenly or something else changed. Next step: fix and rerun.

Be careful with slicing the result by device or country after the fact. Slice enough ways and one will look like a winner by chance. Treat slices as ideas for the next test, not as findings.

Our view: keep a simple log of every test. This is our working practice, not an industry standard, and it costs a spreadsheet.

Record the losers and the inconclusive ones. They stop you running the same bad idea again, and they show how many tests your traffic really supports.

For ZFD: we start with what your traffic can detect. If it cannot support tests, we fix clear problems on evidence and say what we are assuming, rather than run a programme that mostly returns "inconclusive".