How to read incrementality test results: the interval, your threshold and 'no lift'

The point estimate is the single best guess. A test says the channel returned, say, 2.0 in incremental ROAS (iROAS). One trade publication defines iROAS as revenue caused by the ads divided by cost¹. Ordinary ROAS also counts purchases that would have happened anyway. iROAS tries to remove them.

The interval is the range of values the data cannot rule out. A wide interval means the test is unsure. A narrow one means it is not. The width depends on how much data you had and how noisy it was.

The certainty band is how confident the method is that the true value sits in that range. Tools label this differently. A trade-media report on Google's incrementality testing says it rates a test's feasibility as high, medium or low before launch, and uses a Bayesian method, which builds prior expectations into the analysis². Whatever the tool calls it, read the same three things: the best guess, the range and how sure the method is.

Statistical significance checks whether a difference is unlikely to be chance. It does not say whether the difference is big enough to matter. That is why the break-even line comes next.

Less than it sounds like. Lift is the difference in results between an exposed group and a comparable control group. "No lift" means the test did not detect a difference.

That is not the same as "the ads do not work". A trade-media overview of conversion lift reports that Google's interface shows "No lift" below a set confidence level, and that Google says this does not prove the ads failed³. One common reason is that the test was too small to see a modest effect. This is where the minimum detectable effect (MDE) matters: the smallest lift your design could reliably detect.

Our convention separates two cases that a platform label does not:

• A flat result from a well-powered test. The design could have seen a real effect and did not. That is evidence against the channel.
• A flat result from an underpowered test. The test could not tell, either way.

In a conversion lift study, the experiment is run by the ad platform, which designs the test and counts the result. That is useful, and it is also a reason to ask how big the effect had to be before the test could see it.

Compare the interval with break-even ROAS: the lowest ROAS at which ads do not lose money. Our convention for the margin behind it is contribution margin before marketing, meaning revenue minus variable costs such as product cost, shipping, payment fees and returns. Break-even is 1 divided by that margin.

Example (illustrative): contribution margin before marketing is 40%. Break-even iROAS is 1 divided by 0.4, which is 2.5. A test returns a point estimate of 2.0 with an interval from 0.8 to 3.2.

Read it in three steps:

1. Point estimate against break-even. 2.0 sits below 2.5. On its own, that says the channel loses money.
2. Interval against break-even. The interval runs from below the line to above it. The true value could be on either side.
3. Verdict. Inconclusive. The test cannot tell you the channel loses, and it cannot tell you the channel pays.

The same point estimate with an interval from 0.5 to 1.5 would be a different story. The whole range sits below 2.5. That is a clear answer: not worth it at this margin.

Always check the baseline too: the level of sales expected without the activity. Every uplift claim is measured against it.

A noisy test has a wide interval and a jumpy point estimate. Noise cuts both ways.

• Too good. A lucky run produces a high point estimate. You scale on a number that will not repeat.
• Too bad. An unlucky run produces a low one. You cut a channel that was fine.

Small volume is the usual cause. Researchers who studied lift experiments on Facebook note that these studies differ from standard A/B tests, so the usual experimentation literature did not say how to calculate their power and minimum sample size; their paper supplies those calculations⁴. In plain terms: work out the sample size before the test, not after the result disappoints.

There is a second reason results confuse people. They read an incrementality result next to platform ROAS and the two disagree. One analytics vendor says platform ROAS can overstate for activity such as brand search and retargeting, and understate for channels such as YouTube⁵. The gap is not a bug in either number. They measure different things. Our view: do not average them. Pick the one that matches the decision.

Our view: a wide interval is information about the test, not a verdict on the channel.

Say it plainly. "Inconclusive" is a result with a definition.

• What we tested and the decision it feeds.
• The point estimate and interval, against break-even.
• What we can say. For example: the channel is unlikely to be far above break-even.
• What we cannot say. For example: we cannot say it pays.
• What we do now. Keep, trim, re-test or extend.

An inconclusive test is still information. It limits what you can claim, and that is worth knowing before you commit more money. Write the decision rule down before the test starts. Otherwise the rule gets written after you have seen the number.

• Extend if the test is close to conclusive and more time will narrow the interval. Check whether the budget and calendar allow it.
• Re-test if the design was underpowered. Increase the volume, the duration or the contrast between groups.
• Decide with what you have if no plausible result would change your budget. Then the test has done its job, and you act on the cheaper evidence.

Before you spend on another round, read when not to run an incrementality test. Not every decision needs one.