Incrementality for mid-size ecommerce brands: what to measure first, when to test, what skipping costs

It is the part of your sales that the marketing caused. The rest would have arrived anyway: returning customers, people who already searched for your name, the usual weekly swing.

Platform-reported ROAS does not separate the two. It counts sales the platform credits to its ads. Incrementality asks how many of those sales the ads actually caused. The metric built on that answer is incremental ROAS (iROAS): incremental revenue divided by ad spend. A trade-media explainer defines it this way, and notes that sources argue about what belongs in the denominator². So when someone quotes an iROAS, ask what sits under the line.

For a mid-size brand the practical meaning is simple. You have a finite budget, a finance director, and several channels that all claim the same sales. Incrementality is how you decide which claim to believe first.

Not all of them. Measure where two things are true: the money is meaningful, and the answer would change what you do.

Candidates that usually qualify:

• Branded search. People who type your name were probably coming anyway. Whether the ads add sales is a real question: see is branded search incremental.
• Retargeting. It reaches people who already visited. See is retargeting incremental.
• A channel you are about to scale. Doubling spend on a number you have never checked is a bet, not a plan.
• A channel you are about to cut. A vendor article on platform versus incremental ROAS says the platform number can overstate some channels, such as brand search and retargeting, and understate others, such as YouTube³. The error can run either way, so a cut based on the platform number can be wrong too.

Candidates that usually do not: a small channel, a budget fixed by contract, or a decision you will not change whatever the result. For that last case, read when not to run an incrementality test.

There is more than one. Each trades cost against how much you can trust the answer.

Cheaper is not the same as equivalent. A white paper on 12 large lift studies at Facebook, written by two academics with two Facebook staff, found that observational methods often failed to reproduce the results of the randomised experiments⁴. Before-after comparisons and matched-market tests were among the methods it checked. A table does not settle the choice, though. The money does. For how the test formats differ, see geo test vs conversion lift test.

If your budget is modest, a lighter option can still tell you something. An agency guide for small and mid-sized advertisers suggests pausing ads in one of two similar regions, blackout periods and customer surveys⁵. The agency sells services, so treat it as a practitioner's view. The methods are reasonable if you say what they cannot prove.

A test costs more than its fee. You pay in held-out sales, in time, and in team attention. Set that against the decision.

A test is worth it when:

• the spend behind the decision is large next to the cost of testing;
• a plausible result would move your budget;
• your volume can detect an effect big enough to matter, given the minimum detectable effect (MDE) of the design.

It is not worth it when none of these holds. Small budgets produce wide intervals. Economists who summarised 25 large advertising experiments at Yahoo! reported that even well-designed experiments typically need over a million people to measure an economically meaningful effect with any precision⁶. Most brands have far less data than a platform experiment. An inconclusive result is therefore a likely outcome, and it is still a result: it tells you the effect is not large enough to see at your volume. Plan what you will do with it before you start. How to read incrementality test results covers that.

If you do run one, how to run a geo holdout test is the practical next read. Europe adds its own limits on consent and small markets: see incrementality testing and GDPR.

Because they describe someone else's business. The incrementality factor is a vendor term. Haus defines it as the ratio of incremental conversions to platform-reported conversions: if the platform reported 500 and a test found 300 incremental, the factor is 0.6⁷. It is a handy idea. It is a poor thing to borrow.

Measurement vendors say so themselves. Haus, which sells incrementality tests, writes that benchmarks exist but will not tell you what you need to know, because what works for one business often does not work for another⁸. Sellforte, which sells measurement software, writes that a universal correction factor is rarely defensible³. A factor from a brand with a different margin, audience and channel mix tells you little about yours.

For ZFD: we do not give you a "typical" factor. If we have not measured your channel, we say we have not.

An incremental result is revenue. The decision is about money. So convert it.

Our convention (a proposal, not a market standard): contribution margin is revenue minus the variable costs that move with each order (cost of goods, shipping, payment fees, returns and discounts), before marketing. After marketing, we call it contribution after marketing. Other sources draw the line differently.

Example (illustrative): A channel spends €100,000. Platform-reported ROAS is 4.0, so the platform credits €400,000 in revenue. A test suggests only half of that is incremental, so incremental revenue is €200,000 and iROAS is 2.0. With a contribution margin of 40% of revenue, the incremental revenue leaves €80,000 before marketing. After the €100,000 of spend, the channel loses €20,000. The dashboard showed a 4.0. The bank saw a loss.

Then add time. How long until the spend comes back as margin, and can the business wait? That is the payback period question, and it belongs next to the iROAS, not after it. For how platform and incremental figures differ, see platform-reported ROAS vs incremental ROAS.

• Which decision does this feed, and what would we do with each possible result?
• Is our volume enough to see an effect that matters?
• What does the holdout cost us in sales, time and team attention?
• If the result is inconclusive, what then?
• Does the answer end in contribution margin and payback, or in a nicer ROAS?
• What is the cheaper alternative, and what is its bias?