Incrementality testing tools: how to choose without a ranking
The wider thinking is in our guide to incrementality for mid-size ecommerce brands. This page is about the tooling.
What do these tools do?
Incrementality is the extra sales your marketing caused, over and above what would have happened anyway. A tool in this category helps you design a test, split the audience or regions, collect results and report them.
The two methods you will meet most:
- A holdout test: part of the audience does not see the ads, and you compare results.
- A geo test: ads change in some regions while others stay as they were.
The headline output is usually incremental ROAS (iROAS): revenue the ads caused, divided by ad spend. Some vendors also report an incrementality factor: incremental conversions measured in the test, divided by the conversions the platform reported for the same period.
Why do e-commerce brands use them?
Platforms report what they credit to themselves. A test with a control group asks a different question: what would have happened without the ads.
A study of 15 advertising experiments on Facebook compared observational methods with the experiments. The observational methods often failed to produce the same effects1. That is a reason to test. It is not a reason to test everything.
How do you choose a tool? Criteria, not a ranking
Ask these, in this order:
- What decision will the result change? If the answer is none, stop here.
- Is your volume big enough? The minimum detectable effect (MDE) is the smallest lift a test design can reliably detect. If your likely lift is below it, the test will say nothing.
- Which method fits? Holdout, geo, or something else. Does the tool support the one your channel needs?
- What does the control group cost you? Withheld ads can mean sales lost. Someone has to price that.
- Can you see the method? Ask how groups are built, what the tool assumes and what it does when results are unclear.
- Who owns the data and the readout? You should be able to take both with you.
- Does it allow "inconclusive"? A tool that never returns an unclear result deserves a second look.
How do we implement it?
Name the decision
Write down what the result will change. If the answer is nothing, stop there.
Check volume and cost
Work out the minimum detectable effect. Price the sales you give up in the control group.
Method, then tool
Pick holdout, geo or another method for the decision. An analyst may be enough, so compare tools on criteria only if you need one.
Keep the readout
The data and the readout should be yours. Accept an inconclusive result, and act on a clear one.
When should you not use it?
Do not run a test when volume is too low to detect an effect, when no decision depends on the answer, or when the cost of the control group is more than the decision is worth. We cover this in when not to run an incrementality test. Not promising a test you cannot afford is part of how we work.
What does it cost in effort?
No figures. Tool fees come from vendors, and we do not quote them. The real costs are:
- Sales you give up in the control group.
- Time: tests need to run long enough to read.
- Analyst effort to design the test and interpret it.
- The decision afterwards. A result nobody acts on is the most expensive kind.
Pitfalls
Picking a tool before a question
You end up with a dashboard and no decision.
Borrowing a benchmark
Haus, a testing vendor, says aggregate incrementality benchmarks exist but should not be relied on, because results depend on the business4. Treat any single result as yours, not the market's.
Reading one test as permanent
A result describes one channel, one period and one set-up.
Ignoring an unclear result
"We could not tell" is an answer too.
Our take: Core for the method, Selective for the tool
This is our view, not a market fact.
The method is Core. A control group is the cleanest way we know to answer "what if we stopped?". For a brand that spends meaningful money on paid media, it belongs in the toolkit.
The tool is Selective. Nobody needs a platform on day one. Some tests can be designed and read by an analyst. A tool earns its place when you run tests regularly and want the design, power checks and reporting handled in one place.
One caution about shopping around. Comparison pages are often written by vendors. For example, one testing vendor publishes a comparison of itself against two others, and a separate "best tools" list23. A comparison written by one of the contenders will favour its author's criteria. We do not rank vendors on this page, and we do not present any vendor as a partner.
For ZFD: Incrementality is one method among several, not our whole service. We pick the method that answers your decision. Sometimes that is not a test at all.
FAQ
Do I need a tool to run an incrementality test?
Not always. A tool helps when you test often. A single test can sometimes be designed and read without one.
What is the difference between a holdout test and a geo test?
A holdout test withholds ads from part of the audience. A geo test changes ads in selected regions and uses others as the control.
Can I trust vendor comparison pages?
Read them as vendor content. Where the author is one of the vendors compared, as in the examples above2, judge the criteria, not the ranking.
Can I use an industry benchmark instead of testing?
Use it with care. Haus notes that aggregate benchmarks should not be relied on4.
Is a test result ever inconclusive?
Yes. It is a result too. It can mean the test lacked volume to detect an effect.
Sources
- Gordon, Zettelmeyer, Bhargava, Chapsky: A Comparison of Approaches to Advertising Measurement (Marketing Science 2019) - strona KelloggPrimary source
- Measured vs Haus vs Sellforte (Sellforte)Vendor
- Best Incrementality Testing Tools 2026 (Sellforte)Vendor
- What incrementality benchmarks exist? (Haus)Vendor
Let’s find what’s actually working.
Next step. Deciding whether a test is the right method for your budget? Talk to us about your measurement question. See also our Measurement & incrementality service.