Summarize this documentation using AI
Key Takeaways
- Incrementality testing measures the revenue a campaign caused, by withholding it from a randomised holdout group and comparing the two.
- It is the correction to attribution, which credits a channel for every purchase it touched, including the ones that would have happened anyway.
- The method is simple: randomise eligible recipients, suppress the message for the control slice, run a full purchase cycle, compare conversion.
- Expect the incremental number to be materially smaller than the attributed number. That gap is not a failure, it is the measurement working.
- Most lifecycle programmes have never run one, which is why so many report returns that finance does not recognise.
Incrementality testing is a measurement method that isolates the revenue a marketing activity actually caused, by randomly withholding that activity from a control group and comparing outcomes against the group that received it. The difference between the two groups is the incremental effect. Everything else would have happened regardless.
It exists because attribution, the default in almost every marketing report, answers a different question and is routinely mistaken for this one.
Attribution against incrementality
Attribution asks which touchpoints a converting customer encountered, then assigns credit among them. Incrementality asks whether the conversion would have happened without the touchpoint at all.
The distinction matters most in lifecycle marketing, where messages are triggered by behaviour that already signals intent. A customer who abandoned a cart and then bought after receiving a reminder gets counted as flow-driven revenue. Some fraction of those customers were always coming back. Attribution cannot tell you which fraction. A holdout can.
This is why a blended email ROI figure of 40:1 is usually fiction. It divides attributed revenue by platform cost, which overstates the numerator and understates the denominator at the same time. We worked the correct model in lifecycle marketing automation ROI in 2026.
How to run an incrementality test
1. Define the eligible population
Everyone who would normally receive the message. Not your whole list, just the people who trigger the flow or match the campaign segment.
2. Randomise the holdout
Assign a slice at random, commonly 10% for an established flow. Random is essential. A holdout drawn from one segment, one region or one signup source measures that segment rather than the flow.
3. Suppress and wait a full purchase cycle
Run it for at least one complete purchase interval for your category. Ending the test at seven days in a category that reorders every ninety measures almost nothing. If you do not know your interval, work it out first with cohort analysis.
4. Compare conversion, then convert to margin
Incremental conversion rate is treatment minus control. Multiply by eligible volume and average order value for incremental revenue, then apply contribution margin. Revenue-based results flatter every programme.
5. Re-test periodically
Incrementality decays. A flow that was highly incremental when launched becomes less so as customers learn to expect it. Annual re-testing on your largest flows is a reasonable cadence.
What good practice looks like
- Test the flows you believe in most. The uncomfortable results are the valuable ones, and the flow nobody questions is usually where the over-attribution hides.
- Hold out at the flow level, not the channel level. Switching off all email teaches you little and costs a lot.
- Accept a permanent small holdout. A standing 5% control on core flows gives you a continuous read rather than a one-off snapshot.
- Size the test honestly. Small flows need long windows to reach a readable difference. If the volume will never produce a signal, say so rather than running a test that cannot conclude.
Why discount flows are the classic case
Win-back and cart recovery flows carrying a discount are where incrementality testing most often changes a decision, because the discount is a direct margin cost applied to a population that includes people who were going to buy at full price.
A win-back sequence can show excellent attributed revenue and negative incremental margin at the same time, if enough recipients would have returned unprompted and simply took the code. That is a real and common result, and it is invisible without a control group. The win-back hierarchy exists precisely to sequence non-discount interventions before the discount.
Common mistakes
- Non-random holdouts. Suppressing a convenient segment measures that segment.
- Contamination. If the holdout still receives the same offer by SMS or on site, you measured nothing. Suppress across channels.
- Stopping early on a good result. Reading a test the moment it looks favourable is how you convert noise into a strategy.
- Reporting revenue instead of margin. The point of the exercise is a number finance will accept.
The bottom line
Incrementality testing is the difference between knowing your lifecycle programme is busy and knowing it is working. It costs a small amount of suppressed revenue and returns a number you can defend, which is a trade worth making on any flow large enough to matter.
Start with one flow, run it for a full purchase cycle, report it on contribution margin, and use the result to re-price everything else. How to audit your lifecycle marketing program covers where it fits in a wider review, and cohort LTV versus blended LTV makes the same argument about lifetime value.
Sources
Frequently Asked Questions
What is incrementality testing?
Incrementality testing is a measurement method that isolates the revenue a marketing activity actually caused, by randomly withholding that activity from a control group and comparing outcomes against the group that received it. The difference between the two groups is the incremental effect. Everything else would have happened regardless, which is the part attribution cannot separate out.
What is the difference between attribution and incrementality?
Attribution asks which touchpoints a converting customer encountered and then assigns credit among them. Incrementality asks whether the conversion would have happened at all without the touchpoint. The gap matters most in lifecycle marketing, where messages are triggered by behaviour that already signals intent. A customer who abandoned a cart and then bought after a reminder is counted as flow-driven revenue, but some share of those customers were always coming back. Only a holdout can size that share.
How do you run an incrementality test?
Five steps. Define the eligible population, meaning everyone who would normally receive the message rather than your whole list. Randomise a holdout, commonly 10% for an established flow. Suppress the message for that group and wait at least one full purchase cycle for your category. Compare conversion rate between treatment and control, multiply by eligible volume and average order value, then apply contribution margin. Finally, re-test periodically, because incrementality decays as customers learn to expect a flow.
How big should an incrementality holdout be?
Ten percent is a common starting point for an established flow with reasonable volume. The right size depends on how much traffic the flow receives and how large a difference you need to detect: small flows need either a larger holdout or a much longer window to produce a readable result. Many mature programmes keep a permanent 5% control on core flows, which gives a continuous read rather than a one-off snapshot.
Why do discount flows need incrementality testing most?
Because the discount is a direct margin cost applied to a population that includes people who were going to buy anyway. A win-back or cart recovery sequence carrying a code can show excellent attributed revenue and negative incremental margin at the same time, if enough recipients would have returned unprompted and simply took the discount. That result is common and completely invisible without a control group.
