Holdout Tests in Customer.io

Overview

Holdout testing in Customer.io is how you prove whether a journey is actually creating incremental revenue, not just taking credit for orders that would have happened anyway. For D2C teams, it is the difference between “our abandoned cart flow drives $X” and “our abandoned cart flow adds $X in lift versus doing nothing.”

A realistic example: you run email and SMS cart recovery, plus paid retargeting. Without a holdout, your journey will claim conversions from shoppers who were already coming back from ads or intent. A holdout gives you a clean baseline so you can decide if the flow deserves more volume, more messages, or a different offer.

Propel helps brands operationalize testing so your optimizations are tied to profit, not vanity attribution. If you want help setting up lift measurement and decision rules, book a strategy call. We implement and optimize Customer.io programs for D2C teams that want measurable incrementality.

How It Works

Holdout testing in Customer.io works by intentionally excluding a fixed percentage of eligible people from receiving messages in a journey step, then comparing outcomes between the messaged group and the holdout group.

In the workflow builder, you add a Holdout Test block where you want to measure impact. Customer.io splits people into two cohorts:

• Test: continues through the journey and receives messages.
• Holdout: is held back from the downstream path you are testing (typically no messages, or a minimal “control” experience if you design it that way).

You then measure conversion and revenue outcomes by cohort to estimate incremental lift. The key is that both cohorts entered the same trigger and met the same eligibility rules, so the comparison is meaningful. For implementation support and measurement design in Customer.io, treat holdouts as a first-class part of your workflow architecture, not an afterthought.

Step-by-Step Setup

Holdout testing in Customer.io is easiest when you decide upfront what you are trying to prove, and where the “moment of influence” is in the journey.

1. Pick one outcome metric (for example, placed order within 72 hours, revenue within 7 days, or repeat purchase within 30 days).
2. Define the eligible population using your trigger and entry filters (for example, “Started Checkout” event, has email or phone, not purchased in the last 24 hours).
3. Place the Holdout Test block right before the messages you want to evaluate (for cart recovery, put it before the first recovery message, not after delays).
4. Set the holdout percentage. Start with 5% to 10% for high-volume flows, and 10% to 20% for lower-volume flows where you still need statistical power.
5. Decide what holdout receives. In most D2C cases, holdout should receive nothing from that flow so you measure the incremental value of messaging, not creative differences.
6. Set a clear measurement window aligned to buying cycle (cart recovery often 24 to 72 hours, post-purchase cross-sell often 7 to 21 days).
7. Publish and run long enough to smooth weekday effects, promos, and ad volatility (usually at least 1 to 2 full business cycles).
8. Read results as lift (conversion rate and revenue per eligible person in Test minus Holdout), then decide whether to scale, tweak, or cut the journey.

When Should You Use This Feature

Holdout testing in Customer.io is most valuable when a journey overlaps with other demand drivers and you need to know what messaging truly adds.

• Abandoned cart recovery with paid retargeting: quantify how much email and SMS adds beyond ads and organic return visits.
• Discount vs no-discount strategy: hold out a portion from receiving the offer to see if the discount is incremental or just margin leakage.
• Post-purchase cross-sell: measure whether a replenishment reminder or bundle recommendation increases second order rate, not just shifts timing.
• Winback and reactivation: confirm whether your reactivation series brings customers back or if they were already returning seasonally.
• Message volume decisions: test whether adding a “final reminder” SMS actually increases revenue or only increases unsubscribes.

Operational Considerations

Holdout testing in Customer.io only stays trustworthy when your data, segmentation, and orchestration rules stay stable during the test window.

• Eligibility rules must be tight: if you let in low-intent traffic, the holdout may look “worse” simply because the audience is noisy. Use intent signals like checkout started, product viewed multiple times, or cart value thresholds.
• Keep other flows from contaminating the test: if holdout users still receive a different cart flow, a broadcast, or a promo SMS, you are no longer measuring the lift of the journey you think you are measuring.
• Define conversion consistently: pick one purchase event source of truth (Shopify order created, paid order, etc.) and keep it consistent across cohorts.
• Watch deliverability and channel eligibility: if Test has more deliverable emails than Holdout due to list hygiene changes mid-test, results will be skewed.
• Plan for promo periods: a sitewide sale can overwhelm the effect size. Either exclude promo windows or run long enough to include them evenly.

Implementation Checklist

Holdout testing in Customer.io runs smoother when you treat it like a measurement project, not just a workflow block.

• Primary metric defined (conversion rate, revenue per eligible person, repeat purchase rate)
• Measurement window defined and documented
• Entry criteria and exclusions locked (including suppression segments)
• Holdout percentage chosen based on volume and expected lift
• Holdout path confirmed to receive no equivalent messaging from other automations
• Tracking events validated (checkout started, order placed, revenue value)
• Reporting plan agreed (weekly readout, decision thresholds)
• Plan for promos and broadcast conflicts documented

Expert Implementation Tips

Holdout testing in Customer.io becomes a revenue lever when you use it to make hard decisions about offers, frequency, and channel mix.

• In retention programs we’ve implemented for D2C brands, the biggest wins come from holdouts on discounting. Many “high performing” cart flows are really just discounting people who would have purchased anyway. Use holdouts to protect margin.
• Place the holdout before any meaningful delay. If you hold out after a 4 hour wait, you are partially testing the delay logic and partially testing messaging, which makes results harder to act on.
• Use revenue per eligible person, not just conversion rate. A small conversion lift with a higher AOV can beat a bigger conversion lift that comes from over-discounting.
• Run separate holdouts for email-only vs SMS-added paths if you are deciding whether SMS is worth the cost and unsubscribe risk.

Common Mistakes to Avoid

Holdout testing in Customer.io can mislead you when execution details are sloppy.

• Testing too many changes at once: if you change creative, offer, timing, and audience while starting a holdout, you will not know what caused the lift.
• Holdout still gets similar messages: overlapping campaigns, broadcasts, or “backup” flows often contaminate the control group.
• Calling it early: cart behavior varies by day of week and pay cycle. Let the test run long enough to stabilize.
• Ignoring unsubscribes and complaints: a lift that comes with a spike in opt-outs can reduce long-term LTV. Track both short-term revenue and list health.
• Measuring the wrong window: a 7-day window for cart recovery can over-credit the flow for unrelated purchases, especially for brands with frequent promos.

Summary

Use holdout tests when you need true incremental lift, especially in cart recovery, winback, and post-purchase upsell. If you cannot prove lift, you cannot confidently scale messaging or protect margin.

Done well, holdouts turn Customer.io into a decision engine, not just a sending tool.

Implement with Propel

Propel can set up holdout tests in Customer.io, align measurement to your buying cycle, and turn results into an optimization roadmap. If you want a testing plan that ties directly to revenue and margin, book a strategy call.