A holdout on 77,000 depositors showed no incremental deposits, so the budget was stopped in four weeks
Post-view attribution credited retargeting to active depositors with hundreds of thousands of deposits a month. The operator wanted to know how many of those would have happened anyway. The answer took 28 days and $818.
Situation
An iGaming operator in South Asia ran retargeting on its active depositors (the "plr" segment) in Ubidex. By post-view attribution, any deposit within seven days of an impression counts for the campaign. Active depositors see many impressions and deposit often, so attribution alone cannot say whether the campaign adds anything.
What we did
- Randomized split of the depositor base by user id: 38,837 in test, 38,649 in holdout, with the registry kept in the operator's own database.
- The holdout was excluded from every retargeting campaign, not only the test ones, through network IP blacklists, with daily monitoring of impression leakage.
- A 28-day pre-period for CUPED, which removes the groups' baseline difference (test deposited 3.1% less than holdout before the test started).
- Sample-ratio checks, a daily summary in Telegram, and a dashboard with lift and confidence interval.
- A decision on day 28 as planned, with no "run until it becomes significant".
Result (14 September to 11 October 2026)
| Metric | Test | Holdout |
|---|---|---|
| Users | 38,837 | 38,649 |
| Deposits per user, pre-period | 8.57 | 8.85 |
| Deposits per user, test period | 7.81 | 8.06 |
| Deposits in the period | 303,396 | 311,474 |
- CUPED lift -0.95%, 95% CI from -4.09% to +2.19%, p = 0.55. Incremental deposits: within noise (point estimate -2,939; upper bound +6,789).
- Even at the upper bound of the interval an incremental deposit would have cost at least $0.12; at the point estimate there is no effect to pay for.
- Registered non-depositors (350,000 per arm): test below holdout by 26%, CI -70% to +17%, not significant.
- Decision: the campaigns were paused on 28 October and the segment is no longer bought in this network.
What to keep in mind
- The test covered one network (Ubidex) and one segment; other directions and networks need their own holdout.
- Spend from regular campaigns that also reached the test group is not included, so the true cost of the test arm is higher than $818.
- A naive post/pre ratio on a longer window gives a positive point estimate; it sits inside the same confidence interval and is not significant, so we do not report it.
What it means for an operator
An answer to "does it work" in four weeks and a few hundred dollars, instead of months of attribution arguments. Directions with no measurable lift get stopped and the budget moves to ones that have it. The same setup (registry, holdout exclusion, CUPED, SRM checks) is reused for every new direction: reactivation, VIP, new geo.
Measure your retargeting: 6-week pilot with a holdout