Product decisions · Worked example
When an add-on replaces revenue you already had
An add-on can sell well while adding very little to the business.
In the hypothetical test below, the add-on contributes $500 per 1,000 assigned users. Other lines lose $450. The add-on must replace that $450; only the remaining $50 is incremental.
Count what the add-on replaces
An add-on creates a new choice inside an existing offer. Some customers pay more. Others replace a subscription, downgrade, or stop buying an older extra. Those movements belong in one readout.
| Source | Control | Offer | Change |
|---|---|---|---|
| Subscriptions | $4,000 | $3,700 | −$300 |
| Existing add-ons | $600 | $450 | −$150 |
| New add-on | $0 | $500 | +$500 |
| Total | $4,600 | $4,650 | +$50 |
Hypothetical figures. Contribution means receipts after refunds, payment fees and direct variable costs.
The new line contributes $500, but the first $450 only replaces contribution the business already had. The table shows the economic mechanism; it does not tell us whether the $50 difference is reliable or whether it survives renewal.
Keep every assigned user in the comparison
A report limited to add-on buyers leaves out people who saw the offer and then bought nothing. Comparing buyers with non-buyers does not fix that: willingness to buy also separates those groups, so their difference cannot tell you what the offer caused.
Assign eligible users before showing the offer, then compare total contribution per assigned user over the same window. Keep zero-spend users in both groups. The control gives you the subscription and older-purchase contribution you would otherwise have had.
Microsoft's guidance on experiments across products makes the same design point: connect treatment assignment across the affected products and keep assigned users even when they produce no activity in one of them.
Decide whether you are testing an add-on or a package
If the treatment reduces an included allowance and sells extra capacity at the same time, a purchase can mean two things: demand for a new benefit, or payment to restore value removed from the subscription. The combined test can answer whether the package earns more, but not which change created the money.
I would define the decision in total contribution per assigned user and write down the maximum acceptable displacement before launch. If renewal matters, the control stays intact through the first relevant renewal; an early readout remains provisional.
If I needed to learn whether extra capacity has demand, I would leave the current allowance unchanged. If I changed the allowance too, I would treat the result as evidence about that complete package and avoid claiming that the add-on itself caused the gain.