Push notifications can be wonderfully direct: a short message, a timely tap, and a user returns to your app or site. But that simplicity hides a lot of strategic complexity. To learn what actually moves users, teams rely on A/B testing, where different versions of a notification are sent to carefully split audiences and measured against clear outcomes.
TLDR: Push notification A/B testing works best when every campaign has a clear schema: audience, timing, message fields, variants, split percentages, and success metrics. For example, an ecommerce app might test “20% off ends tonight” against “Your cart is waiting” with a 50/50 split and find that urgency drives a 14% higher click through rate. Strong campaign fields keep the test clean, while thoughtful variant splits prevent biased or misleading results. The goal is not just to pick a winner, but to build a repeatable learning system.
Why Schema Matters in Push Notification Testing
A push campaign schema is the structured set of fields that defines how a campaign is created, delivered, tested, and measured. Think of it as the blueprint behind the notification. Without a consistent schema, one marketer may label a variant “A,” another may call it “Control,” and a third may forget to record the audience rules. The result is messy reporting and unreliable insights.
A good schema makes every test auditable. If a campaign performs unusually well, the team can trace exactly what was tested, who received it, when it was sent, and which metric determined the winner. This is especially important when push notifications are part of a larger lifecycle program involving email, SMS, in app messages, and ads.
Core Campaign Schema Fields
While schemas vary by platform, most high quality A/B testing setups include several essential fields. These fields help separate creative decisions from delivery logic and analytics.
- Campaign ID: A unique identifier for tracking, reporting, and debugging.
- Campaign name: A human readable label, such as “Cart Recovery Weekend Test.”
- Objective: The main purpose, such as increasing purchases, reactivating dormant users, or boosting feature adoption.
- Audience segment: The users eligible for the test, for example “users with abandoned carts in the last 24 hours.”
- Exclusion rules: Users who should not receive the notification, such as recent purchasers or unsubscribed users.
- Send time: The scheduled delivery time or trigger condition.
- Timezone handling: Whether messages are sent at a fixed global time or each user’s local time.
- Variant definitions: The different message versions being tested.
- Traffic split: The percentage of eligible users assigned to each variant.
- Primary metric: The main measure of success, such as open rate, click through rate, conversion rate, or revenue per user.
- Secondary metrics: Supporting indicators, including opt out rate, session length, or downstream purchases.
- Attribution window: The time period after delivery in which user actions count toward the campaign result.
These fields may sound operational, but they directly affect the quality of your conclusions. If the attribution window is too short, conversions may be missed. If exclusions are unclear, users may receive overlapping messages. If the variant split is undocumented, analysts may not know whether the test was balanced.
Message Fields: What You Actually Test
In push notification A/B testing, the visible message is only one part of the experiment. Still, creative fields are often the most frequently tested because small wording changes can produce meaningful differences.
Common message level fields include:
- Title: The headline or first line of the notification.
- Body copy: The main message text.
- Call to action: The implied or explicit action, such as “Shop now” or “Read more.”
- Image or rich media: Optional visual content that may increase engagement.
- Deep link: The destination opened when the user taps the notification.
- Personalization tokens: Dynamic fields such as first name, product name, city, or loyalty status.
- Sound or badge behavior: Device level elements, often used carefully because they can affect user irritation.
For instance, a streaming app might test two titles: “New episodes are here” versus “Continue the series you loved.” The first version emphasizes novelty; the second emphasizes personal relevance. If the second title wins, the lesson is not merely that one sentence worked better, but that personalized continuity may be a stronger motivator for that audience.
Understanding Variant Splits
A variant split determines what percentage of the eligible audience receives each version of the campaign. The simplest setup is a 50/50 split, where half the users get Variant A and half get Variant B. This works well when risk is low and the team wants a clean comparison.
However, not every test should use equal distribution. If one variant is experimental or potentially risky, teams may use a conservative split such as 90/10, where 90% of users receive the proven control and 10% receive the new idea. This is common when testing aggressive discount language, new personalization logic, or unfamiliar creative formats.
Another approach is a holdout group, where a percentage of users receive no push notification at all. This is useful when the team wants to measure true incremental impact. For example, if Variant A produces a 6% purchase rate and the holdout group produces a 4.8% purchase rate, the notification appears to create a 1.2 percentage point lift. Without the holdout, the team might wrongly assume all purchases were caused by the campaign.
Choosing the Right Split Strategy
The best split strategy depends on the campaign’s purpose, audience size, and risk tolerance. A small audience may require an even split to gather enough data for both variants. A very large audience gives teams more flexibility to test multiple versions or reserve a holdout group.
- Use 50/50 when testing two low risk creative alternatives and you need a straightforward winner.
- Use 80/20 or 90/10 when introducing a bold new message to a sensitive audience.
- Use multivariate splits such as 40/30/30 when comparing several message styles.
- Use a holdout when measuring incremental lift rather than simple engagement.
One common mistake is testing too many variants with too little traffic. If 2,000 users are divided among five variants, each version may receive too small a sample to produce reliable results. The campaign may identify a “winner,” but the result could be random noise. Better testing often means fewer variants, sharper hypotheses, and cleaner measurement.
Key Metrics Beyond Open Rate
Open rate is easy to understand, but it is not always the best measure of success. A clever notification can earn taps without driving valuable behavior. For commercial campaigns, conversion rate, revenue per recipient, or average order value may be more meaningful. For content apps, session depth, article completion, or repeat visits may matter more.
It is also important to monitor negative signals. A variant that increases clicks by 8% but doubles opt outs may not be a sustainable winner. Push notifications occupy a personal space on the user’s device, so short term engagement should be balanced with long term trust.
A Practical Example
Imagine a food delivery app testing a dinner reminder campaign. The audience includes users who ordered at least twice in the past month but have not ordered in seven days. The campaign compares two variants with a 45/45/10 split: Variant A says, “Dinner solved in 30 minutes”, Variant B says, “Your favorite restaurants are nearby”, and 10% of users are held out.
After 24 hours, Variant A has a 9.5% click through rate, Variant B has an 8.1% click through rate, and the holdout group has a 3.2% order rate without any notification. Variant A produces a 5.1% order rate, while Variant B produces 4.4%. The team concludes that convenience focused messaging creates stronger incremental lift than location based messaging for this segment.
Best Practices for Reliable Tests
- Start with a hypothesis: Define what you expect to learn before launching.
- Change one major element at a time: If title, image, timing, and offer all change, you will not know what caused the result.
- Randomize assignment: Users should be randomly placed into variants to avoid audience bias.
- Respect frequency caps: Over messaging can distort results and increase opt outs.
- Document everything: Store schema fields, results, and learnings for future campaigns.
Push notification A/B testing is most powerful when it becomes an ongoing discipline rather than an occasional experiment. Campaign schema fields provide the structure, variant splits provide the comparison, and analytics provide the evidence. When these pieces work together, every notification becomes more than a message; it becomes a chance to understand users better and communicate with them more intelligently.