Push Notification A/B Test
Overview
Due to the new iOS 14 update, successful funnel experiments I conducted at Mighty Health started performing worse due to unpredictable fluctuations in audience quality targeted by ads. Through comparative analysis and quick iterating, I was able to increase trial conversions by 58.2% thanks to a unique approach to our push notification strategy.
About
Mighty Health is one of the leading health and wellness apps for older adults. They pair you with a personal health coach to keep you accountable towards your goals.
Problem
We were seeing trial conversions starting to fluctuate with changes to the new iOS updates, causing Facebook ads to worsen audience targeting. With the unpredictability of audience quality, it was more important than ever to make bigger swings on our funnel A/B experiments.
Process
Research
While exploring similar comparators, I discovered a unique pattern that kept coming up: the push notification timeline. Case studies on the success of this pattern (such as Blinkist) sparked a lot of experiments with similar app-based subscriptions. We hypothesized identical results on our own paywall seeing the vast improvement it made in these other platforms.
First Iteration
Our first A/B experiment was to mimic the timeline on our actual paywall to directly compare to our control. While normally we conduct a more thorough design process, our goal for this experiment was to test our hypothesis quickly in hopes of improving dropoff as soon as possible. While prospects were promising, the experiment ultimately ended up underperforming. Our biggest theory on why it failed was due to our target audience needing more upselling to happen on the paywall itself, highlighting more of the features being offered rather than a free trial.
Second Iteration
A/B experiments like these typically need a few iterations before gaining significant insights. While I was hoping for instant success, I was still hopeful that we would find our winning formula using this pattern. I revised my hypothesis based on the theory that our current paywall does a better job of upselling the product and proposed turning the paywall into a notification opt-in that appears before the paywall.
Results
The experiment ended up being a huge success, improving trial conversions by 58.2% and notification opt in by 175%. Our hypothesis around reassuring users of upcoming trials before the paywall ended up creating more trust and confidence to start a free trial that once felt daunting due to the higher price point.
What I learned
It can take multiple iterations of an A/B experiment if you don’t have the number count to test 3-4 variants at a time and reach statistical significance in a relatively short period. With the constraints of a small startup environment, sometimes it’s in better interest to create incremental iterations for faster results and increase the likelihood that an experiment will improve conversions.