We Doubled Our Referral Program Bonus and it Backfired

Most mobile app growth marketers I’ve spoken to agree, referral as a channel tends to be one of  — if not the — most efficient channel to acquire new customers and high-quality app users. It makes sense, there have been dozens of studies showing consumers trust recommendations from friends and family over any form of paid media. This tends to hold true across verticals, whether you’re marketing a consumer fintech, casual gaming, health & fitness, social networking, or any other type of app 

The most successful word-of-mouth programs typically include either a one-sided or two-sided reward for participants, whether that’s some form of in-app currency, a discount, a few months free of a subscription-based app, or straight-up cash.

Despite referral being a consistently high-performing user acquisition channel, rarely do I hear about growth teams rigorously testing their referral programs as aggressively as they test new paid channels with fresh creatives and copy, or onboarding user flows, or their ASO. It seems as though most teams launch a referral program, carefully optimized with CAC in mind, but rarely test or tweak it outside of the occasional referral promotion every now and then.

Savings Buckets

Hacking the psychology of referrals

I lead the growth team at Yotta, an aspiring neobank and mobile app with a big twist. For every $25 deposited into a users’ FDIC-insured Yotta account, they receive a recurring ticket into our weekly random number drawings with chances to win prizes ranging from $0.10 to the $10 million jackpot. We’re hacking the psychology behind playing the lottery to help people save money and make healthier financial decisions.

While we are constantly testing new mobile user acquisition channels our word of mouth/referral channel is our most scalable, highest converting, and CAC-efficient channel by far. Unlike other consumer financial service mobile apps that most frequently offer cash bonuses ranging from $1 to $100 to users for referring friends and family to download, register for an account, and take an in-app action, we offer our users a referral bonus in the form of tickets. Our referral program scales from 100 bonus tickets earned per new user referred up to 250 tickets earned per new user referred. The invitee also receives a one-time bonus of 100 tickets if they end up downloading, creating an account, and making a first-time deposit.

In user interviews, we’ve found that Yotta users overwhelmingly prefer to earn bonus tickets over a cash referral bonus. Users say they have a strong emotional connection to our tickets — although the odds of winning the $10 million jackpot are very slim, we hear users say that every extra ticket they earn could be that one lucky ticket that wins them a life-changing amount of money. 

Testing our referral strategy

Our growth team hypothesized that if we were to substantially increase the amount of bonus tickets users referring friends and family earned per successful referral we would see an equally sizable lift in daily invites sent and ultimately a lift in our viral coefficient. It seemed like a solid hypothesis, considering the strong emotional attachment to our weekly prize drawing tickets, who wouldn’t want to earn more tickets for referring new users?

We settled on increasing our current referral bonus by 2.5x, from 100 bonus tickets per referral to 250 bonus tickets per referral. As a team we strongly wanted to avoid running this as a before-and-after test, finding they typically yield imprecise results that could be easily skewed by seasonality or by shifts in spend in other acquisition channels. Instead, we onboarded over 4,000 users split real-time into either the control experience with 100 bonus tickets per referral or into the variation experience with 250 bonus tickets per referral. This required changing referral bonus references in several places across our mobile app, creating custom FAQs, and applying several conditional referral logic sequences all dependent upon which experience a user had been segmented into.

Despite the heavy lift required to truly A/B test our referral program, and knowing we would be 2.5x’ing our CAC on referred users, we knew if the opportunity existed to 2x or 3x our new user acquisition numbers from referral as a channel we’d be stoked. A few members of the team thought we’d even see a much higher lift, 5x or 10x in invites sent in the variation experience.

A failed hypothesis reminds us to test everything

Here’s what actually happened. The variation experience with 250 bonus tickets in place of 100 bonus tickets per referral wildly underperformed the control experience. The control outperformed the variation by 15.2% in unique invites sent per user with >90% statistical significance and we saw a similar drop in our viral coefficient. 

The team was shocked. Even if we didn’t see a huge lift in invites sent there was no way we thought we’d actually see a drop in invites and referrals made. 

My hypothesis as to why more than doubling our referral bonus didn’t lead to a statistically significant lift in invites sent, or in viral coefficient, and in fact lead to a significant drop in invites sent is that 100 tickets per referral might be a sweet spot where users feel that it’s a significant number of tickets (equivalent to $2,500 in deposits) and users don’t think twice about it, but at 250 tickets per referral users might feel like tickets are relatively worthless or have a diminished value. But that’s just my best guess. 

In light of these test results we plan on doing a lot more referral program testing and channel optimizing, and I encourage other mobile marketers to test their referral program as aggressively as they test their other user acquisition channels.

 

Trevor Ford is the Head of Growth at Yotta, a mobile app and neobank hacking the psychology behind the lottery to help Americans make healthier financial decisions.  His background is in marketing and growth for consumer-facing brands in the personal finance space, having previously worked at LendingTree and Student Loan Hero.