Felipe Ruiz, home

Writing · 3 min

How to know if gamification works.

Players who use your reward feature always look better than players who do not. That tells you nothing. Here is how we built a comparison fair enough to defend.

The first number that came out of our daily rewards wheel looked great, and I did not trust it. Players who spun the wheel deposited far more than players who did not. Any feature that rewards people for showing up will show that gap, because the people who show up were already the best players.

The trap: the group chose itself

A player who logs in every day finds the daily spin. A player who opens the app once a month never does. If you compare the two after launch, you are mostly measuring who they were before.

A dashboard that compares “users of the feature” with “non-users” will show a big gain for almost anything. If you decide the budget on it, you will build the budget on a number that is inflated.

Step one: a real test, while you still can

The cleanest answer is a controlled test. We rolled the wheel out in waves, and in the third wave 30% of players had it and another 30% did not, split by the last digits of the player ID, with the same rules and the same season. After 38 days the group with the wheel had about 12% more turnover per player. That was a good first signal, and it was enough to take the wheel to everyone.

But once everyone has the feature, there is no control group left. And a test with two groups does not say what the feature earns once it is part of the product.

Step two: find a twin for each player

After the full launch, the data team looked for a twin for every player who used the wheel: a player in the same value segment, with similar turnover, deposits, active days and time since sign-up. They checked afterwards that the two groups looked alike, and found a twin for about 90% of players.

Then they compared the change, not the level. How much did each group change between the 14 weeks before the launch and the 14 weeks after? Growth that came from the season and the year-end campaigns happened to both, so it cancels out. What is left belongs to the feature.

Step three: try to break your own result

Two checks mattered most. First, the two groups had to move together before the launch. They did. Second, a placebo: invent a launch date in a quiet period and run the same comparison. A fair method should find nothing, and it did.

I like the placebo because it forces you to look for the way the method can fool you, not only for the way it can please you.

What it found, and what it could not

Against their twins, players who used the wheel had 40% more turnover, 39% more deposits and were active on 55% more of the days. The effect grew with use: it was invisible below five spins and grew with every band above that.

The return on the cost was a range, not a number. Depending on which costs we counted, it went from about 1.7x to about 5.7x. Every case was above 1, and the gap between them was real. I would not quote the best one alone.

It also had limits that I would say out loud. It covered about a third of the players who used the wheel, those who existed before launch. It was not a randomised test, so something we did not measure could still explain part of it. And other things were happening in the same weeks.

What I would do on the next one

  • Keep a group that never gets the feature, or a small standing test, even after the rollout. It is the cheapest way to answer the next question.
  • Decide how you will measure before you launch, not after the first good number.
  • Report ranges and limits, next to the headline. A number you can defend is worth more than a bigger one you cannot.

The full method, with the charts, is in part 2 of the rewards wheel case.

Language