When the wheel reached everyone
Most of what went wrong happened after the wheel worked. Small rollouts hid problems that only showed up at full scale. I owned the product, so I own the list that follows. Nobody on it was careless. Each incident was a system that had a limit nobody had hit yet, and a configuration step that had one pair of eyes on it.
Opening the wheel to everyone overloaded the rewards platform. Part 1 tells the day itself. When the whole base came in, the platform had to create a large number of wheels at the same moment, and delivery wobbled. I do not have a clean timeline of that day in my notes, so I will not invent one. What I do have is what we changed. The wheels now go out in batches a few minutes apart, in order of importance, so the wheels for the most valuable players are the first to arrive if anything slows down. And we tell the platform team before every big launch, so they expect the spike.
Then a bonus ran out, and the support desk felt it first. Each prize slice of the wheel gives a bonus created on the betting platform, and every bonus has a limit of free spins as a safeguard. When we moved the older wheels to 70% of the base, one of those bonuses was used up fast. The second most used bonus had taken about 20% of its limit and two others about 10% each. Players who won that prize could not use it on the site, and they called support.
One bonus at its limit while the others were nowhere near. The same shape explains the cause.
The cause was my own configuration: two slices shared one bonus. On the wheel that failed, two different slices each paid 10 free spins, each with a 41.24% chance, and both drew from the same bonus. So about 82% of all spins on that wheel took from one pool. The limit had been sized as if the load was spread over the whole wheel. It was not.
The fix was one bonus per slice, sized for 30 days. On the three wheels with the most pressure on a single result, we went from two slices of the smallest prize to four, each with its own bonus. We also started to compute each limit so that it would last at least 30 days at the expected volume, instead of picking a round number. Operations created a new bonus right away for the players who were complaining, a second one to replace the exhausted bonus on the wheel, and later renewed bonuses twice, raising the limit each time. That task force spent its time on work that a better first setup would have made unnecessary. I count that time as part of the cost.
The support desk was the first to feel it, and the first to tell us. Calls rose, and the tickets dropped only after the bonuses were renewed and the segments fixed. I try to keep the story honest: a weekly report from early October noted that call volume had not moved far from the average of the previous three months, even though it felt much worse inside the team. So some of the pain was real and some was the noise of a launch week. Either way, the desk needed structural work, not only a quick fix.
Auditing the segments found a second cause. Some players were getting no wheel at all. When we checked, 5 of the 17 segments were misconfigured. Four were found and corrected on a Monday and the last one on the Tuesday. Around the same time, players complained that they fell from level 7 to level 0. The flag the platform used for bonus hunters was not reliable, so it had pushed people out of their level. We removed that block. From then on the wheels that pay only go to players with turnover in the last 30 days.
One button was calling the rewards platform on every page load. It should have called only when the player clicked. With the whole base on the wheel, those calls added up as extra load on the platform. Engineering moved the call to the click, and the calls from the button to the platform fell 95%.
The same screen, the same players. The only change was when the call happened.
Three vaults in one weekend
The weekly vault went out three times in a weekend because of one setting. The Vault started on Black Friday and became a weekly send, with the same rule: a bonus worth 10% of the player’s net revenue over the last 7 days, with a 1x rollover. The first weekly send was on 19 December and had a mechanism error that we fixed on the 23rd. On Friday 26 December we sent it again, now with the mechanism working. But the trigger was set to run every day at 17:00 instead of Fridays at 11:00. That is the whole mistake. It produced two extra vaults, one on Saturday 27 and one on Sunday 28.
Five days. The send on the 23rd was planned. The two extras were not.
Finance noticed on Monday, not us. The finance team saw that the bonus spend was above what was expected and alerted us. We inspected the configuration, found the trigger, and fixed it before 17:00 the same day, which was before a fourth send could go out. I am grateful to finance for that. The risk team’s monitoring had not been set up for this case yet, so we had no alarm of our own.
What it cost, in plain terms. The design rule for a weekly vault is 10% of the last 7 days’ net revenue. Each extra send cost 8.05% (Saturday) and 7.44% (Sunday) of the gross revenue from the 7 days before it. The two bases are not the same measure, so I read the match as an order of magnitude, not an exact check. The point holds: each extra was close to a full vault. In round terms, the weekend spent about three vault budgets where one was planned.
The overlap is what made it worse than it looks. Of the players counted in the post mortem, a little over half got only one vault. Close to a quarter got two. And more than one in four got all three.
Share of players by number of vaults received.
Overlapping bonuses with rollover are easier to clear. A rollover bonus asks the player to bet a multiple of the bonus before the money is released. When several of them are active at once, the same bets count toward all of them. So a player with three vaults needed to bet roughly what a player with one needed, to release three times the money. The cost came from the bonuses created and from how easily they were cleared.
The analysis was hard to do, and that was part of the problem. There was a gap between three systems. The rewards platform knew how many vaults were sent and opened, and who opened them, under its own player ID. The betting platform knew when each bonus was created, redeemed and completed, under a different ID. The data team’s analytics tool had neither. On top of that we had sent the same bonus code every day. A player who opened the vaults of the 23rd, 26th, 27th and 28th had four bonuses that all showed the creation date of the 28th. To tell them apart we would have had to recalculate the 7-day revenue for each day. That can be done with time, but not quickly.
The overlap on the 23rd was my decision, and I think it was wrong. It was intentional. I argued that the vault sent on the 23rd should expire on Friday the 26th. The support desk required that it last 7 days, as we had promised players. In the end we did what the desk asked. Looking back, the promise was the right thing to protect, but two vaults on top of each other should have been avoided in the design, not accepted.
What we changed. The post mortem listed the learnings, and I kept them short.
- A second person, never the one who built it, reviews every configuration before it goes live. A vault has many details inside the rewards platform, and the person who built it is the worst person to find its mistake. CRM was the obvious team for it.
- A different bonus for each send. Creating one new bonus every week is a small cost and it gives clean data per week.
- Fixed expiry dates on weekly bonuses, and terms of use that say so, so that a vault opened on day 6 cannot overlap the next one.
- No overlap of vaults, even when the pressure to do it is reasonable.
- Data to track completion: we need to see how many rollover bonuses were completed and paid, not only how many were created.
- Give the risk team what it needs to catch a case like this on Saturday, not on Monday.
A queue that backed up in March
In March 2026 the platform told us its time-based activities were delayed. The service that runs activities at a set time had a backlog. Activities and lifecycle entry conditions that should fire at a given hour fired late, as the system caught up. The product team had already seen wheel sends wobble the day before. The backlog was fully cleared the next morning. The platform said it could not confirm a root cause, and it pointed to what looked like the trigger.
The trigger was a heavy field in the segment. The platform counted 56 activities that used the same calculated field for the player’s turnover over the last 30 days, instead of a lighter native field. All of them ran at about 17:00 UTC, on different days of the week. Many of them at the same hour, each asking an expensive question, built the queue. The platform asked us to use the lighter field at least, and to spread the activities over the day.
The investigation found something I did not expect: the field was wrong. Switching to the native field was not possible as it stood, because the two fields did not mean the same thing. The product team looked inside the calculated field. It summed casino and sportsbook turnover, but the casino part counted live casino, which we had agreed not to include, and the sportsbook part used the average per bet instead of the total wagered. The average pulled the number down, and it was still the wrong number. The question put to me was whether it was meant to be like that. The answer was that the rule had to be agreed again first. The rule had never been written down in one place, and that is how a wrong definition survives.
Our answer was in two parts. First, ask the platform to create a native field that matches the rule we want: casino without live plus the total of sportsbook. That would remove the need for the heavy calculation. Second, spread the timing of the activities so they do not all fire in the same window. Operations and product agreed that the concept had to be aligned first and the rule adjusted second. I left before I could confirm that the new field was live.
Three months earlier the platform had a different kind of failure. On 29 December its delivery of wheels dropped, which affected several players, and it was restored on the 30th. I mention it because it is the other side of the same lesson. Part of the system sits outside our control, so the plan has to include how we find out and how we explain it to players and to the support desk.
What the support cases told us
About 60 case files from September and October showed what players complained about. I read them as a set, with names and personal data removed. The most common reason by far was “I am outside the segmentation”: about 13 of the cases with a recorded reason, including several bonus hunters who complained about not getting what others got. The check was always the same: go over the criteria of the player’s level, such as deposits over 90 days, average bet in sports and casino over 7 days and the bonus received over 7 days, and look at the tags the player had on the platform.
The other cases fall into a few types. The player could not redeem the bonus, redeemed it but could not use it, never received the wheel, was excluded on the platform, tried after the promotion ended, or saw a screen that did not load.
Of 67 cases in the log, 14 were closed and 53 were not. That is 79% left open. Almost always the answer had been given and the record was never closed. I read that as a gap in the process more than 53 unsolved problems. It still has a real cost: we cannot measure which problem type is shrinking, or how long a fix takes, if nothing is closed with a resolution.
Two lessons came out of it. The eligibility check has to be quick and done by the people who answer players, not by the product team. And every new case must be closed with a type of resolution, or we cannot measure anything afterwards. Both are in the list of rules below.
The rules we kept, and what was still open
The rules the team adopted or agreed came from the incidents, not from a plan. Each one has an incident behind it.
How a configuration reaches production now. The red step is the one the vault weekend added.
- A second person reviews every configuration before it goes live (the first item of the list above).
- One bonus for each slice of a wheel, with a limit sized for at least 30 days.
- A new bonus code for each weekly send, with a fixed expiry date.
- Never two vaults at the same time.
- Warn the rewards platform before a big launch, and send the wheels in batches, most important first.
- Calls to the rewards platform happen on click, not on page load.
- Time-based activities use light fields and are spread over the hours of the day (agreed with the platform, not finished).
- Every support case is closed with a type of resolution, and the eligibility check sits with the people who answer players.
- Watch bonus cost as a share of revenue every day, together with risk (proposed to the risk team).
Some things were still not fixed when I left. I want to say it plainly.
- The data gap between the rewards platform, the betting platform and the analytics tool. We could report how many vaults were sent, but not cleanly how many rollover bonuses were completed.
- The risk team’s monitoring for the vault case and the daily cost view. I cannot confirm either was ready.
- The turnover rule behind the heavy field. The new native field was asked for but not confirmed live, and the 56 activities were not yet moved.
- The support log. The rule to close each case was agreed. The backlog of 53 open cases was not cleared in the log.
- The fixed expiry and the updated terms of use for weekly vaults. I cannot confirm they were done.
What I would tell the next person. Most of these incidents came from a limit nobody had tested meeting a configuration that one person had checked. The cheap defenses are plain: a second reviewer, a bonus per slice, a different code every week, a rule written down in one place. I wish I had put all of them in place before the first launch, not after the incidents.