Simulated data. Share of daily players who bought.
Ad revenue
Simulated data
D1 by install day
Simulated data. Bars per install day, line is the 7-day average.
Retention curve
Simulated data. Install weeks observable on the replay day.
Retention by install week
Simulated data. Blank cells are not yet observable.
New player funnel
Simulated data. All installs in the soft launch.
Simulated data. Events per daily player by build, last 14 days.
Simulated data. Events per daily player by build, last 14 days.
Data alerts
Simulated data
Revenue reconciliation
Simulated data. Client purchases vs store settlements, whole soft launch.
LiveOps calendar
Events Offers Experiments Releases Today
Simulated data
Upcoming
Live and past
Offer groups by placement
Simulated data. All platforms and markets, to the replay day. Conversion: viewers who bought, where the offer is shown as a popup or event; otherwise players who bought.
Revenue by offer group, per week
Simulated data
Conversion to date
Simulated data
Spend tiers
players active today, by lifetime spend
Simulated data
Lifecycle
whole player base today
Simulated data
Active payers by spend tier
Simulated data. Average per day, by week.
Markets
soft-launch gates at the end of the install window
Simulated data
Experiments
status on the replay day
Test
Status
Players
Primary metric
Effect (95% interval)
Decision
Simulated data. Effects appear on the read day. A failed quality check stops the read before any effect is shown.
Starter pack at $1.99 instead of $4.99 for day-2 non-payers
Iterateregression avoided
Replay day is before the read on 2026-07-26: this case shows the final read.
1
Goal
Should the $1.99 starter pack replace $4.99 for players who have not paid by day 2?
Primary
14-day revenue per exposed player, smallest effect worth finding 10%
Guardrails
D7 retention no worse than 5%; Crash rate in 14 days no worse than 30%
2
Context
Who
Installs from 2026-06-15 to 2026-07-12, not under 13, no purchase on day 0, active on day 2
Read window 2026-06-15 to 2026-07-25 overlaps reverse_season (Reverse Season pin album, 2026-07-20 to 2026-08-02). Both arms see the event equally, so the comparison stays fair, but effects measured that week may not generalize to a quiet week.
6 weekend mini events fall inside the read window; the test spans full weeks so each arm sees the same mix.
Two list prices for the same pack can be compared in the community. Framed as a limited welcome offer and kept within a narrow band; watch support tickets for price complaints.
3
Data
Sample ratio balanced (p = 0.053, flag below 0.001)
Tracking in the read window ok
D7 retention guardrail within the 5% margin
Overlapping events Reverse Season ran inside the read window; both arms saw it equally
Early-value model not used: it predicted $0.28 at 30 days, the test showed -$0.06
4
Test
Arms
control 50%, treatment 50%
Unit
user
Players
33,528
Window
2026-06-15 to 2026-07-12, read 14 days later
Exposure
Starter pack popup at the first session on day 2
Variance reduction
post stratify
Sample size
62,539 needed for a 10% effect; 8 weeks gives 13.7%
5
Read
More players buy (+4.2 pts conversion), but each pays far less (-45% revenue per payer), so 14-day revenue per player falls -14.0%. The cheap pack wins only with low-engagement players (+$0.105 per player) and loses most with highly engaged ones (-$0.201).
Conversion up, revenue per payer down, revenue per player down
Treatment vs control, relative change with 95% interval
The conversion gain is real, but it comes from players who pay less. Revenue per player is -14.0% at 14 days and still -$0.050 per player at 30 days.
Simulated data
The cheap pack wins only with low-engagement players
14-day revenue per exposed player, treatment minus control, with 95% interval
One price does not fit everyone. The low-engagement third responds to $1.99; the rest would have paid $4.99.
Simulated data
6
Decide
Keep $4.99 as the default. Re-test $1.99 as a targeted offer for low-engagement non-payers, with the same guardrails.
Rolling $1.99 out to everyone would cost about $32,873 a year (range $10,361 to $55,385).
Next steps
Targeted $1.99 test for low-engagement non-payers (ranked first in the backlog)
Keep the D7 retention guardrail and the 30-day revenue read
Control players get the treatment pack's bonus coins after the test
Reverse Season pin album (7 days) vs the persistent LiveOps holdout
Extendunpowered
Replay day is before the read on 2026-08-09: this case shows the final read.
1
Goal
Did the 7-day pin album add revenue, or move it from other packs and from the following week?
Primary
Net revenue, event week plus the week after, smallest effect worth finding 10%
Guardrails
Active days in the week after no worse than 5%
2
Context
Who
Players installed before 2026-07-20; the 10% LiveOps holdout sees no LiveOps events
3 weekend mini events fall inside the read window; the test spans full weeks so each arm sees the same mix.
3
Data
Holdout 16,324 players never saw LiveOps events
Holdout contamination 0.0% of holdout players joined
Pull-forward the week after fell -$0.019 per player (p = 0.063)
Economy coins created per player rose +25%, and players spent them on districts
Power one event can detect $0.217 per player; the net effect is smaller
4
Test
Arms
holdout 10%, exposed 90%
Unit
user
Players
162,046
Window
2026-07-20 to 2026-07-26, read 14 days later
Exposure
First session during the event week
Variance reduction
cuped
Sample size
1,997,113 needed for a 10% effect; 13 weeks gives 35.1%
5
Read
Event packs sold $16,867. After displaced regular packs (-$3,548) and a dip in the week after (-$2,924), the net gain was $10,336. Event-week revenue rose significantly; the two-week net is positive but too small for one event to confirm.
Event packs sold $16,867, but only $10,336 was incremental
Exposed players vs the 10% holdout, total over the event week and the week after
Event revenue overstates the gain. Event-week revenue rose $0.031 per player, but regular packs and the following week gave part of it back.
Simulated data
The event creates more coins per player, and players spend them
Coins created and spent per daily active player
The economy absorbed the extra coins, so the event did not leave players over-supplied.
Simulated data
6
Decide
Run the next season against the same holdout and read both seasons pooled before setting the event cadence.
Reading event-pack revenue alone would overstate the gain by $6,531.
Next steps
Read the next season against the same 10% holdout and pool both seasons
Watch weekend events: join rates fall with each event joined in a row
Price event packs across the same points as regular packs
Where growth comes from
average day; levers sized per year
Revenue per day$4,279
Daily active players16,746
New3,507+10% installs: $212,566
Current12,956+1 pt retention: $128,029
Returning282+1 pt new-player retention: $21,150
x
Revenue per daily player$0.256
Payer rate1.29%+0.1 pt: $169,815
Revenue per payer$18.25+5%: $109,157
Ads per player$0.021rewarded and interstitial
Simulated data
Test backlog
sized revenue x confidence / effort
#
Test
Sized / year
Confidence
Weeks
1
$1.99 starter pack only for low-engagement day-2 non-payers
$84,907
70%
1
2
Raise the energy cap from 50 to 80
$61,891
40%
2
3
Prefer friends as raid targets friend groups
$128,029
30%
4
4
$4.99 season pass with pin rewards
$65,494
40%
3
5
Shorter first-session tutorial
$21,150
50%
2
Blended D1 moved -3.0 pts after day 30; -2.9 pts of that is install mix, not the game.
Simulated data
From a goal to a decision in six steps
Every feature, event or offer goes through the same steps. The rule is written before launch, the data is checked before it is read, code computes every number, and a person makes the call.
Rule before dataMetric, guardrails and ship rule fixed in the brief.
Trust before readNo effect is shown until the quality gate passes.
Code computes, people decideNumbers come from code; an owner approves.
Every decision feeds the nextResults are frozen and the backlog re-ranked.
What decision are we making, and what result would change it?
What we do
One brief per change: hypothesis, the KPI-tree node it moves, one primary metric
Guardrails with stated margins; every monetization test carries D7 retention
The smallest effect worth finding, the horizon and the ship rule, fixed before launch
Why
Writing the rule first stops the data from changing the question after the fact.
Output
Pre-registered brief
Gate to the next step
The brief validates: known metrics, margins on every guardrail, a reason for the randomization unit.
In this run
The starter pack brief fixed 14-day revenue per exposed player as the primary metric, 10% as the smallest effect worth finding, and D7 retention and crash rate as guardrails before any player saw the offer.
What is normal right now, and what else could move the number?
What we do
Baseline from growth accounting: new, current, reactivated and resurrected players
Soft-launch gates by market: retention, engagement, early monetization, crash rate
Calendar overlap and interference checks: other tests, events, players affecting each other
Why
A lift means little without a baseline, and an event running at the same time can fake one.
Output
Baseline and context notes
Gate to the next step
No unplanned overlap with another test or event; baseline set.
In this run
2 overlap notes on the starter test, including the Reverse Season event; both arms saw it equally, so the read stays fair. 4 of 7 markets pass every gate and 3 are on watch.
A versioned tracking contract that every build is checked against
Health monitors per build and platform, so a silent zero cannot pass
Exposure logged when the player sees the change; revenue reconciled to the store
Why
A broken event looks exactly like a change in player behavior; check the pipe before the result.
Output
Health report and one metric registry
Gate to the next step
Contract and monitors clean for the read window.
In this run
Monitors ran 1,592 build x platform x day checks and flagged build 1.5.0 on android the day it shipped. Revenue reconciles to the store with $0.00 unexplained after removing 88 duplicate rows.
Quality gate first: sample ratio, tracking health in the window, holdout contamination
Variance reduction (CUPED, post-stratification), capped and raw revenue side by side
For events: incrementality against the holdout, cannibalization and the week after
Why
Revenue is heavy-tailed and events borrow from next week; the read has to account for both.
Output
Standard readout
Gate to the next step
The quality gate passes, or no effect is reported.
In this run
The gate blocked the Jackpot screen test (sample ratio p = 2.4e-06) before any effect was shown. The starter test passed and read -14.0% on 14-day revenue per player (-22.8% to -5.1%).
The ship rule in code returns an action and one of six outcome classes
Annualized impact with a range, and a person who approves
The decision is frozen in the learning library; the backlog is re-ranked
Why
A result only matters if it changes what the team does next, and the reason has to be kept.
Output
Decision memo and learning record
Gate to the next step
Rule outcome logged with an owner and the next test.
In this run
The rule returned Iterate (regression avoided) for the starter pack: $1.99 for everyone would cost about $32,873 a year. 4 decisions are frozen in the learning library.
Every finished test is checked against the same questions, in this order. The first question answered yes decides the action. A no moves the test down to the next question. Pick a test to see the path it took.
1Was the test stopped early by its owner?
Yes
Stopaborted early
No
2Did the data fail the quality gate? sample ratio, tracking or holdout contamination
Yes
Reruninvalid: no effect is reported
No
3Is the primary metric or a guardrail significantly worse?
Yes
Iteratea pre-registered segment wins
Rerunonly a borderline guardrail
Stopotherwise
all three: regression avoided
No
4Is the primary significantly better, with every guardrail within its margin?
Yes
Shipwin
No
5Is the primary better, but a guardrail not yet shown safe?
Yes
Extendunpowered: collect more data
No
6No change, and could the test have detected the target effect?
Yes
Stopneutral, powered: a real "no effect"
No
7Otherwise the test was too small to tell
then
Extendunpowered: extend or redesign
Simulated data. Paths come from the rule trace each readout recorded.
Around the six steps
MonitoringBetween tests, the overview and agent signals watch every release, event and live test each day, and bring a person a case with research and options.Agents
Growth viewDaily players split by state and revenue broken into levers, each sized in dollars, so the next test goes where the money is.Opportunities
MemoryEvery decision is frozen with its numbers, owner and reason. Win rate and learning rate come from that record.Experiments
GovernanceUnder-13 players never leave the warehouse, only aggregates reach any AI service, and cuts outside the brief are labeled exploratory.
ReuseThe engine knows nothing about this game. A new game or strategy is a new config folder: economy, tracking plan, metrics, briefs, calendar and console settings.
Agent flow
Data to decision. The highlighted path is the selected signal.
Signal inbox
to the replay day
Gaps this closes
where each gap is handled in the flow
Gap today
Step
What the flow adds
Tests are read once at the end, so harm that shows up late is found weeks later.
Signal
Rules run every day on each live test, release and event.
Monitors raise too many alerts, so people stop reading them.
Signal
One alert per incident, only past a pre-set threshold, with a severity.
A blended KPI moves and the team changes the game for the wrong reason.
Research
Mix vs rate decomposition and metric lineage run before any option is proposed.
AI tools narrate dashboards and can state numbers that are wrong.
ResearchAction
Agents call a fixed set of checked analyses; every number comes from code and cites its analysis.
Each event is read alone, and one small holdout cannot settle anything.
Research
Past cases in the decision log are pooled with the current one.
Decisions and their reasons get lost in chat threads.
PersonLog
Each approval or dismissal is logged with its signal, research and chosen option.
Simulated data. A proposal: the agents never change the game or stop a test on their own.