Make your Linux desktop and Android beautiful.
We design artworks for your Linux desktop, icon themes and applications for your Android devices.
We design artworks for your Linux desktop, icon themes and applications for your Android devices.
Last updated: 2026-06-16
We ran a hero banner test. Variant B beat A by +6% in click rate. It looked great on day four. We shipped it fast. Then weekly revenue dipped. Sign‑ups rose, but first deposits fell. Our source mix had shifted. A big partner pushed low‑intent traffic mid‑test. We also found SRM (sample ratio mismatch) on day two due to a bad flag. The “win” was not real.
This is common at scale. More traffic means more edge cases, more bots, more drift. It is not enough to see a green bar. We need stable data, solid math, and strict ship rules. Below is the playbook we use when the market is hard, the noise is high, and the stakes are real.
Small tests forgive small flaws. Big tests make small flaws big. At scale, a tracking gap, a late pixel, or a mis-set bucket can skew a whole quarter. Channel mix moves from hour to hour. Geo splits shift with promo waves. Holidays act like new products. Bots do not sleep.
This is why we treat tests like products. We log, monitor, and audit. We pre‑register goals. We do not ship on clicks alone. And if your tests touch content pages, keep users first. See Google’s guidance on creating helpful, reliable content. Helpful pages plus clean tests make lasting gains.
One more change at scale: cost of error. A false win can push a team to build the wrong thing for months. We lower that risk with guardrail metrics, SRM alarms, and a simple stop plan. You will see these themes below.
Think in systems, not hacks. Our loop is simple: hypothesis → sizing (power/MDE) → design → QA → launch → monitor → decide → check after ship. We write each part down in a decision log. No guesswork later.
For the mental model, we like the short, clear book Trustworthy Online Controlled Experiments. It shows how to avoid common traps and how to make tests part of daily work.
For sizing, we use a sample size calculator for A/B tests. We set power at 80–90% and alpha at 0.05 by default. We pick a realistic MDE (minimum detectable effect) based on past data and cost to build.
Where things break: user IDs change across devices, events fire twice, bots slip in, QA traffic is not filtered. So we do server‑side assignment when we can, log raw events with version tags, and exclude test IPs and known bot ASNs up front.
We care about variance. High variance hides real wins and fakes false lifts. If you have pre‑period data, use CUPED variance reduction. It trims noise without bias when set right. Keep the math in code review. CUPED with wrong covariates can harm.
Do not peek without control. Sequential looks need a plan. Peeking until you see p < 0.05 inflates false wins. A good intro to common errors is A/B testing mistakes. If you must look early, use a group sequential design or a calibrated Bayesian rule. Write the rule before launch.
Watch SRM. If your 50/50 split shows 53/47 with big N, stop and debug. SRM often means a bug in assignment, a traffic filter, or a late script. No decision until SRM is clear.
When you run many tests at once, control false discovery. Teams like Uber wrote on FDR (false discovery rate) and how they apply it in practice. See controlling the false discovery rate in their platform notes. It helps you ship wins without a wave of false greens.
This cheat sheet shows how MDE, baseline, and traffic change the run time. Numbers are rounded. Two‑sided test, alpha 0.05, power 0.90.
| 2.0% | 10% | 90% | 0.05 | ~160,000 | 20,000 | ~8 |
| 2.0% | 5% | 90% | 0.05 | ~630,000 | 20,000 | ~32 |
| 3.0% | 8% | 90% | 0.05 | ~210,000 | 25,000 | ~9 |
| 5.0% | 10% | 90% | 0.05 | ~65,000 | 30,000 | ~3 |
| 5.0% | 5% | 90% | 0.05 | ~255,000 | 30,000 | ~9 |
| 8.0% | 5% | 90% | 0.05 | ~150,000 | 40,000 | ~4 |
Note: Sequential testing can cut time if you use proper alpha spending. If you do it ad hoc, you raise Type I error. Write the plan or use a tested engine.
In high‑pressure niches (gambling, loans, telecom), users face risk and rules. Words matter. So does trust. Small UX copy shifts can move deposits, but only if they do not break rules. One safe win we use a lot: clarify who is eligible for an offer near the CTA. This lowers false clicks and raises real intent.
On our own review work, we had a clean case. If you want to see the kind of sites we test on, you can visit asiaonlineslot.com. On bonus compare pages, we A/B tested a short “Who qualifies” line above the button. The line listed key limits in plain words: age, geo, KYC. CTR went down 2%, but verified first deposits went up 3.1% (95% CI: +0.8 to +5.4). Support tickets on “bonus not paid” dropped. Net revenue rose.
Microcopy helps clarity. See this simple guide on microcopy that improves clarity. It pairs well with tests on forms, KYC tips, and T&C hints. Keep copy short. Put the hard parts near the action. Test with real users, not your team.
Flows also matter. If you send users to a checkout or a long form, follow solid patterns. The team at Baymard has deep notes on checkout UX guidelines. Use them to spot friction before you test. You will waste less traffic.
And bots. In some niches, bot traffic can reach double digits. Do not assign bots to variants. Filter them out before bucketing. Start with IP ranges, ASNs, headless flags, and device mix checks. Learn the basics in bot management best practices. Your SRM chart will thank you.
Client‑side tests are fast, but flicker and ad blockers can bias them. At scale, prefer server‑side flags and stable IDs. A dull, safe flag system beats a flashy UI that drops events. A short intro to this topic is here: feature flags at scale.
For markets with geo limits or strong cross‑talk, try geo experiments. They split markets, not users. This is great for ad mix tests, brand changes, or price tests. See Google’s write‑up on geo experiments in practice.
Ship rules matter. We use a slow ramp: 1% → 10% → 25% → 50% → 100% with guardrails. Guardrails include error rate, refund rate, KYC fail rate, and support tickets per 1,000 sessions. If a guardrail trips, we pause and inspect. No heroes, just a checklist.
Keep logs immutable. Keep event schemas under version. Add a change log to your SDK. These small bits make post‑mortems fast and fair.
Think in bets. Some tests have low risk and fast reads (copy, order, prompts). Others are heavy (new bonus logic, KYC steps). Place small, fast bets to fund bigger ones. Spread risk across areas: acquisition, activation, trust, and pricing. Track the mix in a weekly view.
When you run tens of tests, the chance of a false green rises. You can cap this using FDR or by raising the bar for weak priors. Uber’s post above is a good start. For harder bets, test “who to treat,” not “treat or not.” A short intro is Stitch Fix’s note on causal inference and uplift modeling. It helps you learn which users gain, not just if the average moves.
When to stop? Stop when you hit your pre‑set power or when a ship/kill rule is met. Stop early for harm if guardrails go red. Do not stop at day three “because p is 0.04.” If the plan said 14 days, run 14 days. If seasonality is wild, use staggered starts or geo tests.
Respect user consent. If you operate in the EU or UK, read the ICO’s note on GDPR consent guidance. Honor choices and log consent state. If a user does not allow tracking, do not assign them.
On iOS, honor App Tracking Transparency. Plan for more noise. Use server‑side events and model gaps with care.
Use server‑side tagging where you can. It lowers loss from blockers and lets you add stable IDs with user consent. Here is the Measurement Protocol as a start point.
Re bots: keep a live bot score per source. Exclude bad ranges pre‑assignment. Re‑check after launch. If bot share jumps, pause the test.
A good test culture is boring. We write a short pre‑reg, we review the plan, we run, we decide once, we move on. We keep a small, shared dashboard with the same few charts per test. We do a post‑mortem if we shipped and missed the goal.
For inspiration, read how big teams do it. Netflix has many notes on experimentation culture. You will see the same core: log well, plan ahead, decide once.
Tools do not fix weak plans. Pick tools that make IDs stable, flags safe, and logs exportable. If you use a vendor stats engine, learn the model. Here is a plain intro to one such engine: Stats Engine overview. Know what it assumes before you ship on it.
What would we do today? First, add SRM alarms. Second, lock the channel mix at start or stratify by source. Third, set the main goal to verified depositors, not clicks. Fourth, use a slow ramp with guardrails. And log the decision in a shared doc. Teams like Booking share lessons often; a nice hub is booking.com experimentation lessons.
The banner may still ship, but only if deposit rate holds across sources and weeks. That is the bar in real markets.
What is a realistic MDE in a hard niche?
Pick 5–10% relative for top‑funnel CTR, 3–6% for activation steps, and 2–5% for deposit or paid. If traffic is low, raise MDE or pool across weeks with the same plan.
How do I detect SRM fast?
Set an SRM alarm on the assignment event. Check split by source, device, geo. If any key slice fails, pause. Fix before you read the test.
Should I use sequential testing?
Yes, but only with a plan. Use group sequential designs or a vetted Bayesian rule. Write the rule pre‑launch. Do not peek and ship on a whim.
When are geo experiments better than user‑level A/B?
When cross‑talk is strong (ads, brand), or you test price, geo holdouts are safer. They need more time, but the read is clean.
How do I handle bots in tests?
Filter bots before assignment. Use IP ranges, ASNs, headless flags, and odd device ratios. Track bot share per source daily. Pause if it spikes.
Compliance and care: follow local laws. Promote play with care. If you need help, see BeGambleAware. This article is for product and data teams. It is not legal advice.