Make your Linux desktop and Android beautiful.
We design artworks for your Linux desktop, icon themes and applications for your Android devices.
We design artworks for your Linux desktop, icon themes and applications for your Android devices.
The lobby hums. Traders watch ten screens. The feed jumps. The in‑play clock ticks. Phones light up, fast. A tiny delay in price makes a big risk. A missed bet path means lost money and angry users. This one minute, right before kick‑off, shows the truth. Observability is not “nice to have.” It is how the book stays fair, fast, and safe.
In a live book, nothing is still. Odds change each second. Bets spike in waves. Wallet calls hit card rails you do not control. A queue lags, a thread starves, a flag flips. You need clear sight on all hops. You need to know the state now, not “five minutes ago.” You need to see cause, not only noise. That is what real‑time observability gives you.
First, odds often drift. A source stalls. A parser drops keys. The market manager keeps old prices. The UI shows a price that is stale by one or two seconds. In a top league, that gap is huge. Users slam “place bet” and you face cash‑out risk or must void. Both hurt trust.
Next, the risk engine slows. CPU spikes. The thread pool fills. The gateway times out. You get 5xx in bursts. Alert noise grows. The true root cause hides. Many teams learn to set error budgets and alert rules from the Google SRE workbook. Clean alerts save minutes, and minutes save money on derby day.
Then, payments. A PSP is slow. Retries fire. Without idempotency, you get double auths. Later, settlement fails quietly. A late event lands. Rules mark a winner twice. The fix is simple on paper: measure the right signals end to end and react in seconds, not hours.
Four goals frame the whole effort:
Turn each goal into SLOs (targets) and use SLIs (measurements) to track them. Spend your error budget on big games with care.
Your stack has clear hops: odds in, price make, bet place, risk check, queue, stream, wallet, settle. At each hop, collect three kinds of signals: traces, metrics, and logs. Use one trace context for the whole path.
Use the OpenTelemetry specification to model spans and pass IDs. Include keys like bet_id, event_id, market_id, PSP_route, and a masked user_id. One trace per bet lets you jump across services fast.
Run a tracer like Jaeger distributed tracing so on‑call can search “why is p99 slow?” in seconds. Tie metrics to traces with exemplars. Keep logs structured. No free text for key fields. Mask PII by design.
Watch queue lag, odds staleness, price drift, CPU and GC time, PSP timeouts, and settlement lag. High‑cardinality tags (like market_id) help find the pain point fast. For deep network or kernel views on hot paths, eBPF can help; see eBPF basics for how to get low‑overhead kernel data.
The table below lists what to watch, target levels, what to alert on, and where to look when it fails. Use Prometheus metrics for fast queries and alerts, and build views with Grafana dashboards that map to real user flows. Keep the names plain. Keep units clear. Add links to runbooks in each panel.
| Odds ingestion/update | Feeds, parsers, market manager | Median/p95 update < 150/300 ms; staleness < 1.5 s | Feed gaps; version skew; parse error spike | Alert if staleness > 1.5 s on top markets; parser error rate > 0.5% | Switch to backup feed; suspend bad markets |
| Bet placement (pre‑auth) | API gateway, risk engine, queue | p99 accept < 250 ms; 5xx < 0.2% | Thread pool saturation; queue lag; rate limit burst | Alert if p99 > 250 ms 5m; Kafka lag > threshold | Enable overload shed; scale risk microservice |
| Wallet/cashiering | Payment svc, ledger, PSPs | Auth success > 99.5%; duplicate tx = 0; T+0 recon | PSP timeouts; idempotency misses; partial auth | Alert on duplicate_tx > 0; PSP timeout > baseline + 3σ | Switch PSP route; replay CDC for recon |
| Settlement pipeline | Event bus, rules engine, DB | Correctness 100%; p95 lag < 2 min | Missing outcome events; late data; schema drift | Alert if DLQ grows; settlement lag > 2 min | Replay partition; patch rule; notify finance |
| Cash‑out/hedging | Pricing loop, exposure calc | Price freshness < 300 ms; fail‑closed < 50 ms | Pricing drift; CPU saturation; cache stale ratio | Alert on drift > threshold; stale cache ratio > X | Freeze cash‑out; hedge on alt venue |
| KYC/AML events | Identity svc, rules, case mgmt | Screening p95 < 1 s; false positive rate stable | Vendor SLA breach; queue backlog; rule loop | Alert on backlog > N; vendor error surge | Switch vendor; pause rule; manual triage |
| Support telemetry | Chat/CRM, incident tags | First reply < 60 s; issue tag accuracy | Spike in “delay” tags; NPS dip on weekends | Alert on tag spike > 3× baseline | Hotline open; status page update |
Idempotency keys: every bet, every wallet call. Store the key with short TTL. If the same key shows up again, return the prior result, not a new one. This kills double charges and ghost bets.
Exactly‑once in streams: use EOS where money is at risk. For Kafka, see exactly‑once semantics in Kafka. For heavy event work, Apache Flink documentation shows how to checkpoint and bound late data. In other places, at‑least‑once plus idempotent sinks is fine.
Backpressure: when risk or price lags, shed load. Slow down bet intake by market. It is better to suspend one hot market than to poison the whole book with bad odds.
Circuit breakers and timeouts: set sane timeouts to PSPs and feeds. Break on error spikes. Do not let a slow vendor block the main path. Study the circuit breaker pattern and add per‑route limits.
Synthetic checks: run scripted in‑play bets on top markets each minute. Verify odds freshness, accept time, and cash‑out. Tag these traces so they stand out in dashboards.
Good data is not enough. You need tight loops. Detect, group, route, act, learn. Cut alert storms with dedupe and route by business impact tags like “High Exposure Market.” Invest in high‑cardinality analytics to find the slice that hurts; read about it in high‑cardinality telemetry. Then move fast: auto‑switch to a backup feed, freeze cash‑out, or cap bet size when p99 jumps. After the fix, run a brief review and update the runbook the same day.
Build the parts that set you apart: pricing logic, risk, exposure, settlement rules, and key flows for in‑play. Buy the parts that are undifferentiated but hard to run: time‑series DB, log storage, alerting pipes, and tracing backends. Keep your data model portable to avoid lock‑in.
When in doubt, map vendor tools to the cloud‑native space. The CNCF landscape shows open choices for metrics, logs, traces, queues, and more. A hybrid stack often wins: managed storage but in‑house dashboards and SLOs.
Embed SRE with traders and risk. Share terms. Run game days with real feed jitter and PSP slowness. Rotate on‑call with devs and product. Keep one shared runbook per user path. On a big night, set a war room with a clear lead and a live “top five markets” board.
You must prove fair play and safe funds. The UKGC Remote Technical Standards call for strong controls and audit. Payments must meet PCI DSS v4.0. User data rules come from the GDPR. Observability makes these traceable. Keep audit logs for key actions. Split PII from traces. Mask fields at the edge. Set a data retention plan and stick to it.
It was a Saturday, near half‑time. p99 for bet accepts went from 180 ms to 1.2 s in two minutes. 5xx stayed low. At first, the team looked at the gateway. Nothing stood out. The real clue came from the traces. The risk span time went up, but only on EPL markets. Odds staleness also rose on the same markets. A feed parser had a version skew after a silent deploy. The risk model waited on a price that did not update.
The playbook kicked in. The alert grouped by market_id and tagged the markets as “High Exposure.” An auto‑action suspended cash‑out on those markets and flipped to the backup feed. p99 fell to 230 ms in two minutes. We rolled back the parser. After the game, we ran a short post‑mortem. We used the NIST incident handling guide steps: prepare, detect, contain, recover, learn. We added a canary for parser schema and a rule to block deploys near kick‑off.
Players feel delay and drift at once. A bet slip that hangs for three seconds feels broken. A cash‑out that freezes makes people leave. Fast payouts build trust; slow ones kill it. Public voice reflects this. Independent operator reviews and reliability roundups, such as this guide, often mirror what real users see on big weekends: payout delays, cash‑out pauses, and odd outage waves. Your SLOs show up as stars and comments one day later.
Take server‑side times from the gateway span start to the final risk decision. Do not include client time. Do not sample on quiet hours only. Keep a rolling window over the last 5–10 minutes, with a cap per market to avoid one hot market hiding others.
Top leagues: aim for under 1.5 s, with alerts at 1.2 s on high exposure markets. Lower leagues may allow up to 2–3 s. Track both median and p95. Alert when p95 shifts fast.
Compute a drift score per market: abs(primary_price − ref_price) / ref_price. Alert when it passes a set band for 60 seconds. Keep one backup feed per sport. Add a canary market that you watch at all times.
Group alerts by market_id and service. Add rate limits. Route high exposure first. Tie auto‑actions to grouped alerts. Keep “quiet hours” rules off on match days.
Use idempotency for APIs and ledger writes. Use exactly‑once in the stream where you settle money or trigger payouts. Elsewhere, at‑least‑once plus idempotent sinks is fine and simpler to run.
Day 1–3: define three SLOs (odds freshness, accept p99, settlement lag). Add two synthetic in‑play checks and wire alerts to on‑call. Day 4–7: add OpenTelemetry spans to bet place and risk. Build one Grafana board per user path. Day 8–14: set idempotency on wallet writes; add a backup feed switch button; run one game day drill with traders.