If your p99 dashboard says 50µs, your p99 is probably 2ms
Instrumentation and alerting for trading systems: SLO design with PnL-equivalent error budgets, Prometheus cardinality traps, HDR histogram vs t-digest, distributed tracing hot paths, and incident response.
The most dangerous systems are the ones that look healthy on dashboards while silently degrading. These seven posts document the observability patterns (HDR histograms over t-digest, SLOs tied to error budgets rather than uptime, tail-based tracing for hot paths) that catch problems averages miss.
Why Google's SRE error budget model is dangerously wrong for trading. Session-aware SLOs, PnL-equivalent budgets, and burn rate alerting for live trading systems.
How Akuna Capital's 500+ test environments caused a Prometheus cardinality explosion - and the label design, recording rules, and Thanos/Mimir tradeoffs that fixed it.
Why Prometheus summary metrics lie about your P99, how HDR histograms caught a hidden 2ms tail at Akuna Capital, and building accurate sub-millisecond latency dashboards.
How a naive OpenTelemetry implementation added 15µs to ZeroCopy's signing hot path, and the head/tail sampling architecture that brought overhead to under 300ns.
From 40 pages/week to under 5 actionable pages: the alerting audit, 3-tier model, dead man's switch, and Prometheus alert rules that rebuilt Upside's on-call culture.
At Upside we discovered a $200K undisclosed net long. The IR playbook that doesn't work for trading, and the one that does - from halt to flatten to post-incident PnL analysis.
A single HashMap→BTreeMap change at Akuna Capital added 3µs to the hot path. Only CI benchmarks caught it. The full system: Criterion.rs, dedicated runners, and regression detection.