/blog / observability-across-200-microservices
Observability Across 200 Microservices Without Drowning
Trace sampling that keeps the interesting requests, cardinality budgets for metrics, and why every service ships the same three RED dashboards.
At 200 services, “add more dashboards” stops scaling and “add more discipline” takes over. Three rules kept the platform observable without a seven-figure telemetry bill.
Tail-based sampling, not head-based
Head-based sampling decides at the ingress whether to keep a trace — before it knows the request was slow or errored. Tail-based sampling buffers spans for a few seconds and keeps a trace if any span errored, exceeded a latency threshold, or touched a flagged endpoint. We keep 100% of the interesting traces and ~1% of the boring ones.
Metrics get a cardinality budget
Every label is a multiplier. http_requests_total{service, method, status, path}
with unbounded path is how you get 2 million series from one endpoint. Paths are
templated (/users/{id}, not /users/8134) at instrumentation time, and each
team gets a series budget the platform enforces in CI.
Every service ships the same three dashboards
Rate, Errors, Duration — generated from a template, not hand-built. When you get paged for a service you have never seen, the dashboard is exactly where you expect it, with the same panels in the same order. Familiarity under stress is worth more than a bespoke layout.