A Zero-Trust Auth Gateway in Rust at 2ms p99

The auth gateway sits in front of everything, so its latency is added to every request in the system. The target was a 2ms p99 with full JWT verification and per-tenant rate limiting. Here is where the milliseconds went.

Verify signatures once, cache the claims

JWT signature verification with RS256 is ~0.3ms. Doing it on every request for a token that is valid for an hour is waste. We cache the verified claims keyed by a hash of the raw token, with a TTL that never exceeds the token’s own exp. Cache hit path is a hashmap lookup and an expiry check — under 10µs.

Rate limiting is a sharded token bucket

One global lock on the rate limiter would serialize every request. Instead each tenant maps to one of 256 shards, each shard is an independent token bucket behind its own lock, and contention drops to near zero. Refill happens lazily on read using a monotonic clock, so there is no background sweeper.

fn allow(&self, tenant: TenantId) -> bool {
    let shard = &self.shards[tenant.hash() as usize % self.shards.len()];
    let mut bucket = shard.lock();
    bucket.refill(Instant::now());
    bucket.try_take(1)
}

Pool connections to the identity store, aggressively

Cold DB connections cost 1–2ms to establish — the entire budget. The pool keeps a warm minimum sized to peak concurrency, health-checks in the background, and never hands a request a connection it has to open synchronously. p99 with a warm pool: 2ms. p99 the first time we forgot to warm it: 40ms.