Designing a Distributed Task Queue for 50k req/s

Most task queues fall over in the same place: the moment producers outrun consumers and nothing pushes back. This post walks through the three decisions that mattered most when building a broker that sustains 50k req/s without losing messages.

Persistence is a latency budget, not a checkbox

Every enqueue does a fsync before the producer gets an ack. That sounds expensive, and it is — until you batch. We group writes into 4ms windows and flush once per window. One fsync amortized across ~200 messages brings the per-message cost under 20µs while keeping the durability guarantee intact.

func (w *walWriter) append(rec record) error {
    w.mu.Lock()
    w.buf = append(w.buf, rec)
    w.mu.Unlock()
    return w.flushTicker.wait() // returns after the next batched fsync
}

Dead-letter routing needs a reason code

A message that fails five times goes to the DLQ — but a DLQ full of opaque blobs is useless at 3am. Every dead-lettered message carries the last error string, the attempt count, and the consumer group that gave up. That metadata is the difference between “the DLQ has 12k messages” and “12k messages failed on the same downstream timeout.”

Backpressure is the whole game

When consumers lag, the broker stops accepting enqueues past a high-water mark and returns 429 with a Retry-After. Producers that respect it degrade gracefully; producers that don’t get shed. Either way the broker’s own memory stays bounded, which is the only property that actually keeps you at four nines.