Distributed rate limits: why local counters overshoot

Not simply that the limit is too generous, which would be easy to correct.

With N nodes each holding a full counter, a client spread evenly across them can send N times the intended amount before anything refuses it. If that were the whole problem you could divide by N and move on. The difficulty is that the multiplier is not stable: it depends on how that particular client's requests happen to be distributed, which depends on routing, on connection reuse, on how many nodes are healthy at that moment, and on the client's own concurrency.

So the effective limit differs per client and changes through the day. One client using a single long-lived connection is held to roughly the configured number. Another opening connections freely is held to some multiple of it. Neither is what you specified, and the gap between them is invisible in the configuration, which is why this is usually discovered from the outside.

The same effect works in reverse when you overcorrect. Divide the allowance by the node count and a client that reaches one node is refused at a fraction of the limit you intended, which reads as an unexplainable refusal to everybody involved.

Each is a defensible answer to a different constraint.

Local counters with a divided allowance. Every node enforces the limit divided by the number of nodes. No coordination, no added latency, nothing to fail. It is correct only when traffic is spread evenly, which is what a distributed limit cannot assume: a client pinned to one node meets a fraction of the intended limit, and the divisor is wrong again the moment you scale up or lose a node.

A shared counter. Every node consults one store before deciding. Accurate, and the accuracy costs a round trip on the path of every request, plus a new dependency: if the store is slow, everything is slow, and if it is unavailable, every node must decide what to do without it. This is the arrangement people reach for first and the one whose failure modes surprise them most.

Local counters with reconciliation. Each node counts locally and periodically exchanges summaries, so every node knows approximately what the others have seen. Overshoot is bounded by how stale the summaries are rather than by the node count, and no request waits for coordination. Approximate by design, and for most limits the approximation is smaller than the uncertainty in the number you chose.

Before you deploy a shared counter, decide what happens when it cannot be reached.

Fail open and the limit disappears at the moment your infrastructure is already unhealthy, which is frequently the moment it was protecting something that could not take the load. Fail closed and a hiccup in a store that has nothing to do with your visitors refuses all of them at once, turning a dependency's bad minute into your outage.

Neither is right in general and both are right somewhere. What matters is that the choice is made deliberately and written down, because otherwise it is made by whatever the library does by default, discovered during an incident, and remembered as a surprise. Where you can, prefer a third option: fall back to local counting with a divided allowance while the store is unreachable, which degrades the accuracy rather than the service.

If every request from a client reaches the same node, its local counter is correct for that client.

That is a real solution for the common case and it does not survive the case you built the limit for. Sticky routing depends on a stable notion of who the client is, so it works well for identified clients and poorly for anything counted by address. It concentrates load unevenly, since one heavy client now belongs to one node. And an actor deliberately spreading traffic to evade a limit is precisely the traffic that will not stick.

Treat it as an optimisation that reduces coordination for well-behaved clients, not as the mechanism that makes the limit correct.

Nodes disagree about when a window starts, and it matters less than people fear.

Small clock differences move only the boundary between periods, so their effect is confined to the seam, which the fixed window already handles badly and the sliding and bucket approaches largely remove. Choosing an algorithm whose behaviour does not hinge on an exact shared moment is a cheaper fix than tightening clock synchronisation, and the differences between those algorithms are set out under rate limiting algorithms.

What does deserve attention is any arrangement where a node's decision depends on a timestamp it received from another node. There, skew becomes an arithmetic error rather than a boundary effect, and whose clock is authoritative needs stating explicitly.

The configured number is not the enforced number, so measure the enforced one.

Take a client identity you control, send a known pattern of requests through your real entry point rather than at a single node, and count how many are accepted before the first refusal. That figure is your actual limit. Compare it with the one in your configuration, and if they differ, you now know the multiplier your architecture is applying rather than the one you assumed.

Repeat it with a second pattern that opens many connections rather than reusing one, since that is the difference between the client that meets your limit and the client that meets several copies of it. Then confirm the refusal itself is well-formed, which is what 429 covers, because a distributed limit that refuses correctly but says nothing useful produces retry storms across every node at once.