Bursts and rate limits: why a flat limit breaks users

The mental model behind most limits is a steady stream, and nothing produces one.

Loading a single page produces a burst by design: the HTML document arrives, and then everything it references is requested almost at once. A visitor clicking through a site produces clusters separated by reading time. An integration polling on a schedule produces a spike at the same moment every period, and an unlucky number of integrations produce it at the same moment as each other.

Averaged over an hour, all of these look modest. Measured in the second when they happen, all of them look like abuse to a limit that expects the average.

A useful limit needs both of them, and having only one of the two is where almost all of the trouble starts.

The sustained rate is what a client may do over time: the average you are willing to serve indefinitely.

The allowance for a moment is how much may arrive at once without being refused, provided the sustained rate is respected over the longer view.

With both, a client can behave the way clients behave: a cluster now, quiet after. With only the first, every cluster is a violation. With only the second, a client that never stops bursting is never caught.

The common implementation of this idea is an allowance that refills over time: you may spend a certain amount at once, and the ability to spend returns gradually. That is a mechanism rather than a recommendation, and choosing values for it is where knowledge of your own traffic replaces general advice.

Three failures, and none of them looks like a rate limit from outside.

Pages break in parts. The HTML arrives and some of its resources do not, because the burst crossed the line partway through. The visitor sees a half-rendered page rather than an error, which is reported as «the site is broken» and diagnosed as anything but a limit.

Retries make it worse. A client refused mid-burst commonly retries, and the retry arrives while the allowance is still exhausted. What was a brief overshoot becomes a sustained one, produced entirely by the defence.

The wrong clients are caught. Ordinary behaviour is bursty and automated abuse is often smooth, so a limit calibrated against bursts catches people and misses what it was aimed at. That inversion is the single strongest argument for shaping rather than flattening.

A fourth failure is invisible: nobody reports it. Visitors who hit a limit meant for somebody else rarely write in. They conclude the site is unreliable and leave, so the cost arrives as absence rather than as a complaint anybody can investigate.

Three questions answer it, and none of them is about a number to start with.

What does one unit of ordinary use look like? A page view, a sync, a form submission: count the requests it actually produces. That count is the floor for any allowance, because refusing below it means refusing the behaviour itself.

How long is the quiet between units? The gap decides how fast an allowance can refill without ever letting sustained abuse through.

What is the expensive thing you are protecting? Where cost varies sharply between endpoints, one shape for everything is either too tight for the cheap ones or too loose for the expensive one, which is the argument for treating them separately rather than raising the single limit.

Once those are answered, values follow from your own traffic, and they need watching before they are enforced, which is the discipline described under monitor mode and operations. What subject the counting attaches to is a separate decision, covered under choosing a subject.

One case where a single shape cannot work, whatever the values.

An interface serving both people and machines is serving two entirely different patterns. Browsers burst and go quiet; integrations arrive on a schedule and keep a steady pace. A shape suited to one is wrong for the other, and averaging the two produces something that fits neither: too tight for a page load, too loose for a client that never stops.

Where both use the same paths, the honest options are to distinguish them by the subject the counting attaches to, or to treat the expensive paths separately from the ordinary ones. Both are decisions about structure rather than about numbers, and both are cheaper to make before enforcement than after somebody's integration stops working.

Stated separately from the design advice, and narrowly.

Rate limiting is applied per domain, which is the unit limits are counted against. Shaping described above is a way of thinking about limits generally rather than a description of controls, and where finer behaviour is needed it lives wherever your application knows about individual clients.