Testing a rate limit before it refuses real traffic

Before generating a single request, check the limit against traffic you already have.

Take your busiest legitimate client from the record: a partner API integration, a mobile application, an internal job. Count how its requests actually arrive, not on average but in the shape they arrive in, and walk that pattern against the proposed limit by hand. Would it have been refused? If so, you have found the problem for the price of an afternoon and nobody noticed.

Do this for at least three clients that differ in character: one that opens a connection and reuses it, one that fetches many things at once, and one that runs on a schedule and does all its work in a stretch. Those three cover most of the ways ordinary traffic fails to look average.

The paper test is also the only one that can be run before the limit exists, which makes it the right place to discover that the number needs to be different rather than the algorithm.

The second-cheapest test is to count without enforcing.

Where the limit can be evaluated and logged while every request is still served, do that first and read the result as a prediction: these are the clients that would have been refused, at these moments. It is the same discipline that makes filtering safe to turn on, and the same reason applies, which is that a prediction costs nothing while a refusal costs a customer.

Read the outcome by client rather than in total. A prediction concentrated on one or two clients is a limit doing its job; a thin spread across many different clients means the number or the subject is wrong, and no amount of load testing will tell you that as clearly.

Eventually you need to see the limit fire. Three constraints make that safe.

Test where it will not cost anything. An environment that matches production in topology matters far more than one that matches it in size: the same number of layers, the same entry point, the same arrangement of nodes. A limit behaves differently across several nodes than on one, which is the subject of distributed limits, and a single-node test can give a confident answer to a question you did not ask.

Do not point load at anybody else. Testing your own limit by hammering a third party's service, a payment provider, a search interface, or a partner's interface is somebody else's incident, and it is the sort of traffic that gets your identity treated as hostile long after the test is over. If the path you want to exercise crosses into somebody else's system, stop at the boundary or arrange it with them first.

Do not run the first real-traffic test at your busiest hour. The purpose is to learn cheaply, and a peak makes every lesson expensive. Announce it, run it when the affected route is quiet, and have the rollback ready before you start rather than after something surprises you.

Five things, and the first is the one people assume rather than check.

The effective limit. How many requests are accepted before the first refusal, measured through the real entry point rather than at one node. In a distributed setup this frequently differs from the configured number, and the difference is the point of measuring.

The boundary behaviour. What happens at the moment the window turns over or the bucket refills. Send traffic deliberately across that edge and see whether the result matches the algorithm you believe you are running.

The shape of the refusal. That it is the right HTTP status, that it carries a value telling the client when to return, and that its body is readable if a person receives it. The details are on 429, and a refusal that omits them produces the retry storm described below.

The recovery. How long after being refused a client is served again. This is what a well-behaved client experiences as the cost of the limit, and it is rarely the number anybody expected.

The retry storm. Refuse several clients at once and let them retry the way your real clients retry, which for most software means immediately. If your own limit sustains itself under that, you have found a defect in the refusal rather than in the client.

Each of these is discovered in production if it is not discovered here.

Two clients behind one address. Where the limit counts by IP address, put two independent clients behind a single one and confirm what happens. This is the office, the campus, and the mobile network, and it is the most common cause of a limit refusing people who did nothing.

The exemption you rely on. If your monitoring, deployment tooling, or an internal job is meant to be exempt, verify the exemption before you need it. An exemption that was never tested is a belief, and the moment it matters is the moment you find out.

The client you forgot. Something of yours arrives at your own site on a schedule and is anonymous to the limit. Find it now rather than during the first incident it causes.

Testing tells you the limit is right in the conditions you tested; enforcement tells you whether it is right in the conditions you have.

Turn it on the way you would turn on anything else that can refuse a customer: one route, watched for a full cycle of that route, with a rollback trigger written down beforehand. That sequence, and how to choose the first slice, is set out under staged rollout.