Bridge WAF
How to block bad bots without blocking your customers
Blocking is a response, not a judgement. Deciding what traffic is happens before this, and it happens under uncertainty, which means every response here will sometimes land on a customer. Choosing between responses is therefore a question about consequences: what happens to a real person who receives this, and what happens to automation that does.
The ladder of responses
Ordered by severity, and the useful default is to start further up it than instinct suggests.
Observe. Record the decision without acting on it. The only way to learn what a rule would have done before it does it.
Slow. Delay responses to a client rather than refusing them. Automation depending on throughput loses most of its value; a person notices a sluggish page and continues.
Challenge. Ask for proof that costs little once and much at scale. The safest active response when confidence is low.
Degrade. Serve something cheaper: a cached version, a smaller result, a page without the expensive part. The client gets an answer, your backend keeps its capacity, and a customer who was misclassified still receives something usable.
Refuse. Return an error and stop. The only response that guarantees a misclassified customer is lost, and therefore the one that needs the highest confidence.
Most deployments jump to the last rung immediately, which is why their false positives are expensive.
What is not a control
Instructions for cooperative crawlers are not protection. The file at the root of your site that says which paths may be crawled is a convention followed by automation that chooses to follow it. Anything hostile ignores it, and nothing enforces it.
That is not a criticism of the mechanism, which does its job well: it manages load from cooperative crawlers and keeps them out of sections that waste their time and yours. But it belongs in the category of communication, not defence, and using it as defence has one specific failure: listing sensitive paths there tells anyone who reads it where those paths are, and the file is public by definition.
Blocking by what a client claims to be removes automation that never tried to hide. The declaration is written by the sender, so a rule matching it stops working the moment someone changes one line, and in the meantime it can catch real browsers that happen to match.
Blocking by address, and why it disappoints
The instinct is to block the source. Three properties make it weaker than it looks.
Addresses are shared. Mobile networks and corporate egress put many people behind one, so a block lands on a population rather than on a sender.
Addresses rotate. Automation that finds an address blocked uses another, and the supply is effectively unlimited.
Addresses outlive the reason. A block added during an incident stays until someone removes it, and the address may be serving a customer months later. Anything address-based needs an expiry attached at the moment it is created.
It is still worth doing where the source is infrastructure rather than a consumer network, and where the block is temporary. As a standing strategy it accumulates cost quietly.
Limits as a response
Constraining how much a client may consume is the most broadly useful active measure, because it does not require deciding what the client is: it only requires deciding what a client may have. The design of such limits, including how to weight them by cost and which algorithms behave well under bursts, is a subject of its own and is covered separately.
Two properties matter here. Limits degrade rather than deny, which suits uncertain classification. And they apply equally to automation you approved, which is why the exemptions below come first.
Operational rules that prevent regret
Five habits, and they matter more than the choice of method.
Start in observation mode. Every rule runs without acting until you have seen what it would have caught.
Attach an expiry to everything specific. Blocks, address lists, and rules written against a particular campaign should end on their own. What stays permanently should be structural.
Maintain an exemption list deliberately. Monitoring, payment callbacks, partner systems, and internal jobs, each identified by a credential rather than by an address wherever possible.
Log what decided, not just what happened. A block with no recorded reason cannot be argued about, and the argument always comes from someone whose integration broke.
Have a way to turn it off quickly. Under pressure, the ability to revert everything in one action prevents the improvised changes that outlive the incident.
The failure you will not see
A customer refused by mistake does not complain. They assume the site is broken and leave, and nothing in your logs distinguishes them from a visitor who never arrived. The same is true of automation you depend on: search crawlers stop fetching, monitoring stops reporting, and a callback starts failing at a partner who tells you about it days later.
That asymmetry, and not the effectiveness of any method, is the reason to prefer responses that ask over responses that refuse.
Blocking as part of the stack
Bridge WAF covers three rungs of the ladder above and not the others. Off, monitor, and block are the modes; monitor is the observation step, and it is the one worth starting on. Rate limiting applies per domain, which is the slowing measure. Flagged requests appear in a near-real-time activity log, so a decision has a record behind it.
What is not there: challenges, degraded responses, and any dial between monitor and block. Those belong to your application or to another layer, and a page that recommends preferring them should say where they come from. The exemption list is yours in either case, since only you know which automation you depend on. See what the platform offers at Bridge CDN.
Questions
Does robots.txt block bad bots?
No. It is a convention observed by cooperative automation and ignored by everything else, so it manages the traffic you were not worried about. Listing sensitive paths in it also publishes those paths to anything reading the file to find them.
Is blocking by user agent worth doing?
Only against automation that is not hiding, which limits its usefulness sharply. The declaration is chosen by the sender and altered in a single line of configuration, so anything with a reason to avoid the rule avoids it in seconds.
Why is address blocking unreliable?
Three properties work against it. Addresses are shared, so one block can catch many unrelated people; they rotate, so the target moves; and blocks outlive the reason for them, so an address blocked today refuses somebody entirely different next month.
What should I do when I am unsure?
Slow the client down, challenge it, or serve something reduced, rather than refusing outright. All three leave a misclassified customer able to continue, which is the whole difference between an inconvenience they forget and a departure you never hear about at all.
How do I avoid breaking my own automation?
Three steps in order: list the automation you depend on before enforcing anything, identify each entry by a credential rather than an address so the exemption survives infrastructure changes, and run every new rule in observation mode long enough to see what it would have caught.
How long should a block last?
For a period decided at the moment it is created, whatever that period is. A block without an expiry becomes a permanent decision that nobody remembers making, applied by nobody to a reason that no longer exists, and it is usually discovered years later during an unrelated investigation.
Three rungs of the ladder and not the others. Challenges and degraded responses come from somewhere else, and this page says so.
off
Nothing is inspected and nothing is recorded.
monitor
The observation rung: every rule runs without acting until you have seen what it would have caught.
block
The refusing rung, with a record behind each decision in the activity log.
The firewall layer is included with Bridge CDN. Start in monitor: nothing is refused until you decide it should be.
Get started