Malicious bots: what actually needs blocking

Three properties separate harmful automation from the rest, and none of them is «it is a bot».

It takes something. Content, inventory, data, advertising budget, or the ability of real customers to be served.

It hides what it is. Legitimate automation identifies itself and can be verified. Harmful automation copies browser characteristics precisely because being recognised would end it.

It ignores what you asked. Instructions about which parts of your site may be crawled are followed by the automation that respects them, which is the automation that was never a problem.

Automation that identifies itself, is verifiable, and behaves within limits is a different subject with a different answer. The catalogue of that kind, including the crawlers that read sites for search and answer engines, is maintained at BotBridge.

Each type has its own signals and its own cost of being wrong, which is why they are documented separately.

Accept this first: it decides which methods are appropriate.

Nothing in a request proves it came from a person. Every characteristic a browser announces can be reproduced, every timing pattern can be varied, and every source address can be changed. What remains is evidence rather than proof: behaviour across many requests, consistency between what a client claims and how it acts, and whether a source has a history.

The consequence is that decisions are made under uncertainty. A method's value is therefore not only how much automation it identifies, but what it costs when it is wrong, and those two frequently point in opposite directions.

Every blocking decision has a failure mode on each side, and they are not symmetrical.

Blocking a customer loses a sale and, more expensively, a person who now believes your site is broken. They do not file a report. They leave, and nothing in your logs records what did not happen.

Blocking legitimate automation is quieter and often worse. Search crawlers stop indexing, monitoring stops reporting, payment callbacks stop arriving, and integrations partners depend on fail silently. Each of these surfaces days or weeks later, disconnected from the change that caused it.

Allowing harmful automation costs whatever that type takes, which is usually gradual rather than catastrophic.

The asymmetry explains the shape of every recommendation in this material: prefer measures that ask rather than refuse, prefer measures that expire, and know what legitimate automation calls you before enforcing anything.

Two pieces of preparation make everything afterwards safer.

Inventory your own automation. Monitoring, uptime checks, payment callbacks, partner integrations, internal jobs, and anything a colleague built years ago. Each looks exactly like the traffic you are about to block: no browser, no session, regular timing.

Give expected automation identity. A credential or a key ties access to a known party, which survives address changes and lets you exempt it deliberately rather than by accident. Guessing from addresses fails the first time a provider renumbers.

Bot traffic rarely announces itself as an availability problem, which is why it runs for a long time before anyone investigates.

In your costs. Bandwidth, database load, and capacity provisioned for traffic that never buys anything. The bill grows smoothly, so it reads as success rather than as a problem.

In your data. Analytics that include automation describe a customer base that does not exist. Decisions get made on those numbers, which makes this the most expensive quiet failure in the list.

In your customers' experience. Inventory that is never available, forms full of spam, and accounts whose owners are locked out. None of these appears in an uptime graph, and all of them are visible to the people you depend on.

That is the argument for treating this as ordinary operational work rather than as an incident response. The costs accumulate instead of arriving.

Instructions for well-behaved automation are a courtesy, not a control: they work only for automation that chooses to read them. Useful for managing load from cooperative crawlers, worthless against anything hostile.

Identification by what a client claims to be is trivially bypassed, since the claim is set by the client. It removes lazy automation and nothing else.

Reputation of the source carries useful signal and misfires on shared addresses, which is common on mobile networks and behind corporate egress.

Behavioural analysis looks at what a client does across requests rather than at what it declares, and it is the most reliable signal available, at the cost of needing enough requests to judge.

Challenges ask the client to do something automation finds expensive, and they are the safest measure at low confidence, because a customer who passes is inconvenienced rather than refused.

Bridge WAF takes a narrower slice of this than the list above, and says so: bot classification works on declared identity — the first and weakest of the five signals — and rate limiting is applied per domain. Matches are recorded in the activity log, so monitor mode shows what enforcement would have refused before anything is refused; reputation scoring and behavioural analysis are not part of this layer. What no system removes is the judgement about which automation you want, which is why the inventory above comes first. See what the platform offers at Bridge CDN.