Content scrapers: how to spot and slow them down safely

Four behavioural differences, none of which depends on what the client calls itself.

It fetches only what it wants to keep. A browser loads a page and its assets. A crawler that renders behaves similarly. A scraper takes the document and nothing else, because images and stylesheets are cost without value to it.

Its coverage is systematic where a crawler's is interest-driven. Walking identifiers in order, exhausting a category, or requesting pages in the sequence a database would produce rather than the sequence links suggest.

It ignores what you published as limits. Instructions for cooperative crawlers are followed by cooperative crawlers, which is precisely the population that was never the problem, as covered on blocking.

It cannot be verified. Legitimate crawlers offer a way to confirm they are who they claim, and that verification is a property of the operator rather than a header. Automation with no verifiable identity is not automatically hostile, but it also has no claim on the access you grant to identified crawlers.

The catalogue of automation that identifies itself and can be verified, including the crawlers that read sites for search and answer engines, is maintained on the main site of this network.

Your capacity, in proportion to how expensive your pages are. A catalogue that assembles from a database is far more costly to serve at scale than a static article.

Your position in search results. A copy of your content competes with you wherever it is published. Whether a copy ever outranks an original depends on many things outside this page, and the honest claim is narrower than the usual scare: the copy exists, it is indexed, and it did not cost its publisher anything to produce.

Your commercial position. Where the data is prices or inventory, the value is in its freshness, and continuous copying transfers exactly that.

Three patterns, and the third is the most reliable.

Coverage without depth. Many distinct pages, each fetched once, with no returning visits and no interest in anything interactive.

Requests for what nobody links to. Pages reachable only by constructing a path, or by reading a machine-readable index of your site, and never reached by a person browsing.

Cost concentrated in one client. The measurement that does not require guessing intent: a single source consuming a share of your rendering budget unlike any plausible visitor.

Responses in order of severity, following the ladder on blocking.

Limit by cost, not by request count. Expensive pages deserve tighter allowances, which throttles bulk copying while leaving ordinary browsing untouched.

Degrade rather than refuse. Serving a cached version, a smaller page, or results without the expensive parts removes most of the value to a copier and remains usable for a misclassified visitor.

Challenge on the paths that matter. Catalogue and listing pages rather than the whole site, so the cost falls where the value is.

Require identity for bulk access, and offer it. Where the data has legitimate demand, an interface with a key is cheaper for both sides than an arms race, and it converts an adversary into something you can rate limit by account.

Legal avenues exist and are outside what this page can advise on: what protection applies to your material, and what your terms permit, depends on your jurisdiction and on the nature of the content. That is a question for a lawyer, best asked before an incident rather than during one.

The obligatory section, and on this page it is the longest one for a reason: the lookalikes are numerous and several of them are commercially important to you.

Aggregators and comparison services fetch catalogues systematically because that is their function, and many of them send you customers.

Partners under contract. Integrations you agreed to, often built years ago by someone who has left, calling you from infrastructure with no identifying credential.

Preview fetchers. When someone shares your link in a chat application or a social platform, something fetches the page to build the preview. Blocking those makes your links look broken wherever they are shared, and nothing in your logs connects the two.

Archives and research. Preservation projects and academic crawlers, which are systematic by design.

Feed readers and accessibility tools, which fetch documents without assets for entirely legitimate reasons.

The practical protection is the one from the mother page: know which automation you expect, give it identity where you can, and prefer measures that slow rather than refuse. A blocked aggregator does not complain; it simply stops sending you customers, and you will attribute the decline to something else.

Bridge WAF applies rate limiting per domain, which bounds what a domain as a whole will serve: bulk copying that pushes past the limit is refused at the edge rather than served by your application, and monitor mode records what a limit would have refused before it refuses anything. What no product decides is which copying you mind: a comparison service and a competitor scraping prices produce identical traffic and opposite business outcomes. See what the platform offers at Bridge CDN.