Data scraping bots: how to spot and slow them down safely

Four behavioural differences, none of which depends on what the client calls itself.

What it fetches

• It fetches only what it wants to keep. A browser loads a page and its assets. A crawler that renders behaves similarly. A scraper takes the document and nothing else, because images and stylesheets are cost without value to it.

How it behaves around the rules

• Its coverage is systematic where a crawler's is interest-driven. Walking identifiers in order, exhausting a category, or requesting pages in the sequence a database would produce rather than the sequence links suggest.
• It ignores what you published as limits. Instructions for cooperative crawlers, robots.txt among them, are followed by cooperative crawlers, which is precisely the population that was never the problem, as covered on blocking.
• It cannot be verified. Legitimate crawlers offer a way to confirm they are who they claim, and that verification is a property of the operator rather than a header. Automation with no verifiable identity is not automatically hostile, but it also has no claim on the access you grant to identified crawlers.

Scraping and crawling

The catalogue of automation that identifies itself and can be verified, including the crawlers that read sites for search and answer engines, is maintained on the main site of this network.

Web scraping vs web crawling comes down to purpose rather than method: both fetch pages automatically, and only what happens to the content afterwards separates them.

Web scraping is the automated extraction of data from websites, a form of data scraping, and a content scraper is the program that does it to your pages. The definition that matters to the owner of the site being read is practical: what the extraction costs, and who is doing it.

Screen scraping reads data from what is displayed: a rendered page, a terminal screen, an application window. Web scraping reads the HTML or the API responses behind a site. The words are often used for the same thing, and for a site owner the difference is mostly in how visible the scraper is.

A scraper that drives a real browser looks like a visitor who never stops reading. One that fetches HTML directly is cheaper to run and easier to recognise, because it skips the assets, scripts and timing a browser produces.

Web scraping tools and web scraping software range from a short script written for python web scraping to hosted services that spread their requests over many addresses. A web scraper can change its defaults in one line, so a rule built on the tool is short-lived; the pattern of requests is much harder for it to hide.

Web scraping tools: what they leave in your logs

A web scraping attack is bulk copying the owner did not agree to, as opposed to the aggregator you did. The tool behind it decides what the sweep leaves in your logs.

• A script or HTTP library takes the HTML and nothing else: documents with no assets and the same request shape on every fetch.
• A headless browser loads the page and runs its scripts. Assets arrive, but nobody scrolls, signs in or comes back, and the paths follow a list rather than the links.
• A hosted scraping service spreads one sweep over many addresses. No single address stands out; the sweep appears as coverage of the whole catalogue in a short span.

Write the limit around the cost of the pages being walked, as in the order of responses below, and turn it on in monitor mode first, so that it records which clients the limit would have refused before any of them is refused.

A page that no link leads to separates a sweep from a visitor, which is the idea behind a cybersecurity honeypot. Whoever requests it did not arrive by browsing.

• Your capacity, in proportion to how expensive your pages are. A catalogue that assembles from a database is far more costly to serve at scale than a static article.
• Your position in search results. A copy of your content competes with you wherever it is published. Whether a copy ever outranks an original depends on many things outside this page, and the honest claim is narrower than the usual scare: the copy exists, it is indexed, and it did not cost its publisher anything to produce.
• Your commercial position. Where the data is prices or inventory, the value is in its freshness, and continuous copying transfers exactly that.

Scraper sites: where the copies end up

A scraper site republishes content taken from other sites, adding little or nothing of its own. It is the output of the sweep, and for the owner of the original it is the evidence that a sweep succeeded.

A copy shows what was taken and roughly when. If it carries a page that exists only at an address nobody links to, the sweep reached that address from a list, not by following links. A distinctive phrase placed on one page you can tell apart does the same job for ordinary text: when it appears elsewhere, you know which page was read.

Getting a copy removed is a request to the site's host or to a search engine, each with its own procedure. This page does not describe those procedures, and the legal side is covered below under legal avenues. What you control is the sweep itself. Limits weighted by cost, reduced responses and identified access for partners narrow what the next copy can contain.

Three patterns, and the third is the most reliable.

• Coverage without depth. Many distinct pages, each fetched once, with no returning visits and no interest in anything interactive.
• Requests for what nobody links to. Pages reachable only by constructing a path, or by reading a machine-readable index of your site, and never reached by a person browsing.
• Cost concentrated in one client. The measurement that does not require guessing intent: a single source consuming a share of your rendering budget unlike any plausible visitor.

To scrape website content in bulk, a client first needs a list of addresses, taken from a sitemap, a category page or a pattern in the URLs. Website data scraping therefore shows up as a sweep through that list, which is also how a client that wants to scrape a website for data differs from a visitor following links.

Responses in order of severity, following the ladder on blocking. Asked how to stop web scraping or how to prevent web scraping, the honest answer is to make bulk copying slow and costly: a page that is public to every visitor is public to a script too.

The order of responses

1. Limit by cost, not by request count. Expensive pages deserve tighter allowances, which throttles bulk copying while leaving ordinary browsing untouched.
2. Degrade rather than refuse. Serving a cached version, a smaller page, or results without the expensive parts removes most of the value to a copier and remains usable for a misclassified visitor.
3. Challenge on the paths that matter. Catalogue and listing pages rather than the whole site, so the cost falls where the value is.
4. Require identity for bulk access, and offer it. Where the data has legitimate demand, an interface with a key is cheaper for both sides than an arms race, and it converts an adversary into something you can rate limit by account.

Legal avenues

Legal avenues exist and are outside what this page can advise on: what protection applies to your material, and what your terms permit, depends on your jurisdiction and on the nature of the content. That is a question for a lawyer, best asked before an incident rather than during one.

The obligatory section, and on this page it is the longest one for a reason: the lookalikes are numerous and several of them are commercially important to you.

• Aggregators and comparison services fetch catalogues systematically because that is their function, and many of them send you customers.
• Partners under contract. Integrations you agreed to, often built years ago by someone who has left, calling you from infrastructure with no identifying credential.
• Preview fetchers. When someone shares your link in a chat application or a social platform, something fetches the page to build the preview. Blocking those makes your links look broken wherever they are shared, and nothing in your logs connects the two.
• Archives and research. Preservation projects and academic crawlers, which are systematic by design.
• Feed readers and accessibility tools, which fetch documents without assets for entirely legitimate reasons.

The practical protection is the one from the mother page: know which automation you expect, give it identity where you can, and prefer measures that slow rather than refuse. A blocked aggregator does not complain; it simply stops sending you customers, and you will attribute the decline to something else.

The same measures against repeated login attempts are covered under brute force attack, and the challenge that sometimes sits in front of them has an captcha page.

Credential abuse by automation is described under what is credential stuffing attack, and bots that fill in forms are listed under spammer.

Bridge WAF applies rate limiting per domain, which bounds what a domain as a whole will serve: bulk copying that pushes past the limit is refused at the edge rather than served by your application, and monitor mode records what a limit would have refused before it refuses anything. What no product decides is which copying you mind: a comparison service and a competitor scraping prices produce identical traffic and opposite business outcomes. See what the platform offers at Bridge CDN.