Good bots: what they are and how to tell they are real

A good bot is automation you expect, or one your business quietly depends on. What makes it good is how it behaves and whether it respects your rules, not the fact that it is not a person. Most advice about good bots stops at a list of names. The harder part is knowing which ones you actually have, checking that a visitor claiming to be one really is, and not switching them off by accident when you start blocking.

Section Bots Updated Question is it real?

Quick facts

Definition
Automation you expect or depend on, that identifies itself and follows the rules you publish
What decides it
Behaviour and compliance, not the fact of automation
The user-agent string
A claim written by the sender; see user agent
Google's check
Reverse DNS then forward DNS, or its published IP ranges; see Verifying Googlebot
Does Bridge WAF verify them?
No. It does not check bots by address or reverse DNS and keeps no catalogue of bots. Exceptions and monitor mode help you avoid breaking them.

What counts as a good bot

The phrase sounds like a category of software. It is really a relationship: a bot is good for you when you would notice its absence. Search crawlers are the familiar case, since a site they cannot reach drops out of results. They are far from the only one.

  • Search and answer-engine crawlers. They fetch pages so that people can find them.
  • Uptime and performance monitors. They request a page on a schedule and raise an alert when it fails. Block them and you lose the alarm.
  • Payment and integration callbacks. A provider sends a notification to a URL on your site, a webhook, when an order is paid or a subscription changes. Refuse it and the order never completes on your side.
  • Link previews. Social networks and messengers fetch a page to build the card shown when someone shares your link.
  • Feed aggregators and readers. They poll your feeds so subscribers see new items.
  • Partner and internal integrations. A supplier pulling stock levels, a reporting job calling your API, a script your own team runs overnight.

Two of those have an uncomfortable property: nobody visits your site because of them, so their traffic never shows up in anyone's analytics, and the first sign of a block is a business process failing somewhere else.

Taking inventory of your own

Lists of well-known bots are easy to find. They do not tell you which ones you have. That list is specific to your site and only you can write it, ideally before any blocking rule is turned on.

  • Start from what breaks. For each business process, ask what calls your site from outside: payment confirmations, shipping updates, form handlers, identity providers, status checks, a mobile app talking to an API.
  • Add what you want crawling. Which search engines and answer engines should reach you? That is a decision, and it differs between sites. For one AI crawler in detail, see GPTBot.
  • Then read your logs. Look for clients that request at regular intervals, hit a single path, or never load images and scripts. Some will be on your list. Some will be a surprise, and a surprise is either a forgotten integration or something you have not judged yet.
  • Record how each one is identified. A credential or signature on the request is better than a name. A published address list is the next best. A name alone is the weakest.

Write each entry down with an owner and a reason it exists. Entries that nobody can explain a year later are the ones that should be reviewed.

Behaviour, not label

A good bot is not a safe bot. A crawler from a legitimate operator can still request pages faster than your origin can serve them, and an aggressive but honest integration can cost real money. Conversely, an automated visitor that names itself, asks politely and stays within limits is no problem even if you never wanted it.

So the useful questions are about conduct. Does it identify itself consistently? Can its claim be checked? Does it respect the instructions you publish, including the paths you ask it to leave alone? Does it slow down when you answer with an error that asks it to? Automation that fails the second test is the most important case, because it is pretending, and bot detection is the page that covers how a client's claim is weighed against what it actually does. The wider line between harmful and harmless automation is drawn on the bots overview.

Where a fast but honest client causes load, the remedy is a limit on how much it may take, not a verdict on what it is.

Checking that a bot is real

Because the name a client sends is a statement it wrote itself, anyone can send a name that belongs to a famous crawler. Sites that wave through "Googlebot" on the strength of the string alone give scrapers a free pass. Verification means checking something the sender cannot simply type.

The method Google documents

Google's page describes two routes. For one-off lookups, run a reverse DNS lookup on the requesting address, confirm the name belongs to googlebot.com, google.com or googleusercontent.com, then run a forward lookup on that name and confirm it returns the original address. For large-scale checks, match the address against the IP ranges Google publishes as JSON lists, with separate lists for its common crawlers, special crawlers and user-triggered fetchers.

Two-step DNS check, as described by Google
1. reverse lookup:  requesting IP  ->  hostname
2. check the hostname ends in an allowed Google domain
3. forward lookup:  hostname  ->  IP
4. the IP from step 3 must equal the requesting IP

The forward step matters. Whoever controls the address space can set the reverse record to say anything, so the reverse result alone proves nothing.

For every other bot

Do not assume that Google's procedure carries over. Each operator documents its own: some publish address lists, some sign their requests, some offer a verification endpoint. Webhook senders in particular usually sign the body so you can check it in your application. Read the sender's own documentation, and treat a bot with no documented way to verify it as unverified, however polite its string.

Verification belongs in the layer that can do it. A check on an address and a signature needs the original client address, the sender's current ranges and, for signatures, the shared secret, which usually means your application or an edge service built for the purpose.

What robots.txt does for them

The robots.txt file is how you speak to the good bots you want crawling: which paths they may fetch and which they should leave alone. A well-run crawler reads it before anything else and follows it. RFC 9309, the standard behind the file, is explicit that its rules are not a form of access authorization, so it is an instruction for cooperative automation and nothing more.

That makes it useful for the inventory too. The groups you write name the crawlers you have decided to talk to, and a group naming a crawler that nobody on your team can explain is worth a look. It does not replace verification: a hostile client can read the same file and ignore it. The failure mode runs both ways, since a stray Disallow: / can cut your own search visibility without any attack at all.

Keeping them working when you block

The moment you start refusing traffic, good bots become the first casualties, because they are regular, identifiable and easy to match by accident. A short routine prevents most of the damage. The ladder of responses, and what each costs a wrongly classified client, is laid out on blocking bad bots.

  • Write the list first. Use the inventory above before enabling any rule.
  • Give each bot its own identity. One exception for "all crawlers" lets an impostor through. One entry per bot, matched by the strongest signal you recorded, means you can see and revoke each on its own.
  • Make narrow exceptions. Limit an exemption to the path the bot needs, such as the webhook endpoint, instead of the whole site. The mechanics are on filtering exceptions.
  • Run new rules in monitor first. Read the log for entries that match something on your list before any rule can refuse. Monitor mode exists for this.
  • Watch the quiet failures. After switching a rule to block, confirm the monitor still reports, the payment provider still shows successful deliveries, and crawl activity has not dropped. False positives describes how to find and fix a wrongful match.

Where a WAF helps and where it does not

Bridge WAF has three modes: off, monitor and block. In monitor, requests that a rule would have refused are recorded in the activity log and nothing is refused. That is the tool for this job: you see whether the rule would have caught your payment callback before it can. In block, a filtering exception lets a request you have decided to trust pass. Those two features, used in that order, are what Bridge WAF contributes to keeping good bots working.

What it does not do is settle whether a bot is genuine. It does not check a client by reverse DNS or published ranges, and it does not hold a catalogue of crawlers and their addresses. It cannot tell you that a request calling itself Googlebot is the real one. The proof described above has to come from you, from your application, or from another service, and the exception you create is only as trustworthy as the signal it matches. An exception keyed to a name alone trusts a statement anyone can forge.

So the honest division of work is this. Verification happens outside the WAF. The WAF's part is to let you start in monitor, read what it would have done, and then exempt what you have already confirmed by other means. Nothing here guarantees that a bot you depend on is never matched by mistake, and no mode can promise that, which is why the inventory comes first.

Questions

What is a good bot?

Automation you expect or depend on, that identifies itself and respects your rules. Search crawlers, uptime monitors, payment and integration callbacks, link-preview fetchers and feed readers are the usual examples. What makes a bot good is how it behaves, not the fact that it is automated.

How do I know a bot claiming to be Googlebot is real?

Do not trust the user-agent string, because the sender writes it. Google documents two checks: a reverse DNS lookup on the requesting address followed by a forward lookup that must return the same address, or a match against the IP ranges Google publishes. The hostname must belong to googlebot.com, google.com or googleusercontent.com.

Can a good bot turn bad?

Its conduct can. A legitimate crawler or integration can fetch faster than your origin copes with, or ignore a path you asked it to leave alone. The remedy is usually a limit on how much it may take, with the operator contacted, rather than treating it as hostile.

Does Bridge WAF verify that a bot is genuine?

No. It does not check bots by address or reverse DNS and it keeps no catalogue of them. Monitor mode shows what a rule would have refused, and an exception lets a request you have already confirmed pass, but the confirmation itself happens outside it.

Should I block good bots?

Only on purpose. Blocking a search crawler removes you from its results, and blocking a payment callback breaks orders. If you want less crawling, say so in robots.txt, which cooperative crawlers follow, and limit the rate. Do not leave it to a rule meant for other traffic.