robots.txt: what it is, how rules work, and how to check it

robots.txt is a small text file at the root of a website that tells crawlers which paths they may request. It manages crawl traffic by courtesy. It does not hide anything, and it does not stop a bot that chooses to ignore it. Reading one takes a single command, and this page explains what the lines mean and where the file stops being useful.

Section Tools Updated Check browser or curl

Quick facts

Location
/robots.txt in the top-level path of the site, served as text/plain, UTF-8 (RFC 9309)
Scope
One file per scheme, host and port; a subdomain needs its own
Standard
Robots Exclusion Protocol, RFC 9309
Enforcement
None. Crawlers choose to follow it (Google)
Access control
Not a form of access authorization (RFC 9309)
Missing file
4xx: a crawler may access any resource. 5xx: it must assume complete disallow
Cache
Crawlers should not use a cached copy for more than 24 hours, unless the file is unreachable
Also searched as
robots txt, what is robots.txt, robots.txt file, * robots txt

The short version: robots.txt asks. Whether a crawler listens is up to the crawler.

robots.txt syntax

The file is a list of groups. Each group starts with one or more User-agent lines naming the crawlers it applies to, followed by Allow and Disallow rules for URL paths. A Sitemap line points to a sitemap file. In RFC 9309 a sitemap record is an optional extra that a crawler may interpret, and it does not end a group.

robots.txt
# Rules for every crawler that has no group of its own
User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help

# A separate group for one named crawler
User-agent: examplebot
Disallow: /

Sitemap: https://example.com/sitemap.xml
LineMeaning
User-agentNames the crawler a group applies to; * matches any crawler not matched by a more specific group
DisallowA path prefix the crawler should not request
AllowA path prefix that is permitted, usually to carve an exception out of a wider Disallow
SitemapThe absolute URL of a sitemap; applies to the file, not to one group

An empty Disallow: value blocks nothing. A lone Disallow: / blocks the whole site for that group. Other fields exist in the wild: Google's own page notes that fields such as crawl-delay are not supported by its crawlers and are ignored.

How rules are matched

Two special characters work inside a path. Under RFC 9309, * stands for zero or more of any character, and $ marks the end of the URL. So Disallow: /*.pdf$ covers addresses ending in .pdf, and Disallow: /private without a $ covers everything that begins with that text. A companion file for vulnerability reports is security.txt. A path listed here is no protection against forced browsing.

When several rules match one URL, the most specific one wins, and the RFC defines that as the rule with the most octets in its match. If an Allow and a Disallow match equally, Allow is used. Order in the file does not decide the outcome.

RulesRequestResult
Disallow: /search
Allow: /search/help
/search/helpAllowed: the longer match is Allow
Disallow: /search
Allow: /search/help
/search?q=aDisallowed: only Disallow matches
Disallow: /*.pdf$/docs/a.pdfDisallowed
Disallow: /*.pdf$/docs/a.pdf?v=2Not matched: the URL does not end in .pdf

A crawler picks the group whose user-agent name matches it best and falls back to * when none does. The RFC also sets a floor on file size: a parser must handle at least 500 kibibytes, so keep the file well below that.

What robots.txt is not

  1. Not protection. Google Search Central says the instructions cannot enforce crawler behaviour; it is up to the crawler to obey them. A scraper that never requests the file sees no rules at all.
  2. Not private. Anyone can read it, and the RFC points out that listing paths there makes them discoverable. A line like Disallow: /admin-backup/ tells a curious visitor exactly where to look.
  3. Not a way out of the index. Google states that robots.txt is not a mechanism for keeping a page out of Google. A blocked URL can still appear in results if other sites link to it. Use a noindex rule or authentication instead.
  4. Noindex needs crawling. A crawler can only read a noindex rule on a page it is allowed to fetch. Blocking the page in robots.txt and adding noindex to it cancel each other out. MDN describes the meta robots tag and the X-Robots-Tag header as the indexing controls for crawlable resources.

The file's real job is to steer polite crawlers away from low-value URLs, such as internal search results or endless filter combinations, so they spend requests on pages you want read.

How to check a robots.txt file

This page does not fetch or scan anything. You read the file yourself, and the same steps work on any site.

  1. Open the address. Go to https://example.com/robots.txt. A file for the apex domain does not cover www or other subdomains.
  2. Read the status code first. A 200 with text means rules exist. A 404 means none do. A 5xx is a problem; see the next section.
  3. Find your group. Look for the User-agent line that names the crawler you care about, then for *.
  4. Test real URLs. Take a few addresses and apply the longest-match rule by hand, or use a parser you trust.
shell
curl -i https://example.com/robots.txt

The -i flag prints the status line and headers before the body, so you see the code, the content type and any cache headers in one view. Another read-only check from the same section is HTTP security headers, and a third answers is my IP blacklisted. All the tools live under Tools.

Response codes and caching

What the server returns for /robots.txt changes how a crawler behaves, and RFC 9309 spells it out.

ResponseWhat a compliant crawler does
2xx with a fileMust follow the rules it can parse
4xx, such as 404Treats the file as unavailable and may access any resource
5xx or unreachableMust assume complete disallow
RedirectsShould follow at least five consecutive redirects; after more than five it may treat the file as unavailable

The 5xx rule is the one that bites. An outage or a misconfigured deploy that makes the file return a server error can pause crawling of the whole site. Crawlers should not reuse a cached copy for more than 24 hours unless the file is unreachable, so a fix may take up to a day to be noticed. Google documents its own handling, including a 500 KiB limit and the way it caches for up to 24 hours, in its robots.txt specification.

AI crawlers and user-agent groups

Nothing in the protocol is specific to AI. Rules are matched by the user-agent name a crawler announces, so a site that wants different treatment for one crawler adds a group for that name, as examplebot is shown above. The operator of each crawler publishes the name to use; take it from their documentation, not from a list of guesses. The crawler OpenAI documents has its own page, GPTBot.

What this page does not claim is which crawlers read the file or follow it. That is each operator's choice, and the file cannot check. The sensible reading is the same as for any bot: robots.txt states your wishes, and your logs show what actually arrived.

Where a WAF helps and where it does not

robots.txt asks, and a WAF decides. The file works only if the crawler cooperates; a filter acts on requests whether or not the sender read your rules.

  • What a filter sees. Each request, its address, its headers and how often it arrives. That is enough to notice a client hammering URLs your robots.txt disallows. See content scrapers and bot detection for the signals involved.
  • What it does not see. Intent. A request that matches no rule looks like any other. A bot with a browser-like user-agent and a normal pace is hard to separate from a visitor.
  • What it does not replace. A WAF does not write or fix your robots.txt, and it does not remove URLs from search. Those stay with your server and your noindex rules.

Bridge WAF has three modes: off, monitor and block. Start in monitor, which records matches while every request is still served, read the activity log, then move rules to block once the matches look right. The steps are in WAF monitor mode. For clients that ignore the file, blocking bad bots covers refusal and rate limits.

Questions

What is robots.txt?

robots.txt is a plain-text file at the root of a site that tells crawlers which paths they may request. It follows the Robots Exclusion Protocol, standardised as RFC 9309. It is a request that well-behaved crawlers honour, not an enforcement mechanism.

Does robots.txt hide a page from Google?

No. Disallow stops a compliant crawler from fetching the page, but the URL can still be indexed if other pages link to it. To keep a page out of search results, use a noindex rule on a crawlable page, or put the page behind authentication.

Is robots.txt a security measure?

No. RFC 9309 says the rules are not a form of access authorization. The file is public, so listing a private path in it advertises that path. Protect sensitive content with authentication and server-side access control.

What happens if a site has no robots.txt?

If the file request returns a 4xx status such as 404, RFC 9309 lets a crawler access any resource on the site. A 5xx or unreachable file is treated differently: the crawler must assume complete disallow.

How do I check a site's robots.txt?

Open https://example.com/robots.txt in a browser, or run curl -i https://example.com/robots.txt to see the status code, headers and rules. Request the exact scheme, host and port you care about, because each has its own file.