robots.txt: what it is, how rules work, and how to check it
robots.txt is a small text file at the root of a website that tells crawlers which paths they may request. It manages crawl traffic by courtesy. It does not hide anything, and it does not stop a bot that chooses to ignore it. Reading one takes a single command, and this page explains what the lines mean and where the file stops being useful.
Quick facts
- Location
- /robots.txt in the top-level path of the site, served as text/plain, UTF-8 (RFC 9309)
- Scope
- One file per scheme, host and port; a subdomain needs its own
- Standard
- Robots Exclusion Protocol, RFC 9309
- Enforcement
- None. Crawlers choose to follow it (Google)
- Access control
- Not a form of access authorization (RFC 9309)
- Missing file
- 4xx: a crawler may access any resource. 5xx: it must assume complete disallow
- Cache
- Crawlers should not use a cached copy for more than 24 hours, unless the file is unreachable
- Also searched as
- robots txt, what is robots.txt, robots.txt file, * robots txt
The short version: robots.txt asks. Whether a crawler listens is up to the crawler.
robots.txt syntax
The file is a list of groups. Each group starts with one or more User-agent lines naming the crawlers it applies to, followed by Allow and Disallow rules for URL paths. A Sitemap line points to a sitemap file. In RFC 9309 a sitemap record is an optional extra that a crawler may interpret, and it does not end a group.
# Rules for every crawler that has no group of its own
User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help
# A separate group for one named crawler
User-agent: examplebot
Disallow: /
Sitemap: https://example.com/sitemap.xml
| Line | Meaning |
|---|---|
| User-agent | Names the crawler a group applies to; * matches any crawler not matched by a more specific group |
| Disallow | A path prefix the crawler should not request |
| Allow | A path prefix that is permitted, usually to carve an exception out of a wider Disallow |
| Sitemap | The absolute URL of a sitemap; applies to the file, not to one group |
An empty Disallow: value blocks nothing. A lone Disallow: / blocks the whole site for that group. Other fields exist in the wild: Google's own page notes that fields such as crawl-delay are not supported by its crawlers and are ignored.
How rules are matched
Two special characters work inside a path. Under RFC 9309, * stands for zero or more of any character, and $ marks the end of the URL. So Disallow: /*.pdf$ covers addresses ending in .pdf, and Disallow: /private without a $ covers everything that begins with that text. A companion file for vulnerability reports is security.txt. A path listed here is no protection against forced browsing.
When several rules match one URL, the most specific one wins, and the RFC defines that as the rule with the most octets in its match. If an Allow and a Disallow match equally, Allow is used. Order in the file does not decide the outcome.
| Rules | Request | Result |
|---|---|---|
| Disallow: /search Allow: /search/help | /search/help | Allowed: the longer match is Allow |
| Disallow: /search Allow: /search/help | /search?q=a | Disallowed: only Disallow matches |
| Disallow: /*.pdf$ | /docs/a.pdf | Disallowed |
| Disallow: /*.pdf$ | /docs/a.pdf?v=2 | Not matched: the URL does not end in .pdf |
A crawler picks the group whose user-agent name matches it best and falls back to * when none does. The RFC also sets a floor on file size: a parser must handle at least 500 kibibytes, so keep the file well below that.
What robots.txt is not
- Not protection. Google Search Central says the instructions cannot enforce crawler behaviour; it is up to the crawler to obey them. A scraper that never requests the file sees no rules at all.
- Not private. Anyone can read it, and the RFC points out that listing paths there makes them discoverable. A line like
Disallow: /admin-backup/tells a curious visitor exactly where to look. - Not a way out of the index. Google states that robots.txt is not a mechanism for keeping a page out of Google. A blocked URL can still appear in results if other sites link to it. Use a noindex rule or authentication instead.
- Noindex needs crawling. A crawler can only read a noindex rule on a page it is allowed to fetch. Blocking the page in robots.txt and adding noindex to it cancel each other out. MDN describes the meta robots tag and the X-Robots-Tag header as the indexing controls for crawlable resources.
The file's real job is to steer polite crawlers away from low-value URLs, such as internal search results or endless filter combinations, so they spend requests on pages you want read.
How to check a robots.txt file
This page does not fetch or scan anything. You read the file yourself, and the same steps work on any site.
- Open the address. Go to
https://example.com/robots.txt. A file for the apex domain does not coverwwwor other subdomains. - Read the status code first. A 200 with text means rules exist. A 404 means none do. A 5xx is a problem; see the next section.
- Find your group. Look for the
User-agentline that names the crawler you care about, then for*. - Test real URLs. Take a few addresses and apply the longest-match rule by hand, or use a parser you trust.
curl -i https://example.com/robots.txt
The -i flag prints the status line and headers before the body, so you see the code, the content type and any cache headers in one view. Another read-only check from the same section is HTTP security headers, and a third answers is my IP blacklisted. All the tools live under Tools.
Response codes and caching
What the server returns for /robots.txt changes how a crawler behaves, and RFC 9309 spells it out.
| Response | What a compliant crawler does |
|---|---|
| 2xx with a file | Must follow the rules it can parse |
| 4xx, such as 404 | Treats the file as unavailable and may access any resource |
| 5xx or unreachable | Must assume complete disallow |
| Redirects | Should follow at least five consecutive redirects; after more than five it may treat the file as unavailable |
The 5xx rule is the one that bites. An outage or a misconfigured deploy that makes the file return a server error can pause crawling of the whole site. Crawlers should not reuse a cached copy for more than 24 hours unless the file is unreachable, so a fix may take up to a day to be noticed. Google documents its own handling, including a 500 KiB limit and the way it caches for up to 24 hours, in its robots.txt specification.
AI crawlers and user-agent groups
Nothing in the protocol is specific to AI. Rules are matched by the user-agent name a crawler announces, so a site that wants different treatment for one crawler adds a group for that name, as examplebot is shown above. The operator of each crawler publishes the name to use; take it from their documentation, not from a list of guesses. The crawler OpenAI documents has its own page, GPTBot.
What this page does not claim is which crawlers read the file or follow it. That is each operator's choice, and the file cannot check. The sensible reading is the same as for any bot: robots.txt states your wishes, and your logs show what actually arrived.
Where a WAF helps and where it does not
robots.txt asks, and a WAF decides. The file works only if the crawler cooperates; a filter acts on requests whether or not the sender read your rules.
- What a filter sees. Each request, its address, its headers and how often it arrives. That is enough to notice a client hammering URLs your robots.txt disallows. See content scrapers and bot detection for the signals involved.
- What it does not see. Intent. A request that matches no rule looks like any other. A bot with a browser-like user-agent and a normal pace is hard to separate from a visitor.
- What it does not replace. A WAF does not write or fix your robots.txt, and it does not remove URLs from search. Those stay with your server and your noindex rules.
Bridge WAF has three modes: off, monitor and block. Start in monitor, which records matches while every request is still served, read the activity log, then move rules to block once the matches look right. The steps are in WAF monitor mode. For clients that ignore the file, blocking bad bots covers refusal and rate limits.
Questions
What is robots.txt?
robots.txt is a plain-text file at the root of a site that tells crawlers which paths they may request. It follows the Robots Exclusion Protocol, standardised as RFC 9309. It is a request that well-behaved crawlers honour, not an enforcement mechanism.
Does robots.txt hide a page from Google?
No. Disallow stops a compliant crawler from fetching the page, but the URL can still be indexed if other pages link to it. To keep a page out of search results, use a noindex rule on a crawlable page, or put the page behind authentication.
Is robots.txt a security measure?
No. RFC 9309 says the rules are not a form of access authorization. The file is public, so listing a private path in it advertises that path. Protect sensitive content with authentication and server-side access control.
What happens if a site has no robots.txt?
If the file request returns a 4xx status such as 404, RFC 9309 lets a crawler access any resource on the site. A 5xx or unreachable file is treated differently: the crawler must assume complete disallow.
How do I check a site's robots.txt?
Open https://example.com/robots.txt in a browser, or run curl -i https://example.com/robots.txt to see the status code, headers and rules. Request the exact scheme, host and port you care about, because each has its own file.