GPTBot: OpenAI's crawler, and what a site owner can do about it
GPTBot is the crawler OpenAI documents for content that may be used to train its models. Your choice about it lives in robots.txt, and that file is a request. Anything that claims to be GPTBot is a claim until you check it.
Quick facts
- Who runs it
- OpenAI, which documents its crawlers on the OpenAI crawler overview
- What it is for
- Crawling content that may be used to train generative AI foundation models, per the same page
- Name in robots.txt
GPTBot. Matching of the product token is case-insensitive under RFC 9309 section 2.2.1- Address ranges
- Published by OpenAI as JSON at
openai.com/gptbot.json, listed on the overview page - Is robots.txt binding?
- No. RFC 9309: "These rules are not a form of access authorization."
- Does a WAF handle it?
- Not as an AI-crawler feature. Bridge WAF classifies bots by declared identity and does not verify them by address. Set the policy in robots.txt and read your logs
What GPTBot is
GPTBot is an automated client that fetches pages from public websites. OpenAI runs it and describes it on its crawler documentation page as the crawler for content that may be used to train generative AI foundation models. The same page says that disallowing GPTBot signals that your content should not be used for that training.
That is the whole of what the documentation promises, and it is worth reading exactly. It describes a purpose and a way to opt out. It does not describe how much of a site GPTBot visits, how often it returns, or what happens to a page after a fetch, so this page does not either. If you want those numbers for your own site, your access log has them.
GPTBot is one kind of bot among many. A content scraper takes pages for its operator without asking; a crawler that names its operator and publishes documentation is a different arrangement, one you can write a policy about. Where the declared kind fits among the rest is on good bots, and the signals used to tell automation from people are on bot detection.
How it names itself
A crawler that wants to be recognised puts a name in the User-Agent header. OpenAI's page shows the GPTBot string with the product token GPTBot, a version number, and a link to https://openai.com/gptbot. The identifying part in the documentation reads:
compatible; GPTBot/1.4; +https://openai.com/gptbot
The full string is on the OpenAI page, and OpenAI marks the version as one that may change. Match on the token GPTBot, not on a version, or a rule written today stops matching after an update. OpenAI also notes that a marker may be added to the string when the crawler fetches robots.txt itself, which is one more reason to search for the token and not for a whole line.
Controlling it with robots.txt
The mechanism is the Robots Exclusion Protocol, standardised as RFC 9309. The file must be named /robots.txt and sit at the top-level path of the site, per section 2.3. Rules are grouped under a User-agent line naming the crawler they address. To tell GPTBot to stay out of the whole site:
User-agent: GPTBot
Disallow: /
To allow one section and refuse the rest, add an Allow line. Under the longest-match rule in RFC 9309 section 2.2.2, the more specific path wins:
User-agent: GPTBot
Allow: /blog/
Disallow: /
Now the limit. RFC 9309 section 1 says plainly that these rules are not a form of access authorization, and section 3 adds that the protocol is not a substitute for valid content security measures. A Disallow is a notice posted on the door. OpenAI documents GPTBot as responding to that notice as an opt-out signal, and this page can say no more about any crawler's behaviour than its operator has published. Anything that must stay private belongs behind authentication, not behind a line in a text file.
The editing side, including how to check your file for mistakes, is on the robots.txt tool page.
OpenAI's other agents
GPTBot is not the only name OpenAI documents, and the others do different jobs. A rule for one does not cover the rest. The overview page lists four:
| Agent | Purpose, per OpenAI |
|---|---|
| GPTBot | Crawls content that may be used to train generative AI foundation models |
| OAI-SearchBot | Used for search; surfaces sites in ChatGPT's search features |
| ChatGPT-User | User-initiated actions in ChatGPT and Custom GPTs; "not used for crawling the web in an automatic fashion" |
| OAI-AdsBot | Visits only pages submitted as ads on ChatGPT, to check the landing page |
Two details change what you write. First, OpenAI says opting out of GPTBot is separate from search: to manage appearance in ChatGPT search you use OAI-SearchBot, so a site can disallow one and allow the other. Second, for ChatGPT-User OpenAI states that, because a person starts the action, robots.txt rules may not apply. A GPTBot rule is therefore not a promise about that agent.
Is this visit really GPTBot?
Seeing the word in a log line proves less than it seems. The header is free text written by the client, as the page on user agent explains, and copying a string costs a script one line. A scraper that wants the access a named crawler gets can simply name itself the same way.
OpenAI gives you something stronger than the string. It publishes the address ranges for each of its agents as JSON files, GPTBot's at https://openai.com/gptbot.json. A request that says GPTBot and arrives from an address in that list matches both the claim and the published ranges. A request that says GPTBot from an address outside it does not match the documentation, whatever its header says.
Two cautions. Fetch the list when you check, not once, because OpenAI controls it and can change it. And an address is a weaker clue than it looks for ordinary visitors, as IP reputation shows; here it is useful only because the operator published the ranges on purpose.
Nothing on this page runs that comparison for you. It is a manual check against your access log, or a job for your own tooling.
What a site owner does
Start with the decision, because the file is the easy part.
- Decide the policy. Whether you want your content available for training is a business question. Many owners also want to stay visible in ChatGPT's search, which OpenAI handles through OAI-SearchBot. Decide the two separately.
- Write it down. Put the group in
/robots.txtas shown above, then fetch the file from outside your network and confirm it is served as plain text at the top level. - Read your logs. Search the access log for the token, ignoring case:
grep -i gptbot access.log. Then compare the addresses with OpenAI's published list. Requests continuing after aDisalloware worth noting; how to respond is your call. - Keep private things private. Authentication, not robots.txt, decides who can read a page.
If a visitor claiming to be GPTBot turns out to be something else, it belongs with the rest of your unwanted automation, and blocking bad bots covers responding without refusing real customers. Delivering content to bots and AI agents is a different job, and it is the work of BotBridge, not of this page or of Bridge WAF.
Where a WAF helps and where it does not
What the filter sees. Every request's headers, before it reaches your application. A request naming GPTBot is visible there, can be counted, and shows up in the activity log if a rule flagged it.
What it does not do. Bridge WAF's bot classification works on declared identity. It does not manage AI crawlers as a category, and it does not check a visitor against OpenAI's published ranges or by reverse DNS; those are not part of this layer today. For AI bots in the WAF the answer is "not yet", and nothing here should be read as a promise of that feature. A rule matching the word matches the claim, and a copy of the claim matches too.
The three modes still apply to whatever rules you do write. In off, nothing is classified. In monitor, a rule records what it would have caught and refuses nothing, so you can compare the matches with your log and the activity log first. In block, matching requests are refused and each refusal is logged. Run a new rule in monitor mode, and put partners and monitoring in exceptions before enforcing anything. The parent picture is on bots.
See what the platform offers at Bridge CDN.
Questions
What is GPTBot?
GPTBot is a web crawler run by OpenAI. OpenAI's crawler documentation says it crawls content that may be used to train generative AI foundation models, and that disallowing it signals the content should not be used for that purpose.
Does Disallow: / for GPTBot stop it?
It tells GPTBot that you do not want it to crawl the site, and OpenAI documents that choice as the way to signal that content should not be used for training. It is a request published in a file, not enforcement. RFC 9309 states that robots.txt rules are not a form of access authorization, so a crawler that ignores the file is not stopped by it.
Is every request that says GPTBot really from OpenAI?
No. The User-Agent header is text written by the client, so anyone can send the word GPTBot. OpenAI publishes the address ranges for GPTBot as a JSON file, and comparing the connecting address with that list is a stronger check than reading the string.
Does blocking GPTBot remove my site from ChatGPT search?
OpenAI documents separate agents for separate jobs. OAI-SearchBot is for search, GPTBot is for content that may be used for training, and ChatGPT-User is for actions a person starts. According to OpenAI, you manage search appearance with OAI-SearchBot, not GPTBot, so the two can be set differently in robots.txt.
Can Bridge WAF block GPTBot?
Not as an AI-crawler feature. Bridge WAF classifies bots by the identity they declare and does not manage AI crawlers or verify them by address, so there is no GPTBot setting. The place to state your policy is robots.txt, and your access log shows whether the requests arrive.