‹ Blog

GPTBot and OAI-SearchBot: Two Different Website Controls

Read the actual crawler rule before assuming a site is hidden from, or visible in, ChatGPT search.

A robots.txt file can name several OpenAI user agents, and they do different jobs. OpenAI's crawler documentation describes GPTBot for content that may be used to train foundation models and OAI-SearchBot for websites surfaced in ChatGPT search. It says a publisher may allow the search crawler while disallowing the training crawler.

That distinction matters when a business reviews an old “block AI” rule. A GPTBot disallow is not the same setting as an OAI-SearchBot disallow. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. It also describes ChatGPT-User as a user-triggered fetcher, for which robots.txt may not apply in the same way.

Check the published rule

Open your site's /robots.txt and identify the exact group and path. Use the Robots Check to inspect selected user agents, then verify an important page's actual response. A missing robots.txt file or failed fetch should be treated as unknown by the tool, not as proof that every platform can reach the site.

Decide which public content the business intends to make available. If a page should be private, use authentication or appropriate indexing controls; Google warns that robots.txt alone is not a privacy mechanism. Avoid copying a generic crawler policy without reviewing the site's purpose and other security layers.

Measure outcomes separately

An allowed rule does not prove that OpenAI visited a URL or named the business. A blocked rule does not describe every possible user-triggered behavior. For actual visibility, save responses to a defined set of ChatGPT search queries with dates and context. Use Search Console for Google's indexing and traffic observations, not as a substitute for ChatGPT results.

The narrow claim a robots check supports is “this published rule appears to allow or disallow this user agent for this path.” That is useful enough without turning it into a recommendation guarantee.

Frequently asked questions

Is GPTBot the crawler for ChatGPT search results?

OpenAI documents GPTBot for potential model training and OAI-SearchBot for ChatGPT search. They can have independent robots.txt settings.

Does allowing OAI-SearchBot ensure my business appears in ChatGPT?

No. Allowing the search crawler removes one access barrier. It does not prove a visit or guarantee a particular answer. Measure mentions with defined live queries and saved responses.

Related reading