The Robots Check fetches a site's published robots.txt file and evaluates rules for selected user agents. It can point to a rule worth fixing. It cannot prove that a crawler visited a page, that the page was indexed, or that an AI assistant mentioned the business.
Read the result at the right level
First confirm which URL and user agent were checked. An explicit disallow rule for an important public page is worth reviewing. If the file is missing or the fetch fails, the tool should report uncertainty rather than treating missing evidence as a perfect score. Hosting firewalls and page responses may impose separate access limits that a robots.txt check does not see.
OpenAI's crawler guide distinguishes OAI-SearchBot, which is used for ChatGPT search, from GPTBot, which concerns model training. A site can set different rules for those purposes. OpenAI also says user-triggered ChatGPT-User fetches are different and robots.txt rules may not apply to them in the same way. For Google's AI features, Google points to Googlebot and ordinary Search requirements.
Avoid a common mistake
A broad Disallow: / can prevent a compliant crawler from fetching public content under that group. But robots.txt is not a login or privacy control. Google explains that a blocked URL can still appear in results without its content; use appropriate access controls or noindex when those are the actual goals.
Do not copy a generic “allow all AI bots” file over an existing site. Review what your business intends to expose, the current rules, and any platform or security layer. After a change, fetch robots.txt again and inspect an important page's HTTP response. Then use the relevant platform's tools and a defined AI-query sample to observe outcomes.
The Robots Check is a starting point for that review. If it returns unknown, investigate the fetch before making an access claim.