The robots.txt has been part of every website’s technical infrastructure for more than 25 years. AI search systems have given this small file a new strategic role. It no longer governs only which areas traditional search engines are allowed to index. AI providers now also follow its directives.
A well-configured robots.txt for AI crawlers creates clarity: should public content appear in AI searches, be available for model training, or be explicitly excluded? GPTBot, ClaudeBot, PerplexityBot and Google-Extended each serve different purposes. A blanket allow or block therefore often falls short. The key is a configuration that balances visibility, data protection, server load and business interests.
Why the question of allowing or blocking AI crawlers matters
Allowing AI crawlers opens additional paths for increasing digital reach. Systems such as Perplexity or ChatGPT Search can retrieve publicly accessible pages, cite them as sources and link directly to them in their answers. Guides, studies and glossaries can thus become visible beyond traditional search results. This opens new avenues for customer acquisition.
A key distinction must be drawn between search and model training. Training crawlers can be blocked when sensitive or licensed content should not be used in AI models. This applies especially to exclusive or paid content.
GPTBot and OAI-SearchBot
OpenAI uses OAI-SearchBot for search functions and GPTBot for content used to improve generative models. Both identifiers can be controlled independently. A company can therefore support discoverability in ChatGPT Search while blocking GPTBot from training data. Anyone choosing to allow GPTBot should make that decision deliberately and separately from search visibility.
PerplexityBot
Perplexity describes PerplexityBot as a crawler for displaying and linking to websites in its search results. According to the provider, this bot is not used to train base models. The key question for a suitable PerplexityBot configuration is therefore which public content should appear as citable sources.
ClaudeBot and Claude-SearchBot
Like OpenAI, Anthropic separates training (ClaudeBot) from search (Claude-SearchBot). ClaudeBot also respects restrictions in the robots.txt. Allowing or blocking can be set independently for each. Which option is appropriate depends on the content and business model.
A full allow makes sense when a website with freely available specialist information wants to create new contact points. Typical advantages include:
- Additional mentions in AI-powered search and answer systems
- Possible references to guides, offers or product pages
- Better machine readability of publicly accessible content
- New measurable traffic from AI platforms
That said, full openness is not appropriate for every area. A block is strategically useful above all for content whose uncontrolled processing could cause disadvantages. This includes internal search pages, filter URLs, test environments, customer areas and large archives with no public value. Exclusive analyses, paid content and sensitive documents also deserve individual review.
Targeted restrictions can also reduce server load. Countless filter combinations, calendar views or automatically generated pages offer AI systems little additional informational value while still triggering server requests. Excluding such paths keeps access focused on relevant content.
One important limit remains: the robots.txt is not an access barrier. It sets rules for cooperative crawlers but does not protect confidential data. Content requiring greater security should be placed behind a login, access control or other technical safeguard. A crawling block also does not automatically prevent all indexing or linking.
The robots.txt for AI crawlers therefore serves primarily as a control tool. It can open up public knowledge areas, exclude unnecessary directories and treat individual bots differently. Precisely this degree of granularity makes it relevant for businesses.
Guide and common strategies for configuring robots.txt for AI crawlers
First, check whether a file already exists. Open the main domain in a browser with the suffix /robots.txt, for example example.ch/robots.txt. If a text file appears, a configuration is already in place. An error message or blank page means the CMS or hosting account should be checked.
Depending on the website, the file is found in the root directory of the web server. In WordPress it may exist as a physical file or be generated dynamically. Changes can be made via the hosting file manager, via FTP, using an SEO plugin or through the responsible web administration. Before any change, backing up the existing version is recommended.
The basic structure is straightforward. “User-agent” identifies the crawler. “Disallow” blocks a path. An empty Disallow entry allows access. “Allow” can explicitly open specific subdirectories. Capitalization and slashes must be exact.
To allow GPTBot, use the following rule:
User-agent: GPTBot
Disallow:
To prevent GPTBot from retrieving content:
User-agent: GPTBot
Disallow: /
For search visibility in ChatGPT, OAI-SearchBot is also relevant. Allowing GPTBot alone does not automatically mean content will appear in search results. A differentiated configuration looks like this:
User-agent: OAI-SearchBot
Disallow:
User-agent: GPTBot
Disallow: /
This keeps the website accessible for ChatGPT Search. At the same time, the second rule signals that GPTBot should not retrieve the content for its intended training purpose. Search presence and training access can thus be managed separately.
The same principle applies to Anthropic’s crawler:
User-agent: ClaudeBot
Disallow:
A full block uses a slash instead of the empty entry. For the PerplexityBot configuration the same applies: an empty Disallow value allows access, “Disallow: /” blocks the entire website.
User-agent: PerplexityBot
Disallow:
For many businesses, a selective strategy works better than a blanket decision. Public specialist areas remain accessible while technical or irrelevant directories are excluded:
User-agent: GPTBot
Disallow: /intern/
Disallow: /suche/
Allow: /ratgeber/
The same pattern can be applied to other crawlers and the PerplexityBot configuration. Clear URL structures are a prerequisite. When public and protected content share the same directory, the risk of incorrect rules increases. In such cases, the site structure should be reviewed first.
Google-Extended is a special case. The control token affects certain uses of content for Gemini systems but not standard Google Search. Blocking Google-Extended does not automatically remove pages from traditional search results. AI functions within Google Search are governed by separate mechanisms.
After any change, a technical review is needed. The following points should be checked:
- Is the file accessible at /robots.txt?
- Does it contain only plain text rules rather than formatted web content?
- Are bot names, paths and slashes correct?
- Do important areas remain accessible to desired search crawlers?
- Do server logs show access from expected bots?
- Has the entire website been accidentally blocked?
Testing robots.txt
A correctly configured robots.txt can be verified directly in Google Search Console. The URL inspection tool shows whether a particular bot has access to a page or is blocked by a rule. This allows configuration errors to be identified and corrected before they affect visibility.
For larger websites, a staged rollout is also recommended. Start by releasing individual content areas, then use log files and analytics to check which crawlers are actually visiting and whether useful references are being generated. Since user-agent strings can be spoofed, verifying against published IP ranges provides additional confidence.
The robots.txt for AI crawlers should also be reviewed regularly. New bot names, changed platform features and restructured directories can make existing rules outdated. A fresh review is especially important after a relaunch, a CMS migration or a site restructure.
Precise configuration forms the basis for controlled AI visibility without opening all content indiscriminately. Does your website need a professional review of its existing rules? Get in touch and let us work out together which crawlers should have access and which areas should remain protected.
