How should you write robots.txt for AI agents and crawlers?

The user agents

Names and behaviour change. Check each company’s crawler documentation before you rely on this list, and note that user-triggered fetches may not consult robots.txt in the same way as crawlers, because a person asked for that page.

A template

# Search and answer engines: allowed, so the site can be found and cited
User-agent: Googlebot
User-agent: Bingbot
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /cart
Disallow: /account

# Agents acting for a person: allowed
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Allow: /
Disallow: /account

# Training crawlers: your choice. This example opts out.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /
Disallow: /cart
Disallow: /account

Sitemap: https://www.example.com/sitemap.xml

Whether to allow training crawlers is a business decision. Allowing them puts your products and brand into future models. Blocking them protects content you sell. Either is reasonable. Blocking the search and user-triggered groups is rarely what a business that wants customers intends.

Content Signals

Content Signals is a newer way to state how your content may be used, in one line of robots.txt, separately from who may crawl:

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

search covers building a search index, ai-input covers using a page to answer a question, and ai-train covers model training. Cloudflare reports about 4% of the 200,000 most visited domains use it (Cloudflare). It states a preference. It does not enforce one.

What robots.txt cannot do

• It does not grant access. If your CDN or bot protection blocks a bot, an Allow line changes nothing. This is the most common mistake: a welcoming robots.txt in front of a firewall that challenges every automated request.
• It does not stop anyone. It is a request that reputable bots honour. Enforcement happens at the firewall.
• It cannot verify identity. Anyone can send a GPTBot user agent. Verify bots by the IP ranges their operators publish, through your bot-management provider’s verified-bot list, or through Web Bot Auth, an IETF draft in which bots sign their requests.

Check the firewall, not just the file

curl -s -o /dev/null -w "%{http_code}\n" -A "OAI-SearchBot" https://www.example.com/
curl -s -o /dev/null -w "%{http_code}\n" -A "ChatGPT-User"  https://www.example.com/products/some-product

A 200 is what you want. A 403, 429 or 503 means something in front of your site is refusing the bot, whatever robots.txt says. In the Axidort Index, 31 of 50 leading commerce sites served a challenge to a declared agent. See why AI agents fail at checkout.

Frequently asked questions