What is agent compatibility, and how do you test for it?
Agent compatibility is whether a website works for the AI agents that visit it. There are four kinds, and each needs something different: crawlers need permission, reader agents need facts in the HTML, browser agents need controls they can operate, and tool-calling agents need an API, an MCP server or WebMCP tools. A site can be compatible with one kind and fail another.
By the Axidort team · Updated
The four kinds of agent
| Kind | What it does | What it needs from you | How it fails |
|---|---|---|---|
| Crawlers | Fetch pages in bulk for search indexes and model training, for example GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot. | robots.txt rules, a sitemap, fast responses. | Disallowed or rate-limited, so the site is missing from AI answers. |
| Reader agents | Fetch one page for a user’s question, without a browser, for example ChatGPT-User, Claude-User, Perplexity-User. | Facts in the server HTML, structured data, no challenge. | Sees an empty shell or a challenge page. |
| Browser agents | Drive a real browser to click, type and navigate for a user. | Semantic controls, labels, stable layout, no automation block. | Cannot find or operate the control, or is blocked as a bot. |
| Tool-calling agents | Call functions you expose instead of using the interface. | An API, an MCP server, or WebMCP tools with clear descriptions. | Nothing to call, or tools that are undocumented. |
Test 1: crawlers
Read your own robots.txt and check that it says what you intend for each AI user agent, then confirm that the sitemap it points to loads and lists your important pages. Remember that a CDN or firewall rule can block a bot that robots.txt allows. Details are in robots.txt for AI agents.
Test 2: reader agents
curl -sL -A "Claude-User" https://www.example.com/pricing | head -c 2000Read the output as if it were all you knew about the page. If the answer a customer would ask for is not in it, a reader agent will not find it either. Then paste the URL into an AI assistant and ask a factual question about the page. Compare its answer with the page.
Test 3: browser agents
Give an agentic browser the task your customers most often want done and watch it. Note where it hesitates, where it clicks the wrong thing and where it is stopped. The fixes are almost always semantic controls, labels and removing overlays, covered in how to make a website agent friendly. An Axidort scan runs this test for a purchase journey and records each step.
Test 4: tool-calling agents
If you publish an API, check that it has a machine-readable description (OpenAPI) and that an agent can discover it. If you run an MCP server, connect to it from an MCP client and call each tool with realistic input. If you register WebMCP tools, test them in Chrome with the testing flag enabled. For most sites this layer is optional today. The first three are not.
A compatibility checklist
- AI crawlers and user-triggered agents are allowed in
robots.txt, and not blocked at the CDN. - A declared agent gets a 200 response, not a challenge.
- Key facts are in the first HTML response and match the structured data.
- Search, add to cart and checkout work in a clean, logged-out automated session.
- Every interactive element has a role and an accessible name.
- There is at least one machine-readable entry point: sitemap, feed,
llms.txt, API or MCP.
The measured version of this checklist is the Axidort Score.
Frequently asked questions
01Is agent compatibility the same as agent readiness?
They describe the same goal. Agent compatibility stresses that there are different kinds of agent to support. Agent readiness is the overall state, often expressed as a score.
02Which AI agents should my website support first?
User-triggered reader agents and browser agents, because they act for a person who is trying to use your site right now. Blocking them loses a visit you would have wanted.
03Does my site need an API to be agent compatible?
No. Agents can read and operate a well-built website. An API or MCP server makes tasks faster and more reliable, and becomes important when agents need to act with a user’s account.