How do you make your website agent friendly?

How agents see your site

An agent uses one or more of three views of a page. It can fetch the HTML and read it as text. It can drive a browser and read the accessibility tree, the browser’s summary of roles, names and states for each control. Or it can look at a screenshot with a vision model, which is slow and expensive and is used as a fallback. Google’s guidance for building agent-friendly sites describes the same three views and notes that modern agents cross-check them (web.dev).

A site is agent friendly when all three views tell the same, complete story.

Access

1. Do not challenge declared agents. A CAPTCHA or block page ends the visit. In the Axidort Index, 62% of leading commerce sites served one to a declared agent. Keep strict rules for abuse and allow agents that identify themselves or are verified by your bot-management provider.
2. Write robots.txt rules for AI agents. Decide separately about training crawlers, search crawlers and user-triggered agents. See robots.txt for AI agents.
3. Keep key pages reachable without a login. Product, pricing and documentation pages an agent cannot open cannot be recommended.

Readable pages

1. Send the facts in the first HTML response. Name, price, currency, unit, availability, opening hours, whatever the page is for. Do not rely on a script to add them.
2. Use one `h1` and a logical heading order, with main, nav and article landmarks, so an agent can tell content from navigation.
3. Say things in words. "In stock" as text, not only a green dot. A price with its currency and unit, not only an image.
4. Add structured data that matches the page. schema.org Product and Offer for things you sell, Organization and FAQPage where they apply. See structured data for AI agents.

Operable actions

1. Use real controls. button and a elements, not clickable div elements. Where you must use a generic element, give it a role and tabindex.
2. Label every input. Connect each label to its field with for, so the agent knows what to type where.
3. Keep the layout stable and uncovered. Reserve space for late-loading content, and do not leave transparent overlays or cookie walls over the controls an agent needs. Google’s guidance also advises that actionable elements be larger than 8 square pixels so they are not filtered out.
4. Make the main task short. Working search, an add-to-cart that responds with a visible result, and guest checkout where you can offer it.

Machine-readable entry points

1. Publish the cheap files, then offer tools. A current sitemap.xml, a product feed if you sell, and llms.txt. When agents need to do more than read, give them something to call: an API, an MCP server or WebMCP tools on the page.

A quick self-test

# 1. Is a user-triggered AI agent served the page?
curl -s -o /dev/null -w "%{http_code}\n" -A "ChatGPT-User" https://www.example.com/

# 2. Are the facts in the HTML, without JavaScript?
curl -sL https://www.example.com/products/some-product | grep -i -c -E 'price|in stock'

# 3. Is there structured data?
curl -sL https://www.example.com/products/some-product | grep -c 'application/ld+json'

Then open Chrome DevTools, switch the Elements panel to the accessibility tree view, and check that the search box, the add-to-cart button and the checkout link each have a role and a clear name.

For a measured result, scan your site: an Axidort agent attempts the journey and reports where it stopped.

What not to do

• Do not serve agents different facts from people. Showing an agent a price or stock level a person would not see is cloaking, and it destroys trust the first time an order fails.
• Do not hide instructions for agents in the page. Text aimed at steering a model is treated as prompt injection by agent developers.
• Do not rely on one file. llms.txt helps a model find pages. It does not get an agent past a bot challenge.

Frequently asked questions