What is llms.txt, and does your website need one?
llms.txt is a Markdown file at the root of a site that gives language models a short summary and a curated list of links to the pages that matter. It is a proposal, not a standard that AI providers have committed to. It takes an hour to add and can help agents that look for it, but it does not improve rankings or get an agent past a bot challenge.
By the Axidort team · Updated
The format
The proposal at llmstxt.org defines a fixed order:
- An
H1with the name of the site or project. This is the only required part. - A blockquote with a short summary.
- Optional paragraphs or lists with more detail, without headings.
- One or more
H2sections, each a list of links in the form[name](url): optional note.
A section titled "Optional" marks links an agent can skip when it is short of context. The file lives at /llms.txt.
An example
# Example Store
> Industrial fasteners for trade customers. 40,000 products, shipped from two UK warehouses. Prices are shown in GBP excluding VAT.
Orders can be placed as a guest or with a trade account. Contract prices appear after sign-in.
## Catalog
- [Product search](https://www.example.com/search): search by keyword, part number or standard
- [Product feed](https://www.example.com/feeds/products.json): all products with price, unit and stock
## Buying
- [Delivery and returns](https://www.example.com/delivery): cut-off times, carriers, return window
- [Trade accounts](https://www.example.com/trade): credit terms and approval limits
## Optional
- [About us](https://www.example.com/about)llms.txt, robots.txt and sitemap.xml
| File | Purpose | Read by |
|---|---|---|
robots.txt | Says which crawlers may fetch which paths. | Search and AI crawlers. Widely honoured. |
sitemap.xml | Lists every URL you want found. | Search engines and some AI crawlers. |
llms.txt | A curated guide: what the site is and which pages to read first. | Some agents and developer tools. Support varies. |
Does it work?
Be realistic. No major AI provider has said its crawler or assistant relies on llms.txt, and it is not a search ranking signal. Cloudflare’s Agent Readiness scanner does not check it by default (Cloudflare). Its real use today is with coding assistants and agents that are pointed at the file, and with documentation sites, where it is common.
It is still worth adding because it is cheap and harmless, and because writing it forces you to decide which ten pages explain your business. In the Axidort Score it is one item within static signals, which together make up 10% of the score. The files that do more work are robots.txt and structured data.
Related conventions
- `llms-full.txt`: a single file with the full text of your key pages, for tools that want everything in one request. A common convention, not part of the proposal.
- Markdown versions of pages: the proposal suggests serving a clean Markdown copy at the page URL plus
.md. - Markdown content negotiation: returning Markdown when a request sends
Accept: text/markdown. Cloudflare reports under 4% of top sites do this.
Mistakes to avoid
- Listing every URL. That is what the sitemap is for. Keep it to the pages that matter.
- Linking to pages a bot cannot fetch. Test each link without cookies or JavaScript.
- Letting it go stale. Generate it from the same data as your pages if you can.
- Putting claims in it that the site does not make. It should describe the site, not advertise it.
Frequently asked questions
01Is llms.txt an official standard?
No. It is a community proposal published at llmstxt.org. It has been adopted by many documentation sites, but no major AI provider has committed to using it for crawling or ranking.
02Does llms.txt improve SEO or AI citations?
There is no evidence that it changes search rankings, and no AI provider has confirmed using it to choose citations. Treat it as a low-cost extra, not a substitute for crawlable, well-structured pages.
03Where does llms.txt go?
At the root of the site, at /llms.txt, served as plain text or Markdown. Sub-paths can have their own file, and the most specific one applies.