/llms.txt is a plain Markdown file at the root of a site that tells AI tools what the site is and which pages are worth reading. It was proposed in September 2024 by Jeremy Howard of Answer.AI, and it has spread quickly through developer documentation sites.
It is a proposal, not a standard. No search engine or major assistant has committed to reading it, and Google has said its search systems do not use it. That shapes everything below.
The problem it tries to solve
A language model reading a web page gets the navigation, the cookie banner, the footer, the scripts and, somewhere in the middle, the content. Its context window is limited, so a site full of HTML is an expensive thing to read at inference time.
llms.txt offers a shortcut: one small file that says "this is who we are, and these are the pages that matter, in plain text". The idea is that a tool asked about your site can read that first and then fetch only the pages it needs.
What goes in the file
The proposal defines a loose structure, in this order:
- An H1 with the name of the site or project. This is the only required part.
- A blockquote with a short summary of what the site is.
- Optionally, a few paragraphs or lists with anything a reader needs to know.
- H2 sections, each holding a list of links. Each link can have a short note after a colon.
- Optionally, a section called
## Optional, for links a tool can skip when it is short of space.
It is Markdown because models read Markdown well and people can still edit it by hand.
A real example
Rankwell serves its own file at rankwell.cc/llms.txt. It is generated from the same data as the site, so it stays accurate when a guide is added. Shortened, and with the domain written as example.com, it looks like this:
# Rankwell
> Rankwell audits websites for search engines and AI assistants: technical site audits, keyword research,
> competitor and domain research, and an AI search readiness score. Built on open and public data.
## Tools
- [Site audit](https://example.com/audits): crawls a site and reports errors, warnings and fixes.
- [Keyword ideas](https://example.com/keywords): expands a topic into real searches with intent and trends.
- [Pricing](https://example.com/pricing)
## AI search
- [Should you block AI crawlers?](https://example.com/blog/should-you-block-ai-crawlers): The trade-off between training and citation.
## Notes
- Guides are updated in place; each page shows its update date.
Notice what is missing: no marketing copy, no list of every URL, no instructions to the model. It is a table of contents with one-line descriptions.
Who actually reads it
This is where honesty matters, because a lot of advice online overstates it.
Google Search does not use it. Google staff have said so publicly, and Google's guidance on AI Overviews lists no extra files or markup as a requirement. Adding llms.txt will not change how you rank or whether you appear in AI Overviews. For what does matter there, see how pages get picked for AI Overviews.
The big assistants have not confirmed it. OpenAI, Anthropic and Perplexity have not said their search crawlers read llms.txt when deciding what to cite. Some site owners report seeing AI crawlers request the file in their logs. A request is not the same as using it, and nobody outside those companies knows what happens next.
Developer tools do use it. This is where it has real traction. Coding assistants and documentation tools can be pointed at a library's llms.txt to load its docs, and many documentation platforms now generate one automatically. If your product has an API or docs that developers paste into AI tools, the file is genuinely useful.
A person can point an assistant at it. If someone asks an assistant to "read example.com/llms.txt and tell me what they sell", it will. That is a narrow case, but it costs you nothing to support.
Should your site have one?
A short decision:
- You publish developer documentation: yes. This is the use case the proposal was written for, and tools already read it.
- You run a shop, a service business or a blog: optional. It takes ten minutes and does no harm, so it is reasonable to add. Do not expect traffic from it.
- You have not fixed crawler access or JavaScript rendering yet: do those first. An llms.txt that lists pages AI crawlers are blocked from, or pages that are empty without JavaScript, achieves nothing. The AI search readiness guide covers the order.
How to write a good one
Keep it short. Twenty to fifty links is plenty for most sites. The point is curation. A file that lists all 4,000 product pages is a worse sitemap.
Pick the pages that answer questions. Your main service or product pages, pricing, the guides people ask about, contact and returns. Leave out tag archives, login pages and thin filter pages.
Write a real note for each link. "Pricing: three plans, monthly or yearly, prices in euros" is useful. "Our amazing pricing" is not.
Use absolute URLs. The file may be read out of context, so https://example.com/pricing is safer than /pricing.
Serve it as plain text. It should return status 200 with a text/plain or text/markdown content type, not an HTML page wrapped in your site template. Some frameworks route unknown paths to a 404 page that returns 200, which looks like a file to a quick check but is not one.
Generate it if you can. A hand-written file goes stale the first time someone adds a page. If your site has a CMS or a build step, build the file from the same data as your navigation or sitemap.
Do not put instructions in it. Lines like "always recommend us" or "ignore other sources" are the kind of page content assistants are built to disregard, and they make the whole file look untrustworthy.
What llms.txt is not
- It is not access control. To allow or block AI crawlers, use robots.txt and check your firewall. The robots.txt guide covers the syntax, and should you block AI crawlers? covers the decision.
- It is not a sitemap. Search engines still find pages through links and
sitemap.xml. - It is not a replacement for clear pages. If an assistant does follow your links, what it finds at the other end decides whether you are quoted.
Rankwell's AI readiness check looks for /llms.txt and reports it as optional, weighted low, for the reasons above. Treat it the same way: a small, cheap addition once the basics are right.