llms.txt: what it is, and whether your website needs one

robots.txt tells crawlers where they may go. sitemap.xml tells them what exists. llms.txt — a proposal that has spread quickly across developer tools and documentation sites — tells language models what your site is about and which pages matter, in plain Markdown they can ingest directly.

The format

An llms.txt file lives at your site root (yourdomain.com/llms.txt). It starts with an H1 naming the site, a short blockquote summary, and then Markdown sections linking to your most important pages with one-line descriptions:

# Acme Plumbing

> 24/7 emergency plumbing across Sydney's North Shore.
> Licensed, insured, on-site within 90 minutes.

## Services

- [Emergency repairs](https://acme.example/emergency): burst pipes, blocked drains
- [Hot water systems](https://acme.example/hot-water): supply and installation

## Company

- [About](https://acme.example/about): licenses, insurance, service area
- [Contact](https://acme.example/contact): phone, email, booking form

Does it actually do anything?

Honest answer: adoption by the major AI labs is still uneven, and no one guarantees a model will fetch it. But the cost is fifteen minutes, the downside is zero, and the upside is a growing set of AI tools that do read it — coding assistants, research agents, and answer engines experimenting with the standard. It also forces a useful exercise: writing down, in one page, what your site offers and which URLs are canonical.

Three rules for a good llms.txt

• Keep it short — it's a map, not a mirror of your sitemap. Ten to thirty links.
• Describe pages in plain language a model can quote: what the page is, who it's for.
• Keep it current. A stale llms.txt pointing at dead pages is worse than none.

Our scanner checks for llms.txt on every scan — along with the robots.txt and sitemap fundamentals it complements.