llms.txt: what it is and whether your site needs one

What is llms.txt?

llms.txt is a proposed convention: a markdown file at your site root (/llms.txt) that tells language models what your site is and which pages matter, in a form they can digest without crawling everything. Proposed by Jeremy Howard of Answer.AI in late 2024, it's the opposite of robots.txt in spirit — robots.txt says "keep out", llms.txt says "if you're going to read us, start here".

It is not an official standard, and it blocks nothing. A companion convention, llms-full.txt, inlines full page content for models that want everything in one fetch.

What goes in it?

The format is deliberately plain: an H1 with your site name, a blockquote summarising what you do in a sentence or two, then sections of links with one-line descriptions. The whole point is curation — your ten pages that matter, not a dump of every URL. A sitemap already does exhaustive; llms.txt does selective.

Ours, for illustration:

# SME Solutions
> Free compliance, AI-visibility and email scanners for UK small businesses.

## Tools
- [AI Visibility Audit](/ai-visibility-scanner): does ChatGPT cite your brand on buyer questions…

Does anything actually read it?

Here's the honest part: adoption by the major engines is unconfirmed. Google has said it doesn't use llms.txt for AI Overviews; OpenAI, Anthropic and Perplexity have made no commitment to fetch it. What does read it today is the long tail — developer tools, agent frameworks, AI browsers and research assistants that fetch it when pointed at your site.

So the case for llms.txt is optionality, not evidence: near-zero cost now, a head start if consumption becomes standard, and immediate value for the agent-shaped visitors that are already growing in number. Anyone selling llms.txt as an AI-visibility breakthrough is selling.

llms.txt vs robots.txt vs ai.txt

They compose: robots.txt decides who gets in, ai.txt states your training policy, llms.txt hands the welcome pack to whoever you let through. Keeping the three consistent matters more than having all three — a welcoming llms.txt behind a robots.txt that blocks every AI bot reads as a site that hasn't decided what it wants.

Should your site have one?

If you have twenty minutes: yes. There's no downside beyond the maintenance of keeping it truthful, and this site has one at /llms.txt.

But keep the proportions straight. llms.txt is the final coat of paint; the load-bearing work is being present in the sources engines cite and having pages worth quoting — that guide matters fifty times more than this one. Add the file after the evidence exists, and check the whole stack with the free training audit, which reads all three files together and flags contradictions.