llms.txt: what it is, who actually reads it, and how to write yours
An honest look at the llms.txt standard, including the uncomfortable part about adoption, and a 20-minute template for writing one anyway.
If robots.txt tells crawlers where they're allowed to go, llms.txt tells AI systems what your site is. It's a plain markdown file at yoursite.com/llms.txt that says, in a format built for language models: here's what this product does, here's who it's for, here are the pages that matter.
The idea, proposed by Jeremy Howard in late 2024, solves a real problem. Your homepage is a soup of navigation, cookie banners, and JavaScript that an AI model has to wade through to figure out what you sell. llms.txt is the same information with the soup drained off.
The honest part first
You'll find plenty of articles implying llms.txt is a magic ticket into ChatGPT's recommendations. Here's the actual state of things:
What's true
- It's a real, published convention with a spec
- Major AI companies publish llms.txt files for their own docs
- AI dev tools and agents do fetch it when pointed at a site
- It costs 20 minutes and can't hurt you
What's oversold
- No major AI assistant has committed to using it for search or recommendations
- Google has said plainly that it doesn't use it
- Having one won't, by itself, get you cited anywhere
- Anyone selling "llms.txt optimization" is selling weather forecasts for a planet we haven't landed on
Treat llms.txt like a fire extinguisher: cheap, probably idle, and exactly what you want in place if things take off.
So why bother? Because the cost is twenty minutes, the downside is zero, and the upside is being early on a standard that's one platform announcement away from mattering a lot. Agents and AI dev tools already fetch it today, and that traffic is small but real. This is the cheapest hedge in your marketing toolkit.
The format
The spec is deliberately simple. One H1 with your project name, a blockquote summary, then sections of links with one-line descriptions:
# YourProduct
> One-sentence description of what you do and who it's for.
Key facts an AI should know: what category you're in,
what you cost, what makes you different.
## Docs
- [Getting started](https://yoursite.com/docs/start): Install in 5 minutes
- [Pricing](https://yoursite.com/pricing): Plans and what's included
## Blog
- [Your best post](https://yoursite.com/blog/best-post): What it covers
There's also a convention for llms-full.txt, which inlines the complete content of your pages instead of linking to them. Start with the short one.
Writing yours in 20 minutes
- 1
Write the one-sentence summary
What you do, for whom. If you can't write this sentence, llms.txt isn't your problem.
- 2
State the facts an assistant needs
Category, pricing, the one thing that makes you different from the obvious alternative. Assistants answer "what does X cost" and "X vs Y" constantly. Feed them.
- 3
Link your five most important pages
Pricing, docs, your best blog posts. One line of description each.
- 4
Serve it as plain markdown at /llms.txt
No HTML, no styling. The whole point is machine-readable simplicity.
Check whether anything reads it
Here's the question almost nobody can answer about their own llms.txt: has any AI system ever fetched it? Browser-based analytics can't tell you, because crawlers don't run JavaScript. The file generates server requests and nothing else.
usrPeek watches AI crawler traffic server-side, so you can see whether GPTBot, ClaudeBot, or PerplexityBot ever requested your llms.txt, and what else they read while they were in. Twenty minutes to write the file, one glance to find out if the bet is paying off yet.
See what your visitors actually do
usrPeek is dead-simple analytics for solo founders. One script tag for visitor journeys, auto-tracked clicks, AI traffic, and a daily email. No bolt-on cookie tooling.
Start for free