What llms.txt is, how it differs from robots.txt, what the evidence says about who reads it, and how to write and test one, with a working example.
Quick answer: llms.txt is a Markdown file at yourdomain.com/llms.txt that gives AI systems a short, curated map of your most important pages. It doesn't block or allow crawlers (that's robots.txt's job), and no major AI assistant has confirmed it reads the file when answering questions. It's cheap to add and harmless to have, but it isn't a shortcut to AI citations.
If you've checked your server logs lately and noticed requests for /llms.txt, you're not imagining it. A growing number of sites now publish one. This guide covers what llms.txt is, how it differs from robots.txt, what the evidence says about who actually reads it, the exact syntax with a working example, the common mistakes, and how to test your file once it's live.
llms.txt is a Markdown file hosted at the root of your domain that gives AI systems a condensed, structured summary of your site: who you are, what you offer and which pages matter most. Jeremy Howard, co-founder of Answer.AI and fast.ai, proposed the format in September 2024. The spec now lives at llmstxt.org.
The idea is simple. A full web page is mostly noise for a language model: navigation, footers, cookie banners and repeated boilerplate. llms.txt strips that away and hands an AI system a clean index instead: here's who we are, here are our docs, here's our pricing page.
robots.txt controls access. llms.txt offers curation. One doesn't replace the other.
robots.txt has been around since the 1990s. It tells crawlers which URLs they may fetch, using directives like User-agent, Allow and Disallow. If you want to block GPTBot or ClaudeBot from your site, robots.txt is the tool that does it.
llms.txt can't block anything. It's a suggestion: a curated list of your best content. It doesn't stop a crawler from reading the rest of your site, and it doesn't override robots.txt. If robots.txt disallows a page, linking to it from llms.txt won't unlock it.
An easy way to remember it: robots.txt says what a crawler can see. llms.txt says what's worth seeing.
robots.txt | llms.txt | |
|---|---|---|
Job | Access control: allows or blocks crawlers | Curation: suggests what's worth reading |
Format | Plain-text directives (User-agent, Allow, Disallow) | Markdown (H1, blockquote, H2 link lists) |
Can it block a bot? | Yes | No |
Around since | The 1990s, widely honored | Proposed September 2024, still emerging |
Location | yourdomain.com/robots.txt | yourdomain.com/llms.txt |
Who uses it | Every well-behaved crawler | Some agent, coding and retrieval tools; no major AI assistant has confirmed using it to answer questions |
This is where a lot of GEO advice oversells the format, so here's the honest picture.
Adoption is growing, but from a low base. Rankability's tracker found that about 9% of the top 1,000 websites publish an llms.txt file (Rankability). A separate analysis reported by PPC Land found the number of llms.txt files grew 8.8x in a year (PPC Land).
Most files are never read by AI crawlers. The same PPC Land report found that 97% of published llms.txt files received zero requests from AI crawlers.
Google doesn't use it. Google has said that llms.txt won't help or harm your visibility in Search (Search Engine Land). No other major AI company, including OpenAI, Anthropic and Perplexity, has published documentation saying its assistant reads llms.txt when answering questions.
So why bother? Two reasons. First, it costs almost nothing, and if the major assistants adopt it later you're already set. Second, some AI coding assistants, agent frameworks and retrieval tools do read llms.txt when pointed at a site, especially for documentation. If you publish developer docs or an API, that's a real audience today.
What it won't do is get you cited on its own. The signals in our citation readiness checklist, such as answer-first structure, entity clarity and accurate schema, matter far more.
The file must sit at the root of your domain, at yourdomain.com/llms.txt, not in a subfolder. That's the only place tools look for it.
According to the llmstxt.org spec, the file follows this order:
An H1 with your site or project name. This is the only required element.
A blockquote with a one- or two-sentence summary of what you do and who you serve.
Optional paragraphs of extra context, with no headings.
Any number of H2 sections, each containing a list of links with short descriptions.
Here's a working example for a fictional company:
# Example Co
> Example Co makes invoicing software for freelancers and small agencies.
> It handles quotes, invoices, payments and late-payment reminders in one place.
Example Co is used by more than 10,000 freelancers in 40 countries.
## Docs
- [Getting started](https://example.com/docs/start): Set up your account and send your first invoice
- [API reference](https://example.com/docs/api): Endpoints for invoices, clients and payments
## Product
- [Pricing](https://example.com/pricing): Plans and feature comparison
- [Integrations](https://example.com/integrations): Accounting and payment tools we connect to
## Optional
- [Blog](https://example.com/blog): Guides on getting paid on timeThe first line is the H1, your project name. The blockquote is the summary, and it should make sense on its own, because some tools only read the top of the file. The plain paragraph after it is optional context. Each "##" heading starts a new group of links.
The "Optional" section has a special meaning in the spec: it marks links a tool can skip when it's short on space. Plenty of auto-generated files get this wrong by dumping every URL into one long list.
Contradicting robots.txt. If robots.txt blocks a page that llms.txt then recommends, you've sent mixed signals. Well-behaved crawlers will follow the robots.txt block, so the llms.txt link is pointless at best.
Incorrect syntax. A missing H1, content placed before the blockquote, extra H1s inside link sections, or link lines that don't follow the "- [Title](URL): Description" pattern. Small slips like these are the most common reason a file that looks fine to a person fails validation.
Treating it as an SEO lever. With most files getting no AI crawler requests at all, llms.txt is housekeeping, not a growth tactic. Put your effort into content structure and schema first.
Letting it go stale. A file that links to a page you deleted months ago tells any tool reading it that your site isn't maintained. Update it whenever you add or remove key pages, the same way you would a sitemap.
Once the file is live at your domain root, check three things:
It loads. Open yourdomain.com/llms.txt in a browser. It should return the raw text, not a 404 or an HTML page.
It validates. Run it through a free llms.txt validator to confirm the H1 and blockquote are present and in the right order, and that no headings are malformed.
It reads well. Read it the way a model would. Does the summary describe your business accurately? Are your most valuable pages, such as pricing, docs and your best guides, actually in the list?
To see whether anything is fetching the file, check your server logs for requests to /llms.txt. A request means a tool downloaded the file, not that it used it in an answer.
Writing an llms.txt file by hand is fine for a five-page site. It gets tedious once you're managing dozens or hundreds of pages and need to keep the file current as content changes.
Serpely's LLM.txt Creator builds the file from the site data already in your Serpely dashboard. You choose a standard or extended format, check the result in a live preview, edit anything before it goes live, then deploy it in one click or download and copy it.
It's one small piece of a bigger GEO workflow. The pieces that move AI citations more are tracked elsewhere in Serpely, such as the schema audit and the AI Citation Monitor, which shows whether AI assistants actually mention you.
llms.txt won't get you cited on its own, but it's a quick, sensible piece of housekeeping. Serpely's LLM.txt Creator builds yours from your existing site data, and the rest of the platform shows you which changes actually move your AI visibility.
Visit serpely.com to see how it works.
No. No major AI assistant has confirmed it reads the file when answering questions. It's a low-cost addition to your GEO strategy, not a requirement for showing up in AI answers.
No. robots.txt controls what crawlers can access. llms.txt suggests which accessible pages are worth reading. If you use both, make sure they don't contradict each other.
At the root of your domain, just like robots.txt: yourdomain.com/llms.txt. Tools won't look for it anywhere else.
No. It has no access-control function. To block a bot such as GPTBot or ClaudeBot, add a Disallow rule for it in robots.txt.
Check your server logs for requests to /llms.txt. A request confirms a tool fetched the file. It doesn't tell you whether the content was used in an answer, since most AI companies haven't disclosed how they use it.
It's optional. An llms-full.txt contains the full text of the pages your llms.txt links to, in one file. It's most useful for documentation sites, where coding assistants and agents may want everything in a single fetch.