What is llms.txt? How to create one and whether AI reads it
llms.txt is a Markdown file that points AI tools to your best pages. Here is the format, an example you can copy, and an honest look at who actually reads it.
The seoscore team 9 min read
On this page
- What is llms.txt?
- What goes in an llms.txt file: the format
- An llms.txt example you can copy
- llms.txt vs robots.txt vs sitemap.xml
- Do ChatGPT, Claude, Gemini and Google actually use llms.txt?
- Should you add one? Who benefits and who can skip it
- How to create and publish llms.txt
- How to check your llms.txt is valid
- llms.txt is one AEO signal, not the whole job
- Check your site with seoscore
- Next steps
llms.txt is a plain Markdown file at the root of a website (for example, example.com/llms.txt) that gives AI tools a short, curated map of the site's most useful pages. It costs little to add and does no harm. However, no major AI search engine has said it reads the file: Google says its search doesn't use it, and server-log studies show AI crawlers almost never request it. Where it does get used is by coding assistants and agents that fetch it on request, so treat it as a small extra, not a route into AI answers.
What is llms.txt?
llms.txt is a proposal from Jeremy Howard of Answer.AI, first published on 3 September 2024. Version 2, the first revision, followed on 10 August 2026 (llmstxt.org changes page).
The problem it tries to solve is simple. A normal web page is mostly navigation, scripts, cookie banners and layout code. A language model can only take in a limited amount of text at once (its context window), so wading through that clutter wastes space. llms.txt offers a shortcut: one clean file that says what the site is and links to the pages worth reading, ideally in Markdown versions without the clutter.
You will often see llms.txt on lists of AEO tasks. AEO (answer engine optimization) is the work of making pages that AI engines can read, trust and quote. Our guide to answer engine optimization covers it in full; llms.txt is one small part of it.
What goes in an llms.txt file: the format
The file is Markdown, laid out in a fixed order. Only the first part is required (llmtxt.info format guide).
- An H1 with the site or project name, on the first line:
# Example Ltd. This is the only required element. - A blockquote summary: one or two sentences starting with
>that say what the site is and who it serves. - Notes: optional paragraphs or bullet points, with no headings, giving context a model should know.
- H2 sections of links: each
##heading holds a list in the form- [Page name](url): short description.
Some validators are stricter than the spec and flag a file with no summary or no section of links, so it is easiest to include all four parts.
The Optional section
In version 1, an H2 called ## Optional marked links a tool could skip when it was short of space. In version 2 the heading no longer has any special machine meaning (llmtxt.info). It is still a tidy place for secondary pages.
llms-full.txt and .md pages
The spec suggests offering Markdown versions of your pages and linking to them from llms.txt. Version 2 accepts two URL patterns: adding .md to the full URL (/pricing.html.md) or using .md in place of the page's extension (/pricing.md) (Search Engine Journal).
llms-full.txt is something else: a single file holding the full text of every listed page. It is a community convention, not part of the spec (Codersera). It suits documentation sites. For most others it is a second copy of every page to maintain, and MonsterInsights advises against Markdown copies of every page because they create duplicate content (MonsterInsights).
What version 2 added
Version 2 kept the file format, so a version 1 file is still valid. It added two ways for a page to point to the right files:
<link rel="alternate" type="text/markdown" href="/pricing.md">
<link rel="describedby" href="/llms.txt">
The first points to the page's Markdown version, the second to the llms.txt that covers it. Both can also be sent as an HTTP Link header. If you followed older advice to use rel="alternate" type="text/plain" for llms.txt, change it to rel="describedby". Version 2 also gives subfolder files a scope: /docs/llms.txt covers everything under /docs/, and where more than one file applies, the most specific wins (llmstxt.org).
An llms.txt example you can copy
Here is a complete file for a small invoicing software company. Swap in your own names, URLs and descriptions.
## Tallybook
> Tallybook is invoicing and expenses software for UK sole traders and small firms. It sends invoices, chases late payments and prepares figures for VAT returns.
Tallybook is a web app with iOS and Android apps. Prices on the site exclude VAT. Support is by email and live chat on weekdays.
## Product
- [Features](https://www.example.com/features): What Tallybook does, from invoices to expense tracking
- [Pricing](https://www.example.com/pricing): Plans, what each includes and how the free trial works
- [Integrations](https://www.example.com/integrations): Banks and accounting tools Tallybook connects to
## Help
- [Getting started](https://www.example.com/help/getting-started): Set up an account and send a first invoice
- [VAT returns](https://www.example.com/help/vat): How Tallybook prepares and submits VAT returns
- [Contact support](https://www.example.com/contact): Hours, email and live chat
## Company
- [About](https://www.example.com/about): Who runs Tallybook and where it is based
## Optional
- [Blog](https://www.example.com/blog): Guides on invoicing, cash flow and tax deadlines
- [Changelog](https://www.example.com/changelog): Product updates
For a blog, a shorter file does the job:
## The Allotment Diary
> A gardening blog about growing fruit and vegetables on a UK allotment, with month-by-month sowing guides.
## Guides
- [Sowing calendar](https://www.example.com/sowing-calendar): What to sow and plant each month
- [Start here](https://www.example.com/start-here): Taking on a new plot, from clearing to first crops
## Optional
- [Recipes](https://www.example.com/recipes): Cooking with seasonal produce
What to include and what to leave out:
- Include the pages that answer what people ask about you: what you do, prices, how it works, help and contact.
- Write factual descriptions, not slogans. A model uses them to decide which link to follow.
- Leave out login pages, baskets, tag archives, thin pages and anything robots.txt blocks.
- Keep it short. The reference file on llmstxt.org is about 637 bytes. A few dozen links is plenty for most sites.
llms.txt vs robots.txt vs sitemap.xml
| llms.txt | robots.txt | sitemap.xml | |
|---|---|---|---|
| Purpose | Points AI tools to your most useful pages | Tells crawlers what they may fetch | Lists the URLs you want indexed |
| Who reads it | AI agents and coding assistants, on request | Search and AI crawlers, before crawling | Search engine crawlers |
| Controls access | No | Yes | No |
| Format | Markdown | Plain-text rules | XML |
| Location | /llms.txt, or a subfolder | /robots.txt | Usually /sitemap.xml |
| Status | A proposal, not adopted by a standards body | IETF standard (RFC 9309) | sitemaps.org protocol |
The row that matters is 'controls access'. llms.txt grants and blocks nothing. If robots.txt blocks GPTBot or ClaudeBot, listing pages in llms.txt won't let them in. Our guide on how to allow AI crawlers in robots.txt covers the user agents and rules.
Do ChatGPT, Claude, Gemini and Google actually use llms.txt?
For search and AI answers, there is no public evidence that they do.
Google Search
Google has been consistent. John Mueller compared llms.txt to the keywords meta tag (Stan Ventures) and said: 'As far as I know no AI system currently uses llms.txt' (Stan Ventures). Gary Illyes said in July 2025 that Google doesn't support it and has no plans to (Optimizely). Google's AI search optimization guide, published in May 2026, lists llms.txt as something you don't need for AI Overviews or AI Mode (Search Engine Journal). A Search Central update in June 2026 said keeping one 'won't harm (nor help)' visibility or rankings (TechWyse). Some Google teams host llms.txt files for their developer docs; Mueller has said that is not an endorsement (Search Engine Roundtable).
Chrome Lighthouse
Google's browser team takes a different view. In May 2026 Lighthouse added an llms.txt audit to an experimental 'Agentic Browsing' category, noting that without the file 'agents may spend more time crawling the site' (Chrome for Developers). The category gives no normal 0 to 100 score, and a site with no file is marked not applicable rather than failed (Search Engine Land). It shows interest from browser agents, not a search ranking signal.
OpenAI, Anthropic, Perplexity and Microsoft
We found no public statement from any of them saying their crawlers fetch llms.txt from other people's sites. Publishing a file for your own documentation, as many developer companies do, is not the same as reading everyone else's.
What server logs show
Optimizely summarises four independent studies covering hundreds of thousands of domains and more than 500 million AI bot visits: AI crawlers almost never request the file (Optimizely). A host with around 20,000 domains reported that the only bot fetching it was BuiltWith, a technology-detection service (Kai Spriestersbach). You can check your own logs: filter requests for /llms.txt and look at the user agents.
Where it does get read
The file was designed to be read on demand, when an agent needs information while helping someone, and that is where it works. Howard writes that coding agents 'use them reliably' and that documentation platforms now generate one automatically (llmstxt.org). Patrick Stox names Claude Code as the biggest AI user of the file (Patrick Stox). If developers point an AI assistant at your docs, a good llms.txt helps it find the right page.
Should you add one? Who benefits and who can skip it
| Site type | Verdict |
|---|---|
| Documentation, APIs, developer tools | Worth doing. Coding assistants are the main readers. Consider .md pages too. |
| SaaS | Worth a basic file, especially if you have help docs. |
| Publishers and blogs | Optional. Cheap, but little evidence of benefit. |
| Local businesses | Low priority. Crawler access and clear service pages matter far more. |
| Ecommerce | Optional. List categories, delivery and returns pages, not every product. |
A basic file takes an hour or less. The risks are small but real:
- It goes stale. Links to deleted or renamed pages send agents to 404s.
- It contradicts robots.txt. Listing pages you block sends mixed signals and helps no one.
- It shows up in search. Mueller has suggested noindexing the file (Stan Ventures).
How to create and publish llms.txt
1. Pick the pages
Choose the 10 to 40 pages that best explain your business or product. Group them under a few H2 headings.
2. Write the summary and descriptions
One factual sentence or two for the blockquote, and a short line per link saying what the page covers.
3. Generate or write the file
The free llms.txt generator creates a file for your site, which you can then edit by hand. Or start from the example above.
On WordPress, some SEO plugins, Yoast among them, generate llms.txt automatically. Check what they produce: in one Lighthouse test, a Yoast-generated file was flagged as not following the recommendations (SEO Kreativ). Don't run two plugins that both create the file.
4. Upload it to the root
Save it as llms.txt in UTF-8 and upload it to your web root, so it loads at https://www.example.com/llms.txt. On hosted platforms without file access, look for a custom file or redirect option.
5. Check it is served correctly
curl -I https://www.example.com/llms.txt
You want a 200 status and a text content type such as text/plain; charset=utf-8. A redirect to the home page, a 404 or an HTML content type means it isn't being served as a file. To keep it out of search results, add a noindex header. On nginx:
location = /llms.txt {
add_header X-Robots-Tag "noindex";
}
6. Link Markdown versions if you have them
If you publish .md pages, link to them in llms.txt and add the rel="alternate" and rel="describedby" links shown earlier.
How to check your llms.txt is valid
The common mistakes:
- Wrong location: in a subfolder by accident, or on the wrong subdomain.
- HTML instead of Markdown: often a CMS page or error page served at the URL.
- No H1 on the first line, or more than one H1.
- No summary or no section of links, or empty sections.
- Broken links and links without descriptions.
- Pages that robots.txt blocks listed in the file.
The llms.txt generator also checks an existing file for format errors, so paste in your URL after publishing and after every big site change. The user guide explains the checks.
llms.txt is one AEO signal, not the whole job
A perfect llms.txt won't help if AI crawlers can't fetch or read your pages. The checks that carry more weight:
- AI crawler access in robots.txt.
- Content without JavaScript, so text is in the HTML a crawler receives.
- Entity markup: structured data saying who you are.
- Direct answers near the top of pages, and FAQ markup where it fits.
- Freshness and citations: dates and sources a model can trust.
The free AEO and GEO checker runs all of these on any page, llms.txt included. In a full SEO audit, AEO is 15 of the 100 points, with each fix ranked by the points it adds.
Getting quoted and named is a separate job: GEO (generative engine optimization), whether AI assistants mention and cite your brand. Our GEO guide explains it, and what GEO is and how to do it goes into the work. To measure it, see how to track brand mentions in ChatGPT.
Check your site with seoscore
Generate or check your file with the llms.txt generator, then run the AEO and GEO checker on your home page and key pages to see what matters more. AI visibility, on every plan, asks ChatGPT, Claude, Gemini and Perplexity the questions your customers ask and shows how often they name you. AI mentions, on Starter and up with a free trial, reports where Google AI Overviews and ChatGPT cite your pages, the questions asked and the other sites cited beside you.
Next steps
- Decide whether your site type benefits. If you run docs or a SaaS product, add a file.
- Create it, upload it to the root and confirm a 200 status with a text content type.
- Check it for format errors and for pages robots.txt blocks.
- Run the AEO and GEO checker on your key pages and fix crawler access and readable content first.
- Track whether AI assistants cite you, and judge results by citations rather than by the file itself.
Questions people ask
Is llms.txt an official web standard?
No. It is a proposal maintained at llmstxt.org and has not been adopted by any standards body. Be wary of articles that describe it as a formal standard.
Will llms.txt stop AI companies training on my content?
No. It is a list of pages, not a set of rules, so it cannot block anything. To limit crawling, add rules for each AI crawler's user agent in robots.txt, bearing in mind that they rely on the crawler honouring them.
How often should I update llms.txt?
Update it whenever you add, remove or rename an important page, and after any redesign or URL change. A quick check every few months catches broken links. If a plugin generates the file, review its output after plugin updates.
Should llms.txt link to HTML pages or Markdown versions?
The spec prefers Markdown versions where they exist because they are cleaner to read. If you don't have them, normal page URLs work, although some validators will flag them. Outside documentation sites, building Markdown copies just for this is rarely worth the upkeep.
Sources
- Changes llmstxt.org
- llms.txt format: complete spec reference llmtxt.info
- How llms.txt works: file structure, syntax, parsing llmtxt.info
- Llms.txt V2 Adds Formal Markdown Linking For AI Agents www.searchenginejournal.com
- llms.txt Example + Format: What It Is & Who Reads It codersera.com
- llms.txt and AI Crawlers: What WordPress Site Owners Need to Set Up Now www.monsterinsights.com
- Google Dismisses LLMs.txt as Ineffective and Unused by AI Bots - Stan Ventures www.stanventures.com
- Google: Noindex LLMs.txt to Avoid Search Clutter www.stanventures.com
- Does llms.txt work? What the AI search data says www.optimizely.com
- Google's New AI Search Guide Calls AEO And GEO 'Still SEO' www.searchenginejournal.com
- Google Says llms.txt Will Not Help Rankings www.techwyse.com
- Google Search Team Does Not Endorse LLMs.txt Files www.seroundtable.com
- Google adds llms.txt check to Chrome Lighthouse searchengineland.com
- The llms.txt is dead. More precisely: a dud. medium.com
- llms.txt patrickstox.com
- Lighthouse SEO Audit with AI: Chrome DevTools MCP Test www.seo-kreativ.de
seoscore
Make your llms.txt in a minute
Create and check the file AI assistants read. Free, no account needed to start.
Open the llms.txt generator