llms.txt: What It Is, How to Write One, and Does It Work?

An llms.txt file is a Markdown file at /llms.txt that gives AI agents a short, curated map of a website: a title, a one-line summary and annotated links to the pages that matter most, ideally in clean Markdown. The llms txt proposal comes from Answer.AI, and it is used mostly by coding agents. As of October 2026, Google Search does not use it.

This guide covers the current spec (including the v2 changes from August 2026), a correct example file you can copy, real files published by OpenAI, Anthropic, Google and others, what the server-log data says about who reads them, and a decision table for whether your site needs one.

What is llms.txt?

llms.txt is a proposed convention, not a formal web standard. Jeremy Howard of Answer.AI published it on 3 September 2024, and the llms.txt proposal on llmstxt.org is still the canonical spec. Its goal is to give an AI agent concise, expert-level information in one predictable place, because HTML pages are full of navigation, scripts and layout that waste tokens.

The proposal has two parts:

• The /llms.txt file itself: a small Markdown index with background, guidance and links.
• Clean Markdown copies of pages: the same URL as the HTML page with .md added (for example page.html.md), so an agent can fetch the text without parsing HTML.

The file is meant for inference, the moment an agent needs information while helping a user, rather than for training. The spec says that is how it has been used in practice, mainly by coding agents reading software documentation.

What changed in llms.txt v2

In August 2026 the proposal was revised to v2. According to the llms.txt v2 change log, the main changes are:

• Discovery via link relations: pages can point to their Markdown version with rel="alternate" type="text/markdown" and to the covering llms.txt file with rel="describedby", either as an HTML <link> element or an HTTP Link: header.
• Two Markdown URL forms: page.html.md (appended) and page.md (extension replaced) are both allowed.
• Defined subpath scope: /docs/llms.txt covers everything under /docs/, and the most specific file wins.
• Consumption model stated directly: agents read or search the file, then follow the links they need. The old llms_txt2ctx context-expansion tool is no longer part of the proposal.
• "Optional" loses its special meaning: an ## Optional section is still a useful convention for secondary links, but tools are no longer told to drop it mechanically.

Most articles ranking for this topic still describe v1 only, so check any generator or validator you use against the current spec.

How the llms.txt format works

An llms.txt file is plain Markdown with a fixed order of sections, so both language models and simple parsers can read it. Only the H1 is required.

The file lives at the site root (/llms.txt) or at any subpath. Every link should ideally point to LLM-friendly content, such as the .md version of a page, because the file is an index and the detail lives behind the links.

llms.txt example: a correct file you can copy

The example below is for a fictional B2B software company. It follows the v2 spec: one H1, a factual blockquote, a short notes section without headings, H2 file lists with a note on every link, and an Optional section.

1# Northwind Analytics2 3> Northwind Analytics is a product analytics platform for B2B SaaS teams. It tracks in-app events, builds funnels and retention reports, and syncs audiences to CRMs. Plans: Free, Team and Enterprise. SOC 2 Type II certified.4 5Important notes:6 7- Northwind is a hosted service; there is no self-hosted edition.8- Pricing and plan limits change; always use the pricing page below as the source of truth.9- Use the canonical URLs in this file when citing Northwind.10 11## Product12 13- [Product overview](https://www.example.com/product.md): What Northwind does, core features and supported integrations14- [Pricing](https://www.example.com/pricing.md): Current plans, limits and what each plan includes15- [Security and compliance](https://www.example.com/security.md): SOC 2 report access, data residency and subprocessors16 17## Docs18 19- [Quickstart](https://www.example.com/docs/quickstart.md): Install the SDK and send a first event in about ten minutes20- [Events API reference](https://www.example.com/docs/api/events.md): Endpoints, parameters, rate limits and error codes21- [CRM sync guide](https://www.example.com/docs/crm-sync.md): Connect HubSpot or Salesforce and map audiences22 23## Optional24 25- [Changelog](https://www.example.com/changelog.md): Release notes by month26- [Customer stories](https://www.example.com/customers.md): Case studies grouped by industry

To help agents find the file and the Markdown pages under v2, add a Link header in your server or CDN configuration. This works without touching page templates:

1Link: </product.md>; rel="alternate"; type="text/markdown", </llms.txt>; rel="describedby"

Write it like a briefing, not an ad: Agents use the blockquote to understand everything that follows. Put verifiable facts there (what you are, who it is for, plans, certifications) and leave out superlatives. A file full of claims the visible site does not support is exactly what makes critics call llms.txt untrustworthy.

Real llms.txt examples from AI companies and docs platforms

The clearest adoption comes from developer documentation, including the AI labs' own docs. We opened each of these files on 22 October 2026.

Two patterns stand out. First, the best files are scoped and modular: OpenAI's developer llms.txt is only about 6 KB and hands off to product-level files instead of listing everything. Second, "full" exports can get very large. Anthropic's docs llms.txt redirects to platform.claude.com/llms.txt, and its companion llms-full.txt was about 46 MB when we downloaded it, far beyond any context window. That is fine for search and retrieval tools, but it is not the "fits in context" index the spec describes.

Note what these examples do and do not prove. OpenAI, Anthropic and Google publishing llms.txt files for their developer docs shows that they expect coding agents to use them. It is not a statement that ChatGPT search, Claude or Gemini use llms.txt to choose which brands to cite.

llms.txt vs robots.txt vs sitemap.xml

These three files solve different problems, and llms.txt replaces neither of the others. robots.txt controls access, sitemaps help discovery, and llms.txt offers curated context.

If your goal is to control or check AI crawler access, robots.txt is the file that matters. Lantern's AI Crawler Checker shows which AI and search crawler tokens your robots.txt allows and the exact rule behind each decision, and our guide on whether AI crawlers can access your content explains how to read the results.

Does llms.txt help AI search visibility?

On current evidence, no measurable benefit for AI search visibility has been shown, and Google has said it ignores the file. The value is real but narrow: helping agents, especially coding agents, navigate documentation.

What Google has said

Google's position is explicit. Its guide to optimizing for generative AI features in Google Search lists llms.txt under things you can ignore, saying you do not need new machine-readable files, AI text files or Markdown to appear in Google Search, including AI Overviews and AI Mode, "as Google Search itself doesn't use them." The same page says maintaining llms.txt for other services will neither harm nor help your Google visibility.

That guidance formalized earlier comments. In June 2025, Google's John Mueller wrote that "no AI system currently uses llms.txt" and that server logs showed chatbots fetching pages, not llms.txt files.

What server logs show

The largest public log study points the same way. Ahrefs analyzed 137,210 domains in May 2026 and found:

• 28% of those domains published an llms.txt file (an upper bound, since Ahrefs customers skew technical).
• 97% of those files received zero requests during the month.
• The roughly 22,000 requests that did occur came mostly from SEO audit tools, crawlers and tech-profiling services; AI retrieval bots made up about 1.1%.
• GPTBot was the largest AI requester at 4.51% of requests, ahead of ClaudeBot (0.8%) and OAI-SearchBot (0.74%).
• AI bots did not request missing llms.txt files, which suggests agents fetch the file when something points them to it, not by default.

What the files themselves look like

Quality is also uneven. Common Crawl's analysis of its July 2026 crawl examined 584,107 llms.txt files and found that only 49.9% had the full H1, blockquote and link-section structure, 22.56% contained no links at all, and 68.27% were template-generated (Wix alone produced 41.34%). Among successful responses at llms.txt paths, 40.36% were HTML pages and 10.61% were robots.txt content served under the wrong name.

Tooling is moving anyway. Chrome's Lighthouse now includes an llms.txt audit under agentic browsing, which fails if your server errors while serving the file and marks a missing file as not applicable, because the file is optional.

The honest verdict

Publish llms.txt if agents genuinely use your content, and treat it as a 30-to-60-minute housekeeping task, not a growth lever. The work that moves AI citations is elsewhere: crawler access, server-rendered HTML, clear answers and third-party evidence. Lantern's research on where new brands earn AI authority found that brand-owned sources accounted for just 1.1% of ChatGPT citation activity and 0.5% of Google AI citation activity for the emerging brands studied, so a file on your own domain cannot carry visibility on its own.

Who should create an llms.txt file?

The answer depends on whether AI agents act on your content. Use this table to decide.

If you are unsure, check your server logs for requests to /llms.txt first. Ahrefs recommends the same step, and it tells you whether anyone is reading the file before you invest in it.

How to create an llms.txt file in 6 steps

Creating the file takes under an hour for most sites. These steps keep it accurate and useful.

1. Pick 10 to 40 pages that answer real questions. Product, pricing, docs, security, policies and your best explanatory content. Skip tag pages, thin pages and anything you would not want quoted.
2. Write the H1 and blockquote. Use your exact brand name and a two-sentence factual summary that a stranger could verify on your site.
3. Group links under H2 file lists. Use plain section names (Product, Docs, Policies) and add a short note after every link explaining what the page answers.
4. Serve Markdown where you can. Point links to .md versions if your CMS or docs platform produces them; otherwise link the canonical HTML URL.
5. Publish at the root with the right content type. Serve it as text/plain or text/markdown with a 200 status, not as an HTML app shell. Run Lighthouse to confirm it loads without errors.
6. Generate it from source and monitor it. Rebuild the file when pages change, check for broken links, and review logs monthly to see who requests it.

llms.txt generators and integrations

You rarely need to write the file by hand. The spec's integrations list includes Mintlify, which generates llms.txt and llms-full.txt for hosted docs, plus GitBook, Wix, the Yoast SEO and AIOSEO WordPress plugins, and plugins for VitePress, Docusaurus and Drupal. Generated files are a good start, but edit the blockquote and link notes yourself: Common Crawl's data shows that most template output never gets a meaningful summary.

llms.txt best practices and common mistakes

Most problems come from treating the file as a dump of URLs or as a place for claims that are not on the site. Avoid these:

• Serving HTML at /llms.txt. Catch-all routes in JavaScript apps often return the app shell with a 200 status. Test with a raw request, not a browser.
• Listing every URL. That is a sitemap. Keep the index small and let links carry the detail.
• Adding policy language. Rate limits, copyright notices and "you must cite us" demands are not part of the spec, and no crawler treats them as rules. Use robots.txt and your terms for that.
• Contradicting the visible site. If the file says something your pages do not, agents and reviewers have reason to distrust both.
• Hidden instructions to models. Common Crawl flagged 10 files with top-severity prompt-injection patterns, four of them genuine. Treat the file like code that others will execute.
• Expecting it to fix rendering. If your key content only appears after JavaScript runs, many AI crawlers will not see it, with or without llms.txt.

If you want to know whether AI systems already describe and cite your brand correctly, start with measurement rather than files. Lantern's free AI Visibility Report shows how ChatGPT, Perplexity, Gemini and Claude surface your brand, which is the baseline any technical change should be judged against. For the full technical and content checklist, see our AEO audit guide.

Frequently asked questions

Is llms.txt needed?

No search engine or AI assistant requires it. Google says its Search features ignore the file, and log studies show most files are never requested. It is worth adding if AI agents use your documentation or product information, and it is harmless otherwise.

Does ChatGPT or Claude use llms.txt?

OpenAI and Anthropic publish llms.txt files for their developer docs, and their crawlers occasionally fetch other sites' files, but neither company has said llms.txt affects which sources ChatGPT or Claude cite. For ChatGPT search, OpenAI's guidance focuses on allowing OAI-SearchBot; see our guide on how to show up in ChatGPT.

What is llms-full.txt?

llms-full.txt is a companion file popularized by documentation platforms such as Mintlify that concatenates the full Markdown content of a site into one file. It is not defined in the llmstxt.org spec. It suits retrieval tools and coding agents, but large sites can produce files of tens of megabytes.

Where should I put my llms.txt file?

Put it at the root of your domain (https://www.example.com/llms.txt). Under v2 you can also place files in subpaths such as /docs/llms.txt to cover just that section; agents should use the most specific file that applies.

Can llms.txt block AI training or crawling?

No. llms.txt has no directives and blocks nothing. To control crawling or training, use robots.txt tokens such as GPTBot, ClaudeBot or Google-Extended, and check how they resolve on your site.

Does llms.txt help with Google AI Overviews?

No. Google states that AI Overviews and AI Mode rely on its normal Search index and that llms.txt files are ignored. What helps is being indexed, eligible for a snippet and genuinely useful; see how to rank in Google AI Overviews.

llms.txt is a sensible, low-cost courtesy to AI agents, and v2 makes it easier for them to find. Just keep it in proportion. Lantern helps teams measure how AI answers actually describe and cite their brand, then turn the gaps into review-ready content and technical work, so effort goes where the evidence points.