llms.txt

Updated November 2, 2026

llms.txt is a Markdown file that a website publishes at /llms.txt, or under a subpath, to give language models and AI agents a short, curated map of the site. It names the project in a top-level heading, sums it up in a blockquote and lists links to the pages an agent should read, each with an optional note.

In short: llms.txt is a proposed Markdown file at a site's root or subpath that gives language models a short site summary and a curated list of links to read on demand.

How llms.txt works

An llms.txt file is plain Markdown served as text from a predictable path. The proposal places it at /llms.txt on the site root, or at a subpath such as /docs/llms.txt that covers only the pages under that path 1. When more than one file applies, agents should use the most specific one 1.

The format is loose enough for people to write by hand and strict enough to parse with ordinary code. A file holds these parts, in this order 1.

An H1 with the name of the project or site, which is the only required section 1.
A blockquote with a short summary that carries the key facts.
Optional paragraphs or lists with more detail, with no headings in them.
Optional H2 sections, each holding a list of Markdown links, with an optional note after a colon 1.

The design goal is a small file that fits in a context window. The proposal expects agents to view or search the file, then follow only the links they need, so the detail stays behind the links until it is fetched 1. So each link should point at clean, model-friendly content. The proposal also asks sites to serve a Markdown copy of each page at the same URL with .md added 1.

Version 2 of the proposal, last modified in August 2026, also covers discovery 1. A page can point to its Markdown copy with rel="alternate" type="text/markdown" and to its covering llms.txt with rel="describedby" 1. Both can go in HTML link elements or in an HTTP Link header 1. By convention, a section titled Optional holds secondary links that an agent can skip when it needs a shorter context 1.

An AI agent uses the file at the moment it needs facts about a site. A common case is a person who points an assistant at a site on purpose and asks how its API works 3. The agent reads the summary, picks the relevant links and answers from those pages instead of from scraped navigation menus.

llms.txt vs robots.txt and sitemaps

llms.txt borrows its naming habit from robots.txt and sitemap.xml, a fixed filename at a known path 1. The job is different. robots.txt lets automated tools know what access to a site is acceptable, such as for search indexing bots 1. llms.txt grants and denies nothing. Its content is used on demand, when an agent needs information while assisting a user 1.

A sitemap lists the indexable pages on a site. The proposal gives three reasons it is no substitute 1. A sitemap often leaves out model-readable versions of pages. It does not include useful external URLs. It usually covers far more text than fits in a model's context 1.

The proposal expected llms.txt to matter most at inference time rather than for training, and says that is how it has been used 1. That shapes what belongs in the file. Write for an agent with one question and limited room. Put the facts a model is most likely to get wrong in the blockquote, such as the base URL, the auth method or the billing rule. Then link the pages that answer the next question.

Adoption is broad. The v2 page says thousands of sites publish the file and that documentation platforms generate one automatically 1. It adds that Chrome's Lighthouse audits sites for one as part of its agentic browsing checks 1. Google's guide to generative AI search says Google Search ignores llms.txt files, so creating one neither helps nor harms visibility or rankings there 2. The same guide says it is fine to maintain one for other services and systems that use it 2. Treat the file as a service to agents rather than a ranking lever.

llms.txt also differs from an MCP server. The file describes a site so an agent can read it. An MCP server exposes tools so an agent can act through it. A site that wants to be agent-readable often needs the first, and an API company usually wants both.

Example of an llms.txt file

RichAPI serves its own file at richapi.ai/llms.txt 4. A trimmed copy shows the shape.

text

# RichAPI
> GTM data API: 65+ endpoints, one key, REST + MCP. Explicit endpoint billing, bundle credits never expire.

- MCP:  https://mcp.richapi.ai/mcp  (65+ tools)

## Docs
- [API reference](https://app.richapi.ai/docs): Base URL, auth, response shapes, endpoint billing rules.
- [Full reference for agents](https://richapi.ai/llms-full.txt): Pricing, endpoint rates, FAQs and policies in one file.

## Product
- [Pricing](https://richapi.ai/pricing): Pay-as-you-go credit bundles, 100 free credits to start.

Each part maps to the spec. The H1 names the site and the blockquote carries the facts a model needs first 1. The bullet line under the quote adds detail without a heading, which the spec allows 1. The live file groups its links under three H2 sections named Docs, Product and Optional 4.

The Docs section links the long-form llms-full.txt. An agent that needs endpoint rates follows that one link instead of crawling the pricing page 4. The notes after each link say what the page answers, which helps the agent choose without opening every page.

To build a file like this for another site, the RichAPI generator takes a site name, a one-line summary and one page per line 3. It writes the name as an H1, the summary as a quote and the pages as a link list under a Pages heading 3. Nothing typed into it leaves the browser 3.

llms.txt vs llms-full.txt

The proposal describes llms.txt as a short index whose detail lives behind links 1. Some sites also publish llms-full.txt, one long file with the detail written inline. RichAPI publishes both, with llms.txt as the short index and llms-full.txt as the long form 3.

	llms.txt	llms-full.txt
What it answers	What is this site and where should an agent look?	What are the full details, in one fetch?
Input	Site name, summary and curated links	Page content, prices and policies written inline
Output	A short Markdown index	One long Markdown or text file
Cost driver	Few tokens per read	Many tokens per read
Main risk	Links that point at stale or missing pages	Too long to fit in a context window
Use it when	An agent needs a map first	An agent needs the whole reference at once

Read the reference →

Our take

llms.txt is cheap to ship and easy to overrate. Google says its search ignores the file, so it does nothing for rankings there 2. It pays off in a narrower moment, when an agent lands on a site with a specific question and a tight context budget. For an API company that moment decides whether the agent calls the right endpoint with the right header or guesses. So the blockquote deserves more care than the link list. Put the base URL, the auth method and the billing rule there, in plain words a model can quote back. Link the long reference, the pricing page and the docs, and leave the blog archive out. Then treat the file like code. Change it when a page moves, and recheck it whenever pricing or auth changes, because a stale llms.txt hands agents the wrong answer with full confidence.

Frequently asked questions

Where does an llms.txt file go?

At /llms.txt on the site root, served as plain text, or at a subpath such as /docs/llms.txt. A file covers the pages under its path, and when more than one file applies, agents should use the most specific one 1.

Is llms.txt an official web standard?

Its author describes it as a proposal to standardize on a /llms.txt file. It was first published in September 2024 and revised as v2 in August 2026 1. Treat it as a widely used convention rather than a formal standard.

Does llms.txt help SEO?

Google says its search ignores llms.txt files, so for Google they neither help nor harm visibility or rankings 2. The benefit is accuracy with other systems. When an assistant reads a site on request, a clear summary and good links make it more likely to repeat the right facts.

Does llms.txt block AI crawlers?

No. llms.txt grants and denies nothing. Access rules belong in robots.txt, which tells automated tools what access to a site is acceptable 1. llms.txt only describes the site and points agents to the pages worth reading.

The bottom line

llms.txt is a short Markdown file that tells AI agents what a site is and which pages to read. Only the H1 is required, but the summary and the links do the work 1. Keep it small, point it at clean pages, put the critical facts in the blockquote and update it whenever pages, prices or auth change.

Sources

The /llms.txt file, v2, llmstxt.org (Jeremy Howard)
Optimizing for generative AI features on Google Search, Google Search Central
llms.txt generator, RichAPI
RichAPI llms.txt, RichAPI