What is llms.txt? The New Standard for AI Crawlers

What llms.txt is

llms.txt is a plain-text markdown file served at /llms.txt. It gives large language models a concise, structured summary of your site, including the canonical URLs they should prioritise when answering questions about your brand, products and documentation.

It was proposed in 2024 as a lightweight standard, modelled loosely on robots.txt and sitemap.xml but designed specifically for LLMs rather than crawlers. Adoption has grown steadily through 2025 and into 2026, with major documentation platforms, SaaS vendors and publishers shipping versions of the file.

Why it exists

HTML pages are noisy. Navigation, ads, cookie banners, modals and scripts make it hard for an LLM to extract the substance of a page within its context window. Even a clean article page often weighs in at tens of thousands of tokens once boilerplate is included.

llms.txt solves that by offering a clean, curated index that the model can read in a few hundred tokens and then follow to specific URLs as needed. It is the difference between handing someone a messy filing cabinet and handing them a table of contents.

Recommended structure

• H1 with the brand or project name at the top of the file.
• Blockquote with a one-paragraph description of what the site is and who it serves.
• Sections grouping the most important URLs by topic — usually product, documentation, pricing and key reference pages.
• Optional /llms-full.txt with the full markdown content of every linked page, designed for models that can ingest it directly.
• Optional notes section flagging deprecated content or pages that should not be cited.

A minimal example

A simple llms.txt for a SaaS company might begin with an H1 of the company name, a blockquote describing the product in one paragraph, and then sections titled Product, Docs, Pricing and Company. Under each section, four to ten links to canonical URLs with a short description after each.

The goal is not completeness but curation. Anything you would not want to see quoted back to a buyer should not appear in llms.txt. Anything you actively want cited belongs at the top.

How to deploy it

• Host the file at the root of your primary domain, served as text/plain or text/markdown.
• Keep it under ~10KB so it fits cleanly into a small context window.
• Update it whenever you ship new key pages or deprecate old ones — at least quarterly.
• Mirror the canonical URLs from your sitemap. Do not introduce alternate URLs only used in llms.txt.
• Link to llms.txt from your sitemap and reference it in your llms-full.txt if you publish one.

What llms.txt does not do

llms.txt is not a substitute for robots.txt. Robots.txt controls whether crawlers may access your URLs at all. llms.txt assumes access is allowed and curates which content matters most. The two files live side by side.

It is also not a guarantee of citation. Models still decide what to cite based on the prompt, the retrieval index and their own training. llms.txt makes citation more likely and more accurate, but it does not force it.

How llms.txt fits with sitemap.xml and robots.txt

Think of the three files as a stack. robots.txt sets crawl permissions, sitemap.xml lists every indexable URL for machines, and llms.txt curates the subset that matters most for AI consumption. They are complementary rather than redundant.

The most common deployment pattern keeps sitemap.xml exhaustive and llms.txt opinionated. The sitemap describes the entire site for crawlers. The llms.txt file surfaces the 30 to 100 URLs that an AI engine should prioritise when answering questions about you.

Frequently asked questions