What Is llms.txt and When Is It Worth Adding One?
llms.txt gives AI tools a curated map of your site, but no major AI answer engine says it uses one. When it's worth adding, plus setup steps.
llms.txt is a proposed markdown file, placed at the root of a website, that gives large language models (LLMs) a short summary of the site and a curated list of links to its most useful pages. It is a voluntary convention, and no major artificial intelligence (AI) search engine has publicly documented using it to choose sources.
It matters now because the file has become trivial to add. Documentation platforms such as Mintlify generate one automatically, several website plugins offer it as a setting, and some agencies sell it as an AI visibility fix. That makes it easy to ship one without knowing what it does, who reads it, or what it can't do.
What problem was llms.txt designed to solve?
llms.txt was designed to give a language model a clean, compact map of a site at the moment the model needs to work with it. The proposal came from Jeremy Howard of Answer.AI, and the specification is published at llmstxt.org.
The problem is mechanical. A model reading a web page can only hold so much text at once, a limit called the context window. A typical page built in HyperText Markup Language (HTML) carries navigation menus, cookie banners, scripts, and layout code that take up space and add noise. If a coding assistant needs to use a library's application programming interface (API), it does better with a short index of the documentation and clean text versions of the few pages that matter.
That origin explains the design choices. llms.txt targets inference, when a model or agent is actively doing a task for someone, rather than the bulk crawling that feeds model training. Its first home was software documentation, where developers point tools like Cursor or Claude at docs and want the model to get function names and parameters right.
What does an llms.txt file contain?
An llms.txt file is a markdown document that follows a fixed order: a title, a short summary, optional notes, and sections of annotated links. The proposal only requires the title, but the summary and links are where the value is.
The parts appear in this order:
A top-level heading (a single # line) with the name of the site or project.
A blockquote (lines starting with >) that summarizes what the site is and who it serves, in a few sentences.
Optional paragraphs or lists with context a model should know, such as product names, supported platforms, or terminology.
Sections marked with ## headings, each containing a list of links written as the page name in square brackets, the page address in parentheses, then a colon and a short description.
A section titled "Optional" for secondary links that a tool can skip when space is tight.
The proposal also suggests publishing a markdown version of important pages at the same address with .md added, so a model can fetch clean text instead of HTML. That part is optional and takes more engineering work than the file itself.
Here is a simplified sample for a fictional analytics company. In the real file, the title line starts with a single #, each section name starts with ##, and each link uses the markdown link format described above. Swap in your own pages and descriptions.
`markdown Title: Northpine Analytics
Northpine Analytics is a product analytics platform for mobile apps. It tracks in-app events, builds retention cohorts, and alerts teams when key metrics change.
Northpine has a free tier and paid plans for teams. It supports iOS, Android, React Native, and Flutter apps. Current plan limits are listed on the pricing page.
Section: Product
Product overview (product page on northpine.example): What Northpine measures and who it is for
Pricing (pricing page on northpine.example): Plans, limits, and what each tier includes
Security (security page on northpine.example): Data handling, retention periods, and compliance
Section: Documentation
Quickstart (quickstart page on docs.northpine.example): Install the software development kit and send a first event
Event tracking guide (events page on docs.northpine.example): Naming conventions and required properties
API reference (API page on docs.northpine.example): Endpoints, authentication, and rate limits
Section: Optional
Changelog (changelog page on docs.northpine.example): Release notes by version
About the company (about page on northpine.example): Team, funding, and locations
`
Notice what the descriptions do. Each one tells a model when the link is useful, which is exactly the information a sitemap leaves out.
How is llms.txt different from robots.txt and sitemaps?
robots.txt tells crawlers which parts of a site they may fetch, a sitemap lists every page you want search engines to index, and llms.txt recommends a small set of pages and explains what each one is for. Only robots.txt affects access, and only llms.txt is written for a model to read as content.
File | robots.txt | Sitemap | llms.txt |
|---|---|---|---|
Main job | Allow or disallow crawling by path | List indexable pages for discovery | Summarize the site and point to key pages |
Format | Plain-text directives | Extensible Markup Language (XML) | Markdown |
Typical size | A few lines to a few dozen | Every indexable page | A handful to a few dozen links |
Who reads it | Search crawlers and most AI crawlers, including GPTBot, ClaudeBot, and PerplexityBot | Search engines such as Google and Bing | Tools and agents that choose to fetch it |
Controls access | Yes, for crawlers that honor it | No | No |
Status | Published internet standard | Long-established protocol | Community proposal |
The overlap people worry about is small. A sitemap aims for completeness so search engines miss nothing, while llms.txt aims for selection so a model with limited space reads the right pages first. Keeping both is normal, and neither replaces the other.
If your actual goal is deciding which AI crawlers can fetch your content, or keeping your pages out of model training, robots.txt is the tool. Our guide on AI crawlers and robots.txt covers the user agents and directives. llms.txt has no allow or disallow rules at all.
Which AI systems have said they use llms.txt?
None of the major AI answer engines has published documentation stating that it reads llms.txt to decide what to retrieve or cite. That covers ChatGPT, Gemini, Claude, Perplexity, Microsoft Copilot, and Google AI Overviews. Google's search team has said publicly that Google Search does not use the file, and Google AI Overviews and AI Mode build answers from the same index that regular search uses. This can change, so check the crawler documentation that OpenAI, Anthropic, Perplexity, and Google each publish before planning around it.
A common source of confusion is that AI companies publish llms.txt files for their own documentation. Anthropic's docs have one, as do Cloudflare's developer docs and many sites hosted on Mintlify. Publishing a file to help developers use your docs says nothing about whether your answer engine reads other people's files.
Where the file does get read is in direct use. A developer can paste an llms.txt address into Cursor, Claude, or ChatGPT and ask the model to work from it. Some documentation tools built for coding agents fetch llms.txt to find the right page before answering. Agents that browse a site on a user's behalf can follow it like any other link. In each case a person or tool chose to open the file, which is very different from an answer engine consulting it on its own.
Server logs can mislead here. Once you publish a file, you may see requests for /llms.txt from assorted bots. A request shows that something fetched the file. It does not show that any answer was built from it.
When is llms.txt worth implementing?
llms.txt is worth an afternoon if developers or AI agents need to use your documentation, and it is a low-priority extra for most marketing sites. The file is cheap to make, so the real question is where it sits in the queue.
Developer tools, APIs, and software libraries: High value. Developers already feed docs to coding assistants, and a good index helps those assistants find the right page and get details right.
Large help centers and knowledge bases: Moderate value. Support content is long and nested, and a curated index helps any tool a customer points at it.
Products that AI agents may operate: Moderate value. If agents read your docs to complete tasks for users, a clean map reduces wrong turns.
Marketing sites for most brands: Low value. Your AI visibility depends on whether your pages rank in search indexes and whether other sources mention you, and llms.txt touches neither.
The best argument for a marketing site is indirect. Writing the file forces you to decide which ten or so pages explain your company and to write a plain summary of what you do. If that exercise exposes a missing pricing page or a vague product description, fixing those pages helps every system that reads your site, including search engines and the AI features built on them.
What llms.txt will not do
llms.txt will not get your pages cited, will not change what models learned in training, and will not block anyone. Set expectations accordingly before it goes on a roadmap.
Citations in AI answers: When ChatGPT, Perplexity, or Gemini searches the web, it retrieves pages from search results and cites them. That retrieval, the focus of answer engine optimization (AEO), depends on your pages being indexed, ranking for the question, and answering it clearly.
[Training data](/resources/what-is-training-data): What a model knows without searching comes from its training data. Influencing that, the focus of generative engine optimization (GEO), runs through sources like Wikipedia, Reddit, and industry publications.
Access control: The file has no directives. Crawlers you want to block need robots.txt rules or server-level blocking.
Tracking: Nothing in llms.txt reports back when a model reads it. The visible signal is referral traffic from people who click cited links, arriving from sites such as chatgpt.com and perplexity.ai.
To see whether AI visibility actually moves, run your buyers' questions across assistants many times and track how often you appear and which pages are cited, since single answers vary with phrasing, model version, and chance. nonBot AI does this across ChatGPT, Gemini, Perplexity, Claude, and Google AI Overviews, and also checks whether AI crawlers can read your site.
How do you set up an llms.txt file?
Setting up llms.txt takes a few hours for most sites: pick the pages, write the file, publish it at the root, and assign someone to keep it current.
Choose the pages: List the pages a model would need to describe you accurately or complete a task, such as product overview, pricing, key documentation, policies, and contact details. Leave out blog archives and campaign landing pages.
Write the summary first: Draft the blockquote as two or three plain sentences stating what you sell, who it is for, and what sets it apart. If your team cannot agree on this paragraph, that is the bigger problem to solve.
Add links with descriptions: Group links under ## headings and give each a short note about when it is useful. Put secondary pages under an "Optional" heading.
Decide on markdown page versions: Documentation platforms often generate .md versions automatically. On a custom marketing site, linking to normal HTML pages is fine.
Publish at the root: The file should load at yourdomain.com/llms.txt as plain text with a 200 status code. Most content management systems (CMS) let you upload a static file, and some plugins generate one.
Check that bots can reach it: Confirm robots.txt does not disallow the path and that firewall or bot-protection rules are not blocking the AI crawlers you allow elsewhere.
Test it by hand: Paste the address into ChatGPT or Claude and ask questions about your product. If the answers come back wrong or vague, the summary or descriptions need work.
Give it an owner: Add the file to the checklist for pricing changes, product launches, and documentation releases, because a stale llms.txt points tools to outdated pages.
Before any of that, open a blank document and write the two-sentence summary of your company that you would want a model to repeat. That paragraph is the core of the file, and it belongs on your homepage and About page as well.
FAQ
Can llms.txt stop AI companies from training on my content?
No. llms.txt has no rules for allowing or blocking crawlers, so it cannot opt you out of anything. To limit training crawlers, add robots.txt rules for their user agents, such as GPTBot for OpenAI or Google-Extended for Google's AI models, and check each provider's documentation for current names.
Should llms.txt list every page on my site?
No. The point is selection, and a file that lists everything becomes a second sitemap that a model has to sort through. Keep it to the pages that explain what you do and help someone complete a task, and move secondary material into the Optional section.
Can adding llms.txt or markdown page copies hurt my search rankings?
The file itself is harmless, since Google has said it does not use it. Markdown copies of pages are a different matter if search engines index them, because they duplicate your HTML pages. Serve them with a noindex instruction in an X-Robots-Tag header, or use a canonical link header pointing to the HTML version.
What is llms-full.txt?
llms-full.txt is a companion convention that puts the full text of a site's documentation into one markdown file instead of linking out. Documentation platforms popularized it because coding tools can load an entire doc set in a single request. It suits compact developer docs and is rarely worth building for a marketing site.
How will I know if a major AI answer engine starts using llms.txt?
Watch the crawler and webmaster documentation that OpenAI, Anthropic, Google, Perplexity, and Microsoft publish, since any support would be announced there. Log entries for /llms.txt are not proof, because many bots fetch common files by default. If support does arrive, a file you have already written and kept current costs nothing to leave in place.
Measure Your Brand's AI Visibility
See how often AI assistants like ChatGPT and Perplexity recommend your business.
Free analysis • No credit card required
About nonBot AI: We help brands optimize their visibility across AI platforms—both retrieval-based and training-based. Our AI Visibility tool tracks your presence across ChatGPT, Perplexity, Claude, and more. If you're ready to build a real AIO strategy, talk to an expert.
