llms.txt: Purpose, Limits, and When to Use It
Learn what llms.txt is, see a working example, understand which AI systems read it, and how it differs from robots.txt.
llms.txt is an emerging convention for helping large language models find the most useful, human-curated content on a site. It is not the same thing as robots.txt, and it is not a reliable legal or technical mechanism for blocking model training.
That distinction matters. Many teams treat llms.txt like a defensive access-control file, but in practice it is better understood as a discovery and guidance layer for AI systems that choose to read it. If you want to restrict crawler access, robots.txt, bot-specific policies, authentication, and server controls still matter more.
This guide explains what llms.txt is for, where it helps, and where teams often overestimate it.
Quick takeaways
llms.txtis best treated as an AI-readable guide to your most useful content.- It does not replace
robots.txt, authentication, or server-side protections. - Use
llms.txtto improve discoverability and context, not as your only defense policy.
What llms.txt Is
The proposed llms.txt format gives site owners a place to point AI systems toward important pages, docs, summaries, and structured resources. Think of it as a curated map for LLM-oriented browsing rather than a crawl-permission file.
That makes it useful for:
- highlighting canonical docs or knowledge-base sections
- surfacing the pages you want models to read first
- providing concise summaries or navigation hints for AI agents
What llms.txt Is Not
llms.txt is not a guaranteed enforcement mechanism. If your goal is to block training crawlers or restrict sensitive paths, you still need to rely on bot-specific robots.txt rules, login walls, rate limits, and server-level controls.
It is also not a replacement for good technical SEO. If your site is hard to crawl, poorly structured, or missing clear canonicals, llms.txt alone will not fix those problems. Google also does not require any special AI-only text files or markup to appear in AI features.
robots.txt vs. llms.txt
Use the two files for different jobs:
| File | Main purpose | Best use |
|---|---|---|
robots.txt | Crawl policy | Allow, limit, or block bots from specific paths |
llms.txt | AI-readable guidance | Highlight the most useful pages or sections for LLM consumption |
If you want AI visibility, these files often work together: robots.txt controls access, while llms.txt helps with navigation and prioritization.
When to Publish llms.txt
Publishing llms.txt makes sense when you have:
- product docs or help centers with clear canonical pages
- long documentation trees where models may benefit from a curated index
- editorial content that is already well structured and source-backed
It is less useful if your site still has weak internal structure, duplicate pages, or unclear ownership signals. Fixing those foundations usually comes first.
Example llms.txt File
Here is a working example for a SaaS documentation site:
# Acme Project Management
> Acme is a project management platform for remote teams.
> It offers task tracking, time logging, and team dashboards.
## Documentation
- [Getting Started](https://docs.acme.com/getting-started): Setup guide for new users
- [API Reference](https://docs.acme.com/api): REST API endpoints and authentication
- [Integrations](https://docs.acme.com/integrations): Slack, GitHub, and Jira connectors
## Guides
- [Workflow Automation](https://docs.acme.com/guides/automation): How to set up automated task rules
- [Team Permissions](https://docs.acme.com/guides/permissions): Role-based access control
## Optional
- [Changelog](https://docs.acme.com/changelog): Recent updates and version history
- [Pricing](https://acme.com/pricing): Current plans and pricingKey formatting rules from the llms.txt specification:
- The file is Markdown, not a config file
- Start with
# Site Nameand a> descriptionblockquote - Use
##sections to group links by type (docs, guides, optional) - Each link includes a short description after a colon
- Place the file at your root URL:
https://yoursite.com/llms.txt
Some sites also publish a companion llms-full.txt file containing the full documentation text in a single Markdown file. This gives AI crawlers a single high-signal ingestion point instead of requiring them to stitch together many separate pages.
For SaaS-specific patterns and more examples, see llms.txt Examples for SaaS.
Which AI Systems Read llms.txt
As of April 2026, adoption by major AI platforms is limited:
| AI System | llms.txt Support | Notes |
|---|---|---|
| ChatGPT / GPTBot | Not confirmed | OpenAI has not committed to reading llms.txt |
| Perplexity | Not confirmed | Uses its own crawling and retrieval pipeline |
| Google Gemini / AIO | Not confirmed | Google does not require AI-specific text files |
| Claude / ClaudeBot | Not confirmed | Anthropic has not announced support |
| Cursor / Windsurf / IDEs | Partial | Some AI coding tools read llms.txt for documentation context |
| Documentation platforms | Yes | GitBook, Mintlify, ReadMe, and Fern generate llms.txt automatically |
Only about 2% of the most-cited domains in AI search currently have an llms.txt file. The specification is still a proposal, not a ratified standard, and no major AI company has formally committed to honoring it.
This is why llms.txt should be treated as a low-cost, low-risk discovery layer — not as a primary strategy for AI visibility. The real visibility drivers remain content quality, crawl access via robots.txt, structured data, and domain authority.
How to Implement llms.txt
Keep the file lightweight and editorially intentional:
- Identify your highest-value pages. Prioritize docs, guides, and reference pages that contain clear, factual information. Skip marketing landing pages.
- Write the file in Markdown following the format above. Use canonical URLs, not campaign or parameterized URLs.
- Host at your root URL. The file should be accessible at
https://yoursite.com/llms.txt. - Keep descriptions concise. Each link description should be one sentence explaining what the page covers.
- Update when content changes. Review the file quarterly or when major docs or navigation change.
- Consider
llms-full.txt. If your documentation is under ~50 pages, a full-text companion file gives AI systems everything in one request.
Limitations
The biggest limitation is adoption. llms.txt is still a convention, not a universal standard with guaranteed support. Some crawlers may ignore it entirely. Others may use it only as a discovery hint.
There is also no enforcement mechanism. Unlike robots.txt, which has decades of established crawler behavior, llms.txt has no legal or technical weight behind it. Publishing the file costs very little effort, but expecting it to meaningfully change AI citation outcomes would be overestimating its current reach.
References
- llms.txt
- Google Search Central: Intro to robots.txt
- Google Search Central: AI features and your website
- OpenAI: Publishers and developers FAQ