Have you ever wondered how large language models (LLMs) like ChatGPT, Gemini, or Claude decide which content from your website to showcase when addressing a query or providing an answer? This is where llms.txt comes into play.
llms.txt is a proposed standard, not an officially adopted system, designed to help AI systems understand which parts of a website are most useful. It is not enforced by OpenAI, Google, or any search engine yet. However, it is gaining attention among AI developers and documentation-heavy websites.
In simple terms, the LLMs.txt file contains a “must-read” list of your website’s important pages for Artificial Intelligence.
Why was the LLMs.txt File Proposed?
The idea was introduced by Jeremy Howard in 2024 to address the inefficiency of LLMs’ crawling and interpreting websites with extensive or deeply nested documentation (source).
The problem it solves: LLMs have context windows (a limit to how much content they can process at once). llms.txt helps AI tools to focus only on the most relevant sections instead of analysing the entire website.
How Does LLMs.txt Differ from robots.txt and sitemap.xml?
There is a significant difference between these three files (robots.txt, sitemap.xml and llms.txt):
| Feature | robots.txt | sitemap.xml | llms.txt |
| Purpose | Blocks or allows bots | Lists all site pages | Curates important content for LLMs |
| Audience | Search engine bots | Search engine bots | AI systems (like ChatGPT) |
| Format | Text | XML | Markdown-style text |
| Use Case | Restrict indexing | Help with indexing | Help AI read better |
In short:
- robots.txt tells search engine bots what to crawl and what not to.
- sitemap.xml tells search engine bots about the pages that are available on a website.
- llms.txt is a proposed file for AI to tell it what really matters – highlighting the important content of a website for large language models, like ChatGPT.
Why Would a Website Use LLMs.txt?
As more people have started using AI tools in their daily lives, these tools are often used to answer queries by browsing public web data. Adding llms.txt helps website owners ensure that they serve the best pages and content for AI tools.
Here’s why it matters:
- Better answers from AI using your content.
- More visibility of your most helpful pages.
- Control over how your content is understood.
Think of it like preparing the best highlights reel of your website for AI.
What Should Go in LLMs.txt?
The llms.txt file should include only your website’s most valuable and helpful content, things you’d want an AI to understand and reference clearly.
Core Pages might include:
- Homepage
- Product or service pages
- Top blog posts or articles
- Pricing page, etc.
Optional Pages could be:
- Privacy Policy
- FAQ
- Career opportunities, etc.
You can also include a second file called llms-full.txt if you want to list every page of your site.
What is the structure of the LLMs.txt file?
According to the proposal, this file follows a simple structure:
- The llms.txt file should be placed at the root of your website (e.g., domain.com/llms.txt).
- It uses basic Markdown formatting.
- It includes grouped links with clear headings like “Guides” or “Docs”.
- A quick summary that explains what your website is about.
- Link only important pages and focus on content that’s most useful or relevant.
- Just basic text is required. No XML is necessary, and no coding is required.
For example:
# Website Knowledge Base
> A collection of educational content and tools designed to help users learn, troubleshoot, and grow with confidence.
## Getting Started
– /guides/introduction – Overview of the platform and its features
– /guides/setup – Step-by-step setup instructions for new users
## Technical Documentation
– /docs/api-overview – General information about available APIs
– /docs/authentication – How to securely access the platform’s APIs
## Tools & Utilities
– /tools/data-analyser – A tool for processing and interpreting uploaded datasets
– /tools/form-builder – Create and customise web forms with ease
## Additional Information
– /about/goals – The mission and vision behind the platform
– /updates/changelog – Record of all major updates and improvements
Where Can I Find a List of All LLMs.txt Files Online?
There are some directories available online that collect live llms.txt examples, including:
These show how developers and websites are experimenting with the format today.
Things to Consider When Creating an LLMs.txt File
There are several things that you have to consider while creating a llms.txt file for your website:
- Stick to key URLs only (Focus on 10–20 key URLs, not a full sitemap).
- Use clear formatting with headings and bullet points.
- Avoid broken or redirected pages as they confuse AI.
- Include a brief site summary in a blockquote.
- Use descriptive anchor text or section labels.
✅ Good example: – /docs/setup – Install and configure your environment
❌ Bad example: – /docs/setup
How Do I Create LLMs.txt in Plain Text?
To create the llms.txt file manually, you can follow these simple steps:
- Open Notepad or any text editor.
- Paste the syntax shown earlier.
- List your URLs under the right sections.
- Save the file as llms.txt.
- Upload it to your root directory (same place as robots.txt).
How to Create LLMs.txt in WordPress?
If you use WordPress, then there are two methods to create an llms.txt file for your website:
Method 1: Manual Upload
- Create llms.txt using a text editor.
- Upload using an FTP client (like FileZilla).
- Place it in the /public_html/ directory.
Method 2: Use a Plugin
WordPress SEO plugins such as Yoast have enabled this feature. You can simply enable this feature by navigating to WordPress Dashboard > Yoast SEO > Settings > Site features > Enable llms.txt feature.
Common Mistakes to Avoid
There are some common mistakes that you should avoid while implementing an llms.txt file:
- Saving the file in the wrong place (it must be in the root directory).
- Adding HTML code: keep it simple, using text or Markdown.
- Including too many low-value URLs.
- Forgetting to update when pages are changed.
Final Thoughts
llms.txt is not a replacement for robots.txt or sitemap.xml file. It is a proposed convention that might help to make your site’s content easier for AI tools to process.
Although it’s still a relatively new concept, it reflects a growing interest in providing LLMs with clear instructions. Many websites are already adopting it. There’s little to lose but potentially a lot to gain.
If AI discoverability is important to you, trying out llms.txt could be a smart move
FAQs
1. What is llms.txt, and why was it created?
llms.txt is a proposed Markdown-formatted file placed at your website’s root that helps AI systems (LLMs) identify and understand the most important pages on your website. It was proposed by Jeremy Howard in September 2024 to address LLMs’ limited context windows, which make it difficult for them to process complex website structures
2. How is llms.txt different from robots.txt and sitemap.xml?
- robots.txt: instructs search engine bots on where they can or cannot crawl.
- sitemap.xml: lists every public URL for indexing.
- llms.txt: provides a curated, context-rich list of your key pages specifically for AI models
3. Do I need both llms.txt and llms-full.txt?
llms.txt contains your most important URLs with a brief description, ideal for guiding LLMs efficiently. llms-full.txt includes the full content of all referenced pages, so AI doesn’t need to fetch each page separately.
4. What is the format or structure of an llms.txt file?
- It’s simple Markdown with:
- An H1 title (#) – your site name.
- A blockquote for a short summary.
- H2 sections grouping links (##).
- Bullet list links with titles and optional descriptions under each H2
6. Who is using llms.txt right now?
Adoption is growing among documentation-heavy platforms. Notable examples: Anthropic, Pinecone, Windsurf, Svelte, and Wordlift. Also indexed in community directories, such as llms.txt. site and directory.
7. Is llms.txt officially recognised by AI providers like OpenAI or Google?
Not yet. While some AI systems (Perplexity, Claude) use it experimentally, major providers such as OpenAI/GPTBot and Google/Bard
have not officially endorsed it. It’s still a proposed standard.
8. What are the common mistakes when creating llms.txt?
- Incorrect placement (must be at your site root).
- Poor formatting or invalid Markdown.
- Too many or broken links.
- Forgetting to update when site content changes.
Validating your file with Markdown tools helps catch errors.
9. How often should I update my llms.txt file?
Update whenever you add or remove key pages or change content focus. A good rule is to check and update it quarterly or when there are major site updates, so your AI signals remain accurate.
10. Will AI tools actually use llms.txt?
Yes, some AI tools already support it, and more are starting to explore it. For example, Perplexity and Claude (from Anthropic) have begun using llms.txt files or at least recognise them.
11. Is llms.txt useful for SEO?
Indirectly, yes. It doesn’t affect Google rankings, but it can improve how your content is shown in AI tools.
12. Does file size matter?
Yes, keep it clean. Under 10KB is ideal for quick AI processing.
