llms.txt Explained: What It Is and Do You Need One?

llms.txt is a plain Markdown file placed at the root of a website (yourdomain.com/llms.txt) that gives large language models a short, curated guide to your most important pages. It is a proposal, not an official standard, and no major AI search engine has confirmed that it reads the file to decide what to cite, so most websites do not need one to be visible in AI answers. It can still be a cheap, low-risk extra in a few specific situations, covered below.

This guide explains what the file is, what goes in it, what the evidence says about whether it works and how to decide for your own site. It is part of our answer engine optimisation resource set, so if you are looking for the things that do move AI visibility, read our guides on getting cited by ChatGPT and what answer engine optimization is alongside this one.

What is llms.txt?

The idea was proposed in September 2024 by Jeremy Howard of Answer.AI, and the proposal is published at llmstxt.org. The reasoning goes like this: a language model has a limited context window, and a modern web page is full of navigation, scripts and styling that waste it. A site owner could instead provide a clean, human-written summary of the site, with links to the most useful pages, in a format both people and machines can read easily.

The name echoes robots.txt and sitemap.xml, but the three files do different jobs:

• robots.txt tells crawlers which parts of your site they may access.
• sitemap.xml lists your URLs so crawlers can discover them.
• llms.txt offers a curated, readable overview with short descriptions, meant to help a model understand what matters on your site.

An important difference is that robots.txt and sitemaps are widely supported conventions that major crawlers honour. llms.txt is a community proposal that anyone may publish and anyone may ignore.

What does an llms.txt file look like?

The proposal specifies Markdown with a particular order of parts. Only the first heading is required:

1. An H1 with the name of the site or project.
2. A short blockquote summarising what the site is.
3. Optional paragraphs of extra context.
4. Zero or more H2 sections, each containing a list of links with a short note on each one.
5. An optional section named "Optional" for links that can be skipped when a model has limited space.

Here is a complete example:

# Example Co

> Example Co publishes product guides and documentation.
> This file lists the pages most useful for understanding the product.

Prices are listed in US dollars. Support is available by email.

## Product
- [Overview](https://example.com/product): What the software does and who uses it
- [Pricing](https://example.com/pricing): Plans, limits and billing terms
- [Integrations](https://example.com/integrations): Supported connections and setup notes

## Documentation
- [Getting started](https://example.com/docs/start): Install and first configuration
- [API reference](https://example.com/docs/api): Endpoints, authentication and errors

## Company
- [About](https://example.com/about): Who we are and how to contact us

## Optional
- [Changelog](https://example.com/changelog): Release history

A companion convention, llms-full.txt, puts the full text of your key pages into a single Markdown file. It is even less established than llms.txt itself, and it can become very large, so treat it as a niche option.

Do AI engines actually use llms.txt?

This is the question that matters, and the honest answer is: there is no confirmed support from the major platforms. The evidence points one way.

• Google says Search ignores it. Google's May 2026 guide to optimizing for generative AI features lists llms.txt under things you can ignore. It says you do not need machine-readable files, AI text files, markup or Markdown to appear in Google Search, including its generative AI features, because Google Search does not use them. It adds that publishing one will neither help nor harm your rankings.
• Google's spokespeople have been blunt. John Mueller has said none of the AI services have announced that they use llms.txt, and that server logs show they do not even check for it. He has also said that a file in which a site describes itself cannot help a model tell one site from another, since every site can claim to be the best.
• Usage data is thin. In a May 2026 study of 137,000 domains, Ahrefs reported that 97 percent of llms.txt files received no requests at all, and that the requests that did arrive came mostly from bots that were not AI answer engines. AI bots did not go looking for files that did not exist.
• Controlled before-and-after tests are unconvincing. Published experiments that tracked sites before and after adding the file have mostly found no measurable change in AI traffic, and where traffic rose, other changes were made at the same time.

Evidence changes, so check current statements from the platforms you care about before you decide. But as of this writing, nobody has shown that adding llms.txt makes you more likely to be cited.

Where llms.txt can help

The same people who doubt its value for search describe one narrow use. Mueller has said that if an automated system is already on your website, a file that explains the site's structure could help it. That points to a few realistic cases:

• Developer documentation. Coding assistants and agents that read documentation can use a clean index of reference pages. This is where the format gained most of its adoption, and it fits the original proposal's aim.
• Sites that expect agent visitors. If tools acting on behalf of users, such as browsing agents, are likely to visit and complete a task, a short guide to where things are can save them time.
• Internal or private use. You can give a model a curated entry point to your own knowledge base or help centre as part of a retrieval project. Here you control the reader, so it works regardless of what public engines do.

Google's Chrome team has also added an llms.txt check to experimental audits for agentic browsing, which suggests some tooling interest. Treat that as a signal about agents, not about search rankings.

Do you need an llms.txt file?

Use this decision path.

1. Is your goal to rank in Google, including AI Overviews and AI Mode? You do not need one. Google says it is ignored.
2. Is your goal to be cited by ChatGPT, Perplexity, Claude or Gemini? Nothing confirms that the file influences citations. Put your effort into crawler access, answer-first pages and evidence, which are known to matter.
3. Do you publish technical documentation that developers read with AI tools? A file is worth considering. It costs little and fits the audience.
4. Are you building an agent or retrieval system over your own content? Yes, a curated index is useful, whether or not you name it llms.txt.
5. Do you have the time to keep it accurate? If not, skip it. A stale file that points to removed pages or old prices is worse than none.

If you answered "no need" to the first two questions and "no" to the next two, skip the file and spend the hour on something with proven impact.

How to create one properly if you decide to

1. Choose 10 to 30 pages. Pick your most important, most stable pages: product or service overviews, key explainers, documentation entry points, contact and policy pages. This is curation, not a dump of the whole sitemap.
2. Write honest one-line notes. Describe what each page contains in plain words. Avoid promotional claims, since the file should read as a neutral map.
3. Use absolute URLs for the canonical version of each page, with HTTPS.
4. Save as plain text with UTF-8 encoding, named llms.txt, and place it at the root of your domain.
5. Check it loads at https://yourdomain.com/llms.txt with a 200 status and a plain-text content type.
6. Keep it in version control and add it to your release checklist, so changes to key pages trigger an update.
7. Review quarterly. Remove broken links and refresh descriptions.

A quick check script can flag broken links in your file:

import re, requests

text = requests.get("https://example.com/llms.txt", timeout=15).text
urls = re.findall(r"\]\((https?://[^)]+)\)", text)

for url in urls:
    status = requests.head(url, allow_redirects=True, timeout=15).status_code
    if status != 200:
        print(status, url)

What matters more than llms.txt

If your goal is to be named and linked in AI answers, these actions have a clearer track record than a text file.

• Let search crawlers in. Check robots.txt and firewall rules for the crawlers behind the engines you care about.
• Write answer-first pages. State the answer in the first two sentences, then explain.
• Publish evidence. Original data, named authors, dates and sources give engines something to verify.
• Be consistent across the web. Keep your facts identical on your site, profiles and listings.
• Measure it. Test a fixed set of prompts every week and log citations. Our guide to how to track AEO shows how, and our Google AI Overviews audit applies the same thinking to Google.

Common llms.txt myths

• "It is the new robots.txt." No. It cannot block or permit crawling. It is only a suggestion.
• "It gives you a ranking boost." There is no evidence of this from any major engine.
• "Every site must have one." The people who proposed it never claimed that, and Google says it is unnecessary for its search.
• "More content in it is better." The point is curation. A huge list defeats the purpose.
• "It protects your content from AI training." It does not. Use robots.txt rules and other access controls for that.

How Insyrge can help you grow

Insyrge helps you spend effort where AI visibility is actually won. We audit your crawler access, restructure key pages to answer questions directly, build the evidence AI engines can verify, and set up weekly testing so you can see what works. If you do have documentation or an agent project that would benefit from a curated index, we can build that too. Start with the free AI visibility check, see our AEO agency service, or contact us to talk through your goals.

Frequently asked questions

What does llms.txt stand for?

The name refers to large language models. The file is a plain Markdown text file at your domain root that lists and describes your most important pages for AI tools.

Will llms.txt improve my rankings or citations?

There is no confirmed evidence that it does. Google says its Search ignores the file, and no major AI answer engine has committed to using it. Treat it as optional.

Can llms.txt hurt my site?

Google says publishing one will neither help nor harm its search rankings. The main risk is maintenance: an outdated file can point tools to pages that are gone or wrong.

Where do I put the file?

At the root of your domain, so it loads at https://yourdomain.com/llms.txt. Files in sub-folders are far less likely to be found.

Is llms-full.txt different?

Yes. It is a related convention that places the full text of your key pages in one Markdown file. It is less established still, and it can be very large.

Should I block AI crawlers with llms.txt?

No. The file cannot block anything. Use robots.txt rules or server-level controls to manage which crawlers can access your site.