What is llms.txt & should you use it?

AI platforms like ChatGPT and Google's AI Overviews rely on crawlers to gather the content they cite. But JavaScript-heavy pages, sprawling navigation, and years of accumulated content make it hard for a crawler to find your most important material — and content a crawler misses can't show up in AI answers.

An llms.txt file is meant to give AI crawlers a curated starting point so they can understand your site’s hierarchy and find relevant content for answers. 

Below, we cover what it actually involves, whether any major AI platform has confirmed using it, and how to build and publish your own version if you want to test it.

What is llms.txt?

The llms.txt file is a proposed standard meant to help large language models (LLMs) better understand and use content from websites.

The idea is pretty straightforward: Instead of letting AI crawlers wander around your site, you give them a curated list of your most important content. This curated list (your llms.txt) tells AI which content on your site it should pay attention to.

We already have standards like robots.txt and sitemaps to help search engines navigate websites more efficiently. What’s different about llms.txt is that it’s specifically designed for AI models that might use your content to answer questions or generate responses for users.

There's speculation that llms.txt could boost visibility in AI-generated responses and drive referral traffic. As of now, there's no proof that it does. 

What is growing is the tooling around it, like Google Lighthouse, which now checks for llms.txt in site audits. That's a sign the file is being taken seriously, even though no model provider has confirmed using it yet.

What problem is llms.txt trying to solve?

An llms.txt file is trying to help AI crawlers browse websites more effectively. Right now, these crawlers run into two major challenges:

• Modern websites are hard to read. Most AI crawlers can only read your pages’ basic HTML — not content that gets loaded by JavaScript. That means llms.txt provides a clear, structured format that helps AI crawlers quickly digest the information.
• Most websites have an overabundance of information. AI crawlers don’t know which content is important when they visit your site. If they spend time scraping pages that aren’t useful (like older blog posts), they may generate responses based on suboptimal information. That's the problem llms.txt helps solve.

The file may also reduce crawl inefficiencies. Bots don't have unlimited patience with your site. If a crawler burns its budget parsing your nav menu and cookie banner, it may never get to your actual product pages — the ones you want it to cite.

That's where llms.txt comes in. It points crawlers straight at your best content, so more of it is likely to get read, indexed, and represented accurately in AI answers.

How are llms.txt files structured?

Your llms.txt files should be structured and formatted in Markdown.

Markdown is a lightweight markup language that uses plain text formatting syntax to create structured documents. (It's the same format developers use in GitHub README files, and it’s easily parseable by AI systems.)

Save the file as lowercase llms.txt, which is the filename the official specification uses.

Some common Markdown elements you'll use in your llms.txt file include:

• # for the H1 heading, ## for H2s, ### for H3s, and so on
• > for blockquotes to highlight important descriptions
• - or * for bullet points in unordered lists
• [text](url) for hyperlinks to your content
• : for adding descriptions next to links to help explain what they lead to 
• ``` for code blocks when sharing technical examples

Most llms.txt files follow the same basic pattern: an H1 with your site or product name, an optional blockquote description, then one or more H2 sections that group related links, each written as a bulleted link with an optional colon-separated description.

The official llms.txt specification provides a very basic example of how your file might look. But if your website is large or complex, you might want to add more structure — using H3s and H4s to create subsections, incorporating tables to organize data, or including code snippets to demonstrate API use.

There's nothing wrong with that. Markdown files are completely readable by AI crawlers, so you’re safe as long as you're using valid syntax. The additional structure might actually give AI crawlers more context.

Here's a simple example following the basic specification:

Are brands using the llms.txt standard?

Yes, brands are already using llms.txt files on their websites, and adoption is growing.

In July 2025, only 951 domains had published an llms.txt. But there have been 16,670 as of September 2026. That's roughly 17.5x growth in about 14 months — genuinely fast adoption for a spec that didn't exist before late 2024.

But zoom out: 16,670 domains is still a small percentage of the hundreds of millions of active websites out there. Which means the growth curve is real, but the window is still wide open. Publishing an llms.txt today doesn't put you in a crowded field — it puts you ahead of the vast majority of your competitors.

You can view any site's llms.txt file by adding /llms.txt to the end of its domain (for example, example.com/llms.txt). Here's how a few brands approach theirs:

Notice how differently each company approaches its llms.txt file.  

There’s nothing wrong with this. As long as they’re using valid Markdown, the file is machine-readable and can be easily processed by AI systems.

Should you use llms.txt on your site?

You can use llms.txt on your site as a low-cost experiment, but it isn't mandatory — your content can still appear in AI answers without one. Having a file ready costs you little if adoption grows. Just don't expect it to move the needle yet.

No major AI platform, including ChatGPT, Google's AI Overviews, or Anthropic's Claude, has confirmed that it reads or relies on llms.txt.

Google’s John Mueller also confirmed this on Bluesky:

That said, there are some interesting signals.

For example, Anthropic has published an llms.txt file on its own website. That doesn’t mean its AI crawler is actually using these files — but it suggests they’re probably at least open to the idea.

Google also released Open Knowledge Format (OKF), an open specification for packaging content for AI agents. But OKF isn't a replacement for llms.txt. An llms.txt file faces outward, toward the crawlers visiting your site, while OKF is built for an organization's own internal agents.

OKF is still early, with the current version at 0.1. But a Google-backed standard for AI agents shows that making content readable for AI is getting real attention.

Tooling has also started to catch up to the format, even without crawler confirmation. 

Google added an experimental Agentic Browsing category to Lighthouse, Chrome's site-auditing tool, and one of its checks looks for an llms.txt file. But that's not evidence that Google's own crawlers are using the file to answer queries, since Lighthouse audits and live crawling are separate systems.

We're still in the early speculation phase where people are implementing the file and hoping it might become useful someday.

Back in March 2025, we implemented this file on one of our sister sites, Search Engine Land, to see whether it offers any meaningful advantages in terms of AI visibility and traffic.

In our testing, we didn't find a correlation between implementing llms.txt and improved performance in AI results.

We also analyzed the server logs to see whether AI crawlers were actually accessing the file. From mid-August to late October 2025, the llms.txt page received zero visits from Google-Extended bot (Google's AI crawler), GPTBot (OpenAI's crawler), PerplexityBot, or ClaudeBot.

While traditional crawlers like Googlebot and Bingbot did visit the file, the file received only a few hits. That means they didn’t treat the file with any special importance.

If you're mainly curious what a finished file looks like, a handful of free llms.txt generator and validator tools exist to help you draft and check one quickly — they don't change the underlying uncertainty about whether any AI crawler reads it, but they can save you time if you decide to experiment.

If you also want to experiment with llms.txt on your own site, follow the step-by-step instructions below on how to implement it.

Further Reading: How can you do LLMO? and How can marketers adapt to LLM-powered search?

How to create an llms.txt file (step by step)

Create an llms.txt file by deciding which content you want to feature, creating the file, and uploading it to your website.

1. Decide which content you want to feature

Decide which content you want to feature by selecting the most important pages for your site to include in your llms.txt file.

At the very least, consider your:

• Product or service pages
• Up-to-date blog posts
• Pricing page
• About us page
• Contact page

These are typically the pages that will give AI a good idea of what your business does and how you help customers.

Favor evergreen, canonical content over anything time-sensitive or gated. Good candidates are pages you'd want an AI system to treat as authoritative months from now. Skip pages that are personalized, login-gated, duplicated, or likely to go out of date soon, like a single event announcement or a limited-time promotion.

2. Create the file

Create the file in a text editor like Notepad or Visual Studio Code and name it llms.txt.

You need to format the file using Markdown. A developer isn't strictly required for this step — it's a plain text file — but one can help make sure the Markdown syntax and structure hold up, especially on a larger site.

Here’s how the file’s structure could look:

3. Upload the file to your website

Upload the file to your website in the right location so AI crawlers can theoretically find it.

The exact location depends on the scope of your llms.txt file:

• If it covers your entire website, upload it to your root directory (i.e., "https://[yourdomain.com]") so it's accessible at "https://[yourdomain.com]/llms.txt"
• If the file is specifically about documentation, place it in the corresponding subdomain (e.g., "https://[docs.yourdomain.com]/llms.txt")

You may need a developer’s help to actually upload the file. This file has to be placed on your server — usually through your web hosting control panel, such as cPanel.

Log in to your hosting provider and then navigate to cPanel > File Manager.

Then, go to the correct directory. If your llms.txt file is meant for the whole site, go to the public_html/ folder. (That’s the root directory for most domains.)

If it’s for a subdomain like “https://[docs.yourdomain.com],” navigate to the folder assigned to that subdomain — often named /docs/ or similar.

Upload your llms.txt file there and save the changes.

Once the file is uploaded, confirm everything is working by opening a new tab and visiting the URL directly.

You can also run a quick audit of your website in Semrush Site Audit to confirm that your llms.txt is being picked up correctly.

If you publish an llms.txt, check your server logs periodically to see whether any AI crawlers are requesting it. And keep the file updated as you add or remove pages.

Help AI systems access your content

An llms.txt file isn't required for AI visibility, but more tools and companies are starting to take it seriously — so adding one is a simple way to hedge your bets if it does become standard.

It's not the only lever, though. Site Audit can show you what else might be limiting your AI visibility right now: JavaScript rendering issues, crawler blocks, thin content, and other barriers that keep AI systems from reading your site at all.

Simply run the audit and click “Issues” > “AI Search” to see issues impacting your AI visibility. 

Then, fix each issue to improve your visibility in AI search.