What Is llms.txt? A Site Guide for AI Bots (2026)

Over the past two years, the fastest-spreading piece of advice in SEO has been: "Add an llms.txt file to your site and watch your AI-search visibility climb."

This article covers both what the file actually is and — more importantly — how much of what's said about it actually holds up. Because a good chunk of what's circulating hasn't been proven.

What Is llms.txt?

llms.txt is a Markdown file that sits at your site's root (yoursite.com/llms.txt). Its purpose is to give large language models a summary of what your site is about and which pages matter most.

The idea was proposed by Jeremy Howard in 2024. It started from a practical problem: language models have limited context windows, and modern web pages are packed with JavaScript, menus, ads, and navigation elements. When a model fetches your page, it has to dig the actual content out of all that noise.

llms.txt proposes doing that extraction yourself: a clean, structured, directly readable summary.

How It Differs From robots.txt

File	Who it addresses	What it says	Adoption
robots.txt	Crawler bots	What not to crawl	A widely honored standard
sitemap.xml	Search engines	Which addresses exist	A widely supported standard
llms.txt	Language models	What the site is about, which pages matter most	Still a proposal

The distinction in that last row is the important one: robots.txt and sitemap.xml are standards search engines formally support. llms.txt has remained a community proposal — there's no explicit, official commitment from major AI providers that they actually read the file.

How Do You Write the File?

The format is plain Markdown:

# AppFellas

> A software and digital marketing company. Provides mobile app development,
> web development, and search engine visibility services.

## Key pages

- [Mobile App Pricing](https://appfellas.dev/mobile-app-pricing):
  pricing research compiled from 47 companies, with budget ranges for 16 app types
- [Services](https://appfellas.dev/services/software): mobile, MVP, custom software
- [Work](https://appfellas.dev/work): published apps

## Blog

- [How to build a mobile app](https://appfellas.dev/blog/how-to-build-an-app)

## Optional

- [Careers](https://appfellas.dev/careers)

Rules are simple:

A single # heading with the site/brand name
A one-paragraph summary marked with > — this line matters the most
## headings for sections
A short colon-separated description next to every link
An ## Optional section for pages that can be skipped if context runs tight

There's also a proposed llms-full.txt variant: the entire site's content in a single Markdown file. Not practical for larger sites.

But Does It Actually Work?

Time to be honest here.

What we know:

The file does no harm — it's a few KB of static text.
The writing process itself is useful: ranking your site's 10-20 most valuable pages forces you to clarify your own information architecture.
Some developer tools and documentation platforms do read and use the file.

What we don't know:

There's no public, verified statement from major LLM providers confirming they read this file during crawling.
There's no published measurement showing the file increases citation rates in AI-generated answers.

Bottom line: adding an llms.txt is a reasonable bet, but it shouldn't sit at the center of your visibility strategy. The work with measured impact lives elsewhere.

The Work That's Actually Been Measured

In a study on GEO published by teams from Princeton, Georgia Tech, and IIT Delhi, the interventions that most increased citation rates in AI-generated answers were:

Adding statistics: a noticeable visibility boost
Citing sources: a similarly sized boost
Adding quotations: a similarly sized boost

By contrast, classic tactics like increasing keyword density showed no measurable benefit.

The ranking makes the point clearly: making your content citable matters more than adding a file. Which is why llms.txt belongs at the tail end of your GEO work, not in place of it.

How to Add and Verify It

Prepare the file in the format above.
Place it at your site's root: yoursite.com/llms.txt.
Make sure your server serves it as text/plain or text/markdown.
Open it in a browser and confirm it returns 200 and the content displays correctly.
Make sure it's not blocked by robots.txt.
Update it whenever your site structure changes — a stale llms.txt is worse than none at all.

Note: there's also a proposal for a second variant, serving a Markdown version of individual pages at /page.md. Given the maintenance overhead, it's unnecessary for most sites.

The Decision on AI Bot Access

There's a more fundamental question to answer before llms.txt even comes up: do you allow AI bots to crawl your site in the first place?

Via robots.txt, you can block user agents like GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. But if you block them, llms.txt loses its purpose too — no one comes to read it.

The trade-off breaks down like this:

Allow them, and your content can get cited in AI-generated answers, boosting brand visibility — at the cost of giving away information without earning a click.
Block them, and your content stays protected — at the cost of shutting yourself out of a discovery channel that keeps growing.

For most businesses, allowing access is the right side to be on. We cover how to configure these settings in our robots.txt and sitemap guide.

Frequently Asked Questions

Does llms.txt affect my SEO rankings?

No. Google doesn't use it as a ranking signal. If it has any effect, it's on the source-selection side of generative engines.

Do I need it if I already have a sitemap.xml?

They do different jobs. A sitemap is a list of addresses; llms.txt provides priority and context. Having a sitemap doesn't create any conflict with adding llms.txt.

Is it worth it for a small site?

It takes about 20 minutes to prepare and does no harm. But do the work with measured impact first: concrete numbers, citing sources, a clear heading structure.

How do I know if adding the file actually worked?

There's no clean way to measure that today. The most practical method we have is periodically checking, by hand, how often your brand name shows up in AI tool responses.

Related Articles: