Who Reads llms.txt? 1.9 Million AI Bot Requests Show Agents Do and Crawlers Don't

We analyzed a sample of 1,877,319 AI bot requests. The total number of llms.txt requests made was 4,647 (not a lot 💀).

Crawlers, which are bots that download web pages for use in training or for inclusion in search results, almost never requested llms.txt. Agents, which are bots that retrieve pages as part of helping a person, on the other hand, did. 92% of the total number of llms.txt requests came from ChatGPT, Claude, Cursor and OpenCode. Most requests came from docs sites and in many cases from docs pages that contained links to llms.txt.

We also checked whether AI answers contain citations to llms.txt. In a sample of 108,417 answers from eight AI assistants, 303 contained citations to llms.txt.

What is llms.txt?

The llms.txt file is a markdown file at /llms.txt (or /docs/llms.txt) that gives AI agents a short description of the site and links to the most relevant pages. It is an informal community proposal from Answer.AI in 2024. No major AI company documents llms.txt support for its crawlers, but our logs show ChatGPT, Claude and Cursor reading it when they fetch pages for a person.

The Chrome Lighthouse tool checks sites for an llms.txt file, and OpenAI, Anthropic, and Google publish llms.txt files for their developer docs.

The file itself is simple. It has an H1 heading with the site name. Underneath that, there is a short summary of the site in a blockquote, followed by some notes. Then, there are some H2 headings with lists of links to other pages. These pages should be clean markdown versions, e.g. the same URL with .md appended.

This is our own file at bydefault.so, cut down to the first two links:

# ByDefault

> The AI visibility and GEO platform. Track where your brand appears in AI search results, see which content earns citations, and publish what wins more answers.

Every blog article below is available as markdown: append `.md` to the URL, or send `Accept: text/markdown`.

## Blog

- [How Does AI Search Work? 7,182 Answers From ChatGPT, Claude, Gemini and Cursor Compared](https://bydefault.so/blog/how-does-ai-search-work.md): How AI search works in ChatGPT, Claude, Gemini and Cursor.
- [AI Bot Traffic in September 2026: Meta's Crawler Grew While ChatGPT Fetches Fell 59%](https://bydefault.so/blog/how-ai-bots-crawled-the-web-in-september-2026-what-our-logs-show.md): We analyzed AI bot traffic to websites in September 2026.

## Product

- [Pricing](https://bydefault.so/pricing): plans and limits

Some sites also include an llms-full.txt file, which contains all the text from the site's pages. This file is not part of the proposal for the llms.txt file, which its author, Jeremy Howard, had originally intended to be used by agents to answer questions, rather than for training. This is what we see in our logs as well.

Do AI crawlers read llms.txt?

Almost never. In our sample, GPTBot, ClaudeBot, OAI-SearchBot and PerplexityBot asked for llms.txt 85 times in 90 days, but they asked for robots.txt 12,059 times. This is consistent with what Prosopo, a bot protection company, found in four days of its own logs for crawlers from verified IP addresses, which read robots.txt 120 times, but llms.txt zero times.

Agents that fetch pages for a person are different. They read the file far more often:

On a per request basis, the four agents are asking for llms.txt 27x as often as the six crawlers. 7.4 vs. 0.28 per 1,000 requests. Most of the agent requests come from one docs site, covered in the next section. But if we exclude that site, the agents are still asking 10x as often. 2.6 vs. 0.27 per 1,000 requests.

Of the crawlers, the most llms.txt requests come from Meta-ExternalAgent. It asked for llms.txt 221 times out of more than a million requests.

John Mueller at Google suggests that llms.txt matters once an AI system that sends you significant traffic checks for it as often as robots.txt. ChatGPT-User, Cursor, and OpenCode pass that test on our sites. They asked for llms.txt 3,497 times and for robots.txt 5 times. This is partly because agents that fetch on behalf of a person rarely ask for robots.txt at all.

Agents read llms.txt on docs that point to it

ChatGPT-User read llms.txt 3,141 times. 3,130 of those reads were on a single site: Upstash's docs. Everywhere else in our sample, it read llms.txt a total of 11 times out of 37,411 requests. Claude-User read llms.txt more broadly, on more than half of the sites, but 655 of the 761 total reads of llms.txt were on Upstash.

Every page in Upstash's docs carries a short note for agents. Readers don't see it, because the HTML hides it on screen. The Markdown version of each page starts with it:

> ## Documentation index
> Fetch the complete documentation index at: https://upstash.com/docs/llms.txt
> Search it with GET https://upstash.com/docs/search?q=<query>.
> Use these to discover all available pages before exploring further.

Another docs site in our sample does not have a note like that. ChatGPT-User read 9,830 of its docs pages in 90 days and didn't read its llms.txt once. It's not like we can say definitively, oh yes, that note made ChatGPT-User read its llms.txt. But we have a sample size of 2, and that's the clearest pattern in our data. Agents read the file if they can see it linked from the page they're on.

ChatGPT and Claude use the llms.txt file differently too. ChatGPT-User mostly read the big file. It read Upstash's llms-full.txt (3.6 MB of text) 2,260 times and the short llms.txt index 828 times. On the same docs, Claude-User read the index 499 times and the full file only 41 times. Claude then follows the links. Upstash's index links to .md versions of each docs page.

After Claude-User read an llms.txt file, its next request to that site within 10 seconds was a .md page 11.8% of the time, compared to 2.4% for any other request.

Claude now reads it more often than ChatGPT

To get a sense for how much the composition of agent requests reading llms.txt changed in the 90-day window, let's take a look at the site that received the most reads, Upstash, and was in our data the whole time. The chart below shows the number of reads per day on average in the first 31 days of the window (July 15 to August 14) and the last 31 days of the window (September 12 to October 12). In July, ChatGPT-User read Upstash's llms.txt files on average 76.6 times per day. By October, it had dropped to 9.4 reads per day. In contrast, Claude-User reads per day increased from 6.0 to 10.8 on average, and Cursor reads per day increased from 1.4 to 5.3.

This mirrors the overall traffic each of these agents has been sending to Upstash, with ChatGPT-User requests per day decreasing by 77% (from 5,391 to 1,240 on average), Claude-User requests per day increasing more than 2x (from 1,258 to 2,926 on average), and Cursor requests per day increasing about 7x (from 179 to 1,269 on average). Live fetches from ChatGPT also dropped across the sites we track in September.

Coding agents use llms.txt as their docs format

We've omitted Context7 from the numbers above. This is the Upstash docs service for coding agents that creates llms.txt files for open-source packages and enables authors to submit their own. Over our 90-day period, bots requested 107,749 distinct llms.txt files on context7.com, with a total of 318,318 requests.

Coding agents spent a big share of their visits there on these files:

GPTBot is a crawler that reads these files as well. On Context7, the llms.txt files are part of the site’s docs, so a crawler will treat them like any other page.

On Context7, Cursor alone asked for llms.txt files 224,025 times. On all the other sites in our sample, it asked for them 277 times. When a coding agent reads a library’s docs, it’s often reading an llms.txt file. This fits the bigger picture: coding agents made 45.3% of all requests to docs hosted on Mintlify in a 30-day study.

Do AI answers cite llms.txt?

Hardly ever. ByDefault runs tracked prompts against eight different AI assistants and stores any pages referred to by an answer. In a sample of 108,417 answers recorded between July 15 and October 12, 2026, 303 answers cited an llms.txt file, which is 0.28% of answers.

Codex, Cursor and OpenCode cited the files most. Gemini never cited one:

Across all four coding agents, llms.txt was referenced in 0.43% of the answers. This is about five times more often than the chat assistants, who referenced it in 0.08% of their answers.

On the citation level, the answer is even smaller. There was a total of 1,619,565 citations in the answers. Of those, only 321 were llms.txt files.

Google explicitly says its search engine doesn't use the file. In its AI optimization guide, Google says, "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them." But then the guide goes on to say you can have an llms.txt if you want for other services that use it.

Google's AI optimization guide on llms.txt

Studies have examined the effectiveness of the file in regards to the AI citations, and they found no effect. Ironically, a model of about 300,000 domains got more accurate when removing llms.txt as a factor. Other tests also found no correlation between having the file and better performance in AI answers.

How to set up llms.txt so agents find it

Making an llms.txt file is cheap and agents read it. We can see from our logs what influences whether they find it and what they can get from it:

1. Host the index where your docs are. Having a /docs/llms.txt file that covers all of the pages under /docs/ and a small /llms.txt file at the root of the site, that covers everything else, works well. Mintlify hosts llms.txt and llms-full.txt for you and GitBook added llms.txt to published docs in January 2025.
2. Link to Markdown pages. Our logs show that Claude-User read a .md page right after reading an llms.txt file about five times more often than for other types of requests. Our own blog has every article as .md and answers requests that ask for text/markdown.
3. Include a note about the llms.txt file on every page of docs. On Upstash, every page of docs has a hidden note with the URL of the llms.txt file. On 137,210 domains, AI bots fetched llms.txt only when a link, an index or a user told them it existed. The proposal also suggests adding a Link header to each page (see below).
4. If you have a full file, keep it. ChatGPT-User read Upstash’s 3.6 MB llms-full.txt almost three times as often as the small index file.
5. Monitor who reads it. You can group server logs by user agent and count the requests to the llms.txt paths. ByDefault’s crawler analytics groups crawler hits per bot and per page for you.

This is the header the proposal suggests, which points from a page to its Markdown version and to the llms.txt file that covers it:

Link: </docs/page.html.md>; rel="alternate"; type="text/markdown", </docs/llms.txt>; rel="describedby"

How we measured

We analyzed ByDefault’s own crawler logs and prompt-run results.

• Time period. July 15 to October 12, 2026 UTC, i.e. 90 full days.
• Read definition. A request for /llms.txt, /llms-full.txt, /docs/llms.txt or /docs/llms-full.txt.
• Citations. An answer counts if any of the cited pages’ URL contains llms.txt or llms-full.txt.