AI Bot Traffic in September 2026: Meta's Crawler Grew While ChatGPT Fetches Fell 59%
Josh
Co-founder
On this page
In this article I want to look at every request AI bots made to the websites we track between September 1 and 29, 2026.
Of all requests (many millions), Meta's training crawler sent more than half of them! ChatGPT's live page fetches are down 59% from August, while OpenAI's search crawler and Claude's user fetches grew. Claude-User and coding agents like Cursor spent a large share of their requests on docs, pricing pages and llms.txt (as expected).
Let's look at how AI is browsing real websites at the moment :]
Which AI bots visited websites most in September 2026?
In September 2026, Meta-ExternalAgent made 57.1% of the AI bot requests to the sites we track. That's more than all the other bots combined. ChatGPT-User came second with 11.1%, then OAI-SearchBot (8.1%), Claude-User (5.7%) and Cursor (4.7%).
AI bots sent 51.6% more requests per day in September than in August. When we looked only at sites we tracked in both months, requests grew by almost the same amount (51.7%). So just adding new sites doesn't explain the growth.
Daily traffic changed very differently for the ten biggest AI bots, for example:
What I find interesting is that these traffic patterns look quite different from e.g. Cloudflare. For the last 7 days of September, Cloudflare Radar ranks Googlebot first at 25.2% of AI-related bot traffic, then ClaudeBot at 12.4% and Meta-ExternalAgent at 10.1%. On our sites, ClaudeBot sent only 1.7%. Most of Anthropic's traffic here came from Claude-User, when people asked Claude questions.
Why did Meta's crawler take over?
Meta-ExternalAgent took over because of one burst. From September 9 to 17, it sent about ten times as many requests as usual, then dropped back to normal. Meta says the bot crawls the web "for use cases such as training foundation AI models" or for indexing content. It does not fetch pages for a live user.
Many of our customers reported massive traffic spikes from Meta at this time.
Meta-ExternalAgent sent 10 times as many requests in the second week of September as in the first (Sep 1 to 7). In the third week, it sent 11.5 times as many. In the last week (Sep 22 to 28), it dropped slightly below its first-week level. AI bots sent 11.8% of the month's requests on September 15 alone, six times as many as on a normal early-September day.
Meta-ExternalAgent visited 76.9% of the URLs it requested just once. But it sent 71% of its requests to the busiest 1% of URLs.
Websites outside of our sample got the same Meta spike in September. For example here, Meta-ExternalAgent was 70% of traffic to their site:
Meta runs a separate bot, Meta-WebIndexer, for Meta AI search. Meta says allowing it helps Meta AI cite and link to your content. On our sites, it sent 2.6% of requests, and its daily volume more than doubled (+114%).
By the way, you can totally block Meta's training crawler and still allow the one tied to citations if you want.
Is ChatGPT fetching fewer pages?
Yes. ChatGPT-User, the bot that fetches a page while someone is chatting with ChatGPT, sent 59% fewer requests per day in September than in August. OAI-SearchBot, which builds the index behind ChatGPT search, grew 25% over the same period.
OpenAI's three bots each need their own robots.txt rule.
- GPTBot collects pages that may be used to train OpenAI's models. Blocking it opts you out of training.
- OAI-SearchBot decides which sites show up in ChatGPT search answers. Sites that block it will not appear in those answers.
- ChatGPT-User visits a page when a user's question needs it. OpenAI says robots.txt rules may not apply, because a user started the request.
ChatGPT-User sent fewer requests to our sites every week in September. By the fourth week, it sent 60% fewer requests than in the first. OAI-SearchBot sent more requests at the end of the month. In the last week, it sent almost three times as many as the week before.
Our logs can't tell us why, but I suspect it's because OpenAI is expanding their own search index. If ChatGPT answers more questions from its own search index, it needs fewer live fetches per answer and more crawling ahead of time which would quite easily explain what we see. We explain how that works in how ChatGPT web search works.
ChatGPT-User and GPTBot also behave differently. ChatGPT-User fetched each URL about 9 times on average, but sent only 27.6% of its requests to its busiest 1% of URLs. That looks like people asking about many different pages. GPTBot fetched 94.9% of its URLs just once, which is what you'd expect from a training crawl. It also sent 17% fewer requests than in August.
Which AI crawlers read llms.txt?
User-triggered and coding bots mostly read docs, pricing and llms.txt. Crawlers mostly don't. Claude-User stands out most: 35.1% of its September requests went to docs pages and 14.7% to pricing pages. Across all AI bots, those two page types made up only 7.7% and 1.4% of requests.
(n/a here is for page types that weren't among that bot's main ones)
Claude-User's requests look like someone comparing products. A user asks Claude which tool fits their needs, and Claude opens the docs, the pricing page and the homepage to answer. Claude-User also grew 63% per day from August, the fastest growth among the big user-triggered bots. Anthropic warns that blocking Claude-User may reduce your site's visibility for user-directed web search.
Coding agents now make up a share of traffic too. Cursor sent 4.7% of requests and opened llms.txt more often than any other bot (18.0% of its requests). OpenCode, another AI coding agent, sent 65.8% of its requests to docs pages. On docs-only sites, the share is much higher: Cursor was 39.8% of AI traffic on docs hosted by Mintlify, a docs platform, in a March 2026 study.
AI bots sent 2.0% of their September requests to llms.txt. Almost all of that came from agents working for a user or a developer. Meta-ExternalAgent, the biggest bot by far, sent only 0.14% of its requests there. So llms.txt is worth having if you want agents to read your docs, but training crawlers mostly ignore it.
Where do AI fetches come from, and when?
Crawlers make almost all their requests from the US, but user-triggered fetches come from wherever the user is. Meta-ExternalAgent sent 99.6% of its requests from US IP addresses and OAI-SearchBot sent 98.4%. ChatGPT-User was the opposite: 51.0% of its fetches came from the UK and only 14.0% from the US.
ChatGPT-User's other requests came from several countries, with Denmark at 5.2%, Japan at 4.7% and Germany at 4.3%. ChatGPT-User's locations tell you more than any other bot in your logs about where people ask about you. A crawler's location only tells you where the data center is.
Timing tells a similar story. At 05:00 UTC, Meta-ExternalAgent made 69.0% of all AI bot requests. At 15:00 UTC, it made only 41.7%. ChatGPT-User peaked at 16:00 UTC with 14.9% of requests and dropped to 7.2% at 23:00 UTC. So human-driven fetches follow working hours in Europe and the US, while crawlers run around the clock.
Weekdays were busier too. AI bots sent about 77% more requests on an average Wednesday than on an average Sunday. We only have four or five of each weekday in the month, so Meta's burst has a big effect on the averages. Treat the weekday gap as a rough pattern.
What should you do with this?
TLDR: The biggest bot in your logs is probably a training crawler that sends no visitors. The smaller user-triggered bots fetch pages for people asking about you.
September's numbers point to a few things:
- Search and user bots decide citations. OAI-SearchBot, Claude-User, Claude-SearchBot, PerplexityBot and Meta-WebIndexer fetch the pages AI answers cite. A site that blocks them drops out of those answers.
- Training crawlers get their own rules. GPTBot, ClaudeBot and Meta-ExternalAgent each have a separate robots.txt token, so a site can block training and still allow search. Meta says robots.txt changes take about 24 hours to apply.
- Docs and pricing pages matter most to Claude-User. It spent half its requests on those two page types.
- An llms.txt file gets read by agents. It got 18.0% of Cursor's requests and 9.2% of Claude-User's.
- ChatGPT-User's countries follow your audience. A jump from a new country is a sign that people there are asking about you.
You can check all of this in your own logs by grouping requests by user agent and page path. ByDefault does that grouping for you and also tracks which AI answers cite your site.
How we measured
We used ByDefault's own logs to count requests that AI bots made to our customers' websites. We label each request by its user agent.
- Time window. September means September 1 to 29, 2026, in UTC, which is 29 full days. August is all 31 days. We compare daily averages, not monthly totals.
- Same-site check. We also measured only the sites we tracked in both months. Their growth (51.7%) matches the full sample (51.6%).
- Page types. We group URL paths by pattern: docs, pricing, homepage, llms.txt and a few others.
Thanks for reading!! :]