# AI Bot Traffic in October 2026: Meta's Crawler Spiked Again and PerplexityBot Grew 149%

> What 4.69 million AI bot requests from October 1 to 27, 2026 show: requests per day rose 83%, Meta's crawler sent 874,904 requests on October 2 (most to Upstash's Vercel previews), search crawlers grew to 30% of requests and GPTBot fell 83%.

Published October 29, 2026 by Joscha.
Canonical: https://www.bydefault.so/blog/ai-bot-traffic-in-october-2026

For this post I analyzed 4.69 million AI bot requests between October 1 and 27, 2026. That's about 174,000 a day, up 83% from 95,000 a day in September. Meta-ExternalAgent still sent more than half of the requests. Search crawlers grew the most, from 20% to 30% of total requests, and PerplexityBot requests were up 149% per day. Live fetches for ChatGPT have also resumed growing, after falling all through September. GPTBot, the OpenAI training crawler, sent 83% fewer requests.

The two observations from [last month's post](https://www.bydefault.so/blog/how-ai-bots-crawled-the-web-in-september-2026-what-our-logs-show) that did not hold up: the bot we identified as Cursor is Exa, and ChatGPT's UK traffic is OpenAI's routing, not users in the UK.

## Which AI bots crawled websites most in October 2026?

In October 2026, Meta-ExternalAgent sent 2.53 million of the 4.69 million AI bot requests to the sites we track (54.0%). Next came ExaSearchBot (689,562 requests, 14.7%), OAI-SearchBot (264,012, 5.6%), ChatGPT-User (243,684, 5.2%) and Claude-User (218,599, 4.7%).

![](https://cdn.bydefault.so/image-rL9DFf5TU_GMtJvyK3OdB.png)

The number of requests per day from AI bots in October was 173,620, up 83% from September’s 94,918. When accounting only for sites tracked in both months, the increase was 82%, so new sites don't explain it. Upstash’s Vercel preview deployments saw 1.11 million AI bot requests in October, which accounts for a large portion of the increase. When excluding those requests, the increase per day was 46%.

AI bot traffic comes from three kinds of bots: training crawlers that collect pages for AI models, search crawlers that build the indexes AI assistants search, and fetchers that open a page while someone is chatting. Here are the ten biggest AI bots and how their daily traffic changed:

| Bot                | Owner      | What it does                           | Oct requests | Share of Oct | Change per day vs Sep |
| ------------------ | ---------- | -------------------------------------- | ------------ | ------------ | --------------------- |
| Meta-ExternalAgent | Meta       | Training and indexing                  | 2,529,789    | 54.0%        | +72%                  |
| ExaSearchBot       | Exa        | Builds Exa's search index              | 689,562      | 14.7%        | +847%                 |
| OAI-SearchBot      | OpenAI     | Builds ChatGPT's search index          | 264,012      | 5.6%         | +27%                  |
| ChatGPT-User       | OpenAI     | Fetches a page when a user asks        | 243,684      | 5.2%         | -12%                  |
| Claude-User        | Anthropic  | Fetches a page when a user asks        | 218,599      | 4.7%         | +48%                  |
| PerplexityBot      | Perplexity | Builds Perplexity's search index       | 201,111      | 4.3%         | +149%                 |
| ShapBot            | Parallel   | Builds Parallel's search index         | 170,895      | 3.6%         | +101%                 |
| ClaudeBot          | Anthropic  | Training                               | 127,455      | 2.7%         | +206%                 |
| Meta-WebIndexer    | Meta       | Builds Meta AI's search index          | 85,057       | 1.8%         | +30%                  |
| ExaUser            | Exa        | Fetches a page when a user or app asks | 72,871       | 1.6%         | new in October        |

GPTBot dropped out of the top ten. It sent 6,635 requests in October (0.14%), or 246 a day, down from 1,448 a day in September.

## Meta's crawler burst again, mostly on preview deployments

Meta-ExternalAgent sent 2.53 million requests in October, 2.0 million (78.9%) of which were sent on October 2-6. On October 2, Meta-ExternalAgent sent 874,904 requests (36x the daily average of October 7-27, and 3x its busiest September day, 286,494 requests on September 15).

![](https://cdn.bydefault.so/image-JMCvxMuTk_gyvI9o3VgJW.png)

Two thirds of the sites Meta crawled got at least three times their usual daily Meta requests during those five days. Most of the volume, though, was directed at one company. On October 2, Meta sent 683,931 requests to 31 of Upstash's Vercel preview deployments (the copies of upstash.com that Vercel builds for every branch, on URLs like website-…-upstash.vercel.app) and 34,010 more to upstash.com itself. All the other sites we track got 156,963 between them.

![](https://cdn.bydefault.so/image-LSrasIahd2xHhF8HmpX6n.png)

The September 29th and October 12th spikes in requests were also seen in Upstash’s previews. On October 12th, Upstash’s previews received 90% of requests from Meta, and the number of requests received from Meta without the previews was only up 4% on average per day compared to September. The October peak without the previews was also lower than the September peak.

This pattern was also observed by Vercel. On October 14th, Vercel’s CTO Malte Ubl wrote that there had been “an internet-scale event where Meta's crawler performed excessive, often repetitive requests” and that Vercel worked with the Meta team to normalize crawling volume:

![](https://cdn.bydefault.so/tweet-NxpGZ73iBKJdq1cO0vaZf.png)

His chart matches the overall shape of our chart, from the September 29 rise to October 12 peak. That makes sense, because most of our spike was on Vercel preview deployments.

Malte is also right that the requests are repetitive. On average in October, each URL got 8.4 requests from Meta-ExternalAgent. 64% of Meta’s requests were to the top 1% of URLs. When visiting Upstash’s previews, Meta repetitively requested marketing pages. For example, it requested /pricing/redis 46,900 times. That traffic was distributed over 45 previews. In total, 47 Upstash previews received 1.10 million requests from Meta in October. Docs requests jumped from 12.0% of Meta’s traffic in September to 28.2% in October. This change is also due to the preview deployments. Ignoring preview deployments, docs requests increased from 8.3% to 10.3%.

If your preview deployments are public, crawlers can find them. If you make sure previews are protected by your host’s deployment protection, or have a robots.txt that disallows everything on preview URLs, that will keep crawlers out of preview deployments.

Meta's training crawl and its AI search crawl use two different bots. Meta-ExternalAgent crawls ["for use cases such as training foundation AI models or improving products by indexing content directly"](https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers). Meta-WebIndexer is the one tied to Meta AI answers, and Meta says allowing it helps Meta "cite and link to your content in Meta AI's responses". So a site can stop the training crawl and still be citable in Meta AI:

```text
User-agent: meta-externalagent
Disallow: /

User-agent: meta-webindexer
Allow: /
```

Note: robots.txt is cached by Meta’s crawlers for up to 24 hours.

## Search crawlers grew the fastest

While training crawlers accounted for 61.0% of requests in September, that figure dropped to 57.4% in October. User-initiated fetches dropped from 18.7% of requests to 12.0%. Meanwhile, the percentage of requests from search crawlers—the crawlers used to build search indexes that AI assistants search before providing answers—increased from 20.0% to 30.3%. Search crawlers sent 52,636 requests a day in October, up 177% from 18,995 a day in September.

![](https://cdn.bydefault.so/image-LW1F3Jy3SfobvpLth2xtr.png)

Much of the search crawlers’ growth was driven by Exa’s crawler, mostly on a single large docs site. Even without Exa, though, daily requests from search crawlers grew by 66%.

PerplexityBot grew most steadily. It sent 201,111 requests in October, 7,449 a day against 2,995 a day in September, and more in each week of October than the week before, with the number of daily requests sent by PerplexityBot in the week of October 19 being four times as many as in the week of September 28. It also grew on almost nine in ten of the sites it crawled in both September and October. PerplexityBot is the crawler used to [surface and link websites in Perplexity's search results](https://docs.perplexity.ai/docs/resources/perplexity-crawlers). Perplexity says the crawler doesn’t collect content to train AI models.

Requests from other search crawlers grew too:

* **OAI-SearchBot** grew 27% per day. The crawler is used to build the index that powers ChatGPT search.
* **Claude-SearchBot** sent about eight times as many requests per day as in September. It still accounted for only 0.2% of requests.
* **Meta-WebIndexer** grew 30%.
* **ShapBot** doubled (+101%). The crawler is used to [index pages for Parallel's web APIs](https://docs.parallel.ai/resources/crawler).

A site that blocks these bots risks being dropped from the indexes used to power ChatGPT search, Perplexity, Claude and Meta AI. Note that each of the search crawlers has its own robots.txt token. These are separate from the robots.txt tokens for OpenAI’s, Anthropic’s and Meta’s training crawlers.

## Exa now names its bots

Last month’s post listed 4.7% of requests as coming from Cursor. Those requests were actually coming from Exa, a search engine that AI apps can query via an API. Cursor is an app that integrates Exa’s search engine, and Exa’s website even says that it “[powers their coding agents with smart token-efficient search across docs and repos](https://exa.ai/)”.

The identification of these requests was based on headers requesting markdown before HTML. Exa’s crawler has identified itself as ExaSearchBot since late July, and its live fetches looked like Chrome until October 9, when requests identifying as ExaUser began. Chrome-looking requests with the same headers had ceased by October 12. Both types of requests have requested llms.txt at about the same rate (45.0% for Chrome-looking requests in September vs. 40.1% for ExaUser requests in October), so they are both identified as Exa from this month on.

Exa has two bots:

* ExaSearchBot [pre-crawls pages for Exa’s search index](https://crawler.exa.ai/). It respects robots.txt directives, falling back to rules for major search engines and then the `*` group if none exist for ExaSearchBot.
* ExaUser [fetches a page live in response to a user’s or app’s request](https://crawler.exa.ai/exauser), and robots.txt may not be respected, as these requests are initiated by users.

![](https://cdn.bydefault.so/drawing-Qf7i9oiHs95GV3qbrkeMW.png)

Both bots use Web Bot Auth, which provides a signature in the request headers that sites can verify against keys Exa publishes. This enables sites to verify the identity of the bot without having to rely on the user agent alone.

ExaSearchBot sent 689,562 requests in October (14.7%). That's 25,539 a day, 9.5 times its 2,698 a day in September. ExaUser sent 72,871 (1.6%).

## Is ChatGPT still fetching fewer pages?

No. ChatGPT-User, the bot that fetches a page while someone is chatting with ChatGPT, sent 9,025 requests a day in October, 27% more than the 7,110 a day it sent from September 15 to 30. The average for the month of October is still 12% lower than the average for the month of September, but that’s because the first half of September was much busier than the rest of the month.

The number of daily requests from ChatGPT-User reached its lowest point in the last week of September and increased again in October. The number of daily requests from OAI-SearchBot, the bot used to create the index for ChatGPT search, increased by 27% per day in October.

GPTBot went the other way. It sent 246 requests a day, 83% fewer than September's 1,448. Most of GPTBot's September traffic came in three days, September 23 to 25 (30,249 requests, 69.6% of its month). GPTBot’s number of daily requests still decreased by half if those three days are excluded. GPTBot now sends 14% of the daily traffic it sent in August.

OpenAI says that when both OAI-SearchBot and GPTBot are allowed to access a site, OpenAI [may use the results from a single crawl for both](https://developers.openai.com/api/docs/bots) to avoid crawling the site twice. So even if GPTBot has been quiet, OpenAI might still be reading just as much of your site. It’s not possible to tell from our logs which crawl OpenAI uses for what.

![](https://cdn.bydefault.so/drawing-EmUAQ9B-ZW5aR622t9s_c.png)

## ChatGPT's UK traffic vanished in one day

In last month’s report, we wrote that ChatGPT-User countries were representative of where users were asking questions about you. The UK accounted for 51.0% of September’s fetches. This reading, however, was incorrect. On September 26, the UK’s share of ChatGPT-User fetches fell from 56.3% to 1.6% in a single day, while daily requests only fell by 6%. This reduction occurred on all of our sample’s large sites on the same day.

![](https://cdn.bydefault.so/image-OR7PYVE5nPw6W5MaZfYK5.png)

This change was not because people in the UK stopped using ChatGPT overnight. Instead, the country in ChatGPT-User logs represents where OpenAI fetches from. In October, 37.2% of ChatGPT-User fetches came from US IP addresses, followed by Japan (8.7%), Poland (5.8%), and Switzerland (4.9%). For Claude-User, 52.3% of fetches came from the US.

An increase in traffic from a specific country in ChatGPT-User logs could simply be due to OpenAI changing its routing.

## What Claude's bots read

ClaudeBot, Anthropic's training crawler, sent 127,455 requests in October, 206% more per day than in September. Over half of them (71,600, or 56.2%) came on October 21 and 22. On more typical days, it still increased requests per day twofold compared to September.

Claude-User sent 218,599 requests, 48% more per day, and still reads the pages people check before they pick a product. On upstash.com, it made 90,419 requests in October, twice as many as ChatGPT-User (44,433). This is the share of each bot's October requests that went to each page type:

| Bot                                   | Docs  | Pricing | Homepage | Blog  |
| ------------------------------------- | ----- | ------- | -------- | ----- |
| Claude-User                           | 33.5% | 13.7%   | 20.3%    | 4.6%  |
| ChatGPT-User                          | 19.2% | 2.0%    | 3.2%     | 4.3%  |
| OAI-SearchBot                         | 11.5% | 0.0%    | 7.1%     | 1.1%  |
| PerplexityBot                         | 6.9%  | 0.1%    | 0.2%     | 4.0%  |
| ClaudeBot                             | 8.9%  | 0.1%    | 1.5%     | 3.9%  |
| Meta-ExternalAgent                    | 28.2% | 2.6%    | 3.0%     | 13.5% |
| Meta-WebIndexer                       | 18.3% | 4.5%    | 3.9%     | 22.5% |
| All AI bots                           | 20.9% | 2.3%    | 3.3%     | 8.9%  |
| All AI bots, without Upstash previews | 11.4% | 1.6%    | 3.6%     | 4.3%  |

Claude-User sent 13.7% of its requests to pricing pages, 6 times more than the average of all AI bots. Of all bots, Meta-WebIndexer sent the most to blogs (22.5%). Meta's training crawler looks docs-heavy here only because of Upstash's previews, which were half docs pages. Anthropic says that if you block Claude-User, [it may make your site less likely to appear in user-directed web searches](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler).

I’ve removed llms.txt from the table this month because most of the llms.txt traffic went to one docs site, so its shares are mostly a reflection of that one site.

## What to check in your own logs

Here's what I'd check in your own logs after this month. Group requests by user agent, then by day and host:

1. **Meta bursting by day.** If you look at monthly totals you can miss bursts of Meta activity. If these bursts are costing you bandwidth, you could block Meta-ExternalAgent and allow Meta-WebIndexer.
2. **Preview and staging hosts.** Meta sent 1.10 million requests to Upstash's preview deployments in October. If yours are public, protect them or disallow crawling there.
3. **ExaSearchBot and ExaUser.** These bots are now sending their own names. ExaSearchBot is the one robots.txt can block.
4. **OAI-SearchBot and GPTBot.** You should be watching OAI-SearchBot more than GPTBot. Sites opting out of OAI-SearchBot won’t appear as answers in ChatGPT searches. If you have a quiet GPTBot, OpenAI might have reused the search crawl.
5. **Bot countries as data center locations.** ChatGPT-User’s share of traffic coming from the UK disappeared from one day to the next, while overall traffic didn’t change much.
6. **Docs and pricing.** Claude-User spends almost half of its requests on docs and pricing.

[ByDefault](https://www.bydefault.so/pricing) groups requests made by AI bots by bot, page, and day. It also tracks the AI answers that mention your site.