ChatGPT Knowledge Cutoff vs. Web Search: How Fast 8 AI Assistants Pick Up a Change on Your Site
Josh
Co-founder
On this page
On October 2, 2026, Neon increased the storage of its free Postgres plan from 0.5 GB to 1 GB per project.
The next morning, ChatGPT mentioned in an answer to someone asking about a free Postgres database that Neon's free plan "currently includes 1 GB storage per project." A week later, Claude was still telling people that it has "0.5 GB storage."
We watched this happen because we ask 8 AI assistants 20 of the same Postgres questions every day for State of AI Search. Neon changed the number in the middle of our data, so we used it to answer the question everyone with a pricing page has: When a fact on your site changes, how long until each AI assistant says the new one?
We tracked 4 real changes through 5,748 recorded answers, then we asked all 8 assistants about 8 dated changes directly - 192 times.
Here's what we found:
If an assistant searches for your number, it'll know the new one within a day.
If an assistant mentions your number offhand, it could keep saying the old one for months because it's in the model weights.
How long until ChatGPT shows a change on your site?
ChatGPT returned Neon's new number in its very first daily run after the change, one day after it provided its last answer with the old number. Codex did the same. Claude and Claude Code gave the old number for the entire week while the other four assistants gave a mix of answers.
Our assistants answer every day at about 03:00 UTC. On October 1 and 2, 51 answers gave a storage number for Neon's free plan. 48 of them said 0.5 GB, three hedged with "0.5 GB to 1 GB", and none said 1 GB on its own. Neon published the change on October 2, and the run on October 3 was the first one after it:
The new number arrived in one day and then stopped spreading. From October 3 to 9, 204 answers gave a number, and 127 of them (62%) still gave the old one.
Which number an answer gave depended on which assistant wrote it. These are the answers from October 3 to 9:
In our previous article about why Cursor recommends Neon, we mentioned that Neon had updated the free storage allowance from 0.5 GB to 1 GB.
The results in the table above are slightly more stringent because we considered the number only when the answer connected it to Neon's free tier. ChatGPT and Codex knew how to look for the updated number. All of their 42 answers mentioning 1 GB cite Neon's announcement, which says "Neon gives you 100 projects for free, with 1 GB of Postgres storage each" in its title. Neon's changelog, also published on the same day, has this line as well:
What is a knowledge cutoff?
The knowledge cutoff date of a model is the date after which a model hasn't seen any training data. After that date, the model only knows what it has seen on the internet through its browsing tool - when it answers the question. OpenAI publishes a knowledge cutoff for its latest models as April 30 or May 18, 2026. Anthropic publishes its training data cutoff date as June 2026 for its latest models. Neon changed its pricing on October 2 - months after those dates.
So, when the assistant is looking for your number, it can pull it from two places:
- What it can remember. This is what it saw when it was being trained - up to its knowledge cutoff date. It's fast and it's fixed.
- What it can find now. This is what it can find on pages on the internet through its browsing tool - when it's answering your question. It could be from 1 hour ago, or 3 years ago.
Now - if the assistant has to browse, it'll only be useful if it can find pages that have the updated number. So it'll try to find pages by searching the internet with the question it's being asked, and it'll read what it finds.
In the week after Neon changed their pricing, the answers that got the new number ran an average of 4.8 searches. The answers that got the old number ran an average of 1.8.
Old numbers fade over months
One week is a short window, so we looked for older changes in the same data. Three vendors in our reports changed a number that the assistants quote often, and each one has a dated announcement:
All these answers are based on nine daily runs from October 1 to October 9, 2026, in our Postgres, auth, Redis and app hosting reports, except Neon which counts rows from October 3 to October 9, 2026.
Even eight months after Clerk increased their free tier, nearly a quarter of answers which mention a number, still say 10,000. And after 19 months and 29 months, 7% still say the old number.
The same split by assistant shows up again. Each cell is the number of answers with only the old number, out of the answers that gave a number:
ChatGPT and Codex were fastest to update their data, with no outdated references across 213 answers for the four changes.
The pair of slowest LLMs were Claude and Claude Code. They provided outdated information for almost all questions at 1 week, roughly half at 8 months, and 3 of 73 at 19 months.
Google AI Overviews is unique in its slowness to update. It is still sharing outdated information that a free Render database is available for 90 days in 11 of 15 answers, 29 months after Render changed it.
Where the old number comes from
For three of the four changes, most answers with the old number do not cite the vendor's site at all. They ran fewer searches, and they cited someone else's page:
0 of the 127 old-number Neon answers in the Neon week restricted their search to Neon's site. 39 of the 69 new-number answers did.
The old number came from comparison pages. The page most cited in the 127 old-number answers was a roundup of free Postgres hosting (46 answers). We opened it on October 9. It says, "Updated July 22, 2026," and lists 0.5 GB for Neon in the table.
The second most cited page, another roundup (42 answers), says it was updated on July 31, 2026 and lists Neon with "0.5 GB storage, 100 CU-hours/month."
These two pages were accurate on the day the author checked. Neon changed the number two months later.
Old numbers can also be found on the vendor's own older pages. On Upstash's docs FAQ, there's a question about "10.000 requests per day" free limit. The limit changed 19 months ago. Now it's 500,000 a month. When we asked ChatGPT about it directly, one answer said, "Some older Upstash docs still mention 10,000 commands per day; the current pricing page lists the monthly limit."
This is how an old number gets to an answer.
Asked directly, 188 of 192 answers had the new value
But we only see the numbers when they're answers inside a bigger question. So on Oct 9, 2026, we did another test where we asked for the number directly.
We picked eight public changes, from one day old to 29 months old. Each one has a dated page that states the old and the new value:
We sent each of the 8 assistants the same 8 questions 3 times with internet search enabled, getting 192 responses. Then, we used a script to look through each response and see if the old or new value was mentioned. For each response that mentioned the old value, we then read through it.
No matter if the change was made one day ago, or 29 months ago, the values come back correctly.
Claude and Claude Code, which gave Neon's old number all week in the stored answers, gave 1 GB in 6 of 6 direct answers. This is one from Codex:
Many of these answers did mention the previous value the way this one does, as a bit of historical context, which we counted as a success.
Four answers did not: Gemini twice answered "No, you cannot" to the question about the Vercel option that launched the previous day (its third answer found the changelog and named the option); Claude Code once said that 300 seconds is the longest a Vercel Function can run on Pro (most other answers to this question provide both 800 seconds and 30 minutes, since the latter is opt-in); and ChatGPT once said that Upstash has both the 500,000 requests per month and 10,000 requests per day limits.
The more recent a change, the more answers are based directly on its announcement page: 46 of the 48 answers about the two most recent changes cite the changelog post, compared to 35 of 72 answers about the three oldest changes (the rest are pricing page and documentation).
This test had its limits. For one thing, we collected only three answers for each assistant and question. Also, we called each of them either through its API or its CLI tool, so the consumer applications may behave differently. Finally, every question mentions the name of the vendor, which encourages the assistant to search for that vendor's name, and all eight changes are in developer tools with public changelogs.
Why a passing mention stays stale
The same eight assistants got two different results for the same fact in the same week. When asking about free storage in Neon, 24/24 answers said 1 GB. However, in the daily questions, when the answer was part of a larger answer, 69/204 answers said 1 GB.
A direct question names the vendor, so the assistant searches for the vendor. Our 20 report questions name no vendor. On October 5, someone asks Claude "which postgres host has the best free tier", and Claude runs these two searches:
These searches return roundups. Claude cited twelve pages in its answer, including the two we mentioned in the section above. All twelve are third-party lists of free Postgres hosting providers. Of note, the answer above cited Neon's free tier as 0.5 GB.
ChatGPT handles the same kind of question differently. On October 3, for "where can i get a free postgres database", it ran one search for the category and then one for each vendor:
This approach turns the buyer's query into a search through each vendor's website. This behavior explains why ChatGPT's search feature was aware of Neon's new limit as early as the next day. Google AI Overviews is probably the clearest example of the gap between the two kinds of question.
Google says free Render databases last 30 days in 3 out of 3 answers when asked how long one lasts. Google says they last 90 days in 11 out of 15 answers to questions like "where can i get a free postgres database". When someone is still choosing a database, they'll ask what database to use, the Google AI Overview will tell them about your limit, and Google will get that number from whatever website page the category search finds.
What to do when a number changes
Each of these comes from a result above:
- Publish a dated announcement with the new number in its title. The search assistants find and cite it within a day.
- Update your own pages on the same day. Old blog posts and docs FAQ entries can keep the old number within search reach.
- Find the comparison pages the assistants are citing for your category questions. Ask the authors to update your row.
- Test the passing mention. Ask the questions a buyer would ask, every day, and count the old vs new number. A direct question won't reveal any problems.
- Think in months. Eight months after Clerk's change, 23% of the answers with a number still had the old one.
The daily runs can only tell us ChatGPT took about a day or less. The next test we want to run is a canary: a new fact on our own pages, published at a known minute, with the same question asked every hour, to see when each assistant says it. For now, the daily count is the measure.
See what number each assistant says for your product, question by question.