Where Does ChatGPT Get Its Information? Training Data, Live Retrieval, and the Crawlers In Between
“Where does ChatGPT get its information?” has three answers, and most confusion about AI visibility comes from not knowing which one applies to a given response.
The model can answer from memory. It can search and synthesize. It can fetch one specific page. These produce different accuracy, different freshness, and different citations — and only some of them are influenced by anything you do to your website.
The three sources
1. Pretraining — what the model absorbed. A large corpus of text the model learned from during training, frozen at a cutoff date. The model does not look anything up here; it recalls from its parameters, compressed and lossy. No citations, because there is no specific document being consulted — just a statistical impression of everything it read.
2. Live retrieval — what search returns now. When search is active, the system turns your question into queries, runs them against a search index, retrieves results, and synthesizes an answer from what comes back. This is where citations come from. It reflects the live web, not the cutoff.
3. On-demand fetching — one specific page. Paste a URL, or ask about a page the assistant decides it needs, and it fetches that document at that moment.
| Pretraining | Live retrieval | On-demand fetch | |
|---|---|---|---|
| Freshness | Stale at cutoff | Current | Current |
| Citations | None | Yes | The URL given |
| Influenced by your site today | No | Yes | Yes |
| What it’s good at | General knowledge | Current, specific questions | Reading one document |
| Failure mode | Confidently outdated | Cites the wrong sources | Fetch blocked or unreadable |
The practical tell is simple: an answer with cited links came from retrieval; an answer with none came from parameters. That second kind can be fluent, confident and two years out of date, which is precisely why it is dangerous when it is about your company.
Why this matters more than it sounds
Almost every “ChatGPT says something wrong about us” problem resolves to a source question.
If the answer came from pretraining, your recent website changes are irrelevant — the model learned what it learned, and nothing you publish enters a model that already exists. The remedy is indirect and slow: build enough current, consistent, third-party presence that future retrieval overrides the stale impression.
If it came from retrieval, the answer is fixable and reasonably fast, because retrieval reads the live web. The diagnostic is the citation list. Whatever it cited is what it believed. That list is your target list.
This is why “how do I get into ChatGPT’s training data?” is close to the wrong question. Training is batched, slow, compressed and outside your control. Retrieval is live, attributable and responsive. Retrieval is the surface worth optimizing.
The crawlers, and why there are several
OpenAI runs distinct crawlers for distinct jobs. They can be allowed or blocked independently in robots.txt, and conflating them is a common and costly mistake.
| Crawler | Job | Blocking it means |
|---|---|---|
| GPTBot | Collects content that may train future models | You opt out of training. You remain citable. |
| OAI-SearchBot | Builds the search index behind ChatGPT search | You lose citation eligibility — the visibility path |
| ChatGPT-User | Fetches a page when a user or the assistant asks | Pasted links from your site won’t load |
The configuration that trips people up: blocking everything OpenAI-shaped in a burst of caution, then wondering why the brand never appears in ChatGPT answers. Blocking GPTBot is a rights and competitive decision with no visibility cost. Blocking OAI-SearchBot is a visibility decision, and it removes you from the surface that actually generates mentions and referrals.
Other engines follow the same pattern with their own agents — Anthropic, Perplexity and Google each run separate crawlers for training and for search. Audit yours deliberately rather than by reflex.
Can AI crawlers execute JavaScript?
Generally not — and this is the most consequential difference from Googlebot.
Googlebot runs a full Chrome rendering engine. It will execute your JavaScript, wait for the DOM to settle, and index what results. Most AI crawlers do not. They fetch what the server returns and parse that HTML directly.
Everything that appears only after client-side rendering is therefore at risk of not existing:
- Prices and availability injected after load
- Specification tables rendered by a framework
- Reviews loaded from a separate call
- Anything behind a tab, accordion or “load more”
- Entire pages on a client-rendered SPA with a thin HTML shell
The failure is silent. Nothing errors. Your page looks correct in every browser you test. The crawler simply stores a shell, and the model answers about your product using a competitor’s listing that happened to be readable.
The test takes thirty seconds: disable JavaScript in your browser and load the page. What remains is approximately what an AI crawler sees. If your key facts vanish, they are invisible to the systems answering questions about you.
We hit a version of this on this very site — a CSS reveal animation left long articles at opacity: 0 when an observer never fired, so pages rendered as a nav bar and nothing else. Every check said 200 OK. The content was there in the HTML. It just wasn’t reachable, and that class of bug is far more common than outright errors.
Do AI crawlers support content negotiation for markdown?
Short answer: not in any documented way you should build on.
The idea circulates because it is elegant — use the Accept header to serve text/html to browsers and text/markdown to AI crawlers, giving models clean text without navigation, ads and layout noise. Some tools and proxies experiment with it.
But there is no established standard that major AI crawlers honour, and no commitment from the main providers to respect a markdown Accept header. Building your content strategy on undocumented header behaviour means it breaks silently whenever a crawler changes, and you will not be told.
What actually gained traction instead is llms.txt — a markdown file at a known path listing your key content with links. Adoption is uneven and no major engine has committed to it as a retrieval input, so treat it as cheap hygiene rather than a lever.
The reliable baseline has not changed and is unglamorous: clean, semantic, server-rendered HTML. One <h1>, real headings, facts as text rather than images, content present in the initial response. That works for every crawler, needs no negotiation, and will not break when someone ships a new user agent.
What to actually do
Six things, in the order they pay off:
- Audit your robots.txt against the table above. Decide training and retrieval separately. If you want AI visibility, OAI-SearchBot and its equivalents need to be allowed.
- Disable JavaScript and read your key pages. Anything that disappears is invisible to retrieval. Server-render it.
- Put facts in text. Specifications, prices, comparisons, key claims. Images of tables are not readable content.
- Structure for extraction. Clear headings that match real questions, direct answers immediately below them, no burying the claim in paragraph four.
- Check the citations, not just the answer. When an engine gets you wrong, the cited sources tell you whether the problem is stale parametric memory or the wrong third-party pages ranking. Different problems, different fixes.
- Publish llms.txt if it is cheap for you. Low cost, uncertain benefit, no downside.
Then measure. Whether engines cite you, and which sources they cite instead, is the feedback loop for all of the above — the method is in how to measure AI visibility, and Sanbi.ai automates the citation side across ChatGPT, Perplexity, Gemini and Claude so you can see which sources are supplying the answers in your category rather than inferring it from spot checks.
The bottom line
ChatGPT answers from memory, from live search, or from a page it fetched — and only the last two are things you can influence this quarter.
Stop trying to get into the training data. Make yourself readable and citable to the retrieval path: allow the search crawlers, serve real HTML that survives having JavaScript switched off, put your facts in text, and earn presence in the sources retrieval trusts.
The most common cause of “the AI doesn’t know about us” is not exotic. It is content the crawler could never read in the first place.
Related reading: llms.txt complete guide · Query fan-out explained · AI engine citation trends · Answer Engine Optimization guide
Frequently Asked Questions
Where does ChatGPT get its information?
From three different places, depending on the question. Pretraining — a large corpus of text the model learned from, frozen at a cutoff date, which the model recalls from parameters rather than looking up. Live retrieval — when search is active, the system runs queries against a search index and synthesizes an answer from the results it retrieves. On-demand fetching — when you paste a link or the model decides it needs a specific page, it fetches that URL at that moment. These behave very differently: pretraining has no citations and a stale cutoff, retrieval cites sources and reflects the live web.
Does ChatGPT search the internet in real time?
Sometimes, and only when search is invoked. If the model answers from pretraining alone, nothing is fetched and the answer reflects the world as of the training cutoff. When search is used, the system issues queries, retrieves current results and synthesizes from them — that is when you see citation links. The practical tell is the presence of sources: an answer with cited links came from retrieval, an answer with none came from the model's parameters and may be confidently out of date.
What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?
They are separate crawlers with separate jobs, and you can allow or block them independently in robots.txt. GPTBot collects content that may be used to train future models. OAI-SearchBot builds and maintains the search index that powers ChatGPT's search results — this is the one that affects whether you can be cited in answers. ChatGPT-User fetches a specific page when a user or the assistant requests it during a conversation. Blocking GPTBot to opt out of training while allowing OAI-SearchBot is a common and deliberate configuration.
Can AI crawlers execute JavaScript?
Generally no, and this is the single most consequential difference from Googlebot. Most AI crawlers fetch the HTML a server returns and parse that, without running a browser engine or waiting for client-side rendering. Content injected by JavaScript after load — prices, specifications, reviews, tabbed content, anything behind an interaction — frequently does not exist as far as the crawler is concerned. The failure is silent: the page looks perfect in your browser and arrives at the model as a shell. Test by disabling JavaScript and reading what remains.
Do AI crawlers support content negotiation for markdown?
Not in any reliable, documented way you should build on. The idea is appealing — serve text/html to browsers and text/markdown to AI crawlers via the Accept header — but there is no established standard that major AI crawlers honour, and building your content strategy on undocumented header behaviour is fragile. The convention that has actually gained traction is llms.txt: a plain markdown file at a known path that points to clean versions of your content. Clean, server-rendered, semantic HTML remains the reliable baseline.
Why does ChatGPT get facts about my company wrong?
Usually because it is answering from pretraining rather than retrieval. Parametric memory has a cutoff, compresses what it learned, and has no mechanism to notice that you rebranded, changed pricing or launched a product last quarter. Two other causes are common: the model is synthesizing from outdated third-party sources that still rank, or your own site presents key facts in a way crawlers cannot read — JavaScript-rendered content, or facts that exist only in images. The fix differs by cause, so diagnose which one before acting.
How do I get ChatGPT to cite my website?
Citations come from the retrieval path, so the target is the search index rather than the model's training. That means: allow OAI-SearchBot in robots.txt, serve content that is readable without JavaScript, structure pages so specific claims are extractable, and build presence in the third-party sources that retrieval favours for judgement-based questions. Being in the training corpus is neither necessary nor sufficient for citation — retrieval decides what gets linked, and retrieval reads the live web.
Should I block GPTBot?
It depends on what you are optimizing for, and it is a genuine trade-off rather than an obvious call. Blocking GPTBot opts your content out of future model training, which some publishers want for rights or competitive reasons. It does not remove you from ChatGPT's answers, because citations come through OAI-SearchBot and the search index. Blocking both removes you from the retrieval path too, which is the one that actually generates visibility and referral traffic. Most brands seeking AI visibility should allow OAI-SearchBot at minimum.
Does being in ChatGPT's training data help my brand?
Less than people assume, and it is largely outside your control. Training happens in batches with long lead times, the model compresses what it learned, and nothing you publish today enters an existing model. Retrieval, by contrast, reads the current web on every search-enabled query, cites what it uses, and responds to changes within crawl cycles. For brand visibility work, retrieval is the surface worth optimizing because it is live, attributable and measurable.
What is llms.txt and does ChatGPT use it?
llms.txt is a proposed convention: a markdown file at your domain root listing your key content with links, so AI systems can find clean versions of your material without parsing navigation and layout. Adoption is uneven and no major engine has committed to it as a ranking or retrieval input, so treat it as low-cost hygiene rather than a lever. It costs little to publish and may help; it will not compensate for content that crawlers cannot read in the first place.
How is ChatGPT's retrieval different from Google's index?
Google crawls broadly, renders JavaScript, and returns a ranked list of links for the user to choose from. ChatGPT's search retrieves a smaller set of results and synthesizes them into a single answer, naming a handful of sources. The consequences differ sharply: in Google, position ten still gets some traffic; in a synthesized answer, sources that are not cited get nothing at all. There is no long tail of partial visibility — you are in the answer or you are absent from it.