Merchant Evidence Brief · Issue 1
Your robots.txt now talks to several different AI systems. Here's what each line actually controls.
Sources checked 22 November 2026 (Korea Standard Time, UTC+9). Published here as a web archive.
Many store owners have added a few “block AI” lines to robots.txt, or been told to add an llms.txt file “for AI SEO.” Both moves can have effects you didn’t intend. Platform documentation separates search crawlers, training crawlers, and user-triggered fetchers, and they don’t all obey the same switch. This issue walks through what three primary sources say, and what they don’t.
1. OpenAI: three agents, three different jobs
OpenAI’s crawler page lists separate user agents and says “each setting is independent of the others” (OpenAI):
- OAI-SearchBot is for search. It is “used to surface websites in search results in ChatGPT’s search features.” Sites that opt out of it “will not be shown in ChatGPT search answers, though can still appear as navigational links” (OpenAI).
- GPTBot crawls content “that may be used in training our generative AI foundation models.” Disallowing it “indicates a site’s content should not be used in training” (OpenAI).
- ChatGPT-User is used when a ChatGPT user’s request causes a page visit. OpenAI says it “is not used for crawling the web in an automatic fashion” and, “because these actions are initiated by a user, robots.txt rules may not apply.” It “is not used to determine whether content may appear in Search” (OpenAI).
OpenAI also notes that for search it “can take ~24 hours from a site’s robots.txt update for our systems to adjust,” and publishes IP lists for each agent (OpenAI).
What this means for a store: “Block GPTBot, allow OAI-SearchBot” is a coherent setting according to OpenAI’s own description. It signals no training use while staying eligible for ChatGPT search results. Blocking OAI-SearchBot is the line that removes you from ChatGPT search answers.
What it does not say: nothing on that page promises that allowing OAI-SearchBot gets you shown, or describes how results are ranked.
2. Perplexity: a search crawler and a user fetcher
Perplexity’s documentation lists two agents and says each setting “works independently,” with up to 24 hours for changes to take effect (Perplexity):
- PerplexityBot “is designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models” (Perplexity).
- Perplexity-User supports user actions. When a user asks a question, it “might visit a web page to help provide an accurate answer.” The docs state: “Since a user requested the fetch, this fetcher generally ignores robots.txt rules” (Perplexity).
The same page has a section most store owners skip: WAF configuration. If a firewall or CDN bot rule blocks unknown bots, robots.txt permission doesn’t help. Perplexity recommends combining User-Agent matching with IP verification against its published JSON IP lists, and warns that Cloudflare’s Bot Fight Mode “can’t be skipped with WAF custom rules” (Perplexity).
What this means: if you want to appear in Perplexity’s search results, check your CDN/WAF as well as robots.txt. If you want to restrict it, know that robots.txt is documented as largely ineffective for the user fetcher.
3. Google: no separate “AI” switch for Search
Google’s guidance on AI Overviews and AI Mode is short and direct (Google Search Central):
- “There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary.”
- To be eligible as a supporting link, “a page must be indexed and eligible to be shown in Google Search with a snippet.”
- “You don’t need to create new machine readable files, AI text files, or markup to appear in these features. There’s also no special schema.org structured data that you need to add.”
- Controls: robots.txt for Googlebot manages crawling for Search, and
nosnippet,data-nosnippet,max-snippet, ornoindexlimit what’s shown. Google points to Google-Extended for limiting AI training and grounding “in some of Google’s other systems” (Google Search Central). - Google also lists “making sure your structured data matches the visible text on the page” and keeping Merchant Center information up to date among worthwhile fundamentals (Google Search Central).
What this means: blocking Googlebot to keep content out of AI Overviews would also remove you from regular Search. Snippet controls are the documented lever.
4. And llms.txt?
llms.txt is a proposal, now in its second version, for a /llms.txt Markdown file “to provide information to help agents use a website” (llmstxt.org). The proposal’s author reports wide adoption on documentation sites (llmstxt.org). OpenAI’s and Perplexity’s docs pages both point readers to their own llms.txt indexes (OpenAI; Perplexity). But Google states that no AI text files are needed for its AI features (Google Search Central), and none of the three crawler pages above says an llms.txt affects search inclusion or ranking.
What this means: an llms.txt may be a harmless convenience for tools that read it. Treat it as optional. It isn’t a ranking lever, and it doesn’t replace robots.txt or snippet controls.
Ten-minute check for your store
- Open
https://yourstore.example/robots.txt. List every AI-relatedUser-agentline, and for each, write down whether it’s search, training, or user-triggered, using the definitions above. - Confirm you haven’t blocked a search agent (OAI-SearchBot, PerplexityBot, Googlebot) by accident while meaning to block training.
- Check your CDN/WAF bot settings. A “block AI bots” toggle may act regardless of robots.txt.
- If you use
nosnippet, check it’s intentional on product pages. - Note the date. OpenAI and Perplexity both say robots.txt changes can take about 24 hours to be reflected (OpenAI; Perplexity).
Quick reference
| Agent | Purpose (per the vendor’s docs) | Honors robots.txt? |
|---|---|---|
| OAI-SearchBot | ChatGPT search results | Yes (it’s the documented opt-out) |
| GPTBot | Foundation-model training | Yes |
| ChatGPT-User | User-initiated visits | “may not apply” |
| PerplexityBot | Perplexity search results; not training | Yes |
| Perplexity-User | User-initiated visits | “generally ignores” |
| Googlebot | Google Search, including AI features | Yes; use snippet controls to limit display |
Next issue: what “machine-readable product data” means in two concrete specs, Google merchant-listing structured data and OpenAI’s product feed.
Merchant Evidence Brief is independent and has no affiliation with OpenAI, Perplexity, or Google. This issue has no sponsor.
Sources - OpenAI, “Overview of OpenAI Crawlers”: https://platform.openai.com/docs/bots - Perplexity, “Perplexity Crawlers”: https://docs.perplexity.ai/guides/bots - Google Search Central, “AI features and your website” (last updated 2025-12-10 UTC): https://developers.google.com/search/docs/appearance/ai-features - llms.txt proposal: https://llmstxt.org/