Home · Newsletter · Issue 1

Merchant Evidence Brief · Issue 1

Your robots.txt now talks to several different AI systems. Here's what each line actually controls.

Sources checked 22 November 2026 (Korea Standard Time, UTC+9). Published here as a web archive.

Many store owners have added a few “block AI” lines to robots.txt, or been told to add an llms.txt file “for AI SEO.” Both moves can have effects you didn’t intend. Platform documentation separates search crawlers, training crawlers, and user-triggered fetchers, and they don’t all obey the same switch. This issue walks through what three primary sources say, and what they don’t.


1. OpenAI: three agents, three different jobs

OpenAI’s crawler page lists separate user agents and says “each setting is independent of the others” (OpenAI):

OpenAI also notes that for search it “can take ~24 hours from a site’s robots.txt update for our systems to adjust,” and publishes IP lists for each agent (OpenAI).

What this means for a store: “Block GPTBot, allow OAI-SearchBot” is a coherent setting according to OpenAI’s own description. It signals no training use while staying eligible for ChatGPT search results. Blocking OAI-SearchBot is the line that removes you from ChatGPT search answers.

What it does not say: nothing on that page promises that allowing OAI-SearchBot gets you shown, or describes how results are ranked.


2. Perplexity: a search crawler and a user fetcher

Perplexity’s documentation lists two agents and says each setting “works independently,” with up to 24 hours for changes to take effect (Perplexity):

The same page has a section most store owners skip: WAF configuration. If a firewall or CDN bot rule blocks unknown bots, robots.txt permission doesn’t help. Perplexity recommends combining User-Agent matching with IP verification against its published JSON IP lists, and warns that Cloudflare’s Bot Fight Mode “can’t be skipped with WAF custom rules” (Perplexity).

What this means: if you want to appear in Perplexity’s search results, check your CDN/WAF as well as robots.txt. If you want to restrict it, know that robots.txt is documented as largely ineffective for the user fetcher.


Google’s guidance on AI Overviews and AI Mode is short and direct (Google Search Central):

What this means: blocking Googlebot to keep content out of AI Overviews would also remove you from regular Search. Snippet controls are the documented lever.


4. And llms.txt?

llms.txt is a proposal, now in its second version, for a /llms.txt Markdown file “to provide information to help agents use a website” (llmstxt.org). The proposal’s author reports wide adoption on documentation sites (llmstxt.org). OpenAI’s and Perplexity’s docs pages both point readers to their own llms.txt indexes (OpenAI; Perplexity). But Google states that no AI text files are needed for its AI features (Google Search Central), and none of the three crawler pages above says an llms.txt affects search inclusion or ranking.

What this means: an llms.txt may be a harmless convenience for tools that read it. Treat it as optional. It isn’t a ranking lever, and it doesn’t replace robots.txt or snippet controls.


Ten-minute check for your store

  1. Open https://yourstore.example/robots.txt. List every AI-related User-agent line, and for each, write down whether it’s search, training, or user-triggered, using the definitions above.
  2. Confirm you haven’t blocked a search agent (OAI-SearchBot, PerplexityBot, Googlebot) by accident while meaning to block training.
  3. Check your CDN/WAF bot settings. A “block AI bots” toggle may act regardless of robots.txt.
  4. If you use nosnippet, check it’s intentional on product pages.
  5. Note the date. OpenAI and Perplexity both say robots.txt changes can take about 24 hours to be reflected (OpenAI; Perplexity).

Quick reference

Agent Purpose (per the vendor’s docs) Honors robots.txt?
OAI-SearchBot ChatGPT search results Yes (it’s the documented opt-out)
GPTBot Foundation-model training Yes
ChatGPT-User User-initiated visits “may not apply”
PerplexityBot Perplexity search results; not training Yes
Perplexity-User User-initiated visits “generally ignores”
Googlebot Google Search, including AI features Yes; use snippet controls to limit display

Next issue: what “machine-readable product data” means in two concrete specs, Google merchant-listing structured data and OpenAI’s product feed.


Merchant Evidence Brief is independent and has no affiliation with OpenAI, Perplexity, or Google. This issue has no sponsor.

Sources - OpenAI, “Overview of OpenAI Crawlers”: https://platform.openai.com/docs/bots - Perplexity, “Perplexity Crawlers”: https://docs.perplexity.ai/guides/bots - Google Search Central, “AI features and your website” (last updated 2025-12-10 UTC): https://developers.google.com/search/docs/appearance/ai-features - llms.txt proposal: https://llmstxt.org/