SEO and Large Language Models: What the Documentation Says

SEO and large language models is the question of how your site’s content is used by AI assistants and search features built on language models. A common name is “AEO” (answer engine optimization): that is an industry term, not the providers’. The main fact from their own documentation: each provider has several different robots — for training models, for search and for user-initiated requests — and they can be managed separately. How exactly a model picks sources for an answer, the providers do not publish, so there is no honest “citation recipe”. Below: what is documented, how to manage it and which popular claims (“Wikipedia links = training data”, “5,000 mentions versus 200”) are not supported.

Different robots for different tasks

The settings are independent: OpenAI, for example, writes that you can allow OAI-SearchBot to appear in search results and disallow GPTBot so content is not used for training. A robots.txt change takes about a day to apply at OpenAI and Perplexity. Anthropic supports the non-standard Crawl-delay directive and honors robots.txt prohibitions.

How to manage access

1. Decide by purpose. If you want to appear in assistants’ answers, allow the providers’ search robots (OAI-SearchBot, PerplexityBot, Claude-SearchBot). If you don’t want content used for training, disallow the training ones (GPTBot, ClaudeBot) and Google-Extended.
2. Remember the “on a user’s request” robots. ChatGPT-User and Perplexity-User act on a person’s request; robots.txt rules may not apply to them, and blocking the regular robots will not stop them.
3. Check your infrastructure. robots.txt files must be placed on every subdomain; a CDN, WAF and bot protection may block robots regardless of robots.txt — Perplexity directly advises allowlisting its robots by user agent and published IP ranges (robots.txt).
4. Check the logs. Look in server logs for requests with the named user agents and reconcile IPs with the published lists: OpenAI and Perplexity publish address ranges. Crawls by your own crawler check page availability as a robot without scripts sees it.

What remains a hypothesis

• “Models find content through training data and RAG”. That is a simplification: providers have separate robots for search, training and user requests. How a model picks sources for an answer is published nowhere.
• “Wikipedia and reference works enter training data, so you must get there”. Not stated in the documentation; and mentions in publications are part of ordinary reputation work (link acquisition tactics) that must not be turned into a scheme.
• “5,000 mentions versus 200 is the reason for absence from an answer”. An individual story; correlation does not prove cause (SEO experiments).
• “There are almost no visits except from Perplexity”. There is no documentation on shares; OpenAI writes that sites are shown in ChatGPT search results, and Perplexity writes that PerplexityBot surfaces and links websites.

What carries over from SEO for Google

Google’s recommendations for AI features say: there is no special optimization; what is needed is indexing, eligibility to be shown with a snippet, helpful and reliable people-first content, important content in text form, structured data that matches the visible text, and no special markup or files “for AI” (AI features). In practice the same fundamentals help in other systems too: clear answers to specific questions, verifiable facts with sources and dates, clear author and editorial information (E-E-A-T, helpful content). But this is practice: we cite no provider other than Google as a source of answer-selection rules here.

Content created with AI

If you yourself use generative models, Google asks you to fact-check manually and add value: mass-creating pages with no value for readers may violate the scaled content abuse policy (the generative AI guide, spam policies).

Checklist

1. The access decision is made separately for search, training and user-request robots.
2. robots.txt is checked on every subdomain; the WAF and CDN do not block needed robots.
3. Server logs show which robots come; IPs are reconciled with the published ranges.
4. Important content is available as text; facts are verifiable, dates and sources are given.
5. There are no “files for AI” or markup “for AI” in expectation of Google: it is not required.
6. Hypotheses about citation are tested on your own data, not taken on trust.

Mistakes

• Blocking a provider’s search robots believing you are blocking only training.
• Assuming a robots.txt prohibition will stop user-initiated requests.
• Treating “AEO” as a set of documented rules.
• Mass-placing mentions for the sake of assistants instead of real value.
• Forgetting about robot blocking at the CDN and WAF level.

What next

Next comes the future of SEO: which changes in search are already confirmed by documentation, which remain forecasts and how to build professional skills without tying yourself to one system.