Detailed definition
AI crawlers are the bots AI companies use to fetch web content for model training, for a search index or on a user's request, and their roles are separate. At OpenAI, GPTBot is for model training, OAI-SearchBot for ChatGPT search results and ChatGPT-User for visits a user triggers; blocking GPTBot does not affect appearing in ChatGPT search. At Anthropic, ClaudeBot is for training, Claude-SearchBot for search quality and Claude-User for access on a user's request. At Perplexity, PerplexityBot feeds its search index and Perplexity-User handles user requests. Google-Extended is not a separate crawler but a robots.txt token that controls whether content may be used to train Gemini models and for grounding in the Gemini apps and Vertex AI; it does not affect inclusion or ranking in Google Search, or visibility in AI Overviews and AI Mode (those are managed through Googlebot and snippet controls). User-triggered bots may not always follow robots.txt rules. That is why bots are assessed one by one according to their purpose, not allowed or blocked wholesale.
Examples
- The lines "User-agent: GPTBot" and "Disallow: /" block OpenAI's crawling for model training; OAI-SearchBot, which affects ChatGPT search results, is managed in a separate block.
- A Disallow rule for Google-Extended limits the use of content for Gemini training and grounding; it does not change how the page appears in Google Search or AI Overviews.
Related terms
Would you like support with this?
If you would like to discuss how AI Crawler (GPTBot, ClaudeBot, PerplexityBot) applies to your business, we can arrange a short initial call. The first 30-minute call is free.
Book a free initial call → +90 543 930 12 72