Pan Innovation House Pan Innovation House
+90 543 930 12 72
Discovery call →
AI & GEO

What is AI Crawler (GPTBot, ClaudeBot, PerplexityBot)?

The bots that AI companies such as OpenAI, Anthropic and Perplexity use to crawl web content.

Detailed definition

For GEO to work, these bots must be allowed in robots.txt. The bot list: GPTBot (OpenAI), ClaudeBot + anthropic-ai (Anthropic), PerplexityBot + Perplexity-User (Perplexity), Google-Extended (Google AI Overviews), Applebot-Extended (Apple Intelligence), CCBot (Common Crawl), cohere-ai (Cohere), meta-externalagent (Meta AI), OAI-SearchBot + ChatGPT-User (OpenAI search). Disallowing all of these bots means you will not appear in AI engines.

Examples

  • User-agent: GPTBot\nAllow: / → allows ChatGPT to crawl your brand.

Related terms

GEO (Generative Engine Optimization) Next-generation optimisation that makes brands visible in the answers given by AI engines such as ChatGPT, Perplexity, Claude and Gemini. llms.txt A standard file published at /llms.txt that briefs AI engines on the brand's structure in Markdown format. E-E-A-T (Experience, Expertise, Authoritativeness, Trustworthiness) The four criteria Google uses to assess content quality: experience, expertise, authoritativeness and trust. Core Web Vitals The three main metrics Google uses to measure site performance: LCP, INP and CLS. LCP (Largest Contentful Paint) How long the page's main content block (usually the hero image or heading) takes to load. Good: <2.5s, poor: >4s.
From the same category

Other terms in the AI & GEO category

GEO (Generative Engine Optimization) Next-generation optimisation that makes brands visible in the answers given by AI engines such as ChatGPT, Perplexity, Claude and Gemini. llms.txt A standard file published at /llms.txt that briefs AI engines on the brand's structure in Markdown format.
See the full glossary →
Call Free strategy call →