Glossary / SEO & AI Search
AI crawler.
A bot that reads websites to train or power AI systems and answers.
What it means
A bot that reads websites to train or power AI systems and answers. Examples include GPTBot, ClaudeBot and PerplexityBot. Whether to allow them is controlled in robots.txt; blocking them can keep your content out of AI answers.
Related terms
More in SEO & AI Search
Questions aboutai crawler?
Ask the team that does this work every day. We reply within one business day.

