A crawler is a program that automatically visits websites and fetches their content. Alongside classic search-engine bots (Googlebot), AI bots now operate – GPTBot, ClaudeBot, PerplexityBot – gathering content for language models.
AI bots play two roles: training (content feeds model training) and search (the model checks sources live to cite them in an answer – e.g. OAI-SearchBot, Perplexity-User). For business visibility the second is crucial: a blocked search bot means absence from AI answers.
Control happens in robots.txt – bots can be allowed or blocked selectively, by agent name. Beware of surprises: anti-bot firewalls (e.g. aggressive CDN settings) can block AI bots without the site owner's knowledge.
Our practice: we deliberately allow AI search bots and monitor their visits in server logs – because presence in ChatGPT or Perplexity answers is a new customer-acquisition channel.