---
title: "Crawlers and AI bots – what they do on a site | Sweet Lava"
description: "A crawler fetches your pages automatically; GPTBot, ClaudeBot and PerplexityBot now gather content for AI models. See who visits and whether to allow them."
url: https://sweetlava.eu/en/glossary/ai-crawlers/
lang: en
alternate_pl: https://sweetlava.eu/pl/slownik/crawler-boty-ai/
---

SEO & AI 

# What is Crawlers & AI bots?

A crawler is a program that automatically visits websites and fetches their content. Alongside classic search-engine bots (Googlebot), AI bots now operate – GPTBot, ClaudeBot, PerplexityBot – gathering content for language models.

AI bots play two roles: **training** (content feeds model training) and **search** (the model checks sources live to cite them in an answer – e.g. OAI-SearchBot, Perplexity-User). For business visibility the second is crucial: a blocked search bot means absence from AI answers.

Control happens in robots.txt – bots can be allowed or blocked selectively, by agent name. Beware of surprises: anti-bot firewalls (e.g. aggressive CDN settings) can block AI bots without the site owner's knowledge.

**Our practice:** we deliberately allow AI search bots and monitor their visits in server logs – because presence in ChatGPT or Perplexity answers is a new customer-acquisition channel.

## Related services

- [ Custom website design & development → ](https://sweetlava.eu/en/services/websites/)

## Related terms

- [llms.txt](https://sweetlava.eu/en/glossary/llms-txt/)
- [SEO](https://sweetlava.eu/en/glossary/seo/)

## Planning a new company website?

Design and code grow in one team, so what you approve in the mock-up reaches production with no surprises. Scope and quote within a few days.

[ Get in touch ](https://sweetlava.eu/en/contact/?source=glossary-ai-crawler)