An AI crawler is a bot that requests your pages on behalf of an AI product. They do different jobs, and the important point is that one company often runs several. For example, OpenAI uses GPTBot for training, OAI-SearchBot for its search index and ChatGPT-User for fetches triggered by a user. Anthropic similarly separates ClaudeBot, Claude-SearchBot and Claude-User, and Google uses the Google-Extended token to control use of content for its Gemini models.
You control the well-behaved ones in robots.txt, with a separate rule per name. Blocking a training bot does not block the search bot, and blocking a search bot can reduce how often you are cited (AI citation). Bots that act on a user's request may not follow robots.txt, and robots.txt is a request, not access control, so use firewall rules if you truly need to block something.
Bot names and behavior change, so check each operator's current documentation before editing your rules.
Related: AI Search, Crawling, AI-Ready Website.