You name crawlers with User-agent and list allowed and blocked paths with Allow and Disallow. Crawlers from AI companies, such as OpenAI's GPTBot and OAI-SearchBot, Google's Google-Extended and Anthropic's ClaudeBot, also check this file.
If you block AI crawlers by mistake, your pages are unlikely to be used as evidence in AI answers. It is one of the first things we check in a website audit.
