What is Crawler?

A crawler is a program that automatically visits web pages and collects their content. It is also called a bot or spider.

Crawler, Bot · Marketing Glossary · Updated

Google's Googlebot, Naver's Yeti and Microsoft's Bingbot are search crawlers, and there are also crawlers used for AI training and search, such as OpenAI's GPTBot. Pages that crawlers can't read are unlikely to be used in search results or AI answers.

Some crawlers can't properly read pages whose content only appears after JavaScript runs, so it is safer to put key content directly in the HTML.

Related terms

Now,
it's time we MAKE you.

Tell us your industry and area,
and the channels where you want to be seen.
We'll start by diagnosing where you're visible today.