Submit a Story

Glossary

Crawling

The process where a search engine or other bot follows links and fetches web pages so it can discover and read them.

Crawling is how a bot finds pages. It starts from known URLs, downloads each page, follows the links it finds and adds new URLs to a list of pages to visit. Search engines, AI crawler bots and other services all crawl.

Crawling is not the same as indexing. A page can be crawled and still not be added to the index, for example if it is thin, duplicated or marked noindex. And a page that cannot be crawled, because it is blocked in robots.txt or has no links pointing to it, will usually not be found at all.

You help crawling along with clear internal linking, a current XML sitemap, fast responses and a clean URL structure. Crawl budget matters mainly for very large sites; for most, the goal is simply to make sure nothing important is blocked or hidden.

Related: Technical SEO, Canonical URL, Search Engine Optimization.

← All terms