The crawl budget refers to the number of URLs that will be crawled in a single session by search engines. The crawl rate is different for different websites and is determined by what Google robots have assessed to be the optimal number of URLs to crawl. The crawl rate is influenced by a website’s traffic and the need to keep search engine results pages fresh.
How can crawl budgets be optimised?
SEO experts use a variety of techniques to optimise the crawl budget for a given website. They include:
- Noindexing unimportant pages so that crawling only takes place on important ones
- Reducing redirections, as every redirect uses up the crawl budget for a website
- Using hreflang tags, which identify when pages are the same but in different languages
- Fixing 404 or 410 errors, which also use up a website’s crawl budget
