Crawl Budget

Crawl budget refers to the number of pages Googlebot can and wants to crawl on your website within a given time period. It’s determined by two factors: crawl rate limit (how many pages Googlebot can crawl without overloading your server) and crawl demand (how often Googlebot deems your content worth revisiting based on popularity and update frequency). For large sites with thousands of pages, optimizing crawl budget is critical—ensure important pages are easily discoverable via internal links and XML sitemaps, while low-value or duplicate pages are blocked via robots.txt or canonical tags. Poor crawl budget allocation can delay indexing of new or updated content.