Crawl Budget
Crawl budget is the number of URLs Googlebot can and wants to crawl on a site within a given timeframe. It combines the crawl capacity limit, set by server health, with crawl demand, set by the site's popularity and staleness.
Google's documentation defines crawl budget through two components. The crawl capacity limit is the maximum parallel fetching Googlebot applies without degrading the server — it rises when the site responds quickly with clean 2xx responses and falls when responses slow down or return 5xx errors. Crawl demand is how much crawling Google wants to do: driven by the size of the known URL inventory, how popular individual URLs are, and how stale Google's stored copies have become. Together they set the URLs Googlebot can and wants to crawl.
Crawl budget is a large-site concern. Google's guidance is that it mainly affects sites with 1 million+ unique pages whose content changes weekly, and sites with 10,000+ pages whose content changes daily. A small site with unindexed pages almost always has a discovery, quality, or directive problem — not a budget problem. The signature of genuine budget pressure on large sites is a growing backlog of URLs in Discovered – currently not crawled.
Budget is fungible: every fetch spent on a worthless URL is a fetch not spent on a page that matters. That is why crawl-budget work is mostly subtraction — eliminating the URL spaces that soak up fetches without adding indexable content.
Quick facts
What are the two components of crawl budget?
Crawl capacity limit and crawl demand — Googlebot crawls up to demand, within capacity.
- Crawl capacity limit — the host-load ceiling. Googlebot continuously adjusts parallel connections and delay between fetches based on server responsiveness; fast, error-free responses raise the ceiling, while slow responses and 5xx errors lower it.
- Crawl demand — the appetite. URLs that are popular and heavily linked get recrawled to stay fresh, stale copies get refetched, and site-wide events like migrations spike demand because the whole inventory must be refetched.
Capacity without demand goes unused: on most small sites Googlebot could crawl far more than it does but sees no reason to.
What wastes crawl budget?
Low-value URL spaces waste budget because Googlebot fetches them at the expense of real content. Google's documentation names the main offenders:
- Faceted navigation and session parameters that generate near-infinite URL permutations
- Duplicate content left unconsolidated under a canonical URL
- Soft 404s — thin error pages returning 200 instead of a real 404
- Redirect chains, where every hop consumes a separate fetch
- Infinite spaces such as auto-generated calendar pages and endlessly incrementing archives
The fixes are blunt: block crawl-trap paths in robots.txt, return real 404/410 codes for dead URLs, collapse redirect chains to a single hop, and consolidate duplicates.
How do you monitor crawl budget?
The Crawl Stats report (Search Console → Settings → Crawl Stats) is the primary surface: total crawl requests per day, average response time, host status, and breakdowns by response code, file type, purpose, and Googlebot type. A rising share of requests hitting parameter URLs, redirects, or 404s is budget leaking away from real content.
Server access logs supply the per-URL detail Crawl Stats aggregates away — exactly which URLs Googlebot fetches and how often. Comparing logged fetches against your XML sitemap inventory shows which sections Googlebot neglects.
Frequently Asked Questions
Does crawl budget matter for small sites?+
Rarely. Google's guidance targets sites with over 1 million pages changing weekly, or over 10,000 pages changing daily. On a few hundred pages, unindexed URLs trace to discovery, quality, or directives — fixing those beats any budget optimization.
Does robots.txt disallow save crawl budget?+
Yes. Disallowed URLs are not fetched at all, freeing capacity for the rest of the site. A noindex tag does not save budget — Googlebot must crawl the page to see the tag.
Is crawl budget a ranking factor?+
No. Crawl budget governs how quickly Google discovers and refreshes content, not how content ranks. A crawl-starved site suffers indirectly: new and updated pages take longer to enter the index.