Crawl Rate
Crawl rate is the number of requests per second Googlebot makes to a site while crawling it. Google sets the rate automatically from server health signals; the manual crawl-rate limiter in Search Console was retired in January 2024.
Crawl rate is the speed dimension of Googlebot's activity: how many requests per second it fires at a host and how long it waits between fetches. It is set per host, not per page, and it is one half of a pair that gets conflated constantly — crawl rate is speed, while crawl budget is volume, the total URLs Google can and wants to crawl over time. Rate feeds the capacity half of budget: a host that absorbs fast fetching cleanly gets a higher ceiling.
Google sets crawl rate automatically. Googlebot continuously adjusts to server health — sustained fast 2xx responses push the rate up, while slowdowns, 5xx errors, and 429 (Too Many Requests) responses pull it down. Search Console's manual crawl-rate limiter, the one user-facing control, was retired in January 2024; since then the only levers are the ones the server itself presents.
The demand side is separate: even a high rate goes unused when Google sees no reason to crawl — no fresh content, weak internal linking, few external signals. Most small sites run far below their capacity ceiling, which is why speed tuning alone rarely fixes discovery problems.
Quick facts
How is crawl rate different from crawl budget?
Crawl rate is how fast Googlebot fetches — requests per second against one host. Crawl budget is how much it fetches — the URL volume Googlebot can and wants to crawl in a timeframe. Rate is an input to budget's crawl capacity limit, not a synonym for budget.
The distinction matters for diagnosis. A backlog of URLs in Discovered – currently not crawled on a fast, healthy server is a demand problem — Google is uninterested, not throttled — and the fix is stronger internal linking and better content, not server work. Falling requests per day paired with rising response times is a capacity problem, and the fix is performance.
Can you still set Googlebot's crawl rate in Search Console?
No. Google retired the Search Console crawl-rate limiter in January 2024, and crawl rate has been fully automatic since. The setting let site owners cap Googlebot's request rate; Google removed it because its automatic adjustment to server health had made the manual cap redundant.
The levers that remain are server responses. Returning 429 or 5xx makes Googlebot back off — appropriate during an incident, dangerous as a policy, because persistent 5xx responses lead to URLs dropping from the index. To keep Googlebot away from specific paths permanently, use robots.txt disallow rules instead of rate suppression.
How do you monitor Googlebot's crawl rate?
The Crawl Stats report (Search Console → Settings → Crawl Stats) is the primary surface: total crawl requests per day, average response time, and host status, with breakdowns by response code, file type, and Googlebot type. Requests per day tracks how much Google is crawling; average response time is the strongest signal you control over whether that number rises or falls.
- 1Watch average response time — sustained increases precede rate cuts
- 2Track the 5xx share in the response-code breakdown — errors are the fastest way to lose capacity
- 3Compare requests per day against publishing volume — flat crawling against a growing inventory means discovery lag
- 4Confirm new sections get fetched after submitting a sitemap to Google
Frequently Asked Questions
How do you increase Googlebot's crawl rate?+
Cut response times and eliminate 5xx and 429 errors — the rate rises when the host absorbs fetching without strain. Demand matters equally: fresh content and stronger internal links give Google reasons to use the capacity. No setting raises it directly.
How do you slow Googlebot down without Search Console?+
Return 429 or 5xx responses and Googlebot backs off. Treat this as an incident tool only — persistent 5xx responses eventually drop URLs from the index. For permanent exclusion of specific paths, use robots.txt disallow rules.
Is a falling crawl rate a problem?+
Only with symptoms. Falling requests per day alongside rising response times or 5xx errors is a capacity problem on your side. Falling requests with a healthy server means falling demand — Google sees little new or changed content worth fetching.