Googlebot
Googlebot is the web crawler Google uses to discover and fetch pages for its search index. Googlebot Smartphone, which crawls with a mobile user agent, is the primary crawler for nearly all sites under mobile-first indexing.
Googlebot is the umbrella name for the two crawler types that feed Google Search: Googlebot Smartphone and Googlebot Desktop. With mobile-first indexing complete, Googlebot Smartphone performs the overwhelming majority of crawling — what it fetches with its mobile user agent is the version of the page Google renders and indexes, with the desktop agent used for a small share of verification fetches.
Googlebot's fetch behavior is documented and predictable. It crawls mostly from US-based IP addresses, supports HTTP/1.1 and HTTP/2, fetches the first 15 MB of an HTML file (referenced resources are fetched separately), and adjusts its request rate to the server — slowing down when responses slow or 5xx errors appear, the mechanism behind the crawl budget capacity limit. Fetched pages pass onward to rendering and indexing.
Because the name carries authority, Googlebot is heavily impersonated: scrapers and vulnerability scanners send its user-agent string to slip through bot protections. Log analysis therefore starts with verification — reverse DNS or Google's published IP ranges — before drawing any conclusion about what Googlebot did on the site.
Quick facts
What user-agent strings does Googlebot send?
Googlebot identifies itself with the token Googlebot — the name robots.txt rules match — inside a full user-agent string that embeds a current Chrome version, because the crawler's rendering engine tracks stable Chrome.
- Googlebot Smartphone:
Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) - Googlebot Desktop:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36
W.X.Y.Z stands for the Chrome build in use, which changes as Chrome updates. Specialized crawlers such as Googlebot-Image carry their own tokens but follow rules for Googlebot when no more specific robots.txt group exists.
How do you verify a request is really from Googlebot?
Verification takes two documented forms: reverse-DNS confirmation or matching the IP against Google's published ranges. The user-agent string alone proves nothing — spoofed Googlebots are a constant presence in server logs.
- 1Run a reverse DNS lookup on the requesting IP (
host 66.249.66.1) - 2Confirm the returned hostname ends in
googlebot.comorgoogle.com - 3Run a forward DNS lookup on that hostname and confirm it resolves back to the original IP
For bulk verification, Google publishes its Googlebot IP ranges as a JSON file (googlebot.json), updated as ranges change, so firewalls and log tooling can match against it directly. Requests that fail both checks are impostors and are safe to block.
Does Googlebot respect robots.txt?
Yes. Googlebot fetches robots.txt before crawling a site and honors its rules: a disallowed path is not fetched at all. Compliance stops at the crawling layer, though — a disallowed URL that other pages link to can still be indexed from those references, without its content. Keeping a page out of the index requires noindex, which Googlebot must be able to crawl the page to see.
Malicious bots ignore robots.txt entirely, which is another reason verification matters: a Googlebot hitting disallowed paths is by definition fake. Test how your rules apply to specific URLs with the robots.txt tester.
Frequently Asked Questions
Is every visitor identifying as Googlebot really Google?+
No. The user-agent string is trivially spoofed, and scrapers routinely impersonate Googlebot to bypass bot blocks. Trust a request only after reverse-DNS verification or an IP match against Google's published Googlebot ranges.
Does Googlebot execute JavaScript?+
Yes. Pages Googlebot fetches are passed to Google's Web Rendering Service, which runs JavaScript in an evergreen Chromium browser. Rendering is queued separately from crawling, so JavaScript-dependent content is processed later than static HTML.
Can you control how fast Googlebot crawls?+
Partially. Googlebot slows down automatically when a server responds slowly or returns 5xx or 429 errors, and robots.txt can remove paths entirely. Google ignores the crawl-delay directive and retired Search Console's crawl-rate limiter tool in January 2024.