Crawling
Crawling is the process by which search engine bots discover URLs and fetch their content for analysis. It is the first stage of search visibility: a page must be crawled before it can be rendered, indexed, or ranked.
Crawling combines two operations: discovery (finding URLs) and fetching (downloading their content). Googlebot discovers new URLs by following links on pages it has already fetched, by reading XML sitemaps, and by receiving direct submissions through Search Console or indexing APIs. Every fetched page yields new links, which enter the crawl queue and keep the cycle running.
Google does not crawl all URLs equally or immediately. The crawl queue is prioritized by demand — how heavily a URL is linked, how often it has changed historically, and how important Google judges it — within the limits of the site's crawl budget. A heavily linked homepage is refetched daily or more often; a deep page with one internal link can wait weeks between visits, and a known-but-unqueued URL shows in Search Console as Discovered – currently not crawled.
Crawling is not indexing. A crawl only retrieves the page; indexing — analyzing the content and storing it in the search index — is a separate, selective stage that follows rendering. Google fetches many pages it never indexes, reported as Crawled – currently not indexed.
Quick facts
How does Google discover new URLs?
Google discovers URLs from three inputs: links on pages it has already crawled, URLs listed in XML sitemaps, and direct submissions.
- Links — the primary channel. Googlebot extracts every followed link, internal and external, from every page it fetches, so a page linked from an already-crawled page is discovered on the next fetch.
- Sitemaps — the declared inventory. Sitemap files hand Google a complete URL list with last-modified dates, covering pages that links alone would miss.
- Submissions — the explicit request. The Request Indexing button in the URL Inspection tool, the Indexing API for eligible content types, and URL submission services all place URLs into Google's crawl systems directly.
Discovery only queues a URL. The crawl itself happens whenever the scheduler reaches it, which is where prioritization takes over.
How does Google decide what and when to crawl?
Google schedules crawling by demand within capacity. On the demand side, URLs that are heavily linked, historically fast-changing, or newly discovered move to the front of the queue; on the capacity side, total fetching stays inside what the site's server sustains without slowing down — together, the site's crawl budget.
The result is sharply uneven crawl frequency. High-authority news sites are recrawled within minutes; stable pages on small sites go weeks between visits. No mechanism forces a crawl on a schedule you choose — sitemaps and submissions move a URL up the queue, but Google's scheduler decides the fetch.
Why is a crawled page not always indexed?
A crawl retrieves the page; indexing stores it — and indexing is selective. After fetching, Google renders the page, evaluates content quality, selects the canonical among duplicates, and only then writes it to the index. Pages that fail the evaluation are fetched and dropped, surfacing in Search Console as Crawled – currently not indexed.
The distinction matters diagnostically. A Googlebot fetch confirmed in your server logs rules out discovery and access problems, but it says nothing about whether the page entered the index — indexed state is a separate check in the URL Inspection tool or a bulk index checker.
Frequently Asked Questions
Does crawling guarantee indexing?+
No. Crawling retrieves the page; indexing is a separate, quality-gated decision. Google states directly that it does not index every page it crawls — Search Console's Crawled – currently not indexed state exists precisely for fetched-but-rejected URLs.
How often does Googlebot crawl a site?+
It varies with demand: news sites and busy e-commerce homepages are fetched many times a day, while stable pages on small sites can go weeks between crawls. The Crawl Stats report in Search Console shows your site's actual daily request counts.
How do you make Google crawl a page faster?+
Link to it from a frequently crawled page, list it in a sitemap with an accurate lastmod date, request it in the URL Inspection tool, or push it through a URL submission service. Each shortens discovery; the fetch itself still runs on Google's schedule.