How Google Indexing Works: Crawling, Rendering, and Indexing Explained (2026)

Google indexing works as a pipeline: Google discovers a URL, schedules a crawl, fetches the page with Googlebot, renders its JavaScript, evaluates the content, and — if the page passes canonicalization and quality checks — writes it into the search index, where it becomes eligible to rank. Every URL on the web moves through the same stages in the same order, and every stage can delay a page or drop it entirely.

This guide walks the pipeline stage by stage: how Google discovers URLs, how crawling is scheduled and executed, how rendering works, how indexing decisions are made, what the index stores, and why an indexed page still has to win its rankings at serving time. It closes with the two maps that make the pipeline actionable: where each stage reports in Google Search Console, and which lever you control at each stage.

TL;DR — Quick Summary

• Google indexing is a seven-stage pipeline: discovery → crawl scheduling → fetching → rendering → indexing decisions → the index → serving
• Google discovers URLs through links on known pages, XML sitemaps, redirects, and direct submission
• Discovered URLs wait in a crawl queue; crawl budget — demand versus server capacity — decides when Googlebot fetches them
• Rendering executes JavaScript in an evergreen Chromium; client-rendered content indexes later than server-rendered HTML
• Indexing is selective: Google clusters duplicates, elects one canonical URL, applies quality thresholds, and declines pages that fail them
• The index is an inverted index of roughly 400 billion documents; indexed means eligible to rank, not ranking
• Every stage maps to a specific Search Console report, and every stage has a lever you control — both tables are below

---

How Does Google Discover URLs?

Google discovers URLs from four sources: links on pages it has already crawled, XML sitemaps, redirects, and direct submission. Discovery is pure bookkeeping — Google records that the URL exists and nothing more. No fetch has happened and no content has been evaluated.

Links from known pages are the primary channel. Each time Googlebot crawls a page, it extracts every <a href> link and checks the targets against Google's URL inventory; new URLs join the discovery list. This makes internal linking discovery infrastructure: a page linked from a frequently crawled page is discovered within hours, while an orphan page — linked from nowhere — is discovered only if a sitemap or an external link announces it.

XML sitemaps are the machine-readable inventory you hand to Google: a file listing the URLs you want crawled, with optional lastmod timestamps. Sitemaps announce URLs; they do not command crawling. Google treats each entry as a discovery hint and schedules fetches on its own priorities.

Redirects create discoveries as a side effect: when a crawled URL returns a 3xx response, the redirect target enters the inventory as a new URL.

Direct submission covers the push channels. The URL Inspection tool in Search Console submits one URL at a time into a priority crawl queue. The Google Indexing API pushes URLs programmatically but is officially restricted to two content types — job postings and livestream pages — with a default quota of 200 requests per day per project.

---

How Does Google Decide When to Crawl a URL?

Every discovered URL enters the crawl queue, where Google schedules its fetch based on crawl budget — the number of URLs Googlebot can and wants to crawl on a site within a given timeframe. Two forces set that number, and they pull in opposite directions.

Crawl demand is how much Google wants your URLs. Demand rises with the number and strength of links pointing at them, with how often your known pages genuinely change, and with the site's track record in search results.

Crawl capacity is how much fetching your server sustains. Googlebot measures response times and error rates continuously and throttles itself the moment fetches slow down or fail, because Google's hard constraint is never overloading a host. Google's guide to managing crawl budget for large sites documents both halves of the mechanism.

URLs wait in the queue when demand is low or capacity is constrained. The wait is visible in Search Console as the never-crawled state covered in our guide to fixing "Discovered - Currently Not Crawled", and its length varies by orders of magnitude with site authority: high-demand news sites see new URLs fetched within minutes, while a new domain with few links waits days to weeks for the same fetch.

---

How Does Googlebot Fetch a Page?

Googlebot fetches a scheduled URL in two steps: it checks the site's robots.txt file first, then requests the page — using Googlebot Smartphone as the primary user agent, because mobile-first indexing is complete and Google indexes the mobile version of every site.

The robots.txt check is absolute: a URL matched by a Disallow rule is never fetched. Blocked URLs can still enter the index from link signals alone — without any content — which is why Search Console has a state named "Indexed, though blocked by robots.txt".

The HTTP response to the fetch then tells Google how to proceed:

Sustained server errors and timeouts do double damage: the failed fetch is wasted, and the crawl capacity limit drops for every future fetch on the host.

---

How Does Google Render JavaScript?

Pages that depend on JavaScript go through rendering: the Web Rendering Service (WRS) — an evergreen Chromium that stays current with stable Chrome — loads the fetched page, executes its scripts, and produces the final DOM that indexing systems evaluate.

Rendering runs on its own queue, and that queue is why JavaScript sites index in two beats. Content present in the server's HTML response is processable the moment the crawl completes; content injected by client-side JavaScript exists only after the WRS pass. For most pages the render follows the crawl quickly, but the render queue is shared infrastructure — on large sites and under load, the gap between crawl and render widens, and client-rendered content indexes later than server-rendered content as a result.

Three kinds of content never make it into the render:

• Content that requires interaction. The WRS does not click, type, or log in — content that appears only after a tap, a form entry, or an authenticated session is invisible to indexing.
• Resources blocked by robots.txt. A JavaScript bundle or API endpoint disallowed for Googlebot is missing from the render, along with everything it would have painted.
• Content that arrives after rendering stops. The WRS waits a bounded time for the page to settle; data that loads after that window is not part of the indexed snapshot.

Google's documentation on how Google Search processes JavaScript describes the crawl–render–index split in detail. The operational rule it implies: put content you need indexed into the server-rendered HTML.

---

How Does Google Decide What to Index?

After rendering, indexing systems make the pipeline's only judgment call: they analyze the content, group the page with any duplicates, select one canonical, extract structured data and entities, and score the result against quality thresholds. Pages that clear every check are written to the index; the rest are declined.

Content analysis parses everything the render produced: body text, the <title> element, headings, image alt attributes, and link annotations. A noindex directive — a meta tag or HTTP header instructing Google not to index the page — is read and enforced at this stage, after the crawl, which is why noindex only works on a crawlable page.

Canonical selection deduplicates the web. Google clusters substantially similar URLs — HTTP/HTTPS variants, parameter permutations, syndicated copies — and elects one canonical URL to represent each cluster. Your rel="canonical" tag is a hint in that election, not a directive: 301 redirects, sitemap inclusion, internal link consistency, and HTTPS preference all vote, and when they outvote your tag, Search Console reports "Duplicate, Google chose different canonical than user". Only the winning canonical is indexed for the cluster; the duplicates pass their signals to it.

Quality thresholds make indexing selective. Google evaluates whether the page adds enough value against what the index already holds — thin, duplicative, and redundant pages are declined, and large sites routinely see 10–40% of crawled URLs excluded. The rejection state is the one covered in our guide to fixing "Crawled - Currently Not Indexed": Google fetched the page, evaluated it, and chose not to store it.

Structured data and entity extraction run alongside: Schema.org markup is parsed, the entities the page discusses are identified and resolved, and both are stored with the document — the inputs for rich results and entity-based retrieval at serving time.

---

What Does Google's Index Actually Store?

The search index is an inverted index: instead of storing pages as documents to scan, it maps every meaningful term to the list of documents containing it, with positions, prominence, and entity annotations attached. That inversion is what makes retrieving candidates for any query in milliseconds possible — at a scale that testimony in the 2023 US antitrust trial put at roughly 400 billion documents.

For each indexed URL, Google stores the rendered content with its term postings, the selected canonical, extracted entities and structured data, and the signals its ranking systems read at serving time.

The index is tiered, not uniform. Documents that are served and updated frequently live in faster storage and get recrawled more often; stale, rarely linked documents sit in slower tiers and get revisited less. Membership is not permanent either — pages leave the index through noindex directives, 404/410 responses, canonical consolidation, and quality re-evaluation.

---

Does Indexing Mean Ranking?

No. Indexing makes a page eligible to appear in results; ranking happens at serving time, per query. When a user searches, Google retrieves candidate documents from the index, and ranking systems order them by relevance, quality, links, and freshness — a computation that runs for that query, at that moment. Nothing about rank order is stored in the index itself.

The distinction explains the most common post-indexing disappointment: a fully indexed page can record zero impressions, because eligibility for every query is not visibility for any query. Google's in-depth guide to how Search works draws the same boundary — crawling, indexing, and serving are three separate systems, and passing one does not guarantee passing the next.

---

Where Does Each Stage Show Up in Google Search Console?

Every pipeline stage reports into a specific surface in Google Search Console, which turns the pipeline into a diagnostic checklist: find the stage where a URL stalled, and you have found the report that explains why.

---

What Can You Control at Each Stage?

You control a different lever at every stage: discovery responds to links and sitemaps, scheduling to server health and link prominence, rendering to where content lives in your markup, and the indexing decision to content quality and signal alignment. The table maps each stage to its lever and its measurement.

The stages compound. A page that is easy to discover, cheap to fetch, and server-rendered reaches the indexing decision days before a page that is orphaned, slow, and client-rendered. For the full implementation sequence — sitemap setup, submission, and the fixes for each blocked stage — work through our guide on how to get your website indexed on Google.

---

Frequently Asked Questions

What is the difference between crawling and indexing?

Crawling is fetching: Googlebot downloads the page's HTML and resources. Indexing is evaluation and storage: Google analyzes the rendered content, selects a canonical, applies quality thresholds, and writes accepted pages into the search index. A page can be crawled repeatedly and never indexed.

How long does the full pipeline take?

Between a few hours and several weeks end to end. Established, frequently crawled sites move new URLs from discovery to indexed within a day; new domains with few links wait 1–4 weeks, usually stuck at the crawl-scheduling stage. Our breakdown of how long Google takes to index a page gives the timeline bands by site profile.

Does Google index every page it crawls?

No. Indexing is selective by design: Google evaluates every crawled page against quality thresholds and stores only the pages that add value over what the index already holds. Large sites routinely see 10–40% of crawled URLs excluded, reported in Search Console as "Crawled - currently not indexed".

Can you submit URLs to Google directly?

Yes, through three official channels: the URL Inspection tool's Request Indexing button (one URL at a time, around 10–12 manual requests per property per day), the Indexing API (200 URLs per day, officially limited to job postings and livestreams), and XML sitemaps (discovery only — no crawl guarantee). URL indexing services add a fourth path by submitting URLs directly into Google's crawl systems at volume.