Robots.txt
Robots.txt is a plain-text file at a site's root that tells search engine crawlers which URL paths they may fetch. It controls crawling, not indexing — a URL blocked in robots.txt can still be indexed from links pointing to it.
Robots.txt is the crawl gatekeeper: before fetching any URL on a host, a compliant crawler requests /robots.txt and obeys the rules matching its user agent. The file implements the Robots Exclusion Protocol (REP), created informally in 1994 and published as IETF standard RFC 9309 in September 2022. It applies per host and protocol — https://example.com, http://example.com, and shop.example.com each need their own file.
The most consequential fact about robots.txt is what it does not do: it controls crawling, not indexing. Google can index a URL it is forbidden to fetch when internal or external links point to it, showing a bare, snippet-less listing. Worse, blocking the crawl hides on-page directives: a noindex tag on a robots.txt-blocked page is invisible, because Google never fetches the HTML that contains it. Removing a page from the index therefore requires allowing the crawl so the noindex can be seen — the opposite of blocking it.
What robots.txt is for is crawl management. Disallowing parameter permutations, internal search results, and infinite calendar spaces keeps Googlebot from spending crawl budget on URLs that should never rank — which matters on large sites, where wasted fetches delay the crawling of pages that should.
Quick facts
What directives does robots.txt support?
Robots.txt supports four directives Google honors: User-agent (which crawler a rule group addresses), Disallow (paths that crawler must not fetch), Allow (exceptions inside a disallowed path), and Sitemap (the absolute URL of an XML sitemap, valid anywhere in the file).
User-agent: *— the group applies to every crawlerDisallow: /search/— blocks the internal search spaceAllow: /search/popular— carves one crawlable page out of the blockSitemap: https://example.com/sitemap.xml— points crawlers to the URL inventory
Matching is most-specific-wins: when Allow and Disallow both match a URL, the longer rule applies. Wildcards (*) and end-of-URL anchors ($) are supported. Google ignores Crawl-delay entirely and sets Googlebot's fetch rate automatically.
Does robots.txt stop a page from being indexed?
No. Robots.txt stops the fetch, and Google indexes some blocked URLs anyway — anchored by the links pointing at them — listing them with a URL-derived title and no snippet. Search Console reports these under "Indexed, though blocked by robots.txt".
This produces the classic robots.txt trap: adding Disallow to remove an indexed section keeps Google from ever seeing the noindex that would actually remove it, freezing the section in the index in its blocked state. Deindexing requires the reverse configuration — allow the crawl, serve noindex or a 404/410, and let Googlebot refetch the URLs.
What are robots.txt's size and location limits?
Google enforces a 500 KiB parse limit: only the first 500 KiB of the file are read, and rules beyond that boundary are ignored — a real ceiling for large sites that generate thousands of disallow lines. The file must sit at the root of the host (https://example.com/robots.txt) and covers only that host; subdomains and other protocols resolve their own file.
Response codes matter: a missing file (404) means unrestricted crawling, while a robots.txt returning 5xx errors is treated as a temporary full disallow — Google pauses crawling the site until the file is reachable again. Validate rules against specific URLs with the Robots.txt Tester before deploying changes.
Frequently Asked Questions
Can a page blocked by robots.txt appear in Google search results?+
Yes. Google indexes blocked URLs it considers link-worthy, showing the URL with an anchor-derived title and no snippet — Search Console labels them "Indexed, though blocked by robots.txt". Removing one requires allowing the crawl and serving noindex or a 404.
Do all crawlers obey robots.txt?+
Compliant crawlers — Googlebot, Bingbot, and every major search engine bot — obey it. The protocol is voluntary: scrapers and malicious bots routinely ignore it, so robots.txt is crawl management, not access control. Protecting sensitive content requires authentication.
Does robots.txt save crawl budget?+
Yes. Disallowing parameter spaces, internal search results, and other infinite URL patterns is the fastest crawl-budget lever, because Googlebot stops spending fetches on URLs that were never going to be indexed.