Robots.txt Tester
Test if your URLs are blocked by robots.txt. Paste your file, enter a URL path, and check access for Googlebot, Bingbot, or GPTBot.
This tool runs entirely in your browser. No data is sent to any server.
Find your robots.txt at: yourdomain.com/robots.txt
URL not blocked but still not indexed?
If your robots.txt isn't blocking the page but Google still hasn't indexed it, use IndexBolt to get it crawled in hours.
Frequently Asked Questions
What is a robots.txt file?+
A robots.txt file is a text file at the root of your website (e.g., example.com/robots.txt) that tells search engine crawlers which pages they can and cannot access. It's a key part of the Robots Exclusion Protocol.
Does robots.txt prevent pages from being indexed?+
Not exactly. Robots.txt prevents crawling, not indexing. If other sites link to a blocked page, Google may still index the URL (showing it in results without a description). To prevent indexing, use a noindex meta tag instead.
What does 'Disallow: /' mean in robots.txt?+
Disallow: / blocks the entire site from being crawled by the specified user-agent. If applied to all user-agents (User-agent: *), no search engine crawler can access any page on your site.
How does robots.txt pattern matching work?+
Robots.txt uses simple pattern matching. * matches any sequence of characters, and $ anchors to the end of the URL. For example, Disallow: /*.pdf$ blocks all URLs ending in .pdf. When multiple rules match, the most specific (longest) pattern wins.
Can I block specific bots like GPTBot?+
Yes. Add a User-agent: GPTBot section with Disallow: / to block OpenAI's crawler. Similarly, you can target Googlebot, Bingbot, or any specific crawler by name. Each bot reads only the rules in its own section (or the * section as fallback).