Robots.txt Tester

Parses your robots.txt and checks whether specific URLs would be allowed or blocked for any user-agent crawler, using Google-style matching rules.

Most specific (longest) matching rule wins. Allow beats Disallow at equal specificity. Supports * wildcards and $ end-of-URL anchors.

Yes — Bulk URL Test tab accepts multiple URLs and shows ALLOWED/BLOCKED verdict per URL with summary counts.

Yes, the Robots.txt Tester is 100% free with no registration, no hidden fees, and no usage limits. All processing happens locally in your browser, ensuring complete privacy.

Absolutely. The Robots.txt Tester processes everything client-side in your browser. No data is uploaded to or stored on any server. Your content remains private on your device at all times.

Yes, the Robots.txt Tester is fully responsive and works on smartphones and tablets. You can use it on any device with a modern web browser -- no app download required.

No account or registration is needed. Simply open the Robots.txt Tester in your browser and start using it immediately. There are no sign-up walls or usage restrictions.

Yes, the Robots.txt Tester is built following current Google Search guidelines and best practices. We regularly update the tool to reflect changes in search engine requirements.

Simply enter your input in the provided field, adjust any settings to your preference, and the tool will process it instantly. You can then copy the result to your clipboard or download it.

The Robots.txt Tester works in all modern browsers including Chrome, Firefox, Safari, Edge, and Opera. For the best experience, use the latest version of your preferred browser.

Robots.txt controls crawling, not indexing. A Disallow rule tells crawlers not to request a URL, but it does not remove that URL from search results — if other pages link to it, Google can still index the link with no description, since it was never allowed to read the page. A noindex meta tag or HTTP header, by contrast, lets the crawler fetch the page and then explicitly keeps it out of the index. The catch is that noindex only works if the page is crawlable, so blocking it in robots.txt actually prevents Google from ever seeing the noindex directive. To reliably hide a page, allow crawling and use noindex; to save crawl budget on truly worthless URLs, use Disallow. This tester shows you the exact ALLOWED or BLOCKED verdict so you can confirm a page is reachable before relying on noindex.

The asterisk (*) matches any run of characters, and the dollar sign ($) anchors a rule to the end of the URL path. Together they let one line cover many URLs. For example, Disallow: /*?sort= blocks any path containing a sort parameter, which is handy for faceted-search and filter URLs that create endless duplicate pages. Disallow: /*.pdf$ blocks only URLs that end in .pdf, so a path like /guide.pdf?ref=email would not match because something follows the extension. Google and most modern crawlers support both symbols, but they are easy to get subtly wrong, and a misplaced wildcard can block far more than you intended. Paste your file and test a sample URL here to see whether your wildcard rule matches the way you expect.

You target each bot by name in its own user-agent group. Search crawlers and AI training crawlers use different user-agent strings, so a rule under one does not affect the other. To keep Googlebot and Bingbot indexing your site while shutting out AI bots, leave User-agent: * set to Allow, then add separate Disallow: / groups for agents such as GPTBot, ChatGPT-User, Google-Extended, anthropic-ai, ClaudeBot, CCBot, PerplexityBot, and Bytespider. Because the most specific matching group wins, each named bot follows its own block while search engines stay on the permissive default. Note that compliance is voluntary — well-behaved bots honor robots.txt, but it is not a security barrier. This tester includes a Block AI Bots preset and lets you simulate each crawler individually to confirm search engines are still allowed in while AI bots are blocked.

The most common reason is location: a robots.txt file must sit at the root of the host it governs, at exactly /robots.txt, and rules are scoped per protocol, subdomain, and port. A file at /folder/robots.txt or one on the apex domain will not control a subdomain. Other silent failures include Allow or Disallow lines placed before any User-agent line (crawlers ignore them), duplicate user-agent groups that get merged unexpectedly, an unreachable Allow rule shadowed by a more specific Disallow, and a missing User-agent: * default group. Typos in directive names and negative Crawl-delay values are also commonly dropped. This tester parses your whole file, flags these structural problems with line numbers, and names the exact directive deciding each URL, so you can pinpoint why a rule is not behaving as you expected.

No — Google does not support the Crawl-delay directive and ignores it entirely, so adding it will not change how fast Googlebot requests your pages. To adjust Google's crawl rate you rely on its automatic adaptive crawling, which responds to your server's speed and error responses. Crawl-delay is, however, honored by some other crawlers, including Bingbot and Yandex, where it sets the minimum number of seconds a bot waits between requests. A negative value is invalid and gets dropped. Because support varies by engine, it is worth confirming which crawler reads the directive and what value applies to it. This tester surfaces the Crawl-delay value for whichever user-agent you select, parses it from your file, and warns you about invalid negative values, so you can see at a glance which bots your delay will actually affect.