Robots.txt Tester
Free robots.txt tester — fetch a site's robots.txt or paste your own rules, then test whether specific URLs are allowed or blocked for a chosen user agent.
About the Robots.txt Tester
The robots.txt file controls which parts of your site search engine crawlers may access. A single wrong rule can accidentally block your whole site from Google. This free robots.txt tester lets you fetch a live robots.txt or paste your own, then check whether specific URLs are allowed or disallowed for a given user agent.
Enter a site to pull its current robots.txt through our proxy, or paste draft rules you are working on. Add the paths you want to test and the tool evaluates each against the Allow and Disallow directives, honouring user-agent groups and the longest-match precedence that real crawlers use.
It is the safe way to confirm a rule does exactly what you intend before you deploy it — verifying that important pages stay crawlable while private or duplicate sections are correctly blocked.
How to Use the Robots.txt Tester
- 1Fetch a site's robots.txt or paste your own rules.
- 2Choose the user agent to simulate (e.g. Googlebot).
- 3Enter one or more URLs or paths to test.
- 4Read whether each URL is allowed or blocked, and why.
Frequently Asked Questions
What is a robots.txt file?
Robots.txt is a plain-text file at your site root that tells crawlers which URLs they may or may not request, using User-agent, Allow and Disallow directives. It guides well-behaved bots like Googlebot, though it is a request, not an enforced barrier — sensitive pages need real access controls.
Does robots.txt stop a page being indexed?
Not reliably. Disallow stops crawling, but a blocked URL can still appear in search results (without a description) if other sites link to it. To keep a page out of the index, allow crawling and use a noindex meta tag, or protect it behind authentication.
How do crawlers resolve conflicting rules?
Google follows the most specific rule — the one whose path matches the most characters — and Allow beats Disallow when they are equally specific. This tester applies the same longest-match logic, so you see the outcome a real crawler would produce for each URL.
Can I test rules for different bots?
Yes. Robots.txt can define separate rule groups per user agent, and this tester lets you choose which bot to simulate — such as Googlebot, Bingbot or the default * group — so you can confirm each crawler is treated as intended.
Where should robots.txt live?
It must sit at the root of your domain, reachable at yourdomain.com/robots.txt — crawlers only look there. A robots.txt in a subfolder is ignored. Each subdomain needs its own file, and it should also list your sitemap's URL for discovery.