Robots.txt Tester
Test crawl rules for any URL.
Checked from our server. One location, so results show reachability from there only.
What the robots.txt tester does
A robots.txt file sits at the root of a site and tells crawlers which paths they may fetch. One misplaced slash can hide a whole section from search engines, or leave open a folder you meant to keep out of results. This tester fetches the live file, works out which rules apply to the crawler you name, and tells you whether a given path is allowed or blocked.
Reach for it when pages drop out of search after a site change, when you’re adding rules for AI crawlers such as GPTBot or ClaudeBot, or when you want to confirm that a new Disallow line catches exactly the URLs you intended and nothing more.
How to use the robots.txt tester
- Type the site, or a full page URL. If you leave the path box empty, we test the path from the URL you typed.
- Choose a user agent (such as Googlebot or GPTBot) and, optionally, a path to test. You can type any crawler name, not just the ones in the list.
- Press Test. You get allowed or blocked, the exact rule that decided it, and the robots.txt with that line highlighted.
Reading the results
The verdict box gives the answer in one line, with a sentence explaining why. Underneath, you’ll see the address the file was read from, its HTTP status, the user-agent group that was used and the matching rule with its line number. If no rule matches, the path is allowed. If no group names your crawler and there is no * group, everything is allowed too.
Group matching uses the first word of the agent name, so Googlebot-Image falls back to a Googlebot group when the file has no group of its own for it. When a result surprises you, look at the highlighted line first: a short pattern such as Disallow: /p also blocks /products and /pricing. Any sitemaps the file lists appear as links you can open in the sitemap validator.
Questions
How is the decision made?
Following RFC 9309: the most specific matching user-agent group applies, the longest matching rule wins, and Allow wins a tie. * and $ wildcards are supported.
What if robots.txt is missing or broken?
A missing file (HTTP 4xx) allows everything. If it can’t be fetched because of a server error, crawlers should assume everything is blocked.
Does robots.txt protect private pages?
No. It is a request that polite crawlers follow. Use authentication to keep pages private.
Can I test a path with a query string?
Yes. Type it as /search?q=shoes, or paste a full URL and we keep only the path and query. We check the path both as typed and with percent-encoding decoded.