How to test a robots.txt file
Testing is worth doing after every change, and whenever pages unexpectedly drop out of search.
- 1Enter your domain and press Fetch to load the live file
- 2Pick the crawler you care about, such as Googlebot
- 3Paste the URLs that matter most: your homepage, key category pages, recent posts
- 4Check each verdict and the rule that caused it
- 5Edit the file in the box to try a fix, then copy it back to your site
Which group does a crawler follow?
A crawler obeys exactly one group of rules: the one whose User-agent line matches its name most specifically. If Googlebot has its own group, it ignores the * group entirely, even rules you meant for everyone.
Sub-crawlers fall back to their parent. Googlebot-Image with no group of its own follows the Googlebot group, and only uses * if neither exists. If a crawler matches no group at all, it may crawl everything.
How Allow and Disallow conflicts are resolved
Within the chosen group, the rule with the longest matching path wins, whatever order the lines are in. With Disallow: /admin/ and Allow: /admin/public/, the URL /admin/public/logo.png is allowed because the Allow rule is longer.
If an Allow and a Disallow match with the same length, Allow wins. An empty Disallow: line means nothing is blocked.
Wildcards and end-of-URL matching
Rules match from the start of the path, including the query string.
- /blog
- Matches /blog, /blog/, /blog/post and also /blogging, because rules are prefixes.
- /blog/
- Matches everything inside the folder but not /blog itself.
- /*?
- Matches any URL with a query string, such as /shop?sort=price.
- /*.pdf$
- Matches URLs that end in .pdf. Without the $, /file.pdf?download=1 would match too.
Allowed doesn't mean indexed, blocked doesn't mean hidden
Robots.txt controls crawling only. An allowed URL can still be left out of Google for quality or duplicate reasons. A blocked URL can still appear in results, without a description, if other sites link to it.
To keep a page out of search, allow it to be crawled and add a noindex tag. Our Canonical Tag Checker shows a page's robots meta tags alongside its canonical.
What the HTTP status of robots.txt means
Google reacts to the status code the file returns, not just its contents.
- 200 OK
- The rules in the file apply.
- 404 or other 4xx
- Treated as if there is no robots.txt, so everything may be crawled. 429 (too many requests) is the exception and is treated like a server error.
- 5xx server error
- Google treats the site as fully blocked for a while and may fall back to a cached copy, so a broken robots.txt can stop crawling across the whole site.
Testing AI crawlers
Most AI companies run more than one crawler. OpenAI uses GPTBot for training and OAI-SearchBot for ChatGPT search, Anthropic uses ClaudeBot for training and Claude-SearchBot for search, and Perplexity uses PerplexityBot. Test each one separately, because a rule for one doesn't cover the others.
If you want to be cited in AI answers, make sure the search crawlers are allowed on the pages you want cited. Our AI Readiness Test checks this and several other AI-search signals for a whole site.