What is a robots.txt file?
Robots.txt is a plain text file at the root of a website (example.com/robots.txt) that tells crawlers which URLs they may request. It follows the Robots Exclusion Protocol, standardised as RFC 9309 in 2022.
It's a request, not a lock. Google, Bing and the major AI companies say their crawlers respect it, but nothing forces a crawler to. Never rely on robots.txt to hide private content. Use a login or a server-side block for that.
Robots.txt syntax explained
A file is made of groups. Each group starts with one or more User-agent lines and lists the rules for those crawlers.
- User-agent
- Which crawler the group applies to. * means every crawler without a more specific group of its own.
- Disallow
- A path the crawler shouldn't request. Disallow: / blocks the whole site. An empty Disallow: blocks nothing.
- Allow
- An exception inside a blocked path. When an Allow and a Disallow both match, the longer, more specific rule wins.
- Sitemap
- The full URL of your XML sitemap. It can appear anywhere in the file and applies to every crawler.
- * and $
- * matches any run of characters and $ marks the end of a URL, so Disallow: /*.pdf$ blocks every PDF.
Robots.txt blocks crawling, not indexing
This is the most common misunderstanding. A URL blocked in robots.txt can still appear in Google if other pages link to it. Google just won't know what's on it, so the result shows up with no description.
To keep a page out of search results, let it be crawled and add a noindex meta tag or X-Robots-Tag header instead. If you block the page in robots.txt, Google can't see the noindex, and the page can stay indexed.
How to block AI crawlers in robots.txt
AI companies run several crawlers with different jobs, and you can treat them differently. Training crawlers collect content to train models. Search crawlers index pages so an assistant can cite them in answers. User-triggered fetchers load a page when someone asks the assistant to.
If you want your content kept out of model training but still want to be cited in ChatGPT, Claude or Perplexity answers, block the training crawlers and leave the search ones open. The "Block AI training, keep AI search" button does exactly that.
- Training
- GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Google-Extended, Applebot-Extended, Bytespider, meta-externalagent.
- AI search
- OAI-SearchBot (ChatGPT search), Claude-SearchBot, PerplexityBot.
- User-triggered
- ChatGPT-User, Claude-User and Perplexity-User. These act for a person, and some operators say robots.txt may not apply to them.
Google-Extended does not affect Google Search
Google-Extended isn't a separate crawler. It's a token Google reads to decide whether content Googlebot already crawled can be used for Gemini models. Blocking it doesn't remove you from Google Search, and blocking Googlebot to opt out of AI would remove you from Search entirely.
Where to put robots.txt
The file must be named robots.txt, in lowercase, and live at the root of the host: https://example.com/robots.txt. A file in a subfolder is ignored. Each subdomain and protocol needs its own file, so blog.example.com reads blog.example.com/robots.txt, not the one on example.com.
On WordPress, an SEO plugin usually manages it. On Shopify, edit the robots.txt.liquid template. On Webflow and most site builders there's a robots.txt field in the site's SEO settings.
Common robots.txt mistakes
Most robots.txt problems come from a short list.
- Leaving Disallow: / in place after a site moves from staging to live
- Blocking CSS and JavaScript files, which stops Google rendering your pages properly
- Using robots.txt to remove pages from Google instead of noindex
- Putting the file anywhere other than the root of the domain
- Expecting Google to obey Crawl-delay or noindex lines, which it ignores
- Assuming paths are case-insensitive. /Admin and /admin are different rules
Testing your robots.txt
After uploading, open example.com/robots.txt in a browser to confirm it's live, then test the URLs that matter with our Robots.txt Tester. Google Search Console's robots.txt report shows the version Google last fetched and any errors it found. Google generally caches the file for up to 24 hours, so changes aren't picked up instantly.