Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot collects training data. ChatGPT's search results are built by a separate crawler, OAI-SearchBot. You can block GPTBot to opt out of training while still allowing OAI-SearchBot so your pages can be found and cited in ChatGPT search.

Does blocking Google-Extended affect Google Search or AI Overviews?

No. Google-Extended only controls whether your content is used for Gemini training and grounding. Google Search and AI Overviews are crawled by Googlebot, which Google-Extended rules don't touch.

Do AI crawlers actually obey robots.txt?

robots.txt is a voluntary standard. The major AI companies listed here publish their crawler names and say they respect robots.txt rules, but it can't technically force any crawler to comply. Blocking at the server or firewall level is the only enforceable option.

Where does robots.txt go?

It must be a plain-text file at the root of your domain, for example https://example.com/robots.txt. Rules in a robots.txt anywhere else are ignored, and each subdomain needs its own file.

AI Crawler robots.txt Builder (GPTBot, ClaudeBot, PerplexityBot)

Choose which AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more — can access your site, and get ready-to-paste robots.txt rules.

Free. Runs entirely in your browser — nothing you enter is sent to our server.

Training, search and user fetches are different crawlers

Most AI companies run more than one crawler, and they do different jobs. Training crawlers like GPTBot and ClaudeBot collect content to train future models. Search crawlers like OAI-SearchBot and PerplexityBot index pages so an assistant can find and cite them in answers. User-requested fetchers like ChatGPT-User retrieve a page when someone asks about it directly.

That split matters. Blocking every AI crawler opts you out of training, but it also removes your pages from the AI answers that increasingly send traffic and brand mentions. For most publishers, the "block training, allow AI search" preset is the balanced choice. See our guide to generative engine optimization for why AI search visibility is worth keeping.

How to use the generated rules

Paste the output into your existing robots.txt rather than replacing it. Each blocked crawler gets its own group, and a crawler that matches a named group follows only that group, so your general User-agent: * rules still apply to every other bot. After you publish, open /robots.txt on your domain to confirm the rules are live.

Blocking AI search crawlers also takes you out of the citation pool entirely. If you want to be cited, the bigger levers are on the page itself — see how to structure content so ChatGPT quotes you.

AI Crawler robots.txt Builder

Your robots.txt rules

Training, search and user fetches are different crawlers

How to use the generated rules

Frequently asked questions