"Block AI" is offered as one switch by many hosts, but it is at least three separate decisions. Getting them mixed up is how sites disappear from AI answers by accident.
The three decisions
1. Training. Should your pages be used to train future models? Crawlers: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent.
Arguments to block: you sell the content itself, or you object on principle. Arguments to allow: models that know your brand mention it more readily, and the cost to you is a crawl.
2. Citation in answers. Should assistants be able to find and quote your pages when someone asks a relevant question? Crawlers: OAI-SearchBot, Claude-SearchBot, PerplexityBot, plus Googlebot and Bingbot, whose indexes feed several assistants.
Blocking these removes you from AI answers. For most businesses that is the same as leaving a growing channel.
3. Fetch on request. Should an assistant be able to open your page when a user pastes the link or asks it to check? Crawlers: ChatGPT-User, Claude-User, Perplexity-User.
Blocking these means a person who explicitly asks about your page gets "I could not access that site", which reads as a fault on your side.
A sensible default
Most sites want: training your choice, citation and fetch allowed.
# Not for training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Welcome to find, quote and open pages
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: *
Allow: /
Sitemap: https://yoursite.com/sitemap.xml
Publishers whose product is the writing itself often block training and allow citation, so their brand still appears with a link back. Shops and service businesses usually allow everything, because being known is worth more than the text.
Do not forget the firewall
robots.txt is a request. Your CDN is a wall. Cloudflare and others added AI-bot blocking that operates before robots.txt is ever read, and it can be on without anyone deciding to turn it on.
Check both. A site can have a perfectly written robots.txt and still return a challenge page to every AI crawler.
What blocking cannot do
- It does not remove content already used in earlier training runs.
- It does not stop scrapers that ignore robots.txt. Only server-side rules and rate limits affect those.
- It does not stop a person copying your page into a chat window.
Review it once a quarter
New crawler names appear regularly. A robots.txt written a year ago probably says nothing about the crawlers that matter now, and silence means "allowed" for the ones you never listed.
What your logs tell you
Before deciding anything, look at what these crawlers are actually doing on your site. Your server or CDN logs show the user agent, the pages fetched, and how often.
Three questions worth answering from the logs:
- Which AI crawlers visit at all? Some will be absent because they never found you, which is a different problem from being blocked.
- What are they fetching? If a crawler is spending its visits on filter combinations and search result pages, tidy those up: they cost you crawl capacity and teach the model nothing useful.
- How heavy is it? A crawler making a few thousand requests a day on a small server is worth rate limiting, whatever you decide about training.
Rate limiting is the moderate option people forget. You can allow a crawler and still cap it, which keeps you visible without letting anyone hammer your origin.
Write the decision down
Whatever you choose, record it in a comment at the top of robots.txt: who decided, when, and why. Six months later, somebody will ask whether the block was deliberate, and the file itself is the only place that answer survives staff changes and agency handovers.
Rankwell checks 16 AI crawlers against your robots.txt, runs the firewall test, and separates "for AI answers" from "for AI training" so you can see which decision you actually made. Run it on your site in domain overview.