AI assistants answer questions by fetching pages, reading them and quoting a few. Every step of that can fail on your site for reasons that have nothing to do with how good your content is.
This guide covers the checks that decide whether you are eligible to be quoted at all.
1. Let the right crawlers in
There are three kinds of AI crawler, and they deserve different decisions:
| Kind | Examples | What blocking them means |
|---|---|---|
| Search crawlers | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot, Applebot |
You cannot appear in those AI answers at all |
| User-triggered fetchers | ChatGPT-User, Claude-User, Perplexity-User |
The assistant cannot open your page when a person asks it to |
| Training crawlers | GPTBot, ClaudeBot, Google-Extended, CCBot, Meta-ExternalAgent |
Your content is not used to train future models |
Blocking training crawlers is a legitimate choice and does not hide you from AI answers. Blocking search crawlers does.
A robots.txt that allows answers but not training looks like this:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: *
Allow: /
Sitemap: https://yoursite.com/sitemap.xml
2. Check your firewall, not just robots.txt
Many CDNs added "block AI bots" switches, and some turn them on by default. Your robots.txt can welcome every crawler while your firewall quietly returns a challenge page.
Test it from a terminal:
curl -s -o /dev/null -w "%{http_code}\n" -A "OAI-SearchBot/1.0" https://yoursite.com/
A 403, 429 or a challenge page means you are invisible to that assistant. Check your CDN's bot settings and allow verified AI search crawlers. Rankwell runs this test for you and reports the result.
3. Put the text in the HTML
Most AI crawlers do not run JavaScript. If your page arrives as an empty shell that fills itself in, they see nothing to quote.
curl -s https://yoursite.com/your-page | wc -w
A few hundred words means you are fine. Twenty means your content is invisible to them. Server-side rendering or static generation fixes it.
4. Say who you are, in a way machines can parse
AI systems connect a brand name to a website through structured data and public records. Add Organization data on the homepage with your name, logo, URL and sameAs links to your official profiles. If your organisation is notable enough to have a Wikidata item, its "official website" property is one of the strongest connections you can make.
5. Write sentences that can be quoted
An assistant lifts a sentence or two. Give it sentences that stand alone:
- Quotable: "A pour over uses about 60 g of coffee per litre of water."
- Not quotable: a table cell containing "1:16".
Put the direct answer near the top of the page, use headings phrased as questions, and keep one idea per paragraph.
6. Keep it current
Assistants prefer recent sources for anything that changes. Show a real update date on the page, keep lastmod accurate in your sitemap, and revise prices and product names rather than leaving them stale.
7. Publish llms.txt, with realistic expectations
/llms.txt is a short Markdown map of your site for AI tools. It is a proposal, not a standard: some tools read it, and the big assistants have not confirmed they do. It costs ten minutes, so it is worth having, but it is not a substitute for anything above.
Check your own site
Rankwell's AI search readiness check runs all of this against any domain: 16 crawlers checked against robots.txt, the firewall test, the no-JavaScript test, structured data, sameAs, llms.txt, sitemap freshness and Wikidata. It returns a score out of 100 with the failing checks first and the fix for each.
Next: how AI assistants choose which pages to cite, and whether to block AI crawlers at all.