robots.txt is a plain text file at the root of your site that tells crawlers which paths they may fetch. It is short, it is read before almost anything else, and one wrong line in it can keep an entire site out of search for months.
This guide covers how the rules are actually read, what a sensible file looks like, and the mistakes worth checking for today.
Where it lives and what it covers
The file must sit at the root of the host: https://yoursite.com/robots.txt. A file at /blog/robots.txt is ignored.
Each host has its own. www.yoursite.com, shop.yoursite.com and the http:// version are separate as far as crawlers are concerned, so each needs its own file if each serves pages.
The format is standardised in RFC 9309. Google, Bing and the major AI crawlers follow it. Paths are case-sensitive, so Disallow: /Admin does not block /admin.
A safe example to start from
For most small and medium sites, this is enough:
User-agent: *
Disallow: /admin/
Disallow: /cart
Disallow: /search
Disallow: /*?sort=
Sitemap: https://yoursite.com/sitemap.xml
Line by line:
User-agent: *starts a group that applies to every crawler not named elsewhere.- Each
Disallownames a path prefix the group should not fetch./admin/blocks everything under that folder./cartblocks/cart,/cart/and/cart-summarytoo, because it is a prefix match. /*?sort=uses a wildcard:*matches any run of characters. This blocks sorted versions of listing pages, which are near-copies of the unsorted page.Sitemapgives the full URL of your sitemap. It sits outside any group and can appear anywhere in the file.
The two special characters are * (any characters) and $ (end of the URL). Disallow: /*.pdf$ blocks URLs ending in .pdf but not /guide.pdf?download=1.
How conflicting rules are resolved
Two rules people get wrong:
A crawler obeys one group only. If there is a group for Googlebot, Googlebot reads that group and ignores the * group entirely. Rules are not merged. So if you add a Googlebot group to allow one extra path, you must repeat every Disallow from the * group in it, or Googlebot is suddenly allowed into your admin and cart pages.
The longest matching rule wins. Given these two lines:
Disallow: /shop/
Allow: /shop/sale
/shop/sale/shoes matches both, and the Allow is longer, so it is allowed. If an Allow and Disallow are exactly the same length, Google picks the less restrictive one, the Allow.
robots.txt is not noindex
This is the most common misunderstanding, and it causes real damage.
Disallow stops a crawler from fetching a page. It does not stop the page being indexed. If other pages link to a disallowed URL, Google can still list it in results, usually with no description, because it knows the URL exists but was never allowed to read it. Search Console reports this as "Indexed, though blocked by robots.txt".
noindex works the other way round. It is a meta tag or HTTP header on the page itself:
<meta name="robots" content="noindex">
For Google to see that tag, it has to crawl the page. So if you block a page in robots.txt and add noindex, the noindex is never read.
The rule of thumb:
- To keep a page out of results, allow the crawl and use
noindex. - To save crawl effort on pages that do not matter (endless filters, internal search), use
Disallow. - To keep something private, use a password. robots.txt is public and anyone can read it, so listing
/secret-launch-plans/in it advertises the folder.
How search engines find your pages explains the difference between crawling and indexing in more detail.
The lines that hide a site
These are the mistakes worth looking for, in rough order of how much harm they do.
Disallow: / under User-agent: *. This blocks everything. It usually arrives from a staging site, where it was correct, and gets copied to production during a launch. It is the first thing to check after any redesign or move, as covered in keeping rankings through a migration.
An empty Disallow read the wrong way. Disallow: with nothing after it means "nothing is disallowed". Disallow: / means "everything is". One character apart.
Blocking CSS and JavaScript. Old advice said to block /wp-includes/ or /assets/. Google renders pages much like a browser does, and if it cannot load your stylesheets and scripts it may see a broken or empty page. Leave them crawlable.
A named group that forgets the basics. As above: a Googlebot group that says only Allow: /special lets Googlebot crawl everything else too, including the paths you blocked for everyone.
A server error on the file itself. If robots.txt returns a 5xx error, Google stops crawling the site, assuming it should not proceed without knowing the rules. A 404 is treated as "no rules, crawl everything". A file that times out or errors because of a firewall rule can therefore stop crawling with nothing visibly wrong on the site.
Blocking AI crawlers by accident. Some hosts and plugins add blanket rules for AI user agents. Whether that is what you want is a separate decision, covered in should you block AI crawlers?.
Testing a change before it goes live
Google caches robots.txt for up to about a day, so a mistake can stay in effect after you fix it, and a fix can take a day to be noticed.
Before deploying a change:
- Fetch the live file and save a copy, so you can roll back.
- List a handful of URLs that must stay crawlable: the homepage, a product or article, a category page, your CSS file.
- Check each against the new rules. The robots.txt report in Search Console shows what Google last fetched and any lines it could not parse, and the URL Inspection tool tells you whether a specific URL is blocked.
- After deploying, fetch the file from outside your network to confirm the CDN serves the new version.
curl -s -o /dev/null -w "%{http_code}\n" https://yoursite.com/robots.txt
That should print 200, or 404 if you deliberately have no file. Anything in the 500s needs fixing first.
Rankwell's site audit reads your robots.txt before it crawls, follows its rules the way a search engine would, and reports how many pages it was kept away from, so a stray Disallow shows up as a number rather than a mystery.