When a page is missing from search results, there are only three possible reasons: it was never crawled, it was crawled but not indexed, or it is indexed but does not rank. Each has a different fix, so it pays to know which one you are looking at.
Crawling: the visit
A crawler is a program that downloads pages, the way your browser does, and follows the links it finds. Googlebot, Bingbot and the AI crawlers all work this way.
Crawlers find your pages through links from pages they already know, and through your sitemap. They do not see your site's menu the way a person does; they see HTML and links.
Things that stop a crawl:
robots.txtdisallows the path.- The page is only reachable by a form, a search box, or a link built by JavaScript after a click.
- Nothing links to it at all, and it is not in the sitemap. These are orphan pages, and they are more common than people expect.
- The server is slow or returns errors when the crawler visits.
How to check: search site:yoursite.com in Google to see roughly what is known, and look in Search Console under Pages for the reasons Google gives.
Indexing: the decision to keep it
After a crawl, the search engine decides whether to store the page. It skips pages that say noindex, pages that duplicate another page, and pages it judges too thin to be worth keeping.
Two mechanisms matter here:
- Canonical links.
<link rel="canonical" href="...">tells the engine which URL is the real one when the same content is reachable at several addresses (tracking parameters, trailing slashes, printer versions). Pointing every page's canonical at the homepage, which some themes do by mistake, tells the engine your whole site is one page. - Duplicate content. Identical titles, descriptions and body text across many pages make the engine pick one and drop the rest.
Ranking: the order
Only pages in the index can rank. Ranking compares your page against others for a specific search: does it answer the question, is it readable and fast, do other sites link to it, is it current.
This is where content work happens, and where most people start, which is why so many sites stall. Fix crawling and indexing first, or you are optimising a page nobody can see.
JavaScript, the common trap
If your page arrives almost empty and fills itself in with JavaScript, Google can usually still render it, but with delay and at a cost. Most AI crawlers do not run JavaScript at all, so they see an empty page.
To check what a crawler sees, view the raw HTML rather than the rendered page:
curl -s https://yoursite.com/your-page | head -100
If your main text is not in there, use server-side rendering or static generation for the content that matters.
Crawl budget, and when to care
Large sites get a rough limit on how much is crawled per visit. You waste it on: redirect chains, broken links, endless filter combinations (?colour=red&size=xl&page=7), and slow responses.
Under a few thousand pages this rarely matters. Over that, tidy the links and block filter parameters that create infinite variations.
A quick diagnosis
| Symptom | Likely stage | First check |
|---|---|---|
Page never appears, even with site: search |
Crawling | robots.txt, internal links, sitemap |
site: finds it, but it ranks for nothing |
Ranking | content, intent match, links |
| Search Console says "Discovered, not indexed" | Indexing | thin or duplicate content |
| Search Console says "Blocked by robots.txt" | Crawling | robots.txt rules |
Rankwell's site audit covers the crawling and indexing side for you: it crawls your site the way a search engine does, then reports blocked pages, redirect chains, duplicates, orphan pages and pages that need JavaScript to show their text.