Free Noindex Checker: Find Pages Blocked From Google's Index
Table of Contents
- The four places a noindex rule can hide
- The robots.txt conflict that cancels your noindex entirely
- Meta robots tag or X-Robots-Tag: picking the right one
- Why noindex in robots.txt stopped working on September 1, 2019
- Reading your noindex checker result: four outcomes and the fix for each
- Where accidental noindex tags actually come from
- Run the noindex checker on one URL, then check the whole site
A page disappears from Google. You check the sitemap, the canonical, the internal links, the server logs. Everything looks healthy. Then you open the page source and find one line in the head that explains the whole thing: <meta name="robots" content="noindex">. A noindex checker answers that in one step, without you guessing which of four places the rule is hiding in.
Paste a URL into the noindex checker above and you get the answer in one step: whether that page carries a noindex rule, which of the four possible places it lives in, and whether Googlebot can even reach it to read the rule. That last part is where most noindex checks go wrong, and it is the reason a page can be tagged noindex and still sit in search results for months.
The four places a noindex rule can hide
A noindex directive is not a single setting. It can arrive from any of these, and a page can carry more than one at once with the strictest rule winning.
The robots meta tag. <meta name="robots" content="noindex"> in the <head>. This is the common case and the one most people check by hand.
A crawler-specific meta tag. Swap robots for googlebot and the rule applies to Google only. A page can look clean to a generic checker and still be noindexed for Google specifically.
The X-Robots-Tag HTTP header. Returned in the response, invisible in the page source. You will never spot this by pressing Ctrl+U. It is also the only way to noindex a non-HTML file such as a PDF or an image.
A rendered tag injected by JavaScript. The raw HTML has no noindex, the rendered DOM does. Google processes the rendered version, so the tag counts.
Checking only the first of those is why "I looked and there was no noindex tag" is such a common and such an expensive mistake.
The robots.txt conflict that cancels your noindex entirely
Here is the failure nobody warns you about, and it bites in both directions.
Say you want a page out of Google. You add a noindex tag, then you also add a Disallow line in robots.txt to be extra safe. That second step undoes the first. The disallow stops Googlebot from fetching the page, so Googlebot never reads the noindex tag, so the URL can keep appearing in search results. Google's own documentation is blunt about it:
"For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler."
Google goes on to spell out the consequence: if the crawler cannot access the page, it "will never see the noindex rule, and the page can still appear in search results." Those results usually show up with no description, which is the classic fingerprint of a URL Google knows about but has never been allowed to read. Source: Google Search Central, Block Search Indexing with noindex.
The reverse case costs more traffic. A page you want indexed picks up a stale Disallow from an old staging rule or an over-broad wildcard. The page has no noindex tag at all, so a basic noindex checker returns a clean result, and you spend a week looking in the wrong place.
So the useful check is not "does this page have a noindex tag." It is "can Google fetch this page, and if it can, what does it find." Two questions, and you need both answers to act.
Meta robots tag or X-Robots-Tag: picking the right one
Situation | Use | Why |
|---|---|---|
A normal HTML page | Meta robots tag | Easiest to set per page and easiest for anyone to verify in the source. |
A PDF, image, or other non-HTML file | X-Robots-Tag header | There is no <head> to put a meta tag in. |
A whole pattern of URLs (search results, filtered views) | X-Robots-Tag header | Set once at the server or CDN layer with a pattern match instead of editing hundreds of templates. |
You need it live in the next five minutes | Whichever your stack ships faster | Both carry identical weight with Google. Neither is stronger. |
One rule that trips up teams: the two are not additive. A page with a meta noindex and no header is exactly as noindexed as a page with both.
Why noindex in robots.txt stopped working on September 1, 2019
Older SEO guides and a surprising number of legacy robots.txt files still contain lines like Noindex: /private/. That directive does nothing today.
Google announced in July 2019 that it was retiring all code handling unsupported and unpublished robots.txt rules, with the change taking effect on September 1, 2019. Google's reasoning was that these rules were never documented, and that their use was contradicted by other rules in all but 0.001% of robots.txt files on the internet, meaning site owners were mostly hurting their own search presence by accident.
If you inherit a site and find a Noindex: line in robots.txt, treat every URL under it as fully indexable until you have verified otherwise. That line has been decorative for years.
Reading your noindex checker result: four outcomes and the fix for each
What the check returns | What it means | Do this |
|---|---|---|
Indexable, no noindex found | Nothing is blocking indexing at the page level. | If the page still is not ranking, the problem is elsewhere: canonical pointing away, thin content, or no internal links. |
Noindex found in the meta tag | Working as configured, intentionally or not. | If unintended, remove the tag, then request indexing in Search Console rather than waiting for a recrawl. |
Noindex found in the X-Robots-Tag header | Set at the server, CDN, or framework layer, not in your CMS. | Check your hosting config, edge rules, and any preview or password mode before touching page templates. |
Blocked by robots.txt | Google cannot read the page, so any noindex on it is irrelevant. | Decide the outcome first. To deindex: remove the disallow and let the noindex be seen. To index: remove the disallow and confirm the page is clean. |
Where accidental noindex tags actually come from
In practice, the same handful of causes account for most of them.
A staging environment shipped to production. Staging sites carry a site-wide noindex, and a deploy occasionally carries it along. This is the single most damaging version because it hits every page at once.
A CMS visibility toggle. WordPress has a "Discourage search engines from indexing this site" checkbox in reading settings. One click, whole site, no warning.
Plugin or theme defaults on secondary templates. Tag archives, author pages, and paginated pages are often noindexed by default. Sometimes correct, sometimes it removes pages that were earning traffic.
Edge or middleware rules. A CDN rule written to protect a preview branch that quietly matches more paths than intended.
An intentional noindex nobody documented. Someone deindexed a section two years ago for a good reason. That reason left the company.
The pattern worth noticing: almost none of these show up on the page a human would think to inspect. They are set globally, at build time, or at the edge, which is exactly why spot-checking one URL by hand keeps missing them.
Run the noindex checker on one URL, then check the whole site
Once a single URL is clear, the same rules apply site wide, and our free robots.txt and sitemap checker will confirm nothing is blocking the crawler before you look at page level tags. Single-URL checks are the right tool when you are debugging one page you already suspect. They are the wrong tool for the failure mode that costs real traffic, which is a rule applied across a template or an entire environment.
If a check comes back with an unexpected noindex, assume it is not alone. Crawl the full site, group the results by template, and look at whether the tag tracks with a URL pattern. A noindex on one product page is a mistake. A noindex on every page under /products/ is an incident.
CrawlTide crawls your whole site and flags indexability problems across every page, including X-Robots-Tag headers and robots.txt conflicts, then scores each by severity so you know which to fix first. Start free at crawltide.com.
CrawlTide Team
Written by the CrawlTide team, SEO practitioners who build the platform.
Suggested Reading
Organization Schema for AI Search: What It Fixes and What It Won't
Organization schema tells AI and search engines who publishes your site. What it declares, a real coverage result, and where it stops helping.
7 min read
Breadcrumb Schema (BreadcrumbList): How It Helps AI and Search Read Your Site Structure
Breadcrumb schema tells AI and search engines where a page sits in your site. What BreadcrumbList encodes, the common mistakes, and where it helps.
7 min read
What Is AI Visibility? How to Check If Your Website Shows Up in AI Search
AI visibility measures whether your brand shows up in AI-generated search experiences like ChatGPT and Google AI Overviews. Here’s how to check it and improve it.
9 min read