The noindex tag tells search engines they may crawl a page but must not show it in search results. Nofollow, at page level, tells them not to use the links on that page; together they are two independent switches in one robots meta tag, <meta name="robots" content="noindex, nofollow">, giving four possible instructions.
It is one of the smallest pieces of code on a website and one of the easiest ways to remove your own pages from Google. A single misplaced noindex — left over from a staging site, or applied too broadly by a plugin default — can take out traffic on a site with nothing else wrong with it. The fix, once you find it, takes seconds.
This guide covers what each switch does, all four combinations with examples, when to use noindex and when not to, how to set it in HTML, HTTP headers and WordPress, how it differs from robots.txt and canonical tags, and what to do when Search Console says a page is "Excluded by noindex tag".
What the noindex tag actually does
Noindex is an instruction: crawl this page if you like, but do not put it in your index. When Googlebot crawls a page and finds a noindex directive, it drops that page from search results, however many sites link to it.
There are two ways to send it, and they do the same thing.
The meta tag, inside the <head> of an HTML page:
<meta name="robots" content="noindex">
The HTTP header, called X-Robots-Tag, sent with the server's response:
HTTP/1.1 200 OK X-Robots-Tag: noindex
The meta tag only works on HTML pages. The header works on anything your server returns — PDFs, images, video files — which makes it the only option for non-HTML resources. If both are present with conflicting values, Google follows the more restrictive one.
One detail trips people up constantly: a page has to be crawlable for noindex to work. If you block a URL in robots.txt, Googlebot never requests it, so it never sees the noindex. Worse, a robots.txt-blocked URL can still be indexed as a bare URL with no snippet if other pages link to it. That is the single most common noindex mistake, and we come back to it below.
The two independent switches
Index/noindex controls whether the page itself can appear in search results. Follow/nofollow, at page level, controls whether Google should use the links on the page to discover other pages. They are separate decisions about separate things: one is about this page's visibility, the other about what this page does for the pages it links to.
If you do not add a robots meta tag at all, the default is index, follow. You only add the tag to opt out of one or both.
Two caveats worth knowing before the combinations:
- The page-level
nofollowin a meta tag is different from the link-levelrel="nofollow"attribute on a single<a>tag. The meta version applies to every link on the page; the attribute applies to one link. - Since 2019, Google treats nofollow — both versions — as a hint, not a rule. It usually respects it, but may still crawl a nofollowed link it finds valuable. Do not rely on nofollow as an access control.
The four combinations
Index, follow — the default
<meta name="robots" content="index, follow">
Google can show the page and follow its links. You almost never need to write this; it is the default. Use it for every normal public page you want to rank: posts, landing pages, category pages, the homepage.
Noindex, follow — hide the page, keep the links working
<meta name="robots" content="noindex, follow">
Do not show this page in results, but still use its links to find other pages. Use it for pages you do not want in search that still sit inside your link structure: internal search results, thin tag archives, a filtered or re-sorted copy of a category. Paginated archives (page 2, 3 and onward of a blog or category) do not belong on this list. Google's pagination guidance is to keep them indexable, each with a canonical pointing to itself, because they are how crawlers reach older posts. Nobody searches for "blog page 3", but Google still needs to crawl it.
Index, nofollow — rare, and usually a mistake
<meta name="robots" content="index, nofollow">
Show the page, but do not follow any of its links. There is rarely a good reason for this at page level. The plausible case is a page full of user-submitted or sponsored outbound links — and even then, rel="ugc", rel="sponsored" or rel="nofollow" on the specific links is more precise. If you find this combination on your site and did not set it deliberately, investigate: it often comes from a plugin rule applied to a whole content type.
Noindex, nofollow — the full block
<meta name="robots" content="noindex, nofollow">
Equivalent shorthand:
<meta name="robots" content="none">
Do not show the page and do not use its links. Use it for pages with no search value that are not useful crawl paths either: a private campaign landing page, an internal tool that ended up crawlable, staging content that leaked into production. For thank-you pages that link back into the site, noindex, follow is usually the better call.
When to use noindex
Noindex earns its place on pages that are useful to visitors or to the site but add nothing in a Google search.
Thin or low-value pages. A tag archive with three posts, an author archive on a single-author blog, auto-generated aggregate pages. They can keep working for navigation without competing for a place in results.
Duplicates you cannot consolidate with a canonical. A print-friendly version, or a legacy page you cannot merge. If the pages are not similar enough for a canonical to be a clean fit, noindex the secondary version.
Staging and development sites. Noindex the whole environment as a standing policy — better still, put it behind a password — and check it after every deployment change.
Thank-you and confirmation pages. Nobody searches for them, and a ranking thank-you page confuses visitors and pollutes conversion tracking.
Faceted navigation. Filter combinations (size × colour × price × sort) multiply fast. Noindex the low-value combinations and keep the ones that match real search demand.
Internal search results. Dynamic, thin and of no value to someone arriving from Google. Most WordPress SEO plugins noindex them by default.
When not to use noindex
Do not noindex pages you want to rank. Obvious, and it happens constantly — a "noindex everything" setting carried over from staging, or a plugin default applied more widely than anyone checked. Spot-check important pages after every migration, redesign and plugin update.
Do not noindex a page instead of fixing it. If a page is thin or outdated, improving it is usually the better long-term move. A page that ranked and lost its noindex months later rarely picks up where it left off.
Do not combine noindex with a canonical pointing elsewhere. One says "consolidate signals onto that URL", the other says "remove me". Google generally prioritizes noindex, so the page drops out and the signals you meant to pass along can be lost. Consolidation: canonical alone. Removal: noindex alone.
Do not also block the page in robots.txt. The block stops Google from seeing the noindex, and the URL can still be indexed without a snippet. To keep something out of the index, let it be crawled and rely on noindex. Save robots.txt Disallow for URLs you do not want requested at all.
Do not use noindex to deprioritize. Noindex is binary. If you want one version of a page preferred over another while both stay live, that is a canonical's job.
How to add a noindex tag
In raw HTML, inside the <head>:
<meta name="robots" content="noindex">
To target only Google and leave other search engines to decide for themselves:
<meta name="googlebot" content="noindex">
With an HTTP header, configure your server or CDN to send X-Robots-Tag: noindex for the relevant paths or file types — in .htaccess or the server config on Apache, in the server block on Nginx, or in your CDN's response-header rules. This is the only option for PDFs and other non-HTML files.
In Yoast SEO, open the post, go to the Yoast panel's Advanced tab and set "Allow search engines to show this content in search results?" to No. For a whole content type or taxonomy, change it in Yoast's settings instead of editing every page.
In Rank Math, open the post, go to the Rank Math meta box's Advanced tab and tick "No Index" under Robots Meta. For defaults by post type or taxonomy, use Rank Math SEO → Titles & Meta.
In AIOSEO and SEOPress, the same control lives in the plugin's panel under the post editor (an Advanced tab with a noindex option), with site-wide defaults per content type in the plugin settings. Still choosing a plugin? See Rank Math vs Yoast, AIOSEO and SEOPress.
Whichever method you use, the plugin is generating the same meta tag or header; it just gives you a UI for it. Keep a short written record of which sections are noindexed and why. Six months from now, "we noindexed those filter pages on purpose" is a much better answer than a confused scramble.
Noindex vs robots.txt vs canonical
These three get lumped together, but they work at different layers.
| Tool | Controls | Google still crawls the page? | Use it to |
|---|---|---|---|
robots.txt Disallow | Crawling | No | Stop crawlers wasting requests on junk URLs |
noindex | Indexing | Yes — it has to, to see the tag | Keep a page out of search results |
rel="canonical" | Which duplicate ranks | Yes | Merge signals from near-duplicate pages onto one URL |
robots.txt controls crawling, not indexing. A disallowed URL can still appear in results, without a snippet, if it is linked from elsewhere. Noindex controls indexing directly, but only works if the page is crawlable. A canonical is a hint for consolidation, not removal: both pages stay live and crawlable, and Google may still index the non-preferred one if it decides they are not duplicates.
On WordPress, the WordPress robots.txt guide covers the default file and how to fix "Indexed, though blocked by robots.txt".
The decision is simple: do not want a URL requested at all → robots.txt. Want a page gone from results → noindex. Want near-duplicates merged onto one URL → canonical. And never combine noindex with either of the other two on the same page.
Mistakes that accidentally deindex important pages
The combinations are simple one at a time. The damage comes from settings applied at scale, inherited from templates, or left over from an earlier state of the site.
Bulk plugin rules applied too broadly. "Noindex all tag archives" is reasonable, until the rule also sweeps up a custom taxonomy that held real landing pages.
Staging settings that ship to production. A sitewide noindex, nofollow meant for staging goes live with the launch. Nobody notices until organic traffic falls and someone finally reads the page source.
If your whole site has dropped out of Google, start with why your website is not showing up on Google, which checks WordPress's "Discourage search engines" setting first.
Switching CMS or SEO plugin. A migration can reset per-page robots settings to the new system's defaults, flipping carefully set pages either way.
Nofollow at page level instead of link level. Someone wants to nofollow affiliate links and sets the whole page's meta to nofollow, so navigation, related posts and breadcrumbs stop passing signals too.
Conditional logic bugs. A rule such as "noindex posts under 300 words" that miscounts content from a page builder can noindex long posts without anyone touching a setting.
Duplicated templates. A new page built by copying an old one that was noindexed for a good reason at the time, with the setting hidden in a field nobody looks at.
Temporary rules that become permanent. Translated sections noindexed during a rollout and never switched back on.
The pattern: the damage rarely comes from deliberately noindexing the wrong page. It comes from a rule applied broadly, a default nobody updated, or logic that misfires on an edge case. That is why spot-checking single pages is not enough.
Why WordPress sites are especially exposed
On WordPress, robots settings live in plugin settings and post meta, not in the visible content, and two or three plugins can often touch them — the SEO plugin, a page builder, sometimes a security or caching plugin. When they disagree, or an update changes a default, the tag on a given page is decided by whichever runs last, and the only way to know is to read the rendered HTML.
Two more details make audits harder than they look:
- Rules for different crawlers combine.
<meta name="robots" content="nofollow">plus<meta name="googlebot" content="noindex">means Googlebot applies both: noindex and nofollow. If your stack can print more than one robots tag, check all of them. - Placement matters. A robots meta tag only counts inside the
<head>. One that ends up in the<body>— it happens with some builders and shortcodes — is ignored, even though it looks right in the source.
"Excluded by noindex tag" in Search Console
Search Console's Indexing → Pages report lists URLs under "Excluded by noindex tag" when Google crawled them and found a noindex. Whether that is a problem depends entirely on the page.
- Export the list and sort it into "meant to be hidden" (thank-you pages, archives, internal search) and "should rank".
- For pages that should rank, open URL Inspection to confirm Google saw the noindex, then view the live page source and search for
robots. - Find where it comes from. Check the post's SEO plugin panel first, then the plugin's content-type defaults (a sitewide rule is the usual cause), then Settings → Reading → "Discourage search engines from indexing this site", which adds a sitewide noindex.
- Remove it, then request indexing in URL Inspection. Google has to recrawl the page before it can return to results; for important pages, requesting indexing speeds that up.
If the list only contains pages you meant to hide, nothing is wrong — the report is just confirming your settings work.
How to verify noindex, page by page and sitewide
View source. Search the rendered HTML for name="robots". This catches templating bugs where a setting looks right in WordPress but never reaches the output.
URL Inspection. Shows whether Google saw a noindex on its last crawl and how the URL is indexed. More reliable than checking your own site, because it reflects what Googlebot actually received.
Site-wide crawl. Screaming Frog or Sitebulb report the robots value for every URL in one export. Filter for noindex and nofollow and compare against your list of important pages. This is the only practical way to catch a bulk rule applied too broadly.
Ongoing monitoring. A crawl catches today's problems; robots tags can change after any plugin update. On a WordPress site, RankHive reads your Search Console data every week and flags pages Google isn't indexing and the noindex and robots mistakes behind them. Where the fix lives in a post, a page or robots.txt, it drafts the change and waits for your approval before anything ships.
Allow for lag. A newly noindexed page does not vanish immediately: Google has to recrawl it, which can take days to weeks depending on how often the page is crawled. The same applies in reverse when you remove a noindex.
Revisit robots settings when a page changes
The right setting for a page is not fixed forever. A campaign page that was correctly noindexed while it was temporary may deserve indexing once it becomes a lasting resource. A thin support article that started as noindex can be indexed once someone expands it. Make "check the robots setting" part of every content refresh. It costs nothing and catches pages that were rightly hidden a year ago and are now worth surfacing.
FAQ
Does noindex remove a page from Google immediately?
No. Google has to recrawl the page to see the directive, which can take days to weeks. You can request a recrawl with URL Inspection, and Google's Removals tool can hide a URL quickly in urgent cases, but the noindex is what keeps it out.
What does content="none" mean?
It is shorthand for noindex, nofollow.
Can people still visit a noindexed page?
Yes. Noindex only affects search results. Anyone with the URL, or following a link on your site, can still reach the page.
Does nofollow stop Google from crawling a page?
No. Nofollow is a hint about passing signals through links. Google can still discover the linked page through a sitemap, another link or an external site.
Should I use noindex or robots.txt for a staging site?
Noindex, and ideally password protection or an IP allowlist on top. A robots.txt block does not reliably keep URLs out of the index; it only stops crawling.
Why does Search Console say "Excluded by noindex tag" when I never added one?
Usually a plugin default, a theme setting, or a setting left over from a migration or staging. Check your SEO plugin's content-type defaults and WordPress's "Discourage search engines" setting before assuming a single page is broken.
Can I set different rules for different search engines?
Yes. Replace robots with the crawler's name, for example <meta name="googlebot" content="noindex">. Where rules for several crawlers apply, they combine.
Will noindex hurt my crawl budget?
Not meaningfully for most sites. Google still crawls noindexed pages occasionally to confirm the tag. On very large sites, noindexing huge numbers of low-value URLs mainly helps by reducing how often Google bothers to recrawl them.
Should thank-you pages and internal search results be noindex, nofollow?
Noindex, yes. Nofollow depends on whether the page links to anything useful; a thank-you page that links back into the site is usually better as noindex, follow.
How do I know if a plugin update changed my robots settings?
Run a fresh crawl after plugin and theme updates and compare the robots values with your last known-good export, or watch Search Console's Indexing report for new "Excluded by noindex tag" entries.
Related reading
- Meta tag noindex vs robots.txt
- Canonical tags and duplicate content
- WordPress technical SEO
- WordPress SEO Audit: 10 Steps That Ship Fixes
Your job is to know which of your pages carry noindex and whether each one should. Check that by hand with the crawl and Search Console steps above, or let RankHive watch your Search Console data for these mistakes every week and ship the fixes after you approve them.