Two words, four combinations, and a surprising number of ways to shoot yourself in the foot. That's the robots meta tag in a nutshell.
Most people learn about noindex and nofollow separately, as if they're two unrelated settings. They're not. They're two independent switches that live in the same tag, and together they form four distinct instructions you can give a search engine. Get the wrong combination on the wrong page, and you can quietly deindex your best content or leak link authority to pages you never meant to.
This post walks through all four combinations, what each one actually tells Google, when you'd use it, and how the syntax looks in both raw HTML and the WordPress plugin dashboards most people actually use.
The two independent switches
Before the combinations, a quick reset on what each word controls, because this is where the confusion starts.
Index/noindex controls whether the page itself is eligible to show up in search results. Follow/nofollow, at the page level, controls whether Google should use the links found on that page to discover other pages. These are separate decisions about separate things — one is about this page's visibility, the other is about what this page does for the pages it links to.
The tag lives in the <head> of your HTML:
<meta name="robots" content="noindex, nofollow">
You can also send it as an HTTP header, useful for non-HTML files:
X-Robots-Tag: noindex, nofollow
If you don't specify a meta robots tag at all, the default behavior is index, follow — Google will index the page and follow its links, no tag required. You only need to add the tag when you want to opt out of one or both defaults.
One more thing worth knowing before we get into combinations: the page-level nofollow in a meta tag is different from the link-level rel="nofollow" attribute you put on an individual <a> tag. The meta tag version applies to every link on the page, blanket. The attribute version applies to just that one link. Also, since 2019, Google treats nofollow — both versions — as a hint, not an absolute rule. It usually respects it, but it reserves the right to still crawl a nofollowed link if it finds it valuable enough. Keep that caveat in mind; it matters for how much you should rely on nofollow as a control mechanism.
Index, Follow — the default (and most common)
This is what every page gets unless you say otherwise. Google can show it in search results, and Google can follow its links to discover other pages on your site.
Syntax:
<meta name="robots" content="index, follow">
You almost never need to write this explicitly — it's the default. If you see it hardcoded in a template, it's usually there just to be explicit, which is harmless but unnecessary.
Use case: every normal, public page you want to rank and want Google to crawl onward from — blog posts, product pages, category pages, your homepage. If you're not sure what a page should be, this is almost certainly the answer.
Noindex, Follow — hide the page, keep the links working
This tells Google: don't show this specific page in search results, but still use the links on it to find and crawl other pages.
Syntax:
<meta name="robots" content="noindex, follow">
In Yoast SEO, this shows up as the toggle "Allow search engines to show this page in search results?" set to No, with the separate "Follow links on this page" setting left on. RankMath has the equivalent under its page-level Advanced tab, with Robots Meta Tag set to No Index while Follow stays checked.
Use case: this combination is exactly right for pages you don't want appearing in search but that sit inside your internal link structure and shouldn't become dead ends for crawlers. A classic example is a paginated archive page — page 3 of your blog listing. Nobody searches for "blog page 3," so it doesn't need to be indexed. But it links to a bunch of your older posts, and you want Google to keep discovering those posts through that link. Noindex, follow lets you hide the archive page from search while still letting its links do their job.
Internal search results pages fall into the same bucket. You don't want yoursite.com/?s=blue+shoes ranking for anything, but if that page happens to link back to your main navigation or related categories, there's no reason to cut off that link equity flow.
Index, Nofollow — rare, and usually a mistake when unintentional
This tells Google: you can show this page in search results, but don't follow any of the links on it.
Syntax:
<meta name="robots" content="index, nofollow">
This is the least common of the four combinations, and honestly, it's rare to have a legitimate reason for it at the page level. It means you're fine with the page ranking, but you don't trust or want to pass authority through any of the outbound links it contains.
Use case: the most plausible scenario is a page whose ranking value you want to keep, but which is stuffed with outbound links you don't control — think a page of user-submitted resources, or a heavily monetized page with a lot of affiliate or sponsored links where you'd rather handle link qualification at the individual link level instead. In practice, most sites handle this more precisely by using rel="sponsored" or rel="nofollow" on the specific links that need it, rather than nofollowing an entire page's worth of links, some of which are probably legitimate internal links you'd want followed.
If you find a page on your site with this combination and you don't remember setting it deliberately, it's worth investigating — it's an unusual setting to apply by accident, but I've seen it happen from a misconfigured plugin rule applied to a whole content type.
Noindex, Nofollow — the full block
This is the strongest combination: don't show this page in search, and don't use any of its links to discover other pages either.
Syntax:
<meta name="robots" content="noindex, nofollow">
Equivalent shorthand, per Google's own spec:
<meta name="robots" content="none">
In Yoast, this is both toggles set to "off" — search visibility off, and "follow links" off. In RankMath it's Robots Meta Tag set to No Index with the Follow checkbox unchecked.
Use case: pages with genuinely zero SEO value that you also don't want treated as a crawl pathway to anything else. Thank-you pages after a form submission, internal admin-facing pages that somehow ended up crawlable, duplicate staging content that leaked into production, or a private landing page built for a single ad campaign that has no organic relevance and links only to internal tools nobody outside your team should ever reach. This is also the combination Google effectively assumes you want for pages you're deliberately walling off from any SEO consideration whatsoever.
Mistakes that accidentally deindex important pages
The four combinations are simple individually. The mistakes happen when settings get applied at scale, inherited from a template, or left over from a previous site state.
Bulk plugin defaults applied too broadly. A common one: someone sets a "noindex all tag archives" rule in their SEO plugin to clean up thin tag pages, but the plugin's category matching is broader than they think, and it accidentally sweeps up a custom taxonomy that actually held real, valuable landing pages.
Staging settings that ship to production. Development environments are often configured with sitewide noindex, nofollow to keep test content out of Google. When the site launches, if that setting lives in a config file or environment variable that doesn't get updated, the live site launches fully deindexed. This is more common than you'd think, and it's brutal because nobody notices until organic traffic craters and someone finally thinks to check the page source.
Migrating CMS platforms. Moving from one CMS to another, or even switching SEO plugins on WordPress, can silently reset per-page robots settings to a new default that doesn't match your old configuration. Pages that were carefully set to noindex, follow can revert to indexed, or vice versa, depending on how the new system's defaults are set up.
Nofollow applied at the template level instead of the page level. Someone wants to nofollow outbound affiliate links, but instead of using rel="nofollow" on those specific <a> tags, they set the whole page's meta robots tag to nofollow. Now every internal link on that page — including navigation, related posts, and breadcrumbs — stops passing link signals, which can quietly choke off internal linking to important pages.
Conditional logic bugs. A theme or plugin sets noindex based on a condition — say, "noindex if this post has fewer than 300 words" — meant to catch thin content. If that logic has a bug, or the word count is being calculated incorrectly (for example, not counting content loaded by certain page builders), legitimate long-form posts can get miscategorized and noindexed without anyone touching a setting directly.
The theme connecting all of these: the damage rarely comes from someone deliberately noindexing the wrong page. It comes from a rule applied broadly, a default that didn't get updated, or logic that misfires on an edge case. Which is exactly why spot-checking individual pages isn't enough — you need visibility across the whole site.
Copy-pasted templates carrying old settings. Someone builds a new landing page by duplicating an old one that happened to be set to noindex for a legitimate reason at the time — maybe it was a test page, or content still being reviewed. The duplicate ships live with the same setting still attached, because nobody thought to check a field that isn't visible anywhere in the normal editing view.
Multilingual or multi-region setups with inconsistent rules. Sites running separate language versions sometimes apply a blanket noindex to non-default-language content during initial rollout, intending it as temporary, and then never circle back to remove it once the translated content is actually ready and worth ranking.
Why this matters more on WordPress specifically
WordPress sites are unusually exposed to this category of mistake, for a structural reason: robots settings usually live inside a plugin's database tables or post meta fields, not in the visible page content, and there are often two or three plugins with the ability to touch them — an SEO plugin, a page builder, sometimes a caching or security plugin with its own indexation controls layered on top.
When these plugins disagree, or one gets updated with new defaults, the resulting robots tag on a given page can end up being decided by whichever plugin's logic runs last in the rendering order — which is not something most site owners have any visibility into without actually inspecting the rendered HTML. This is a large part of why "it used to rank fine and now it doesn't" investigations so often end with someone discovering a stray noindex tag nobody remembers setting.
Reading the tag in context: multiple crawlers, multiple rules
One detail that surprises people the first time they run into it: you can specify different robots instructions for different crawlers on the same page, and when you do, the rules don't simply override each other — they combine.
<meta name="robots" content="nofollow"> <meta name="googlebot" content="noindex">
In this example, the general robots rule says nofollow, and the Googlebot-specific rule says noindex. Google's crawler doesn't pick one or the other — it applies the sum of the restrictions that apply to it, which means Googlebot treats this page as noindex, nofollow even though neither individual tag said both. This is a deliberate design choice on Google's part, meant to keep sites from accidentally under-restricting a page by only remembering to update one of several tags.
This matters practically because it means auditing a page's true robots behavior isn't always as simple as reading a single meta tag and calling it done. If your CMS or a plugin is capable of emitting more than one robots-related tag — which happens more often than people realize, especially on sites that have layered multiple SEO tools over the years — you need to check for all of them and understand how they stack, not just glance at the first one you find in the source.
A word on the HTML placement itself
The robots meta tag only counts if it's placed correctly: inside the <head> section of the document, not the <body>. A robots meta tag that accidentally ends up in the body — which can happen with some page builders or when content is inserted via certain shortcodes — is simply ignored by search engines. It looks correct to a human skimming the page source, but it does nothing.
This is a good reason to verify your robots meta implementation with an actual crawler tool rather than trusting a plugin's settings screen alone. A plugin can report "noindex: on" in its dashboard while a theme conflict or a caching layer causes the actual served HTML to omit the tag entirely, or to render it in the wrong place. The only way to know for certain is to check what's actually being sent to a crawler, which is exactly what site crawl tools and cache-bypassing header checks are for.
Choosing the right combination isn't a one-time decision
It's worth saying plainly: the right robots meta combination for a page isn't fixed forever. A page's purpose changes, and its indexation setting should change with it.
A landing page built for a single seasonal campaign might reasonably carry noindex, follow while the campaign is live and the page has no long-term SEO value of its own, but still links into your permanent site structure. Once the campaign ends and the page becomes a historical artifact with genuinely useful content, it might be worth switching to index, follow if you decide it's worth ranking on its own after all — or leaving it as-is if you're planning to retire it and prefer to redirect it instead.
Similarly, a thin support article that started as noindex because it barely had any content can graduate to index, follow once someone expands it into something genuinely useful. Treating robots meta settings as a decision you revisit occasionally, tied to a page's actual current value, catches a category of missed opportunity that a "set once and never look again" approach misses entirely — pages that were correctly hidden a year ago but have quietly become worth surfacing since.
The practical habit worth building: whenever you do a content refresh or a substantial rewrite of an existing page, treat checking its current robots meta setting as part of that same pass. It costs almost nothing to check, and it's one of the few places where a stale setting can be actively costing you traffic without any visible symptom pointing you toward it.
How to check current tag combinations sitewide
Checking one page at a time by viewing source is fine for a single URL, but it doesn't scale, and it won't catch the bulk-rule mistakes described above.
View source on individual pages. Right-click, View Page Source, and search for name="robots". Quick, but only tells you about the one page in front of you.
Google Search Console's Page Indexing report. Look at the "Excluded" section for the category "Excluded by noindex tag." This tells you exactly which URLs Google has seen carrying a noindex instruction. If a page you expect to be indexed shows up here, that's your signal something's wrong.
Site-wide crawlers. Tools like Screaming Frog or Sitebulb crawl your entire site and report the meta robots value for every URL in one export. This is the only practical way to catch the "bulk rule applied too broadly" problem — you can filter the export for any page that's noindexed or nofollowed and eyeball whether it belongs on that list. Cross-reference it against a list of your important landing pages and top organic traffic pages; if any of those show up as noindexed, you've found your problem before it costs you months of lost traffic.
Ongoing monitoring, not just one-off audits. A one-time crawl catches today's problems, but robots meta tags can change after a plugin update, a theme switch, or someone fat-fingering a setting in the CMS. This is a case where continuous monitoring beats periodic audits — RankHive checks indexation settings across your WordPress site on an ongoing basis and flags the moment a page's robots tag changes unexpectedly, so a bad setting gets caught in a day instead of a quarterly review.
FAQ
What does "content=none" mean in a robots meta tag?
It's shorthand for noindex, nofollow — the strongest combination, blocking both search visibility and link-following on that page.
Does nofollow stop Google from crawling a page?
No. Nofollow, whether at the page level or the individual link level, is a hint about whether to pass ranking value through a link and whether to prioritize following it — it doesn't guarantee Google won't ever discover or crawl that linked page through some other path, like a sitemap or an external link.
Can I use different robots instructions for different search engines on the same page?
Yes. You can target a specific crawler by replacing "robots" with its user-agent token, like <meta name="googlebot" content="noindex">. If you specify rules for multiple crawlers with different instructions, most engines combine the most restrictive rules that apply to them.
Is nofollow the same as rel="nofollow" on a link?
Related but not identical. The page-level meta tag nofollow applies to every link on the page. The rel="nofollow" attribute applies to one specific <a> tag. You can use both, or just the link-level one if you only want to flag a handful of specific outbound links.
Will noindex hurt my crawl budget?
Marginally, and rarely in a way that matters. Google still needs to crawl a page occasionally to confirm the noindex tag is still there. For most sites this isn't a meaningful drain. It only becomes a real concern on very large sites with tight crawl budgets, where combining noindex with an eventual robots.txt block — after confirming deindexation — can help.
Should thank-you pages and internal search results always be noindex, nofollow?
Noindex, yes, almost always — there's no reason for these to appear in search. Nofollow depends on whether the page links to anything worth passing signal through. A thank-you page usually links back to your homepage or related content, so noindex, follow is often the better call than the fully closed noindex, nofollow.
How do I know if my SEO plugin changed my robots settings during an update?
Run a fresh sitewide crawl after any plugin or theme update and compare the meta robots values against your last known-good export. This is tedious to do manually every time, which is the reason ongoing monitoring tools exist — catching a settings regression the day it happens is a lot cheaper than catching it after three months of lost rankings.