Indexed, though blocked by robots.txt
Where you see it: Search Console → Indexing → Pages, listed as a warning on indexed pages
Google added this URL to its index without ever reading the page. Your robots.txt keeps Googlebot from fetching it, but links on your site or elsewhere told Google the address exists, so it can appear in search, usually with little or no description. What you do next depends on one question: should this page be in Google at all?
Act on it: decide, URL by URL, whether it belongs in Google.
If the page should rank, remove the block so Google can read it. If it should not, remove the block anyway and add noindex: a noindex only works on a page Google is allowed to fetch.
Check the URL Search Console listed
Type your domain into Site URL, leave the robots.txt box blank and test the listed address as Googlebot. The validator reads the file your server sends now and quotes the rule standing between Google and the page.
Robots report
Fetch or paste robots.txt to test crawler access.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
# The WordPress core sitemap. SEO plugins serve their own instead,
# such as /sitemap_index.xml for Yoast SEO and Rank Math.
Sitemap: https://rankhive.app/wp-sitemap.xmlReading the result
- Blocked, with a Matched rule: that line stops Google reading the page, and with it any noindex the page carries. It has to go, or be narrowed, before either fix can work.
- Allowed, Matched rule None: the block is already gone. Google is still working from an earlier visit and will update the listing after it fetches the page again.
- Blocked, with Matched rule Disallow: /*? or a similar pattern: the listed URL carries a query string such as ?add-to-cart= or ?replytocom=. Read the section on when to leave it alone before you unblock a whole pattern.
The same tool with its full guide: Robots.txt
Why it happens on WordPress
The warning needs a Disallow rule over the URL and a reason for Google to know the address. WordPress supplies plenty of reasons: menus, widgets, theme templates and sitemaps all print links, including links to URLs a robots.txt template blocks.
Pages blocked to hide them
Someone added Disallow: /thank-you/, /members/ or /old-landing/ to keep a page out of Google. That stops the crawl, not the listing, so the address stays findable through its URL and the link text pointing at it.
Cart and account links in every header
Store robots.txt files often block /cart/, /checkout/ and /my-account/, while the theme links to them from every page. Google meets those links on each crawl. WooCommerce already marks these pages noindex, so the block adds nothing and hides that tag from Google.
Parameter links your templates print
Add-to-cart buttons in product grids (?add-to-cart=), comment reply links (?replytocom=), sort menus (?orderby=) and filter widgets all write crawlable links. Block those patterns and some of the URLs get indexed from the links alone.
A block added after the page was indexed
Block a folder of posts that Google already had, perhaps to “remove” them, and Google can keep the old listing with no way to read the page again. A staging rule that ran on the live site for a few weeks does the same.
A sitemap that lists blocked URLs
Yoast SEO, Rank Math and core list what their settings tell them to, not what robots.txt allows. A sitemap naming URLs that robots.txt forbids hands Google addresses it may index but not read.
How to fix it
Settle the question for each group of URLs first. The fix for a page you want ranked is the opposite of the fix for one you want gone, apart from the first move.
1. Mark each URL keep or drop
Go through the warning’s example URLs one by one. Keep: posts, pages, products and categories you want found. Drop: thank-you, account, cart, members and parameter URLs.
2. For pages you keep, delete the rule
Find the matching line with the validator and remove or narrow it in the file the server sends: a physical robots.txt in the site root, the physical file behind Yoast SEO → Tools → File editor, or the virtual file under Rank Math SEO → General Settings → Edit robots.txt. Google can then crawl the page and replace the bare listing with a real title and snippet.
3. For pages you drop, unblock and add noindex
Remove the rule first, then mark the page so Google drops it on its next visit:
- Yoast SEO: in the post’s Yoast SEO box, open Advanced and set “Allow search engines to show this Post in search results?” to No (Page, on a page).
- Rank Math: open the Rank Math panel in the editor, go to Advanced and tick No Index under Robots Meta.
- WooCommerce cart, checkout and my-account pages: they already carry noindex, so removing the robots.txt rule is the whole fix.
- Anything private, such as a members area or client files: put it behind a login. Robots.txt is a public list of the paths you would rather hide.
4. Wait for the drop, then decide about the rule
When URL Inspection shows the page as excluded by noindex, it is out of the index. You can block crawling again afterwards if the crawl volume bothers you, but any new link can bring the bare address back, which is why Google recommends noindex over robots.txt for this job.
5. Clean up links and sitemaps
Remove links in menus, widgets and old posts that point at pages you dropped. In Yoast SEO and Rank Math, a post set to noindex leaves the XML sitemap on its own; check that nothing you block is still listed there.
Why robots.txt can’t keep a page out of Google
Robots.txt answers one question for crawlers: may I fetch this URL? Google’s robots.txt guide says the file is mainly for managing crawler traffic and is not a way to keep a page out of Google, because a disallowed address can still be indexed from the links that point at it.
The Page indexing help gives the same advice for this warning: to keep a page out, remove the robots.txt block and use noindex.
When the warning can wait
If the list is ?add-to-cart=, ?orderby= or ?replytocom= addresses, check the Performance report for impressions on them. None means nobody sees the bare listings, and unblocking a whole parameter pattern sends Googlebot to many near-empty URLs. Act when a URL with impressions appears, or a page you meant to hide.
Check the fix in Search Console
Google updates this warning one URL at a time, as it fetches each page under your new rules.
- In URL Inspection, run Test live URL on a fixed address. Crawl allowed? should read Yes. For a page you dropped, Indexing allowed? should now read No, because Google can see the noindex at last.
- Use Request indexing on kept pages that matter, so Google reads them sooner.
- Click Validate fix on the warning in Indexing → Pages only once every listed URL is unblocked. Google’s help allows up to about two weeks for a typical validation.
- Dropped URLs move to Excluded by ‘noindex’ tag, kept ones become plain indexed pages.
Validation fails as soon as Google rechecks a URL that is still blocked. If you left ?add-to-cart= or ?orderby= addresses blocked on purpose, skip it and let the warning update as Google recrawls.
Can RankHive fix this?
Partly. RankHive can add an Allow line that unblocks a URL, as long as WordPress serves the robots.txt, and it can repoint links inside your posts. It can’t set noindex, so for pages you want out of Google, that step stays in your SEO plugin.
What it writes, once you approve
- An Allow rule in the virtual robots.txt, or an edit to a line RankHive added before.
- Edits to links inside posts and pages that send Google to URLs you dropped.
What it can’t do here
- Set or remove noindex on any URL.
- Edit a physical robots.txt file, which hides RankHive’s lines while it exists.
- Remove links your theme, menus or WooCommerce print, such as header cart links.
- Change your XML sitemap or click Validate fix for you.
Every change is drafted next to the current value and waits for your approval. Each one written is logged in the Change Log, with Revert.
Check the rest of your site while you’re here
The free site check reads your public pages for titles, meta descriptions, headings, canonical tags and schema, and shows what needs fixing. If you sign up, RankHive reads your Google Search Console data every week, drafts the fixes, and writes the ones you approve to WordPress. Nothing changes on your site until you say yes.
Related issues
- Blocked by robots.txtFix it only for URLs you want found in Google.
- Excluded by ‘noindex’ tagKeep it on pages you hid on purpose; remove it from pages you want found.
- URL Inspection toolNothing to fix: it’s the tool you diagnose and confirm fixes with.
- URL is not on GoogleFix it if you want this page in search, after reading the reason.
Common questions
Is “Indexed, though blocked by robots.txt” a problem?
Sometimes. Google files it as a warning: the URL is indexed, with a thin listing, because Google never read the page. It matters for pages you want ranked and for pages you meant to hide.
Can I keep the Disallow rule and add noindex?
No. With the rule in place, Googlebot never fetches the page, so it never sees the noindex. Remove the rule, let Google read the noindex, and decide about blocking again only after the URL has left the index.
Can I put noindex in robots.txt instead?
No. Google stopped supporting noindex lines in robots.txt in 2019. Use the robots meta tag your SEO plugin sets, or an X-Robots-Tag header for PDFs.
How do I get a page out of Google quickly?
Unblock it, add noindex, and use the Removals tool in Search Console to hide it temporarily while Google recrawls. The removal ends on its own, so the noindex has to be in place before it does.
Read next
Sources
Checked against these pages on .