Blocked by robots.txt
Where you see it: Search Console → Indexing → Pages → Why pages aren’t indexed
Googlebot checked your robots.txt before fetching this URL, found a Disallow rule covering it, and stayed away, so the page was never read or indexed. On WordPress that is often deliberate: admin screens and internal search results are common blocks. Fix it only when a page you want found is on the list.
Fix it only for URLs you want found in Google.
Export the examples and read the paths. /wp-admin/ and ?s= searches are blocks someone chose. A rule over /cart/, /checkout/ or /my-account/ does no harm but adds nothing: WooCommerce already noindexes those pages, and the block hides that tag from Google. Posts, products, categories or your homepage mean a rule reaches too far, or a staging block went live.
Check the URL Search Console listed
Replace the sample Site URL with your domain and leave the robots.txt box empty, so the validator fetches your live file. Put the address Search Console listed in Test URL and keep Googlebot as the user agent.
Robots report
Fetch or paste robots.txt to test crawler access.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
# The WordPress core sitemap. SEO plugins serve their own instead,
# such as /sitemap_index.xml for Yoast SEO and Rank Math.
Sitemap: https://rankhive.app/wp-sitemap.xmlReading the result
- Blocked, with a Matched rule such as Disallow: /shop: that line is the cause. Rules are prefixes, so without a closing slash it also covers /shop-guide/.
- Allowed, Matched rule None: the live file no longer blocks the URL. Google may hold yesterday’s copy, or it read the file on the http:// or non-www host, so test that too.
- A Matched rule you can’t find among your * rules: look it up in the Fetched robots.txt box and see which User-agent group holds it. Under User-agent: Googlebot it binds Google alone, and Googlebot then skips every * rule.
- Source ending in HTTP 500 or 503, or 429, which Google handles like a server error: the file itself is failing, and an erroring robots.txt can make Google pause crawling of the whole site. Fix that first.
The same tool with its full guide: Robots.txt
Why it happens on WordPress
WordPress keeps no robots.txt on disk by default. Core prints one on request that blocks only /wp-admin/, so when real content shows up as blocked, the rule came from a file, a plugin, a host or a migration.
A staging Disallow: / that shipped with the site
Staging copies are often set to refuse every crawler, and a migration can carry that file to production. Every URL Google tries afterwards lands here at once. Look for a robots.txt in the site root dated around your launch.
A physical file that overrides your plugin
A real robots.txt next to wp-config.php is sent by the web server before WordPress runs. Delete a rule in Rank Math’s or All in One SEO’s editor and nothing changes, because they edit the virtual file the server no longer uses. Yoast SEO’s File editor, by contrast, edits the physical file.
A prefix or wildcard that reaches too far
Disallow: /blog blocks /blog-tips/ as well as /blog/. Disallow: /*? keeps ?add-to-cart= links out but also hides filtered product listings you may want found. “Optimised WordPress robots.txt” templates are a common source of both.
A Googlebot group that cancels your general rules
Once a group is headed User-agent: Googlebot, Google stops reading the User-agent: * group. A Disallow: / added there during a crawl spike locks Google out while Bing keeps crawling.
Another host serving its own file
Each protocol and host has its own robots.txt. If http:// or the non-www address answers with an old file instead of redirecting, URLs found there show as blocked. On a subdirectory multisite, only the main site’s file counts.
How to fix it
Sort the list before you edit anything. Most sites need one or two rule changes, not a fix per URL.
1. Pick out the URLs that were meant to rank
Open Blocked by robots.txt in Indexing → Pages and scan the example paths. Admin and search URLs can stay blocked, and so can WooCommerce’s cart, checkout and account pages. Whatever posts, pages, products and categories remain are your work list.
2. Confirm the block in URL Inspection
Inspect one URL from that list and expand Page indexing. Crawl allowed? answers No when robots.txt is the cause. Ignore Indexing allowed? here: Google’s help says it always reads Yes on a blocked page, since Google can’t see a noindex it may not fetch.
3. Find the line, then the file it lives in
The validator above names the matching rule. Then open /robots.txt on your domain, such as example.com/robots.txt, in a private window to see what the server sends, and edit it at the source:
- A physical file: look next to wp-config.php by SFTP or your host’s file manager. If it exists, it is the only file that counts.
- Yoast SEO: Yoast SEO → Tools → File editor edits that physical file, or creates one if none exists. If the menu item is missing, file editing is off on your install; use SFTP.
- Rank Math: Rank Math SEO → General Settings → Edit robots.txt edits the virtual file, and locks while a physical file exists.
- All in One SEO: All in One SEO → Tools → Robots.txt Editor, with its custom robots.txt option on.
4. Remove or narrow the rule
Delete a Disallow: / that came from staging. Close a prefix with a slash: Disallow: /shop/ instead of Disallow: /shop. To reopen one path inside a blocked folder, add an Allow line for it, since the rule with the longer path wins. Fold a one-rule Googlebot group into User-agent: *.
5. Retest the live file
Purge your page cache and any CDN copy of /robots.txt, then run the validator with the box empty. The URL should read Allowed, and /wp-admin/ should still read Blocked.
6. Ask Google to fetch the new file
In Search Console Settings, open the robots.txt report, use the menu beside your file and click Request a recrawl. Google refreshes its copy within about 24 hours anyway, but this helps after you unblock pages that matter.
Same rule, three different outcomes
What matters is whether Google read the page. A Disallow rule stops the reading but not the listing, so a blocked URL with links pointing at it can still be indexed. A noindex tag works the other way round: Google must read the page to find it.
| Status | Did Google read the page? | In search results? | Act? |
|---|---|---|---|
| Blocked by robots.txt | No | No | Only for pages you want found |
| Indexed, though blocked by robots.txt | No, it indexed the address from links | Yes, with little or no description | Usually |
| Excluded by ‘noindex’ tag | Yes, and found noindex | No | Only if the noindex is a mistake |
Check the fix in Search Console
Each URL clears only after Google fetches it again under the new file. Confirm one by hand before you ask for a wider recheck.
- Run Test live URL on a URL you unblocked. Crawl allowed? should read Yes and Page fetch Successful.
- Click Request indexing on the few pages that matter most; the quota is small.
- If every listed URL should now be crawlable, open Blocked by robots.txt in Indexing → Pages and click Validate fix. By Google’s estimate, most validations finish within about two weeks.
- Validation stops at Failed the moment Google rechecks a URL that is still blocked. The details name it; look for a second rule.
Blocks you keep on purpose, such as /wp-admin/, stay in this report, so a list mixing them with reopened pages can’t pass. Skip Validate fix then; the report catches up as Google recrawls.
Can RankHive fix this?
Yes, when WordPress serves your robots.txt, which is the default: RankHive adds an Allow line that reopens the blocked path, or edits lines it added before. It can’t delete a blocking line written in a physical robots.txt file, by another plugin or by your host, and while a physical file exists none of its lines reach Google.
What it writes, once you approve
- An Allow rule in the virtual robots.txt that reopens a path a broader Disallow blocks.
- Edits to, or removal of, lines RankHive added earlier.
What it can’t do here
- Edit or delete a physical robots.txt file.
- Delete lines Rank Math, All in One SEO or another plugin adds; it only writes its own.
- Change files served by your host, your CDN or another host such as a subdomain.
- Request a recrawl or click Validate fix: its Search Console access is read-only.
Every change is drafted next to the current value and waits for your approval. Each one written is logged in the Change Log, with Revert.
Check the rest of your site while you’re here
The free site check reads your public pages for titles, meta descriptions, headings, canonical tags and schema, and shows what needs fixing. If you sign up, RankHive reads your Google Search Console data every week, drafts the fixes, and writes the ones you approve to WordPress. Nothing changes on your site until you say yes.
Related issues
- Indexed, though blocked by robots.txtAct on it: decide, URL by URL, whether it belongs in Google.
- Excluded by ‘noindex’ tagKeep it on pages you hid on purpose; remove it from pages you want found.
- URL is not on GoogleFix it if you want this page in search, after reading the reason.
- Blocked due to access forbidden (403)Fix it if the page should be public; keep Google away if it never should be.
- Sitemap could not be readFix it if the file is broken; give a new submission a few days.
Common questions
Is “Blocked by robots.txt” bad for SEO?
Not for URLs you blocked on purpose. It is bad when the list includes pages you want to rank, because Google can’t read a word of them.
Why does Search Console say blocked when my robots.txt allows the page?
Usually timing or host. Google generally caches robots.txt for up to 24 hours and reads a separate file for each host and protocol. The robots.txt report shows which files Google fetched and when.
Does “Discourage search engines from indexing this site” cause it?
No. Since WordPress 5.3 that checkbox adds noindex instead of a robots.txt rule, so it shows up as Excluded by ‘noindex’ tag. A live Disallow: / comes from a plugin, the host or a physical file.
Should I block /wp-content/ or /wp-includes/?
No. They hold the stylesheets and scripts Google needs to render your pages, and /wp-content/uploads/ holds the images you want in Google Images.
Read next
Sources
Checked against these pages on .