What the validator reads and decides
Give it a site URL and it fetches /robots.txt from the root of that host, following up to four redirects. Paste text into Robots.txt content instead and it uses your text, even if a site URL is also filled in; the paste box holds up to 20,000 characters. Either way it parses the file line by line, then answers one question: may this user agent crawl this test URL?
Test URL starts filled in with an example, so replace it with the address you care about. User agent defaults to Googlebot, and you can type a short name such as Bingbot or a full user-agent string; the validator reduces either to the product token crawlers match on.
When it fetches, the HTTP status decides first, the way Google reads it. A 4xx other than 429 means the site has no robots.txt, so every URL is Allowed. A 5xx or a 429 means Google stops crawling the whole site for the first 12 hours, so every URL is Blocked. Either way the error page is not read as rules. A 200 that serves an HTML page, usually the theme’s not-found page, is reported as one error instead of a parse error for every line.
Until you run a check, the box under the report shows a starter robots.txt for WordPress, built from your site URL. After a run, it shows the text that was checked, labeled Fetched or Pasted. If the site has no readable file, the box goes back to the starter.
What it does not do
- It refuses a file larger than 500 KiB. Google reads only the first 500 KiB and ignores the rest, so a file that size needs trimming anyway.
- It does not model crawlers with special rules. Google’s AdsBot crawlers, for example, ignore the * group and obey only a group that names them.
- It does not tell you whether a page is indexed, only whether it may be crawled. The Indexability Checker covers the rest.
How a crawl decision is made
The validator follows RFC 9309, the robots.txt standard, and the behavior Google documents on top of it. As of October 2026 these are the rules Googlebot applies, in this order:
- 1.Find the group. A group is one or more User-agent lines followed by rules. The crawler uses the group that names it most specifically: a Googlebot-Image group beats a Googlebot group, which beats User-agent: *. Groups naming the same crawler are merged.
- 2.Use only that group. If a group names Googlebot, the rules under User-agent: * do not apply to Googlebot at all.
- 3.Match paths from the start of the URL path, including any query string. Matching is case-sensitive. An asterisk matches any run of characters, and a dollar sign at the end means the URL must end there.
- 4.If several rules match, the one with the longest path wins.
- 5.If an Allow and a Disallow match with paths of the same length, Allow wins.
- 6.If nothing matches, or no group applies, the URL may be crawled. An empty Disallow line blocks nothing, and /robots.txt itself is always allowed.
Sitemap lines sit outside the groups and apply to whichever crawler is reading, so it does not matter where in the file they appear.
Reading the report line by line
Source is the robots.txt URL that was fetched and the HTTP status it answered with, or Pasted text. Crawl decision is Allowed or Blocked for the test URL and user agent, or No test URL checked if there was nothing to test. Matched rule is the line that decided it. Matched rule: None with Allowed means no rule in the applicable group matched, which is normal for most pages.
Sitemaps lists every Sitemap line the file declares. Warnings and Errors list problems with the file itself, independent of the test URL. The messages are fixed, so you can look each one up here.
| Message | What it means | What to do |
|---|---|---|
| URL returned HTTP N, so the site has no robots.txt. | The file answered with a 4xx other than 429. Google crawls every URL | Nothing, if you want no rules. Otherwise create the file, or check that your SEO plugin’s robots.txt is turned on |
| URL returned HTTP N. … Google treats the whole site as disallowed. | The file answered with a 5xx or a 429, often a firewall or rate limit catching the request | Fix it first. Google stops crawling the whole site for the first 12 hours, then uses its last good copy for up to 30 days |
| URL returned an HTML page, not a plain-text robots.txt. | The URL answered 200 with a web page, so no rules apply | Open /robots.txt in a browser. Make it serve a real file, or a 404 if you want none |
| URL is served as text/html. Serve robots.txt as text/plain. | The rules are fine but the content type is wrong | Fix the server, CDN or plugin rule that sets the header |
| URL is empty, so it blocks nothing. | The file exists but has no rules, the same as having none | Nothing, unless you meant to publish rules |
| Line N is missing a colon. | The line is not a directive, usually a typo or a stray line of text | Fix the typo or delete the line |
| Line N uses an unrecognized directive: X. | Not a directive Google uses. Noindex and Host in robots.txt are ignored by Google | Remove it, or leave it if it is meant for another crawler or service |
| Line N has allow/disallow before any user-agent, so crawlers ignore it. | A rule above the first User-agent line belongs to no group | Move it under the group it was meant for |
| Crawl-delay is ignored by Googlebot. | Googlebot sets its own crawl rate. Bing does honor Crawl-delay | Keep it if it is aimed at Bingbot or another crawler that reads it |
| No Sitemap directive was found. | Crawlers cannot discover your sitemap from this file | Add a Sitemap line with the full URL, or ignore it if you submit sitemaps in Search Console and Bing Webmaster Tools |
| At least one rule blocks an entire site section with Disallow: /. | Some group blocks everything, for one crawler or all | Check which group. Under User-agent: * on a live site it is an emergency; under an AI crawler it may be intended |
When the HTTP status decided, Matched rule reads None, decided by the HTTP status, and Sitemaps is empty, because an error page holds no rules.
Google’s own parser forgives a few common misspellings, such as User agent without the hyphen. The validator reports those as unrecognized directives and does not start a group for them, so its answer for that crawler can differ from Google’s. Correct the spelling; other crawlers may be less forgiving.
Worked example: the Googlebot group trap
Illustrative file from a WooCommerce site. The owner added a Googlebot group to keep Google out of checkout, and expected the general rules to keep applying too.
The file
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /*?s= Disallow: /cart/ User-agent: Googlebot Disallow: /checkout/ Sitemap: https://example.com/sitemap_index.xml
Googlebot obeys lines 7 and 8, its own group, and the Sitemap line, which belongs to no group. Lines 1 to 5 are the * group, and Googlebot skips all of them.
What the validator showed
Test URL https://example.com/?s=shoes. Googlebot: Allowed, Matched rule: None. Bingbot: Blocked, Matched rule: Disallow: /*?s=.
Same URL, opposite answers. Matched rule: None for Googlebot, on a path that line 4 clearly matches, is the giveaway that a more specific group took over. Googlebot was also free to crawl /wp-admin/ and /cart/.
The fix
Copy lines 2 to 5 into the Googlebot group under Disallow: /checkout/, or delete the Googlebot group and add Disallow: /checkout/ to the * group.
Re-run with Googlebot: /?s=shoes is now Blocked by Disallow: /*?s=, and /wp-admin/admin-ajax.php is Allowed by its Allow line. Unless a crawler genuinely needs different rules, one * group is easier to keep right.
Worked example: prefixes, wildcards and ties
Each case below was run through the validator. The URLs are illustrative; the results are what its matching rules produce.
A missing slash blocks more than you meant
Before: Disallow: /blog blocks /blog/, /blog-tips/ and /blogroll
After: Disallow: /blog/ blocks /blog/ and everything under it, and nothing else
Rules are prefixes. Without the trailing slash, every path that starts with those five characters is blocked.
A tie goes to Allow
Before: Disallow: /shop/, Allow: /shop/sale/ and Disallow: /*?orderby= leave /shop/sale/?orderby=price Allowed
After: Disallow: /shop/*?orderby= blocks /shop/sale/?orderby=price
Allow: /shop/sale/ and Disallow: /*?orderby= are both 11 characters long, so Allow wins the tie. Writing the Disallow with the longer /shop/ prefix makes it the more specific rule. /shop/sale/shoes/ stays Allowed and /shop/hats/ stays Blocked.
The end anchor is literal
Disallow: /*.pdf$ blocks /files/guide.pdf but not /files/guide.pdf?download=1
The dollar sign means the URL ends there, so any query string escapes the rule. Add a second rule such as Disallow: /*.pdf? if you need both.
Case matters
Disallow: /Private/ does not block /private/
Paths are case-sensitive, while directive names such as Disallow are not.
A bare query wildcard
Disallow: /*? blocks /guide/?utm_source=newsletter and leaves /blog/page/2/ Allowed
It stops crawling of every URL with a query string, including the ones you link to on purpose, such as filtered product lists you want found. Test those before you ship it.
Testing an edit before it goes live
Paste the draft into Robots.txt content and test it against the URLs that matter before you save it anywhere. On WordPress that list is short and predictable.
- 1.The homepage, one post, one page and one category: all should be Allowed.
- 2.A product and a shop category if you run WooCommerce, then the cart and checkout if you block them.
- 3.A stylesheet or script under /wp-content/themes/ or /wp-includes/, and an image under /wp-content/uploads/ with Googlebot-Image as the user agent: all should be Allowed so Google can render the page and index the images.
- 4./wp-admin/admin-ajax.php: Allowed, because some themes and plugins load front-end content through it.
- 5.Anything under /wp-json/ if your theme or plugins load content from the REST API on the front end.
- 6.Each URL again with Bingbot, and with any AI crawler you name in the file, if their groups differ.
Then save the file, clear the Robots.txt content box so the validator fetches the live file, and repeat the two or three most important tests. That catches a cache, a CDN or a second robots.txt source serving something other than what you saved.
Pointing the starter file at your sitemap
The starter file holds the rules WordPress core generates on its own, plus a Sitemap line for the core sitemap at /wp-sitemap.xml on your site URL. On a site without an SEO plugin it can go live as it is. With a plugin, change the Sitemap line, as the comment in the file says.
Starter file, then adjusted for Yoast SEO or Rank Math
Before
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php # The WordPress core sitemap. SEO plugins serve their own instead, # such as /sitemap_index.xml for Yoast SEO and Rank Math. Sitemap: https://example.com/wp-sitemap.xml
After
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://example.com/sitemap_index.xml
The comment lines are optional; crawlers skip everything after a #. The Allow line keeps admin-ajax.php open, because some themes and plugins load front-end content through it.
Use the sitemap address your setup actually serves: /sitemap_index.xml for Yoast SEO and Rank Math, /sitemaps.xml for SEOPress, and /wp-sitemap.xml for WordPress core without an SEO plugin. Open it in a browser first; a Sitemap line pointing at a 404 helps no one.
Do not block /wp-content/ or /wp-includes/. Those folders hold the CSS, JavaScript and images Google needs to render the page, and a blocked stylesheet can make a page look broken to Google.
Which file your WordPress site is actually serving
WordPress has no robots.txt file by default. When a request for /robots.txt reaches it, core generates one: User-agent: *, Disallow: /wp-admin/, Allow: /wp-admin/admin-ajax.php, plus a Sitemap line for the core sitemap when the site is public. SEO plugins add to or replace that output. All of this is a virtual file, and it only works when WordPress runs at the root of the domain.
A real robots.txt file in the site root wins over all of it. The web server answers with the file before WordPress sees the request, so every virtual edit is silently ignored.
| Source | Where it is edited | File type |
|---|---|---|
| WordPress core | Not editable without code or a plugin | Virtual |
| Yoast SEO | Yoast SEO → Tools → File editor, which creates a robots.txt if none exists | Physical file |
| Rank Math | Rank Math SEO → General Settings → Edit robots.txt. The editor is locked while a physical file exists | Virtual |
| SEOPress PRO | SEO → PRO → robots.txt tab, after enabling the virtual robots.txt | Virtual |
| A physical file | SFTP or the host’s file manager, in the web root, usually the folder that holds wp-config.php | Physical file |
RankHive’s plugin adds the robots.txt rules you approve to the virtual file in the same way, so a physical file hides those too. Some CDNs and hosts can also add lines of their own; a Content-Signal line at the top of a file on Cloudflare is one example, and the validator reports it as an unrecognized directive.
One more trap: Settings → Reading → Discourage search engines from indexing this site has not changed robots.txt since WordPress 5.3. It adds a noindex tag to every page instead. If your live file says Disallow: /, a plugin, a host or a physical file put it there.
What a robots.txt rule cannot do
It cannot take a page out of Google. A blocked URL that other pages link to can still be listed, without a description, because Google knows it exists but may not read it. To remove a page, let Google crawl it and serve noindex, return 404 or 410, or use the Removals tool in Search Console, which hides it for about six months while a permanent fix takes effect.
It does not protect anything. The file is public and lists the paths you would rather people not look at. Anything private needs a password.
It covers one host and protocol. https://www.example.com/robots.txt says nothing about https://example.com, http://www.example.com or shop.example.com; each serves its own file, or none.
Changes are not instant. As of October 2026, Google’s documentation says it generally caches robots.txt for up to 24 hours. A 4xx response, other than 429, means no restrictions. A 5xx or 429 response makes Google treat the whole site as blocked for a while, then fall back to the last good copy it has, so a robots.txt that errors can stall crawling of a healthy site.
robots.txt problems by symptom
The cases that bring people to this tool, with the usual cause on WordPress.
| Symptom | Usual cause | Fix |
|---|---|---|
| Search Console says Blocked by robots.txt, the validator says Allowed | Google read an older copy, or another host or protocol’s file | Check the copy Google fetched in the robots.txt report, and test the exact host and protocol |
| Indexed, though blocked by robots.txt | The URL is linked from elsewhere, so Google listed it without reading it | To keep it out, remove the block and serve noindex; to get it indexed properly, remove the block |
| Rules saved in Rank Math or SEOPress do not appear | A physical robots.txt file is in the site root | Move any rules you need into the plugin, then delete the file |
| Every page blocked right after launch | A Disallow: / from the staging site, in a physical file or a plugin | Remove it, fetch the file again here, then request a recrawl of robots.txt |
| URL Inspection’s screenshot looks unstyled | A rule blocking /wp-content/ or /wp-includes/ | Remove the rule and run the live test again |
| The response is too large for this free check. | A file over 500 KiB, usually generated rules that piled up | Replace long lists with a few wildcard rules |
Confirming the change with Google
In Search Console, Settings → robots.txt shows the robots.txt files Google found for the top hosts of your property, when each was last fetched, whether the fetch worked and which issues Google saw. It replaced Google’s old robots.txt Tester. After an urgent fix, such as removing a Disallow: / that blocked the whole site, use Request a recrawl there instead of waiting for the cache to expire.
For a single page, URL Inspection shows Crawl allowed under Page indexing. After the file is fixed and recrawled, Test live URL should show crawling allowed. Pages that were listed as Blocked by robots.txt can then be validated from the Page indexing report.
Some results are fine to leave as they are. Disallow: /wp-admin/ is the WordPress default. A Disallow: / under an AI crawler you chose to block will trigger the entire-site warning, and that is expected. Crawl-delay for Bingbot, a Content-Signal line from your CDN, and a missing Sitemap line on a site that submits sitemaps in Search Console all produce warnings that need no action.
FAQ about the Robots.txt
Can robots.txt remove pages from Google?
No. It stops crawling, not indexing, and a blocked URL can stay listed if other pages link to it. To remove a page, allow crawling and add noindex, or return 404 or 410. For something urgent, the Removals tool in Search Console hides a URL for about six months.
Should WordPress block wp-content?
No. It holds your theme’s CSS and JavaScript and every uploaded image. Blocking it stops Google from rendering pages properly and from indexing images. Blocking part of it, such as /wp-content/plugins/, does the same to every page that loads files from there.
Why does Googlebot ignore my User-agent: * rules?
Because the file also has a group for Googlebot. A crawler obeys only the most specific group that names it, so once a Googlebot group exists, every rule Googlebot should follow has to be repeated inside it.
Which wins when Allow and Disallow both match a URL?
The rule with the longer path. If both are the same length, Allow wins. The order of lines in the file does not matter to Google, which is why lengthening a Disallow path is the way to beat a broad Allow.
Where is robots.txt on a WordPress site?
Usually nowhere on disk: WordPress generates it when /robots.txt is requested, and SEO plugins edit that virtual output. If a real robots.txt file exists in the site root, the server sends that instead, and plugin edits stop showing.
How quickly does Google pick up a robots.txt change?
Google generally refreshes its cached copy within 24 hours, as of October 2026. If the change is urgent, request a recrawl from the robots.txt report in Search Console, then check the fetched version there.
Should I block AI crawlers in robots.txt?
That is a business choice, and the file can express it per crawler. As of October 2026, GPTBot and ClaudeBot collect training data, while OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch pages for AI search answers, so blocking those can keep you out of their results. Google-Extended controls use of your content for Gemini training and grounding, and does not affect Google Search.
Do Bing and other search engines read robots.txt the same way?
Mostly. The group and longest-match rules come from the same standard, and Bingbot follows them. The main difference is Crawl-delay, which Bing honors and Google ignores. Test with Bingbot as the user agent if you give it a group of its own.