WordPress robots.txt: Defaults, Edits and Fixes

WordPress robots.txt is a virtual file: unless you upload a real robots.txt to your web root, WordPress generates one on the fly when anyone requests yoursite.com/robots.txt. The default blocks /wp-admin/ (except admin-ajax.php) and points to your sitemap, and for most sites it is the right setting — the edits worth making are few, and the wrong ones can quietly block Google from your theme files or whole sections of the site.

That virtual file is why so many site owners cannot find it. Load yoursite.com/robots.txt and something appears. Connect over SFTP, look in the web root, and there is no such file. This confuses people into pasting rules into a file that does not exist, or into copying a 40-line "optimised" ruleset from a blog post that blocks the CSS their theme needs.

This guide covers what WordPress actually serves, how to edit it with or without Yoast, Rank Math, AIOSEO or SEOPress, the handful of rules worth adding, how to fix "Blocked by robots.txt" and "Indexed, though blocked by robots.txt" in Search Console, and the four edits that quietly cost sites traffic. Every technical claim here is checked against Google's robots.txt specification and the WordPress core source rather than the received wisdom, because on this topic the received wisdom is several years out of date. To check a live file, or build one, use the free robots.txt generator and validator.

Last updated: October 13, 2026.

WordPress serves a virtual robots.txt

When a request hits /robots.txt, WordPress checks whether a real file exists in the web root. If one does, the web server returns it and WordPress never gets involved. If one does not, WordPress handles the request itself and prints a file from memory through its do_robots() function.

That virtual file is why the directory looks empty. It is also why FTP-based edits appear to do nothing until you create the file rather than edit it.

The default output is three lines:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Since WordPress 5.5, core appends a sitemap reference as well, using WP_Sitemaps::add_robots():

public function add_robots( $output, $is_public ) {
	if ( $is_public ) {
		$output .= "\nSitemap: " . esc_url( $this->index->get_index_url() ) . "\n";
	}

	return $output;
}

So a stock modern WordPress install serves:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml

Note the $is_public guard. The sitemap line is only added when Settings → Reading is set to allow indexing. A site marked private serves the two rules with no sitemap.

This default is well judged. /wp-admin/ is a login-walled area with nothing worth crawling, and admin-ajax.php is explicitly re-allowed because plenty of themes and plugins route front-end functionality through it — blocking it can stop Google rendering parts of your pages. Most WordPress sites need no changes to this at all.

The "Discourage search engines" setting no longer touches robots.txt

This is the fact that most WordPress robots.txt tutorials still get wrong.

Settings → Reading has a checkbox: Discourage search engines from indexing this site. For years, ticking it added Disallow: / to the virtual robots.txt. Many guides published today still say so.

It stopped doing that in WordPress 5.3, released in November 2019. The setting now outputs a meta tag instead:

<meta name='robots' content='noindex,nofollow' />

The core team's reasoning, from the development announcement, is worth internalising because it explains the single biggest conceptual trap in this whole topic:

A site doesn't have to be crawled to be listed. If a link points to a page … Google follows that link. If the robots.txt prevents crawling … it'll still show the URL in the results.

In other words, the old behaviour was counterproductive. It stopped Google reading the page while doing nothing to stop Google listing the URL — producing exactly the naked, snippet-less search result the setting was meant to prevent.

If you are on WordPress 5.3 or later and want a site kept out of search, the checkbox now does the right thing. You do not need to add anything to robots.txt, and adding Disallow: / would make the outcome worse.

Crawling is not indexing

Everything else in this guide follows from one distinction, so it is worth stating plainly.

Robots.txt controls crawling. It tells a compliant bot which URLs it may request. It has no authority over what appears in an index.

Google's documentation is unambiguous:

Google can't index the content of pages which are disallowed for crawling, but it may still index the URL and show it in search results without a snippet.

That produces the result everyone has seen: a bare URL, no title worth reading, and the line "No information is available for this page."

The practical rule:

The middle row is the only legitimate use of a Disallow rule. The first row is where people reach for robots.txt and cause the problem they were trying to solve — a page blocked in robots.txt can never be de-indexed by a meta tag, because the crawler is not permitted to fetch the page and read it.

Google also dropped support for noindex: directives inside robots.txt in 2019. If you see that syntax in a ruleset you are copying, the ruleset is stale.

The syntax, precisely

Robots.txt has a small grammar, and almost every real-world bug comes from one of four misunderstandings about it.

Groups, and why a bot may ignore your rules entirely

A robots.txt file is a series of groups. Each group starts with one or more User-agent lines and is followed by the rules that apply to them.

A crawler obeys exactly one group: the most specific one that names it. A named group is not merged with the * group. This is the subtle one:

User-agent: *
Disallow: /*?s=
Disallow: /cart/

User-agent: Googlebot
Disallow: /internal/

Googlebot here is subject to only Disallow: /internal/. It ignores the wildcard group completely, so it will happily crawl /cart/ and every internal search URL. Any rule you want Googlebot to follow has to be repeated inside its own group.

This is the most common way a carefully written robots.txt does nothing. If you add a named group for any bot, audit what that bot just stopped obeying.

Wildcards

Two special characters, both supported by Google:

• * matches any sequence of characters.
• $ anchors the match to the end of the URL.

Disallow: /*.pdf$        # any URL ending .pdf
Disallow: /*?            # any URL containing a query string
Disallow: /private*/     # /private/, /private-notes/, /privateers/

That third example is a reminder that * is greedy and does not respect path boundaries. /private*/ matching /privateers/ is exactly the kind of accident that quietly de-indexes a section.

Precedence

Rule order in the file is irrelevant. Google resolves conflicts by specificity: the rule with the longest matching path wins, and when two rules are equally specific, the least restrictive one wins — which means Allow beats Disallow on a tie.

Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

The Allow is longer and therefore more specific, so admin-ajax.php is crawlable while the rest of /wp-admin/ is not. That is precisely how the WordPress default works, and it is why you can safely write broad Disallow rules and carve out exceptions afterwards.

Case and paths

Paths are case-sensitive. Disallow: /Downloads/ does not block /downloads/. Directive names are not case-sensitive, so disallow: and Disallow: are equivalent, but there is no reason to be inconsistent.

An empty Disallow: with no path means allow everything — it is the conventional way to write a permissive file, and it is what Yoast writes by default.

How to edit robots.txt in WordPress

There are three routes — a physical file, your SEO plugin's editor, or code — and the one thing to know before choosing is precedence: a physical robots.txt in the web root always wins. If one exists, the web server returns it and nothing WordPress or a plugin generates is ever seen.

In descending order of how much I would trust them:

A physical file in the web root

Upload a file named exactly robots.txt to the root of your site — the same directory as wp-config.php and wp-load.php. A physical file takes precedence over the virtual one and WordPress stops generating output entirely.

This is the most predictable option. It is also the easiest to forget about, because nothing in the WordPress admin will tell you it exists.

Editing robots.txt with Yoast

In Yoast it is at Yoast SEO → Tools → File editor → Create robots.txt file. Despite living in the admin, this does not create a virtual override — it writes a real file to your web root, with the same precedence as uploading one yourself.

Two caveats from Yoast's own documentation: the editor needs the web root to be writable, and it will not appear at all if your install has DISALLOW_FILE_EDIT set — a common and sensible hardening measure on managed hosts. If the editor is missing, that is usually why, and SFTP is the answer.

Note also that Yoast replaces the WordPress default with a more permissive User-agent: * / Disallow: plus a sitemap line. That is fine, but it does mean you silently lose the /wp-admin/ rule unless you re-add it.

Editing robots.txt with Rank Math, AIOSEO or SEOPress

These three edit the virtual file rather than writing one to disk. In Rank Math the editor is under General Settings → Edit robots.txt (with Advanced Mode on); in AIOSEO under Tools → Robots.txt Editor; in SEOPress it is part of PRO.

The catch is the precedence rule above. If a physical robots.txt already sits in your web root — left by an old plugin, a host, or a previous Yoast install — these editors cannot override it: Rank Math locks its editor and tells you so, and the others' rules simply never get served. Delete or rename the physical file first, then check the live URL. Still deciding which plugin to run? See Yoast vs All in One SEO, Rank Math and SEOPress.

Editing robots.txt without a plugin

Without an SEO plugin you have two options: upload a physical file over SFTP, or use the robots_txt filter below. The physical file is simpler; the filter keeps the rules in version control and leaves WordPress's sitemap line intact.

The robots_txt filter

If you maintain the site in code and want the rules version-controlled with everything else, filter the virtual output instead of creating a file:

add_filter( 'robots_txt', function ( $output, $public ) {
	if ( ! $public ) {
		return $output;
	}

	$output .= "Disallow: /*?s=\n";
	$output .= "Disallow: /*?attachment_id=\n";

	return $output;
}, 10, 2 );

Respect the $public argument. Appending rules to a non-public site's output is how you end up with contradictory instructions on a staging environment.

Rules actually worth adding

The honest answer for most sites is none. The default is fine, and every rule you add is a rule that can be wrong later. But four additions earn their place on larger or busier WordPress sites.

Internal search results

WordPress search URLs (/?s=keyword) are infinite, generated on demand, and worth nothing in an index. On a site with any traffic, bots will find and crawl thousands of them.

Disallow: /*?s=
Disallow: /search/

This is the clearest win in the list. Google has treated internal search results as low-value for search users for a long time, and crawling them consumes requests that could go to your actual content.

Attachment and feed clutter

Disallow: /*?attachment_id=
Disallow: /*/feed/$
Disallow: /*/trackback/

The $ anchors the feed rule to URLs ending in /feed/, so it does not accidentally match a page whose slug contains the word.

Faceted and filtered URLs

This is the one that matters on WooCommerce stores. Layered navigation generates a combinatorial explosion of URLs — colour × size × price × sort order — that are all near-duplicates of a category page.

Disallow: /*?filter_
Disallow: /*?orderby=
Disallow: /*?min_price=

Check your own parameter names before pasting this; they vary by theme and plugin.

Your sitemap

If you run Yoast or Rank Math, core's wp-sitemap.xml is usually disabled in favour of the plugin's own index. Make sure the Sitemap: line points at the one that actually exists:

Sitemap: https://example.com/sitemap_index.xml

A Sitemap: line pointing at a 404 is a small, common, and entirely avoidable error. It is worth checking after any plugin change.

AI crawlers: the decision robots.txt is actually for now

The interesting robots.txt question in 2026 is no longer about Googlebot. It is about which AI systems may read your content, and for what.

These are separate crawlers with separate tokens, and — this is the part most templates get wrong — blocking a company's training crawler is a different decision from blocking its search crawler. Conflating them is how sites remove themselves from AI search answers while believing they only opted out of model training.

The tokens that matter

The trade-off, stated honestly

OpenAI's documentation is explicit that sites disallowing OAI-SearchBot will not appear in ChatGPT's search answers. Anthropic and Perplexity say blocking Claude-SearchBot or PerplexityBot can cost you visibility in their search results. If a meaningful share of your audience now asks an assistant instead of typing into Google, blocking those crawlers can take you out of that surface — and unlike a Google ranking drop, nothing will alert you.

Google-Extended is the cleanest decision on the list, because Google has removed the ambiguity:

Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.

So blocking Google-Extended costs you nothing in Search. Blocking Googlebot costs you everything. They are not the same token and the difference is not cosmetic.

One more nuance worth knowing: ChatGPT-User and Claude-User fetch a page because a specific person asked a question. OpenAI notes that robots.txt rules may not apply to ChatGPT-User's user-initiated fetches; Anthropic says Claude-User follows robots.txt, and blocking it stops Claude fetching your pages for those questions. Treating them as crawlers to be blocked is usually a category error — that is a visitor, arriving through an unusual door.

A defensible default

For a business that wants AI visibility but not to donate a training corpus:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Note what is absent: OAI-SearchBot, Claude-SearchBot and PerplexityBot are all left crawlable, because those are the ones that can send you traffic. That asymmetry is the whole point, and it is the opposite of what a copy-pasted "block all AI bots" snippet does.

If you would rather be cited in AI answers than withhold training data — a defensible position, since citation is distribution — add none of this. The one option with no upside is blocking the search crawlers while leaving the training crawlers open, which a surprising number of hand-edited files manage to do by accident.

Also worth being clear-eyed about: robots.txt is a request. Well-behaved operators honour it. Scrapers that are not interested in your preferences will not, and no file at your web root changes that.

Four edits that quietly cost traffic

Each of these appears in widely-copied "optimised WordPress robots.txt" templates. Each is a mistake.

1. Blocking /wp-content/ or /wp-includes/

Disallow: /wp-content/
Disallow: /wp-includes/

This is the most damaging entry on the list and the most frequently recommended.

Google renders pages before assessing them. It needs your CSS to understand layout and mobile usability, and your JavaScript to see content that loads client-side. /wp-content/ contains your entire theme and every plugin's front-end assets. /wp-includes/ contains core's bundled JavaScript, including jQuery, which a great many themes still depend on.

Blocking them means Google fetches your HTML and renders it without styles or scripts. The damage is invisible in a normal browser and obvious in Search Console's URL Inspection tool, which is where you should check after making any robots.txt change.

If your concern is that uploaded files are being indexed, that is an indexing problem. Solve it with noindex, or with X-Robots-Tag headers for non-HTML files.

2. Using Disallow to hide private content

Disallow: /secret-client-portal/

Robots.txt is a public file at a predictable URL. Anyone can read yours right now, and so can anyone scraping for interesting paths. Listing a sensitive directory there advertises it to precisely the audience you were worried about, while providing no access control whatsoever — nothing obliges a bot to honour the file.

Private means authenticated. There is no robots.txt rule that makes something private.

3. Crawl-delay

Crawl-delay: 10

Google ignores this. Its documentation lists the supported fields as user-agent, allow, disallow and sitemap, and states that other fields such as crawl-delay are not supported. Bing does honour it, so it is not useless — but if you added it hoping to reduce Googlebot's load, it did nothing. Crawl rate for Google is managed by Google, which slows down on its own when your server returns errors or responds slowly; Search Console's crawl rate limiter was retired in January 2024.

4. One ruleset copied across environments

A staging site that inherits production's permissive robots.txt gets indexed and competes with the real site. A production site that inherits staging's Disallow: / disappears.

The second failure is worse and more common than people expect — it typically happens during a migration, and because nothing visibly breaks, it can run for weeks before anyone connects the traffic decline to the deploy. Make the robots.txt environment-aware, or make the first item on your go-live checklist fetch /robots.txt on production and read it.

Testing and validating

Three checks, in order.

Read the file. Open yoursite.com/robots.txt in a private window. Not the admin editor — the actual served URL. This catches the physical-file-overriding-your-plugin-edit case immediately.

Test specific URLs. Search Console's URL Inspection tool reports whether a given URL is crawlable and shows you the rendered page as Googlebot sees it. If you have just changed a rule, inspect one URL that should be blocked and one that should not.

Watch for the accidental global block. Search Console raises an "Indexed, though blocked by robots.txt" warning when it has listed a URL it was not allowed to crawl. That message is the symptom of the crawling/indexing confusion described earlier, and it means a page you tried to hide is in the index anyway, with no snippet.

Google retired its old robots.txt Tester. Its replacement is the robots.txt report under Search Console → Settings, which shows the robots.txt files Google found for your site, when each was last fetched, and any lines it could not parse — and lets you request a recrawl of the file after a fix. To test a specific URL against your rules, paste both into the robots.txt validator.

A few operational details worth knowing:

• Google caches robots.txt for up to 24 hours, so a fix is not instant.
• The parse limit is 500 KiB; content past that is ignored. You will not hit this by hand, but generated rulesets occasionally do.
• A 4xx response other than 429 is treated as no restrictions at all. A missing robots.txt is not a problem.
• A 5xx response is treated as a full block for the first 12 hours, then the cached version is used for up to 30 days. A robots.txt that errors during an outage can suspend crawling of the whole site.

That last point is the argument for the virtual file over a physical one on sites with shaky hosting: WordPress generating the file means it is served by the same process as everything else, so it fails at the same time as the rest of the site rather than independently.

Fixing "Blocked by robots.txt" in Search Console

Search Console's Indexing → Pages report lists "Blocked by robots.txt" under the reasons pages are not indexed. It is not automatically a problem: on a WordPress site, /wp-admin/, internal search, cart and checkout URLs belong there. It is a problem when a post, page or product you want in Google shows up in the list.

1. Decide whether the URL should be crawled. Export the list and sort it into "meant to be blocked" and "should be in Google". Only the second group needs work.

2. Confirm the block on one URL. Open URL Inspection for an affected page. Under crawl details it says whether crawling is allowed; "No: blocked by robots.txt" confirms it.

3. Find the rule. Read the live yoursite.com/robots.txt, then test the URL against it in the robots.txt validator, which shows the rule that matches. The usual culprits on WordPress are a broad wildcard (Disallow: /*? also blocks every URL with a query string), a folder rule broader than intended, a leftover Disallow: / from staging, or a named group for Googlebot that ignores the rules you wrote under User-agent: *.

4. Fix it where the file actually lives. If a physical file exists, edit that — a plugin edit will not show. Otherwise use your SEO plugin's editor or the robots_txt filter. Then reload the live URL to confirm the change is served.

5. Ask Google to look again. In the robots.txt report, request a recrawl of the file. Then use URL Inspection → Request indexing on your most important affected pages, and click Validate fix on the "Blocked by robots.txt" issue so Search Console tracks the recovery.

"Indexed, though blocked by robots.txt"

This one is the opposite case: Google listed the URL even though it was not allowed to crawl it, usually because other pages link to it. The result is a search listing with no proper title or snippet.

• If the page should rank, remove the block. Google can then crawl it and show a real snippet.
• If the page should not be in Google, remove the block and add a noindex robots meta tag. Google has to be able to crawl the page to see the noindex — a blocked page can never be de-indexed by one. Once it has dropped out of the index, you can decide whether blocking crawling again is worth it.

Noindex, nofollow and the robots meta tag covers the noindex side in detail.

When traffic drops and you suspect robots.txt

A robots.txt mistake has a recognisable signature: impressions fall across a whole section at once rather than page by page, the decline starts on a deploy date rather than a Google update date, and nothing about the pages themselves changed. Work through it in this order.

1. Fetch the live file. curl -s https://yoursite.com/robots.txt. Use the production hostname, not a staging alias, and not the admin editor. If you see Disallow: /, you have found it — that is a staging file that shipped.

2. Check what changed and when. If the file is physical and in version control, git log on it. If it is not in version control, that is the real finding. A plugin update, a host migration, or a security tool can all replace this file without announcing it.

3. Inspect a specific affected URL. Search Console → URL Inspection. It states plainly whether crawling is allowed; test the URL against the live file in a validator to see which rule matches. Do this before changing anything, so you can prove the fix.

4. Look for "Indexed, though blocked by robots.txt". In Search Console's Pages report, this means Google listed URLs it could not read. Those pages are in the index with no usable snippet, which converts far worse than not ranking at all.

5. Fix, then wait. Google caches robots.txt for up to 24 hours, and re-crawling the affected pages takes longer than that — days to weeks depending on how often the section is normally crawled. There is no way to make this instant. Requesting indexing on a handful of your most important URLs is the only accelerant available, and it is a modest one.

The prevention is unglamorous: put robots.txt in version control, and make "fetch the production robots.txt" an explicit line on the go-live checklist. Both of the expensive failures in this article — the inherited staging block and the blocked asset directory — are caught in the ten seconds that check takes.

On a WordPress site, RankHive runs the Search Console half of this every week. Its audit looks for pages Google isn't indexing and for robots.txt mistakes, and drafts the robots.txt edit through its plugin; nothing changes until you approve it. Every robots.txt change it ships is logged, and Revert in the Change Log restores the previous file. Its lines are added to the virtual file, so a physical robots.txt hides them just as it hides a plugin editor's; the integration pages for Yoast, Rank Math and SEOPress cover how they sit next to your SEO plugin's rules.

Subdomains, multisite and protocol

Robots.txt applies to exactly one host, protocol and port. https://example.com/robots.txt governs https://example.com and nothing else.

That means:

• shop.example.com needs its own robots.txt. The root domain's file does not apply.
• http:// and https:// are technically separate origins for this purpose, though in practice your redirect to HTTPS handles it.
• In subdomain multisite, each site serves its own virtual robots.txt. This works correctly by default.
• In subdirectory multisite, only the network's root site can serve /robots.txt, because that is the only site at the top level. example.com/site-two/robots.txt is not a robots.txt file — it is just a page at that path, and no crawler will look for it. Rules for the whole network have to live in the root file.

The subdirectory-multisite case catches agencies out regularly: rules added through a sub-site's Yoast editor appear to save and have no effect whatsoever.

A sensible starting point

For a standard content site, the WordPress default plus internal search:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /*?s=
Disallow: /search/

Sitemap: https://example.com/sitemap_index.xml

For WooCommerce, add your faceted-navigation parameters and the cart and checkout paths, which are user-specific and have no business being crawled:

Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*?filter_
Disallow: /*?orderby=

Leave /wp-content/uploads/ alone. Product images belong in Google Images.

Change the sitemap URL to whichever one your setup actually serves, and verify it returns 200 before you move on.

Where this fits in a WordPress SEO setup

Robots.txt is a crawl-budget and tidiness tool. On a site with a few hundred pages, a perfect robots.txt is worth close to nothing, and a bad one can cost you real traffic — which makes the expected value of aggressive editing distinctly negative. The default is the right answer more often than any tutorial will admit.

The asymmetry is the point. There is no version of this file that makes a small site rank better. There are several versions that make it rank worse.

If you want the related directives covered properly, noindex, nofollow and the robots meta tag explains which tool does what, and the technical SEO checklist puts robots.txt in sequence with the other crawl and indexation checks worth running before you publish more content.

Your job with robots.txt is mostly to keep the default and notice fast when something changes it. Check it by hand with the steps above, or let RankHive watch Search Console for robots.txt problems each week and ship the fix after you approve it.