Robots.txt Fix in WordPress Multisite

The Problem

WordPress Multisite is an invaluable tool for creating multilingual websites. However, if you’ve ever worked on SEO for such websites, you may have noticed a significant issue: when Multisite operates in subdirectory mode, WordPress fails to correctly generate the robots.txt file. Specifically, it does not include sitemaps for subsites, which can negatively impact your site’s SEO.

Here, I’ll share a code snippet that addresses this issue effectively.

The Code

In WordPress, the robots.txt file is dynamically generated – it doesn’t physically exist on the server. Therefore, to modify it, we need to use WordPress hooks.

To fix the issue, open the functions.php file of your theme and insert the following code:

/**
 * Modifies the robots.txt file.
 *
 * Includes sitemaps for all subsites in the network.
 */
add_filter( 'robots_txt', function ( string $output ) : string {
	foreach ( get_sites() as $site ) {
		if ( get_main_site_id() !== intval( $site->blog_id ) ) {
			$output .= sprintf(
				"Sitemap: %s\n",
				esc_url( get_home_url( $site->blog_id, 'wp-sitemap.xml' ) )
			);
		}
	}

	return $output;
}, 999 );

This code appends lines in the format: Sitemap: https://site.com/subsite/wp-sitemap.xml to the end of the robots.txt file for all subsites in the network, excluding the main site.

Explanation

• Dynamic robots.txt: WordPress doesn’t store robots.txt as a physical file but generates it dynamically. To modify its content, we hook into the robots_txt filter.
• Excluding the Main Site: The code ensures that the sitemap for the main site isn’t duplicated, as it’s already included by default.
• Subsite Sitemaps: The function iterates over all sites in the network and generates sitemap URLs for subsites, appending them to the robots.txt.

Why It Matters

This solution ensures that search engines like Google can properly index all subsites in your WordPress Multisite network. Without these sitemap entries, subsites may remain under-indexed, leading to poor SEO performance.