Generating robots.txt dynamically

by

I just tested the ‘robots_txt’ filter on a single installation to modify the output of the virtual /robots.txt that WordPress displays and it worked fine for me:

add_filter('robots_txt', 'wpse_248124_robots_txt', 10,  2);

function wpse_248124_robots_txt($output, $public) {

  return 'YOUR DESIRED OUTPUT';
}

What is really happening when you try to reach /robots.txt? Does it display the default robots.txt content or a 404? If you’re getting a 404, then you might have Apache or Nginx rules that are not allowing the /robots.txt request to go through PHP. It is very common to have something like this on a nginx configuration:

# Don't log access to /robots.txt
location = /robots.txt {
    access_log    off;
    log_not_found off;
}

You should replace it with something like:

# Don't log access to /robots.txt
location = /robots.txt {
    try_files     $uri $uri/ /index.php?$args;
    access_log    off;
    log_not_found off;
}

You should also check if the WordPress own internal rewrite rules are working correctly using the Rewrite Rules Inspector (or any other method available) by making sure the following rewrite rule exists:

robots\.txt$ index.php?robots=1

If it doesn’t, you should try to deactivate plugins, activate a default theme and flush the rewrite rules to check if the rewrite rule comes back, but if you have no time for that, just add this rewrite rule to your .htaccess:

RewriteRule robots\.txt$ index.php?robots=1

Leave a Comment