For those who are using WordPress as CMS for their site, you can bypass your web hosting server rules by simply removing your robots.txt file and instead modifying the virtual one generated by WordPress. You just have to add a filter to the functions.php file of your theme. Here’s the code snippet: //* Append directives … Read more
I just tested the ‘robots_txt’ filter on a single installation to modify the output of the virtual /robots.txt that WordPress displays and it worked fine for me: add_filter(‘robots_txt’, ‘wpse_248124_robots_txt’, 10, 2); function wpse_248124_robots_txt($output, $public) { return ‘YOUR DESIRED OUTPUT’; } What is really happening when you try to reach /robots.txt? Does it display the default … Read more
Straight from the source, (line 1845 wp-includes/functions.php, 3.3.1): function do_robots() { header( ‘Content-Type: text/plain; charset=utf-8’ ); do_action( ‘do_robotstxt’ ); $output = “User-agent: *\n”; $public = get_option( ‘blog_public’ ); if ( ‘0’ == $public ) { $output .= “Disallow: /\n”; } else { $site_url = parse_url( site_url() ); $path = ( !empty( $site_url[‘path’] ) ) ? … Read more
Well those are two different things: A redirect means they are not accessible any more at all. noindex just means that search engines ignore it while it is still accessible if you access the URL. So I’d recommend option 1. This is a simple way of doing this that you can improve an. (E.g. this … Read more
WordPress generate a dynamic robots.txt which does not physically exists. To remove/disable it you have two options: Option 1: Remove do_robots action in your theme functions.php or plugin remove_action(‘do_robots’, ‘do_robots’); The action do_robots is still available to be added again by other plugins. Option 2: Create a real robots.txt file, put it the root folder … Read more
Thought this was a great question so I went digging. In default-filters.php on line 208 there’s add_action(‘wp_head’, ‘noindex’, 1); as of WordPress 4.1. The noindex() function in turn checks to see if you have set blog_public option to 0. If you have, it calls wp_no_robots() which is simply: function wp_no_robots() { echo “<meta name=”robots” content=”noindex,follow” … Read more
Once I experienced the same issue, this is what I did to fix the issue. Edit the robots.txt file directly (using FTP/SSH), User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ There are two reasons if the robots files not updated when you edited using a plugin. File permission. Some other plugin is reverting the changes. Also try … Read more
FWIW, trackback URLs issue redirects and have no content, so they won’t get indexed. And at the risk of not answering the question, RE your points 2 and 3: http://googlewebmastercentral.blogspot.com/2008/09/demystifying-duplicate-content-penalty.html Put otherwise, I think you’re wasting your time worrying about dup content, and your robots.txt should be limited to: User-agent: * Disallow: /cgi-bin Disallow: /wp-admin … Read more