WordPress generate a dynamic robots.txt which does not physically exists. To remove/disable it you have two options: Option 1: Remove do_robots action in your theme functions.php or plugin remove_action(‘do_robots’, ‘do_robots’); The action do_robots is still available to be added again by other plugins. Option 2: Create a real robots.txt file, put it the root folder … Read more
Thought this was a great question so I went digging. In default-filters.php on line 208 there’s add_action(‘wp_head’, ‘noindex’, 1); as of WordPress 4.1. The noindex() function in turn checks to see if you have set blog_public option to 0. If you have, it calls wp_no_robots() which is simply: function wp_no_robots() { echo “<meta name=”robots” content=”noindex,follow” … Read more
Once I experienced the same issue, this is what I did to fix the issue. Edit the robots.txt file directly (using FTP/SSH), User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ There are two reasons if the robots files not updated when you edited using a plugin. File permission. Some other plugin is reverting the changes. Also try … Read more
FWIW, trackback URLs issue redirects and have no content, so they won’t get indexed. And at the risk of not answering the question, RE your points 2 and 3: http://googlewebmastercentral.blogspot.com/2008/09/demystifying-duplicate-content-penalty.html Put otherwise, I think you’re wasting your time worrying about dup content, and your robots.txt should be limited to: User-agent: * Disallow: /cgi-bin Disallow: /wp-admin … Read more