You should use the robots_txt filter to exclude all of the URLs of posts written by specified author (untested): add_filter( ‘robots_txt’, static function ( $output ) { $query = new WP_Query( array( ‘author’ => 3, ‘posts_per_page’ => -1, ‘no_found_rows’ => true, ‘fields’ => ‘ids’, ) ); if ( ! $query->have_posts() ) { return $output; } … Read more
The WP robots.txt file is a virtual file, created on demand. You can see the created file by entering a URL like this https://www.example.com?robots=1 . The virtual file is generated and used if there is not an actual robots.txt file. If you have an actual one, the virtual file is not generated or supplied on … Read more
You can disable the robots.txt file by putting this code in an mu-plugin: remove_action( ‘do_robots’, ‘do_robots’ ); However, I don’t recommend that. It will make it harder for search engines to index your content, and it won’t really do anything to hide the fact that you’re using WordPress. There are a dozen ways that someone … Read more
The current/computed directives that WP generates for a virtual robots.txt file are not available with any function call or filter. The directives are only available with a page request similar to https://www.example.com?robots=1 . That page request will result in WP generating and returning the contents of the virtual robots.txt file, along with a Header. The … Read more
Try to go Settings –> Reading, take a look if Search Engine Visibility is checked, and in this case unchecked. Now the robots.txt has disallow only wp-admin path
Today I got a mail from Google webmasters, that Google bots are unable to find robots.txt file of my website. This is not actually a problem (from crawling perspective at least, it might be for excluding things from crawling, which the file primarily used for). The robots.txt is neither mandatory on necessary for your site … Read more
If they are 404, you can’t put a noindex on them! You can put rel=nofollow on the links that link to the removed pages. It also helps to have a sitemap so that Google can track changes more easily.
Please go to Settings > Reading or wp-admin/options-reading.php If it’s not unchecked it will not allow search engines to reach. For further modification information related to robots you can use the below plugin to modify robots.txt file: Plugin – Multipart robots.txt editor To see your robots.txt file you would visit the link as: http://www.example.com/robots.txt
Here’s a programatic method. What it does it checks if an ajax request is running and if the useragent to see if it is in the defined whitelist and if it is then it continues loading wp as normal, else it dies. I used registered_taxonomy because it’s the second hook in the hook loading processes: … Read more
You can always noindex those pages from your robots.txt file. User-agent: * Disallow: /all-bots/noindex-this-page.html