You can disable the robots.txt file by putting this code in an mu-plugin: remove_action( ‘do_robots’, ‘do_robots’ ); However, I don’t recommend that. It will make it harder for search engines to index your content, and it won’t really do anything to hide the fact that you’re using WordPress. There are a dozen ways that someone … Read more
The current/computed directives that WP generates for a virtual robots.txt file are not available with any function call or filter. The directives are only available with a page request similar to https://www.example.com?robots=1 . That page request will result in WP generating and returning the contents of the virtual robots.txt file, along with a Header. The … Read more
Try to go Settings –> Reading, take a look if Search Engine Visibility is checked, and in this case unchecked. Now the robots.txt has disallow only wp-admin path
Today I got a mail from Google webmasters, that Google bots are unable to find robots.txt file of my website. This is not actually a problem (from crawling perspective at least, it might be for excluding things from crawling, which the file primarily used for). The robots.txt is neither mandatory on necessary for your site … Read more
If they are 404, you can’t put a noindex on them! You can put rel=nofollow on the links that link to the removed pages. It also helps to have a sitemap so that Google can track changes more easily.
Please go to Settings > Reading or wp-admin/options-reading.php If it’s not unchecked it will not allow search engines to reach. For further modification information related to robots you can use the below plugin to modify robots.txt file: Plugin – Multipart robots.txt editor To see your robots.txt file you would visit the link as: http://www.example.com/robots.txt
Here’s a programatic method. What it does it checks if an ajax request is running and if the useragent to see if it is in the defined whitelist and if it is then it continues loading wp as normal, else it dies. I used registered_taxonomy because it’s the second hook in the hook loading processes: … Read more
You can always noindex those pages from your robots.txt file. User-agent: * Disallow: /all-bots/noindex-this-page.html
Here is some info about robots.txt. It looks like your example wouldn’t work correctly with the * but, it looks to me like you’d be ok if you just removed the *. You can always check the robots.txt at with a validator like this one
I was editing stuff in default-filters.php and then cleared my wp cache and it finally went away. Then I undid my change to default-filters.php and it stayed away. So I’m inclined to think, this was a caching issue?