You should use the robots_txt filter to exclude all of the URLs of posts written by specified author (untested): add_filter( ‘robots_txt’, static function ( $output ) { $query = new WP_Query( array( ‘author’ => 3, ‘posts_per_page’ => -1, ‘no_found_rows’ => true, ‘fields’ => ‘ids’, ) ); if ( ! $query->have_posts() ) { return $output; } … Read more
The WP robots.txt file is a virtual file, created on demand. You can see the created file by entering a URL like this https://www.example.com?robots=1 . The virtual file is generated and used if there is not an actual robots.txt file. If you have an actual one, the virtual file is not generated or supplied on … Read more
You can disable the robots.txt file by putting this code in an mu-plugin: remove_action( ‘do_robots’, ‘do_robots’ ); However, I don’t recommend that. It will make it harder for search engines to index your content, and it won’t really do anything to hide the fact that you’re using WordPress. There are a dozen ways that someone … Read more
The current/computed directives that WP generates for a virtual robots.txt file are not available with any function call or filter. The directives are only available with a page request similar to https://www.example.com?robots=1 . That page request will result in WP generating and returning the contents of the virtual robots.txt file, along with a Header. The … Read more
Try to go Settings –> Reading, take a look if Search Engine Visibility is checked, and in this case unchecked. Now the robots.txt has disallow only wp-admin path
Here is some info about robots.txt. It looks like your example wouldn’t work correctly with the * but, it looks to me like you’d be ok if you just removed the *. You can always check the robots.txt at with a validator like this one
I was editing stuff in default-filters.php and then cleared my wp cache and it finally went away. Then I undid my change to default-filters.php and it stayed away. So I’m inclined to think, this was a caching issue?
User-agent They are Web Robots (also known as Web Wanderers, Crawlers, or Spiders), are programs that traverse the Web automatically. Search engines such as Google use them to index the web content, spammers use them to scan for email addresses, and they have many other uses. Disallow It tells the robots that it should not … Read more
If you changed your site path, the urls previously indexed by search engines are no longer valid. Google Webmaster Tools has a change of address tool: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=83106 In general, you need to : setup 301 redirects for your old urls create a new sitemap be patient
Here’s the thing There are multiple ways to do what you want, from adding meta tags, to passing headers, but because, you tagged your question with robots.txt So i consider it off-topic to discuss any other solutions. Considering your demand you need to have this as your robots.txt This disallowed access to wp-admin, but depending … Read more