robots.txt

Here is some info about robots.txt. It looks like your example wouldn’t work correctly with the * but, it looks to me like you’d be ok if you just removed the *. You can always check the robots.txt at with a validator like this one

User-agent They are Web Robots (also known as Web Wanderers, Crawlers, or Spiders), are programs that traverse the Web automatically. Search engines such as Google use them to index the web content, spammers use them to scan for email addresses, and they have many other uses. Disallow It tells the robots that it should not … Read more

If you changed your site path, the urls previously indexed by search engines are no longer valid. Google Webmaster Tools has a change of address tool: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=83106 In general, you need to : setup 301 redirects for your old urls create a new sitemap be patient

Here’s the thing There are multiple ways to do what you want, from adding meta tags, to passing headers, but because, you tagged your question with robots.txt So i consider it off-topic to discuss any other solutions. Considering your demand you need to have this as your robots.txt This disallowed access to wp-admin, but depending … Read more

Robots.txt with WordPress and Joomla in same server

you can just make a file in your root named robots.txt if you don’t already have one User-agent: * Disallow: /myPage is the format for that one page there is no allow, only disallow. disllowing pages you follow the above format WP may not have generated a robots.txt, I believe it only would have if … Read more

From WordPress v3.5 this menu has moved to “Settings” -> “Reading” From here you can easily update the text of your robots.txt file. See https://wordpress.org/support/topic/wp-robots-txt-privacy-doesnt-appear-in-settings for further info

Two options: Create a static file robots.txt. Highly recommended. Filter ‘robots_txt’: add_filter( ‘robots_txt’, ‘wpse_77969_robots’ ); function wpse_77969_robots() { status_header( 204 ); return ”; }