It seems to be a WP default setting, as many Webmasters have gotten this warning and never edited the robots.txt. Removing all the disallows is the easiest solution, but I assume you want some or all of those directories blocked. Google is only concerned about the .js and .css files, so you could in theory … Read more
Just delete file robots.txt. WordPress is generating virtual robots.txt file, and Yoast SEO plugin (and other plugins too) allows you to customize it. But, if you have an actual robots.txt file, it will prevent virtual one to work.
Okay, after digging into various areas, I noticed I have a security plugin that is blocking these changes. Problem solved by whitelisting, 🙂 in case anyone else has this issue, check your security settings.
You can safely place your robot.txt file as well as the sitemap.xml file into your websites root directory and NOT the WordPress directory. When a robot visits your site, the first place it checks is http://sitename.com/robots.txt, hence you want to keep these files at the root of your domain. You can also disallow what the … Read more
The file robots.txt doesn’t exclude all the search engines to crawl your site… Most search engines crawl the hole site without taking the robots.txt file in consideration. I think that the best way is to use meta tags in the theme of the blogs. Use the next meta tags for post and categories template files: … Read more
If you changed your site path, the urls previously indexed by search engines are no longer valid. Google Webmaster Tools has a change of address tool: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=83106 In general, you need to : setup 301 redirects for your old urls create a new sitemap be patient
Here’s the thing There are multiple ways to do what you want, from adding meta tags, to passing headers, but because, you tagged your question with robots.txt So i consider it off-topic to discuss any other solutions. Considering your demand you need to have this as your robots.txt This disallowed access to wp-admin, but depending … Read more
the truth is that probably nothing should be blocked in robots.txt by core (this is IIRC was joost’s position abot the matter) as wordpress is an open platform and front end content and styling might and is generated in all kinds of directories which might not make much sense to you and me. WordPress is … Read more
Robots.txt with WordPress and Joomla in same server
Nevermind. It just doesn’t work when wordpress is in a subfolder. (robots.txt should be on website root so just create your own if wordpress is in a subfolder)