Writing a robots.txt is an easy process. Follow these simple steps: Open Notepad, Microsoft Word or any text editor and save the file as ‘robots,’ all lowercase, making sure to choose .txt as the file type extension (in Word, choose ‘Plain Text’ ). Next, add the following two lines of text to your file: User-agent: … Read more
Disallow with no trailing slash is the same as “allow all” – see here – nice find!
If you have an actual robots.txt file in the root of your site then WP/plugins will be unable to override it. Its related functionality works with “virtual” file, when actual file doesn’t exist. Otherwise it might be plugin conflict, try one related plugin at a time.
User-agent They are Web Robots (also known as Web Wanderers, Crawlers, or Spiders), are programs that traverse the Web automatically. Search engines such as Google use them to index the web content, spammers use them to scan for email addresses, and they have many other uses. Disallow It tells the robots that it should not … Read more
It seems to be a WP default setting, as many Webmasters have gotten this warning and never edited the robots.txt. Removing all the disallows is the easiest solution, but I assume you want some or all of those directories blocked. Google is only concerned about the .js and .css files, so you could in theory … Read more
Just delete file robots.txt. WordPress is generating virtual robots.txt file, and Yoast SEO plugin (and other plugins too) allows you to customize it. But, if you have an actual robots.txt file, it will prevent virtual one to work.
Okay, after digging into various areas, I noticed I have a security plugin that is blocking these changes. Problem solved by whitelisting, 🙂 in case anyone else has this issue, check your security settings.
You can safely place your robot.txt file as well as the sitemap.xml file into your websites root directory and NOT the WordPress directory. When a robot visits your site, the first place it checks is http://sitename.com/robots.txt, hence you want to keep these files at the root of your domain. You can also disallow what the … Read more
The file robots.txt doesn’t exclude all the search engines to crawl your site… Most search engines crawl the hole site without taking the robots.txt file in consideration. I think that the best way is to use meta tags in the theme of the blogs. Use the next meta tags for post and categories template files: … Read more
If you changed your site path, the urls previously indexed by search engines are no longer valid. Google Webmaster Tools has a change of address tool: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=83106 In general, you need to : setup 301 redirects for your old urls create a new sitemap be patient