Google is unable to crawl Robots.txt file of my website and please help me

Today I got a mail from Google webmasters, that Google bots are unable to find robots.txt file of my website.

This is not actually a problem (from crawling perspective at least, it might be for excluding things from crawling, which the file primarily used for). The robots.txt is neither mandatory on necessary for your site to be crawled.

I also tried to locate the file by http://www.mydomain.com/robots.txt but the file isn’t there. Instead the message is displayed as below.

This actually is robots.txt. 🙂 That is what rules in it look like.

If your installation doesn’t have physical robots.txt file than WordPress will be serving virtual one, using do_robots() to generate it.

It is hard to say why Google wasn’t to access the file (since you seemingly are able to). It might have been temporary issue that will clear itself, otherwise you might want to contact hosting and inquire if something might be interfering with Google bot accessing your site.

Related Posts:

  1. Google saying is blocked by robots.txt without robots.txt on my website
  2. Old robots.txt file not changing, can’t update to the current robots.txt
  3. How can I modify a post depending on the source of the traffic?
  4. Correct way to hide pseudo pages from being shown?
  5. Preventing Search Engines Indexing Pages 2, 3 and More?
  6. Disallow author with robots.txt
  7. Google Adwords Tracking Code causing too many redirects error
  8. Google maps .gm-style div now displays as block, thus breaking the map
  9. Rich Snippets for WordPress
  10. Automatically post to Google+ on new post
  11. Remove Query String from Google jQuery
  12. Code syntax error, “userfullName undefined”
  13. How to remove Google Analyticator in WordPress site?
  14. Google disapproved Ads due to malware, but no malware in site? [closed]
  15. Disallow: /wp-* in robots.txt?
  16. Google Apps Domain for WordPress
  17. How to fetch csv file and show in shortcode
  18. How to download finished WordPress project from Google Cloud?
  19. How find what plugins load google Font and block it
  20. Will dequeueing in child theme functions.php file prevent Google Fonts from loading?
  21. Google flagged a wp-admin redirect as phishing
  22. How to non index my website page links?
  23. How to view the code for a certain page?
  24. Google Webmaster Tools error “Url blocked by robots.txt.”
  25. Is there a blank theme framework compatible with WP 3.0? [closed]
  26. How to add plugin to WordPress Repository?
  27. Plugin SVN & update API – how are plugins identified?
  28. Change admin bar to default:off
  29. Custom Search Query
  30. How do I unlock a post programmatically?
  31. Add custom objects/entities to WordPress
  32. Template for individual post designs
  33. Managing Images for a WordPress Blog: Picasa or Flickr?
  34. Search results with custom loop don’t update when paged
  35. Extra User Profile Field Upload File / Image
  36. wp_head() remove redundant scripts?
  37. query_posts sort in multiple directions
  38. Type hinting and void return question
  39. Custom Post for Locating by Google Maps API
  40. Leveraging WordPress Comments on Custom Plugin Object page
  41. Remove All, Published and Trashed Post Views in Custom Post Type
  42. custom headers for static home page and posts page
  43. How to check what plugins used to be on a WordPress installation?
  44. Programmatically adding product attributes to WP / Woo commerce
  45. List Recent Post Titles from Custom Taxonomies?
  46. Best way to implement a thumbnail-based gallery in WordPress?
  47. How to remove buttons from the WP Editor on the edit post page in the admin area
  48. How to make theme elements customizable in wordpress?
  49. Conditional Query of Custom Post Type and custom taxonomy
  50. WordPress Locale Switching on Accept-Language Header
  51. className not populating in custom block
  52. Google Tag Manger – no-script Code Not Found
  53. How do I call in a category title on a category page in a child theme?
  54. Include custom JS without child theme
  55. Customization API for plugin development
  56. I need a “Choose from existing content” popin
  57. Custom Log In Screen – Disable password recovery [duplicate]
  58. For homepage images (for small business website), is it better to use custom fields or post_thumbnail?
  59. WordPress 3.1 .htaccess contents keep dissappearing?
  60. How to get every custom taxonomy names and urls?
  61. Best structure for entering recipes in a WordPress theme?
  62. Two description meta tags All in One SEO WordPress [closed]
  63. Create page as a file in my theme?
  64. Need custom picture field for pages
  65. How can I hide custom field from users used for caching response from external api?
  66. How to clean up the theme for production?
  67. How to include custom PHP file both in header and footer files
  68. Changing the the_author_posts_link() hover title
  69. Save button is not working on custom widget
  70. register_post_type permalink
  71. How to add custom ‘layout’ buttons to wordpress editor?
  72. How to handle a WordPress blog with articles in different languages?
  73. Call javascript functions from each page
  74. Custom Register Link on Backend Form
  75. Prevent WordPress from automatically installing a new theme each year
  76. How to remove font awesome from loading in wp theme
  77. customer specific email attachment
  78. How do I run a web app in my WordPress theme without the “wp-content/theme/…/webapp” in URL?
  79. Use Theme CSS in custom code
  80. Highly stylized pages
  81. How should I design subcategory login site?
  82. How do I change the button text color on my header?
  83. Remove Domain Mapping
  84. Redirecting the lost password page request when using a custon login page
  85. Execute function with jQuery if widget added or removed?
  86. Customizing WooCommerce with PHP
  87. Check get_post value after wp-admin login
  88. Multiple Owl Carousels on the same page with navigation problem
  89. Multiple Custom Post Type loop logic
  90. How to disable jQuery tabs when customizing the theme
  91. Gettind 404 not found wen click the view button for a given category custom post type portfolio
  92. How Does One Create a Global Variable Repository
  93. Multiselect value in wp_query
  94. featured images – custom links
  95. wp_redirect and search engine bots
  96. Return $post_id when DOING_AUTOSAVE?
  97. Date when the custom field was added
  98. How to have more post in a page than in your home page
  99. How can i move my product name & price from below thumbnail to be the rollover content in Avada & Woocommerce?
  100. ACF select box css color change