Disallow: /wp-* in robots.txt?

It seems to be a WP default setting, as many Webmasters have gotten this warning and never edited the robots.txt. Removing all the disallows is the easiest solution, but I assume you want some or all of those directories blocked.

Google is only concerned about the .js and .css files, so you could in theory edit the robots.txt to include:

User-Agent: Googlebot
Allow: /.js
Allow: /.css

However, being that specific could require future changes to the user agent, in case more search crawlers follow Google’s example.

You want to make sure you know how robots.txt work so you don’t accidentally block your entire site or important sections. Here is a good reference for more details about robots.txt:

http://www.robotstxt.org/robotstxt.html

Related Posts:

  1. Remove Google Fonts Which Are (Probably) Added By Plugins
  2. Looking for a solution – slightly convoluted problem
  3. install robots.txt to wordpress website admin
  4. Restrict access to content in conjunction with facebook connect & google login
  5. Is there a way to embed a Google Docs form in a page without using plugins?
  6. Stop wordpress from requesting external jquery from googleapi
  7. Getting 400 Error: redirect_uri_mismatch when trying to grant permission to Gmail SMTP plugin
  8. Hide plugins and theme from public
  9. How to add height and width to images on a page(using WP Bakery) to improve Google’s CLS score?
  10. Google Maps for Woocommerce Checkout
  11. Bing/msn bots is heavily requesting random of my website
  12. Accessing data from Google Cloud Datastore in WordPress site
  13. Different function call depending on shortcode?
  14. Speeding up WordPress on a Local Network
  15. Adding google authenticator and use only email address of user
  16. Google adsense stats plugin? [closed]
  17. How to enable suggested edits?
  18. How to change a user’s password programatically
  19. wordpress on localhost lamp doesn’t let me install plugins
  20. WP_User_Query and non-unique usermeta data
  21. How to delete custom taxonomy terms in plugin’s uninstall.php?
  22. To Disable WordPress Rest API or Not To Disable?
  23. how to include other plugins css files in a shortcode?
  24. Redirect to settings page after install
  25. Email Notifications of new posts to users – suggestions [closed]
  26. What is the correct way for a theme to support plugin UIs?
  27. What is the ‘admin_action_’ . $_REQUEST[‘action’] hook used for?
  28. Get URL of a specific file
  29. Building a plugin that doesn’t break when deactivated
  30. How does WordPress determine the primary plugin PHP file?
  31. translation does not work with text_domain
  32. Data sanitization escaping HTML apostrophes
  33. Custom comment type maybe?
  34. Remove Adminstrator Hyperlink from a user having role to add and see users
  35. Enable comments for post with comments meta box removed
  36. Overide a function that is inside a plugin
  37. Having separate plugins and themes folder for multi-site setup
  38. Is there a plugin to record querystring parameters for a page (for customer tracking)? [closed]
  39. Many plugins making WordPress run slow by design?
  40. Why are some of my thumbnails not being generated?
  41. Remove caching from wp_remote_get calls from custom plugin
  42. Detect the activated theme from plugin code
  43. Transient unique names
  44. Add keywords meta to index.php page
  45. Can’t delete Draft Post [closed]
  46. How to check if tables in WordPress still exists after activations
  47. Looking for a way that will collect visitors emails before letting them view the blog posts [closed]
  48. Force file download on plugin activation
  49. Unicode characters displaying as ? after import using WP Clone
  50. How to change the name of the “edit my profile” link in the WordPress admin backend
  51. Why are my queries interfering with the global post variable?
  52. Overwrite a plugin’s CSS?
  53. Problem with isset and $_POST
  54. WooCommerce product and terms count [closed]
  55. MediaElement.js works on one site but not the other?
  56. Moving to a WordPress blog. What do I need to know?
  57. How can I support plugins in a custom theme?
  58. is it possible different visitors location to show different informations?
  59. Couple of beginner’s questions [closed]
  60. Can I filter a function created by a theme or a plugin?
  61. Get all posts, regardless of status
  62. Best way to retrieve data in custom api extended from wp-api
  63. WordPress rewrite rules not working
  64. Ajax request sends url rather than data
  65. All In One SEO Plugin – Category INDEX
  66. Edit post image attributes on fly?
  67. Grouping imported products and showing variations based on selected product
  68. WordPress fatal error from php protocol codes
  69. Replace only the first CSS class
  70. Ultimate Members Default Post Layout problem
  71. General Term for this form Field
  72. how to enable/activate a plugin per Cpanel
  73. Gutenberg Block: How to reload a ServerSideRender?
  74. How to define the slug for my custom plugin?
  75. Is it possible to create duplicate post on other site (either push, on publish, or pull, periodically)?
  76. Run a Change Role Cron Job on WordPress via cPanel
  77. How to display variables from plugin?
  78. using wordpress login details for other website / application / forum?
  79. ACF Repeater Field Question [closed]
  80. Publish check if the edited content has HTML “div”
  81. ACF repeater field
  82. How to remove ‘View Details’ page & updates from third-party plugin?
  83. Which wordpress plugin increases the size of tag as per more number of posts on it?
  84. Redirect default login page to a custom page [duplicate]
  85. appointment plugin doesn’t display all appointments
  86. How do I convince this button to do something when it is clicked?
  87. Doing action based on input from options menu
  88. Images not showing and plugins not installing on wordpress server move
  89. W3 Total Cache and IIS7 not doing much [closed]
  90. Mystery URL http://#/ Listed in XML Sitemap Created by WordPress SEO by Yoast Plugin
  91. “The Events Calendar” Issues in WP 3.2
  92. How can I be a plugin developer? [closed]
  93. How to generate a responsive output from plugin?
  94. save_post hook is not working after changing priority
  95. The plugin generated xxx characters of unexpected output during activation
  96. My wordpress site wont load, it gives header error warnings
  97. Shortcode returning specific content of a post
  98. How to add custom html to the Media > Attachment Details modal?
  99. Unable to access WPadmin, website down
  100. ‘Active lotteries’ only custom query for woocommerce lottery plugin and elementor
techhipbettruvabetnorabahisbahis forumutaraftarium24edusedusedusedueduedusedueduedusedu