Can you please verify my robots.txt for wordpress

The file robots.txt doesn’t exclude all the search engines to crawl your site… Most search engines crawl the hole site without taking the robots.txt file in consideration.

I think that the best way is to use meta tags in the theme of the blogs.

Use the next meta tags for post and categories template files:

<meta name="robots" content="index, follow" /> 
<meta name="googlebot" content="index, follow" />

Use this meta tags for the rest of the template files

<meta name="robots" content="noindex, nofollow, noarchive" />
<meta name="googlebot" content="noindex, nofollow, noarchive" />

Related Posts:

  1. Create unique robots.txt for every site on multisite-installation
  2. How to use WordPress Multisite With Different Domain Names?
  3. Can i merge 2 new WP_Query($variable) ‘s?
  4. How to reliably flush rewrite rules on multisite?
  5. Convert a Multisite WordPress install into a Single site
  6. Using wp_insert_post() with Networking enabled
  7. Sharing Dynamic Sidebars across Multisite Blogs
  8. WP Multisite – multilanguage – swich to other language by click, but “same” content
  9. Multisite: How to store global options vs. site options?
  10. Hide a theme on list of themes in wp-admin without editing core files
  11. Lock one of WP Network sites for not registered users
  12. How can I manage all multisite pending comments in one place?
  13. WordPress Multisite Network installation and dev questions
  14. List all sites in multisite
  15. Migrating WPMS from apache to IIS7 causes Database Error
  16. Turn off trackback/pingbacks and comments on all existing multisite sites
  17. My subsites accidentally went from one multisite network to another. How do I change it back?
  18. How to change a WordPress network domain
  19. Subfolder multisite issue. Themes and plugin files 404
  20. Creating Admin Options Page where users can make changes to a theme
  21. Creating multisite in wordpress using php
  22. How to use both British and American English?
  23. Multisite list all pages in site
  24. WordPress Multisite Sites show in Dropdown but not in Sites menu
  25. WP Multisite Static Frontpage
  26. Disable domain redirect
  27. Multisite setup on IIS with subdomains
  28. Partially translated dashboard in one of network’s site
  29. How can I fix a Virtual Multiblog install that has no errors, but won’t show all my sites?
  30. Depth > 2 possible with multisite?
  31. What are Attributes used for in a WordPress Multisite?
  32. Make a user administrator to a sub directory site and a contributor to main site in multisite network
  33. How to store custom user data on the database?
  34. Redirecting non www to www ONLY for domain, subdomain without www
  35. NGINX rewrite rules for sub subdomains
  36. wordpress multisite dashboard link in admin bar
  37. WordPress Multisite Question
  38. Multisite – user email activation links broken
  39. Wrong Redirect/Rewrite Rules?
  40. WP MU Register User across all blogs
  41. multisite subdomain behind reverse proxy
  42. Is this the right set of records for Route53 domain to point to a mapped domain in a multisite? [closed]
  43. Multisite Subdirectory with same domain subdomain
  44. Initiate multisite with blog_id outside of wordpress
  45. wp_mail() not sending email in multisite subdomain
  46. What are good approaches to routing some urls to a second WordPress install?
  47. Show media-frame-menu on custom Media Frame
  48. How to run a WordPress site across multiple ccTLDs?
  49. Why is MySQL receiving such a large serialized object?
  50. Google Apps SMTP for WP MultiSite?
  51. How to Display BuddyPress Profile Field Data In Multisite Blogs [closed]
  52. Set up Network Locally on a Flash Drive
  53. Help with multisite redirect issues using Cherokee OR solid nginx config?
  54. Get users registered 30 days ago(or more)
  55. Pull Custom /wp-admin/css for Non-Admin in Multisite
  56. Being Forced To Use Sub-Domains on Multisite
  57. WordPress in subdirectory, Multisite using subdomains Nginx
  58. sub-page settings saving in NETWORK DASHBOARD
  59. Check if current site ID and value exist in WPDB
  60. How to delete user from MU site when the user is removed from their site?
  61. Multi-vendor with custom shop site for each vendor
  62. You need to login with a user that has edit posts capacity?
  63. How to publish some default pages in Multisite creation?
  64. Change WP-LOGIN and WP-SIGNUP for any request
  65. unique identifier for the same pages in a multilanguage/site context
  66. more tables created when create a new site
  67. How to automate setup of theme, pages, widgets, etc for a new site in multi-site?
  68. Multisite ‘Welcome User Email’ SITE_NAME returns ‘network’ name, not the name of the blog
  69. Unified login page for a Multisite/BuddyPress site
  70. Problem with new Multisite Installation
  71. Merge posts of more than one site by using one database with the WordPress network
  72. User registration on wordpress multisite
  73. WP-CLI Error Requesting CURL Extension
  74. Website creation for multiple clients but from a single account on multiple host name
  75. Is mature Multisite
  76. What is the optimal solution: WP Multisite vs InfiniteWP
  77. Best way to have multiple custom domains associated with a site within a multisite installation
  78. Subdomain multisite instances having same content
  79. Can I add a rewrite rule in htaccess to remove the multisite subdirectory from the URL?
  80. allow multisite in the installation process
  81. Error establishing a database connection after moving local wordpressmultisite from one directory to another
  82. WordPress single sign on using cookies with shared user role functionality between more than 2 wordpress subdomains
  83. How to split database of WordPress multi-site network into multiple database
  84. Multisite /files/ URLs broken after server move
  85. WordPressMU: URL change and broken permalinks/links
  86. Extended URLs created using multisite feature not loading CSS/JS
  87. Backend freezing on certain pages of a subsite
  88. Modifying get_author_posts_url() in Multisite
  89. Multisite – User creation for second site from first site?
  90. Get users from all/specific blog by user_role and current_user role
  91. How do I enable Multisite on WP 3.7.1?
  92. Multiple domains for with a single instal
  93. Pages redirect me to the homepage
  94. Using common slider in all sites
  95. Letting clients use visual builders plugins/themes on WordPress Multisite installation
  96. WordPressMu network with private sites
  97. Moving a WordPress Multisite site to a new install
  98. HyperDB, Multisite – on new site creation put site’s tables to specific database
  99. Change a subsite Admin role of a WordPress Multisite after 24 hours registering
  100. different image for mobile template block theme