Questions about Robots.txt

you can just make a file in your root named robots.txt if you don’t already have one

User-agent: *
Disallow: /myPage

is the format for that one page
there is no allow, only disallow. disllowing pages you follow the above format

WP may not have generated a robots.txt, I believe it only would have if you wanted your site not indexed

I’m not sure if WP has inbuilt mechanism for not indexing your admin stuff….but mine’s never been indexed. If you are worried, to block entire directories you add

Disallow: /wp-admin/

it’s one entry per line/disallow

Related Posts:

  1. Stuck on my server root folder, robots.txt file not deleting
  2. Unwanted “crawl delay: 10” line added to my robots.txt
  3. Why does Google Webmaster Tools say my whole site is blocked by robots.txt when I have no robots.txt?
  4. How to stop WordPress showing a virtual Robots.txt?
  5. Where is the “Privacy” menu for the “WP Robots Txt” plugin?
  6. Robots.txt with WordPress and Joomla in same server
  7. How to prevent Google from indexing the /wp-content/ directory?
  8. Is this robots.txt making Google not to show some of my content?
  9. What is the meaning of User-agent and Disallow in robots.txt file in wordpress
  10. Why are some user agents still getting a meta robots tag with noindex from my site?
  11. Question about robots.txt file on WordPress site
  12. Google Webmaster Tools error “Url blocked by robots.txt.”
  13. What is a good robots.txt?
  14. Robots.txt not updating
  15. NoFollow Entire Website
  16. Correct way to hide pseudo pages from being shown?
  17. Create unique robots.txt for every site on multisite-installation
  18. Generating robots.txt dynamically
  19. Preventing Search Engines Indexing Pages 2, 3 and More?
  20. Add robots.txt to root
  21. Google saying is blocked by robots.txt without robots.txt on my website
  22. Hide pages from google and visitors
  23. “No sitemap linked in your robots.txt file”
  24. Disallow author with robots.txt
  25. Worried I have a funky .htaccess for WP site in light of strange search engine behavior
  26. Set noindex page-comment from Pages 2, 3 and More?
  27. robots.txt Not Updating
  28. How to solve Robots.txt problem with WordPress fresh install?
  29. Why does Google Image Search still display my images?
  30. install robots.txt to wordpress website admin
  31. Virtual robots.txt missing
  32. Why does do_robots() Allow: /wp-admin/admin-ajax.php by default?
  33. Can you please verify my robots.txt for wordpress
  34. Where to put robot.txt and sitemap.xml for wordpress installed in its own directory
  35. How to remove WordPress Virtual Robots.txt that keeps overriding my root robots.txt? [closed]
  36. Old robots.txt file not changing, can’t update to the current robots.txt
  37. Disallow: /wp-* in robots.txt?
  38. Trying to update robots.txt [closed]
  39. auto generated robots.txt problem
  40. Setting up robots.txt with Yoast’s Seo plugin
  41. How to dissallow all wordpress posts in robots.txt?
  42. Hide plugins and theme from public
  43. Use a filter for wp_robots to block CPT/feed/
  44. WordPress site using Yoast SEO being blocked from indexing by robots.txt
  45. How to stop google from indexing spammed internal search results?
  46. Bing/msn bots is heavily requesting random of my website
  47. How to noindex, follow a specific category wordpress?
  48. Robots.txt file not updating
  49. wp_redirect and search engine bots
  50. Disable Ajax for Spiders
  51. No exposure on search engines
  52. How to view the code for a certain page?
  53. Google is unable to crawl Robots.txt file of my website and please help me
Hata!: SQLSTATE[HY000] [1045] Access denied for user 'divattrend_liink'@'localhost' (using password: YES)