Here’s the thing There are multiple ways to do what you want, from adding meta tags, to passing headers, but because, you tagged your question with robots.txt So i consider it off-topic to discuss any other solutions. Considering your demand you need to have this as your robots.txt This disallowed access to wp-admin, but depending … Read more
the truth is that probably nothing should be blocked in robots.txt by core (this is IIRC was joost’s position abot the matter) as wordpress is an open platform and front end content and styling might and is generated in all kinds of directories which might not make much sense to you and me. WordPress is … Read more
Robots.txt with WordPress and Joomla in same server
Nevermind. It just doesn’t work when wordpress is in a subfolder. (robots.txt should be on website root so just create your own if wordpress is in a subfolder)
There are two possible solutions, you can try: Solution 1 – Changing Uploads Directory In WordPress Settings If you have moved your WordPress to a new hosting recently. There is great chance that path to your WordPress uploads directory is incorrect. Follow below steps to correct/verify settings. Sign in as admin on WordPress click on … Read more
This took the index some more time, but after around six weeks, the blocked images are no longer displayed as search results. I am not sure if this will help for future uploads for photos of the same people with different filenames, but I had added strings with placeholders like these: User-Agent: Googlebot-Image Disallow: /wp-content/uploads/*adam*smith*.jpg … Read more
you can just make a file in your root named robots.txt if you don’t already have one User-agent: * Disallow: /myPage is the format for that one page there is no allow, only disallow. disllowing pages you follow the above format WP may not have generated a robots.txt, I believe it only would have if … Read more
Your robots file as of right now: User-agent: Googlebot Disallow: User-agent: * Disallow: Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /wp-includes/ Sitemap: http://www.yadfaeq.com You aren’t specifying anything for Googlebot so you should remove it. Additionally, Disallow: could be interpreted as Disallow: / which would block your entire site. What your robots file should be: User-agent: * Disallow: … Read more
From WordPress v3.5 this menu has moved to “Settings” -> “Reading” From here you can easily update the text of your robots.txt file. See https://wordpress.org/support/topic/wp-robots-txt-privacy-doesnt-appear-in-settings for further info
Normally, if there’s a WordPress file on disk, that’ll be served first directly by Apache or Nginx, before WordPress gets involved. This is done in your virtualhost config, e.g. in Nginx you’ll typically find the following, which tells it to try actual files first before letting index.php handle the URL and generate a page on … Read more