Two options: Create a static file robots.txt. Highly recommended. Filter ‘robots_txt’: add_filter( ‘robots_txt’, ‘wpse_77969_robots’ ); function wpse_77969_robots() { status_header( 204 ); return ”; }
Add a simple checkbox to the post editor to toggle the generation of a meta field that disallows search engines to index your content. Then hook into wp_head, check for the setting and print that field. Sample code for a plugin: add_action( ‘post_submitbox_misc_actions’, ‘show_noindex_checkbox’ ); add_action( ‘save_post’, ‘save_noindex’, 10, 2 ); add_action( ‘wp_head’, ‘insert_noindex_meta’ ); … Read more
Despite the fact your site lacks a robots.txt file, it might be worth making one to allow for Googlebot and other bots to crawl your website. Please use this example: User-agent: * Allow: / Disallow: /wp-content Note: Please remove or amend the /wp-content disallow on the robots.txt, if for whatever reason, you’d like your images … Read more
First of all, in order for WordPress to generate a robots.txt for you you must be using a non-default permalink structure. Make sure you’ve selected an option in the Settings > Permalinks menu. Also, if a robots.txt file exists at your root directory it will override the setting in WordPress. It looks like you already … Read more
How exactly are you setting up your home page? I think the problem is with it having unwanted pagination in first place and not that pagination being indexed. In general robots.txt file is good way to prevent indexing in bulk. I think it would be following directive in your case (please test it so it … Read more
Similar problem occurred to me, and I got the solution following these: Step 1: Take a backup of your .htaccess file and then remove it (Don’t worry, on next refresh WordPress will create one for you) Step 2: If there’s no robots.txt exists, create one with blank page Step 3: Resubmit the sitemap to google … Read more
For those who are using WordPress as CMS for their site, you can bypass your web hosting server rules by simply removing your robots.txt file and instead modifying the virtual one generated by WordPress. You just have to add a filter to the functions.php file of your theme. Here’s the code snippet: //* Append directives … Read more
I just tested the ‘robots_txt’ filter on a single installation to modify the output of the virtual /robots.txt that WordPress displays and it worked fine for me: add_filter(‘robots_txt’, ‘wpse_248124_robots_txt’, 10, 2); function wpse_248124_robots_txt($output, $public) { return ‘YOUR DESIRED OUTPUT’; } What is really happening when you try to reach /robots.txt? Does it display the default … Read more
Straight from the source, (line 1845 wp-includes/functions.php, 3.3.1): function do_robots() { header( ‘Content-Type: text/plain; charset=utf-8’ ); do_action( ‘do_robotstxt’ ); $output = “User-agent: *\n”; $public = get_option( ‘blog_public’ ); if ( ‘0’ == $public ) { $output .= “Disallow: /\n”; } else { $site_url = parse_url( site_url() ); $path = ( !empty( $site_url[‘path’] ) ) ? … Read more
Well those are two different things: A redirect means they are not accessible any more at all. noindex just means that search engines ignore it while it is still accessible if you access the URL. So I’d recommend option 1. This is a simple way of doing this that you can improve an. (E.g. this … Read more