Similar problem occurred to me, and I got the solution following these: Step 1: Take a backup of your .htaccess file and then remove it (Don’t worry, on next refresh WordPress will create one for you) Step 2: If there’s no robots.txt exists, create one with blank page Step 3: Resubmit the sitemap to google … Read more
For those who are using WordPress as CMS for their site, you can bypass your web hosting server rules by simply removing your robots.txt file and instead modifying the virtual one generated by WordPress. You just have to add a filter to the functions.php file of your theme. Here’s the code snippet: //* Append directives … Read more
WordPress generate a dynamic robots.txt which does not physically exists. To remove/disable it you have two options: Option 1: Remove do_robots action in your theme functions.php or plugin remove_action(‘do_robots’, ‘do_robots’); The action do_robots is still available to be added again by other plugins. Option 2: Create a real robots.txt file, put it the root folder … Read more