WordPress ignore init for crawlers

Just added this to functions.php:

// Returns TRUE if it's a crawler
function check_is_crawler() {
    if (isset($_SERVER['HTTP_USER_AGENT']) && preg_match('/bot|wget|crawl|google|slurp|spider/i', $_SERVER['HTTP_USER_AGENT'])) {
        return true;
    } else {
        return false;
    }
}

And I’m using it on critical functions to lower resource usage.

Also, created a robots.txt with the following content:

User-agent: *
Crawl-delay: 10

It puts a halt on crawlers, so they don’t “spam” your website and consume all your resources

Be warned ! However, google does not like this AT ALL. When accessing your page, if google notices different behavior for crawlers and visitors from your website, it will possibly consider your website as spam.

Thanks for the tip @Jack Johansson, I’ll use it only on internal functions. It’s an ads website, and there’s a lot of things going under the hood that don’t output to the user.

Related Posts:

  1. when should an action be added to init and when should it be added to wp_head
  2. What is the entry point of a WordPress theme?
  3. Init hook function to call other hook in function.php
  4. How to fix “Attempted relative import in non-package” even with __init__.py
  5. git add . -> still “nothing to commit” with new files
  6. How to fix “Attempted relative import in non-package” even with __init__.py
  7. Inheritance and init method in Python
  8. What is the very earliest action hook you can call?
  9. Which hook should be used to add an action containing a redirect?
  10. Marking future dated post as published
  11. Difference between after_setup_theme and init action hooks?
  12. WordPress Theme activation hook?
  13. Use wp init hook to call other hooks?
  14. Are there any action like ‘init_frontend’
  15. Init action hook running late after PayPal’s return url?
  16. When we register a custom taxonomy or post type, does the WP database modified at all?
  17. Change page title from plugin
  18. Using get_terms for custom taxonomy in functions.php
  19. set_query_var doesn’t seem to work on init hook
  20. Intercept request to /wp-content/uploads/random.file
  21. add_feed and flush_rewrite_rules
  22. wp_redirect() not working on form submission with init hook
  23. WordPress Plugin Boilerplate – add_action hook in static “activate” function
  24. What is the best filter where to use register_block_type?
  25. Cron While Editing Post
  26. Best action hook to create custom post and assign taxonomy terms to it on plugin activation?
  27. How to insert a text in all pages and posts before or after specific places?
  28. Stop WordPress Entirely and Run My Own Code
  29. Get log of function/method calls made by do_action() on WordPress initialization
  30. Load ajax if is_home()
  31. Plugins rewrite rules the right way
  32. How to setcookie if is_page(‘page’) ? should I use add_action(‘init’) or there is another action?
  33. Problem with is_page & wp_redirect
  34. Contact Form 7 – Uncaught TypeError: wpcf7.initForm is not a function [closed]
  35. wp_redirect on base wp-admin and login
  36. Init plugin again after ajax call finish
  37. Fire on widgets_init only on dashboard
  38. $wpdb->insert running multiple times?
  39. Hooking action once but called twice
  40. PHP header() gives headers already sent
  41. Post submit using shortcode and init
  42. Form Shortcode not saving data to WP database
  43. Which functions/classes are available during SHORTINIT?
  44. add_action in a custom plugin
  45. Plugin init hook fires repeatedly
  46. Add a filter inside an action init
  47. Why can’t a custom postype be registered with “admin_init” hook?
  48. How to hook WP initialization without modifying functions.php
  49. Basic function call on init failing
  50. How to do a task only once for logged in users
  51. add_action(‘init’) not work
  52. Init action and refresh page after form action
  53. Action ‘init’ function doesn’t return variable
  54. Can i use init hook for API purpose?
  55. Why doesn’t global $wp_query not get hooked?
  56. How to translate Widget Description in constructor?
  57. Why do I need to register my custom post type a second time when flushing rewrite rules?
  58. Developing plugin, where to place action hooks etc?
  59. Preset Widgets ONLY after site is initially created
  60. Custom Post Type Dashboard Tab not displaying any posts
  61. Session management issues with WordPRess 404 Error page
  62. Using get_terms for custom taxonomy in functions.php
  63. Set user role, if an specific role created an user
  64. Store a value in global scope after init hook is fired
  65. Misleading behaviour on Menu & Theme location
  66. When/where would want to attach other classes to the base class in a WordPress plugin?
  67. wp-cron event doesn’t run when custom login API is enabled
  68. Why should I need to add init action to include PHP file to WordPress Plugin?
  69. adding an action inside if condition not working
  70. add_action init breaks site on PHP 7.1
  71. How to use add_action(‘wp_ajax_[action name]’,…) for a specific page with condition?
  72. WP_mail() Issue. Duplicate emails if $_GET[‘A’] == email
  73. Function Hooked on Init Running Multiple Times
  74. Can I use a hook other than ‘init’ to handle form submissions?
  75. API calls on plugin activation or init?
  76. Custom setting field value not working inside add_action init hook
  77. update_post_meta returning false when called during init
  78. Init hook for header send
  79. do_action( ‘init’ ) in wp-settings.php breaks my website. All plugins and themes disabled
  80. add_action not working in header?
  81. IE 9 Clears Form Fields
  82. remove an action hook function on a plugin from a theme
  83. plugin: ‘init’ hook, check if we’re not in admin
  84. theme or plugin or server is outputting html before the init hook fires, how is it possible to prevent this
  85. How to execute init or woocommerce_init only for checkout page
  86. Best hook for a function which adds shortcodes to the system?
  87. Best way to disable swap in Linux
  88. Setcookie not setting cookies anywhere except admin
Hata!: SQLSTATE[HY000] [1045] Access denied for user 'divattrend_liink'@'localhost' (using password: YES)