WordPress ignore init for crawlers

Just added this to functions.php:

// Returns TRUE if it's a crawler
function check_is_crawler() {
    if (isset($_SERVER['HTTP_USER_AGENT']) && preg_match('/bot|wget|crawl|google|slurp|spider/i', $_SERVER['HTTP_USER_AGENT'])) {
        return true;
    } else {
        return false;
    }
}

And I’m using it on critical functions to lower resource usage.

Also, created a robots.txt with the following content:

User-agent: *
Crawl-delay: 10

It puts a halt on crawlers, so they don’t “spam” your website and consume all your resources

Be warned ! However, google does not like this AT ALL. When accessing your page, if google notices different behavior for crawlers and visitors from your website, it will possibly consider your website as spam.

Thanks for the tip @Jack Johansson, I’ll use it only on internal functions. It’s an ads website, and there’s a lot of things going under the hood that don’t output to the user.