Create unique robots.txt for every site on multisite-installation

by

Straight from the source, (line 1845 wp-includes/functions.php, 3.3.1):

function do_robots() {
    header( 'Content-Type: text/plain; charset=utf-8' );

    do_action( 'do_robotstxt' );

    $output = "User-agent: *\n";
    $public = get_option( 'blog_public' );
    if ( '0' == $public ) {
        $output .= "Disallow: /\n";
    } else {
        $site_url = parse_url( site_url() );
        $path = ( !empty( $site_url['path'] ) ) ? $site_url['path'] : '';
        $output .= "Disallow: $path/wp-admin/\n";
        $output .= "Disallow: $path/wp-includes/\n";
    }

    echo apply_filters('robots_txt', $output, $public);
}

So to customise it:

function my_custom_robots( $robots )
{
    if ( my_condition() )
        $robots .= "\nDisallow: /something/else/";
    return $robots;
}
add_filter( 'robots_txt', 'my_custom_robots' );

Leave a Comment