Changing a WordPress core function without hacking core

You aren’t filtering correctly. Firstly, you aren’t passing variables to the function, so your function has no way of knowing what $comment_author is, etc. If you had debug mode enabled you’d probably get errors about undefined variables. Secondly, you need to return a value.

Untested, but seems like it ought to work:

add_filter('pre_comment_approved', 'custom_blacklist', 10, 2 );

function custom_blacklist( $approved, $commentdata ) {
    extract($commentdata, EXTR_SKIP);
    if ( wp_blacklist_check($comment_author, $comment_author_email, $comment_author_url, $comment_content, $comment_author_IP, $comment_agent) )
        $approved = 'trash';

    return $approved;
}