Cancel comment filter?

You are certainly on the right track. You can create a plugin (create a directory in “wp-content/plugins” and create a php file with the same name as the folder) with the following code:

function wpse_33944_filter_handler( $approved , $commentdata ){
    if(strlen($commentdata[comment_content]) > 5) {
        return false;
    }
    return true;
}
add_filter( 'pre_comment_approved' , 'wpse_33944_filter_handler' , '99', 2 );