If you examine the wp_insert_user() function, you can see there are a myriad of filters and actions that are called throughout the process. The first is a filter called pre_user_login on the username.
Line 1304 of wp-includes/user.php:
$user_login = apply_filters('pre_user_login', $user_login);
You could hook onto that and throw your own custom error message.
Edit: The email filter is just a few lines below that.
Second Edit: Added example
add_filter( 'pre_user_login' , 'my_username_block' );
function my_username_block( $user_login ) {
$black_listed = array( 0 => 'satan' , 1 => 'hitler' );
if( in_array( strtolower( $user_login ) , $black_listed ) ) {
wp_die('That username is blacklisted');
}
return $user_login;
}
add_filter( 'pre_user_email' , 'my_email_block' );
function my_email_block( $user_email ) {
$black_listed_emails = array( 0 => '[email protected]' , '[email protected]' );
if( in_array( strtolower( $user_email ) , $black_listed_emails ) ) {
wp_die('That email address is blacklisted');
}
return $user_email;
}