Can I hook into user registration *before* a user is created?

You’re looking in the wrong place.

When a user first attempts to register, their username and email is processed and sanitized inside the register_new_user() function in wp-login.php. This is where you want to do your filtering.

Before the user is created, WordPress will pass the sanitized user login, email address, and an array or errors through the ‘register_post’ action. If there are any errors after that, then the user is not added and they will see the errors in the UI.

So the following untested function might help:

function prevent_email_domain( $user_login, $user_email, $errors ) {
    if ( strpos( $user_email, '@baddomain.com' ) != -1 ) {
        $errors->add( 'bad_email_domain', '<strong>ERROR</strong>: This email domain is not allowed.' );
    }
}
add_action( 'register_post', 'prevent_email_domain', 10, 3 );