Which hook should be used to validate custom form fields on the login form?

Use the action 'login_init' to catch all calls to wp-login.php.

Sample code:

add_action( 'login_init', 'wpse_51227_validate_custom_field' );

function wpse_51227_validate_custom_field()
{
    if ( ! isset ( $_POST['special_custom'] )
        return;

    if ( ! is_numeric( $_POST['special_custom'] )
        // handle the error
}

There are two important global variables available:

• $errors is an instance of WP_Error. Maybe you want to use it to store error data.
• $action is the current action. Maybe you want to rest it to login to prevent a redirect if the user input is invalid.