Can I hook into the invite user process to verify their email address is from a certain domain?

It is possible to hook into wpmu_validate_user_signup, which returns the $result of the sign-up process. Add another check for the email domain whitelist and add an error if not allowed.

add_filter( 'wpmu_validate_user_signup', 'whitelist_registration_wpse_82859' );

function whitelist_registration_wpse_82859( $result )
{
    // Test array
    $whitelist = array( 'gmail.com', 'mydomain.com' );

    // http://php.net/manual/en/function.explode.php
    $user_name_domain = explode( '@', $result['user_email'] );

    if( isset( $user_name_domain[1] ) && !in_array( $user_name_domain[1], $whitelist ) )
        $result['errors']->add( 'user_email',  __( 'Email domain blacklisted' ) );

    return $result;
}

PS: nice trick with the fake filter wpmu_signup_user 😉