Refine user taxonomy query based on role

There is more generic version of current_user_can()user_can() which takes user and capability or role to check for.

So your pseudo code would become something like:

if ( ! user_can( $user_id, 'invisible' ) )