Limit roles displayed in users.php depending on custom role

If you look at the source of that file at the very top there is capability check:

if ( ! current_user_can( 'list_users' ) )
    wp_die( __( 'Cheatin’ uh?' ) );

So list_users seems like a perfect starting point (at least) to remove from your custom role. However it is hard to say what else it might affect without thoroughly testing out all parts of admin your user should and should not be able to use.