You can use the views_users
filter to alter it. Here’s an example:
/**
* Remove the users view
*/
add_filter( 'views_users', '__return_empty_array' );
if you want to remove it for every user.
Here’s another example how you could remove it for non-admins (PHP 5.4+):
/**
* Remove the users view for non-admins
*/
add_filter( 'views_users', function( $views )
{
return current_user_can( 'manage_options' ) ? $views : [];
} );