How to query 5 users in random who have published more than 10 posts

I tried this code and it should work for you

In your function.php to make random order available in WP_User_Query

function my_random_user_query( $class ) {
 if( 'rand' == $class->query_vars['orderby'] )
 $class->query_orderby = str_replace( 'user_login', 'RAND()', $class->query_orderby );
 return $class;
}
add_action( 'pre_user_query', 'my_random_user_query' );

Source

And a query like this

// Get all users id
$args = array (
 'has_published_posts' => true,
 'fields' => 'ids',
);
$wp_users = new WP_User_Query( $args );
$ids_users = $wp_users->get_results();

// Return users id with more than 10 posts
$users = array_filter($ids_users, function($id){
  $count = count_user_posts( $id );
  if($count >= 10) return $id;
});

// Get users previously filtered 
$args = array (
 'orderby'    => 'rand',
 'include'    => $users,
 'number' => 5
);
$wp_users = new WP_User_Query( $args );
$result = $wp_users->get_results();

Codex query about users