Please try this:
$args = array( 'post_type' => 'custom',
'posts_per_page' => 5,
'orderby' => 'comment_count',
'order' => 'DESC',
);
$my_query = new WP_Query( $args );
to order by the comment count.