Your arguments for the comment query is empty, so it returns every comment on the planet. You should specify the post ID, to get comments that belong to that post.
$args = array(
'post_id' => get_the_ID(),
);
Take a look into the codex page for more information about arguments.