Did you actually find a case where there are any real comments in the wp-query object after running a normal query?
If you examine the code in class-wp-query.php, you should find that the comments field only is populated with comments when a “comments-feed” is being queried for.
In normal operation, comments are not retrieved by the WP_Query, but instead retrieved by using WP_Comment_Query, and this does not happen until your theme calls the comments_template() function.
As such, you can use the ‘comments_template_query_args’ filter to filter the arguments passed to that class before the comments are displayed on the page. This won’t affect the comments feed, because those are indeed populated by the main WP_Query, but only for the comments feed case.