do i need to fetch comments from all sites, one by one ?
Yes and you might try to do some json-merge in javascript for few sites, like with posts in this article by Josh Pollock, but here are some alternative ideas:
For few sites you might try to create a custom endpoint where comments are fetched from each site via PHP and merge them for the REST response.
An alternative approach, for larger number of sites, I think you could use the same technique sometimes used to collect tags or posts from all sites:
Create a site that also collects sitewide comments through hooks like wp_insert_comment.
Then you could use the REST API to get all comments from that site.
The first plugin that I know of that used this technique was the plugin WordPress MU Sitewide Tags written by Donncha Ó Caoimh.
A quick search gave e.g. WDS Multisite Aggregate that seems to be inspired by that plugin (I’m unrelated to it and haven’t used it).
I also wonder if GraphQL could help here, as it seems to be a possible future replacement for the REST API?
Hope it helps.