Is it a bad practice to use switch_to_blog in the frontend?

For 4 blogs – most likely no.

The problems start to appear when you have tens of thousands of posts+ across many blogs, and you want to do queries like getting the latest 10 posts from the network.

You should test this yourself, and see if the use of that function (and the others that followed it) significantly affected site performance in some way