Few months ago @tom-j-nowell (one of the mods here) wrote an article explaining the issues with the abuse of meta queries by many WP plugins: https://tomjn.com/2016/12/05/post-meta-abuse/
Among other things he says there:
[…] sites have been brought down by this, and it’s the reason a number of popular plugins don’t scale to high traffic […]
So, yes, using ACF, or any other plugin which make too much use of meta data for “searchable” data can seriously compromise the performance of your website.
And yes, using taxonomies for that, is just much better.
Register a taxonomy, prefill it with some given terms, and provide an UI to choose between those terms (which is pretty much what WP itself does with post formats) is not that hard, and if the code is done in a flexible enough way, could be used for many taxonomies and is possible to avoid ACF and any other such plugins.
However, to be honest, in my experiece the only viable way to obtain decent performant queries in a big WP websites, expecially if full-text search is involved, is to use a search engines like Elastic Search or Apache Solr.
Even because there are things (take as example product prices) that can’t be really handled with taxonomies and if query for taxonomy is easy, order by taxonomy not very so.