You can’t order by taxonomy terms, the very concept breaks down when you try to handle posts that have multiple terms in a taxonomy. You may know that only 1 term will ever be selected but WP_Query isn’t built that way.
Instead, keep your taxonomy so that you retain all the performance/theming/Admin UI benefits, but also save the information in post meta so that you can sort. This way you get the best of both worlds, and you can automate updating the post meta when the terms change. This also lets you use post meta sorting with tax_query.
While this does lead to duplication the cost is minimal, and the performance gains are very real. You may even decide to store data in the future that’s only ever used to improve performance, e.g. intermediate pre-calculated data, or buckets ( e.g. storing a price in post meta, but putting the posts into predefined buckets such as $100-$150 )