Comments are kind of “limited” in WordPress. You have three different types for them as get_comment_type() shows nicely and you can not extend that list.
So, adding meta values to your comments is the easiest, and probably “most standard” way, to be able to distinguish different “types” of comments.
Handling the comment in- and output stays basically the same, except that you have to add a meta value to some of them. As you have already shown this is done quite easily.
Regarding your questions:
Is it worth it?
I think your solution is the best possible way without tinkering too much in the database, creating new structures and stuff like that. Your solutions just simply doesn´t need that. I like that because it improves compatibility.
Would it perform much worse than default comments?
I don´t think it will perform worse at all. You are basically just adding another parameter to the database query/doing another join and databases are fast as hell for things like this. Try to implement caching when you get more load on your site and it should work fine.