Set unique term_id from custom meta keys

The last (fourth) argument that you pass to wp_set_object_terms() is true, which, “If true, tags will be appended to the object. If false, tags will replace existing tags”. It sounds as though you are instructing wp_set_object_terms() to do, if I understand you, exactly what you don’t want it to do. I think that you need to set the last argument to false or just leave it out, as false is the default.