Removing a taxonomy term

Your issue is last argument in wp_set_object_terms() call. You are setting $append to true, so it essentially checks if input is already present and does nothing else.

You want it set to false (which is also default) so that terms are forced to be same as input and difference (what you removed) is deleted from post.