Dealing with terms is known to be very heavy operation.
The common tweak recommended is to use wp_defer_term_counting()
to temporarily disable updating of terms count (which is stored persistently in database and so must be updated when you change terms).