If we look at the official documentation, this is how wp_set_object_terms is used:
wp_set_object_terms( int $object_id, string|int|array $terms, string $taxonomy, bool $append = false )
Notice the final parameter that isn’t being used in your code, it has a default value of false:
bool $append = false )
So everytime your loop runs, it wipes the slate clean and sets the terms rather than adding them. You need to pass the final parameter.
Also consider deliberatly calling this with a blank array before your loop so that terms can be unset. Alternatively, add them to an array and do a single call after the loop
Make sure in these situations to always read the official API docs at https://developer.wordpress.org