Adding terms to custom post type – incorrect term being applied

Did you try explicitly setting the term array?
And set append to true?

$terms = array( 35, 36 ); 
wp_set_post_terms( $this->postId, $terms, "my_tax", true );

tech