You need to make sure you cast all the terms as integers. Passing in strings will compare the id’s to the slugs, and end up creating new terms with the ids as the name.
Edit
Specifically, alter these two lines as follows (note the intval):
wp_set_object_terms( $post_id, intval( $term ), $field['id'] );
...
wp_set_object_terms( $post_id, intval( $category ), 'category' );