After much trial and error, I have found a solution that will allow wp_set_object_terms to be used. Despite already checking if my strings were integers, apparently I needed to convert the array explicitly as well. So I changed this:
$customtax = $customtaxarray;
to this:
$customtax = array_map('intval',$customtaxarray);
and suddenly wp_set_object_terms now works. That does not explain why wp_set_post_terms was working instead (unless perhaps it does this conversion automatically?), but at least this is now working as expected.
