According to The Codex on wp_set_object_terms() get set by passing in a slug instead of the actual Term Name:
wp_set_object_terms( $object_id, $terms, $taxonomy, $append );
$terms
(array/int/string) (required) The slug or id of the term (such as category or tag IDs), will replace all existing related terms in this taxonomy. To clear or remove all terms from an object, pass an empty string or NULL. Integers are interpreted as tag IDs.
This may work for you ( not sure if WP slugs strip # or not so this slug is just a guess ).
wp_set_object_terms( $product_id, 'test-model-#', 'pa_model' , false);