For non-hierarchical terms (such as tags), you can pass either the term name or id. If you pass the id there is only one caveat: You must pass it as an integer, and it must be in an array. This is necessary because any non-array value passed will be converted to a string, which will be interpreted as a term name.
$tag = '5'; // Wrong. This will add the tag with the *name* '5'.
$tag = 5; // Wrong. This will also add the tag with the name '5'.
$tag = array( '5' ); // Wrong. Again, this will be interpreted as a term name rather than an id.
$tag = array( 5 ); // Correct. This will add the tag with the id 5.
wp_set_post_terms( $post_id, $tag, $taxonomy );
This function will only work on the native post type.