Right there is no big difference between them, actually wp_set_post_terms() uses wp_set_object_terms() but does few extra check for you. That’s also noted on wp_set_object_terms() Codex page:
Perhaps the wp_set_post_terms() is a more useful function, since it
checks the values, converting taxonomies separated by commas and
validating hierarchical terms in integers.
http://codex.wordpress.org/Function_Reference/wp_set_object_terms#Notes