How can I get more out of `$post` in an `add_action` callback?

Simply use the post ID, provided by $post_id (and stored at $post->ID) to use the various functions WordPress provides to get any extra data related to that post (get_the_terms,get_post_meta etc).

Performance wise you suffer no loss by using these functions as opposed to trying to force more data into the $post object (probably the reverse in fact), since WordPress caches data related to posts (such as taxonomy terms, post meta,…)