How can I auto-assign a CPT post to category in a custom taxonomy when published?

As Documentation says, wp_set_object_terms accepts 4 arguments:
$object_id, $terms, $taxonomy, $append

• object_id – ID of post, page, attachment or anything which is in post table
• terms – slug(s) or id(s) or term(s)
• taxonomy – slug of taxonomy – in your case it will be you custom taxonomy slug
• append – whether delete all existing terms assigned to object or append new on end

Usage: wp_set_object_terms($post_id, 'magazine', 'your_taxonomy_slug', true);