Send an email when custom post type category is changed

Use the hooks added_term_relationship and deleted_term_relationships. These only fire when the relationship changes, as opposed to set_object_terms which always fires:

function wpse_181090_object_terms_updated( $object_id ) {
    static $did = array(); // This function might fire multiple times for the same object, ensure it only runs once

    if ( ! isset( $did[ $object_id ] ) ) {
        $did[ $object_id ] = true;

        // Your notification code
    }
}

add_action( 'deleted_term_relationships', 'wpse_181090_object_terms_updated' );
add_action( 'added_term_relationship',    'wpse_181090_object_terms_updated' );