taxonomy/category term in URL slug irrelevant for post?

I wouldn’t call it a bug, you haven’t done anything to place that restriction on the query for your post. It also seems like a fairly unlikely case where random strings will end up in place of your taxonomy terms for this to even present itself.

Anyway, the answer is to modify the query via pre_get_posts to add in whatever taxonomy data you need. I don’t know your specifics, so this isn’t a copy-paste solution, but it shows the general concept of what you need to do.

function wpd_single_cpt_queries( $query ){
    if( $query->is_singular()
        && $query->is_main_query()
        && isset( $query->query_vars['your_tax'] ) ){
            $tax_query = array(
                array(
                    'taxonomy' => 'your_tax',
                    'field' => 'slug',
                    'terms' => $query->query_vars['your_tax'],
                )
            );
            $query->set( 'tax_query', $tax_query );
    }
}
add_action( 'pre_get_posts', 'wpd_single_cpt_queries' );