Show ACF field from custom taxonomy and display on the single template

I would try something like this…

// Assume there is only one category.
$product_category = get_the_terms($post, 'product-category')[0];

$cat_fields = get_fields("term_$product_category");

$color = $cat_fields['category_colour_primary'];

You could also do this…

// Assume there is only one category.
$product_category = get_the_terms($post, 'product-category')[0];

$color = get_field('category_colour_primary', "term_$product_category");