Adding a custom PanelColorSettings control to a core block, and using the color slug in a custom className

Your code looks ok and would work in any other component other than colors, because colors need the higher-order-component withColors. In the past I struggled also dealing with the behaviour, that your are describing and found these two solutions:

1. If you want to insert your own color component, check the second example from here https://awhitepixel.com/blog/add-custom-settings-to-existing-wordpress-gutenberg-blocks/.

2. Otherwise you can just add color support in the blocks.registerBlockType. This is also what i’m using right now and is working great:
   if( ['core/quote'].includes(name) ){ const supports = { ...props.supports, color: true, }; props = { ...props, supports }; }

Here is also a link https://css-tricks.com/a-crash-course-in-wordpress-block-filters/