How to move custom gutenberg block controls from settings to styles tab?

Set the group property of InspectorControls to styles:

<InspectorControls group="styles">
    // etc.
</InspectorControls>

The other possible groups are described in this article.