No, this is not possible with the current implementation of the format library and FormatToolbar component.
Formats have no order other than the order they are registered in, and this cannot be changed once set, or influenced in core formats.
Notice here the format toolbar component being added to the inline toolbar control slot:
// Render regular toolbar
return (
<BlockControls group="inline">
<FormatToolbar />
</BlockControls>
);
Inside Format Controls you can see that the text color has special hardcoded handling to force bold/italic/etc to always show:
{ [ 'bold', 'italic', 'link', 'text-color' ].map( ( format ) => (
<Slot
name={ `RichText.ToolbarControls.${ format }` }
key={ format }
/>
) ) }
Each is given a dedicated slot allowing them to show in predefined locations when active. The rest are rendered in a dropdown control. There are no flags or options to move items between these areas.
You could try to add your own button on the RichText.ToolbarControls.text-color slot, but implementing it would require very advanced proficiency in gutenberg internals, would be likely to break in the future, and would result in duplicate buttons if a format was applied.
The only remaining hopes you might have are to unregister the text-color format, then re-register it with modified code so that it always shows:
const hasColorsToChoose = ! isEmpty( colors ) || ! allowCustomControl;
if ( ! hasColorsToChoose && ! isActive ) {
return null;
}
Note that comments indicate this is considered a temporary workaround in the code, expect it to change in a future update. Also note that unregistering and re-registering may cause problems. E.g. unregistering and re-registering a block will fail at the unregistering stage if the block is in use and crash the block editor.