How to add custom table style for table block in gutenberg editor

The register_block_style function is what you’ll want.

The basic implementation is

register_block_style(
    'core/table',
    array(
        'name'  => 'my-table-style',
        'label' => 'My Table Style',
    )
);

When you select the style, it will add the is-style-my-table-style class to the block, which you can style using your style.css or other stylesheet.

There are two other implementations, where you can declare an inline style or a whole stylesheet for that specific style. You can find more details in the documentation: https://developer.wordpress.org/block-editor/developers/filters/block-filters/#register_block_style