How can I get the selected string when using a toolbar button in Gutenberg?

You get the selected content with getTextContent(). So you should add your HTML before and after this content.

A small code example that will not work because I added variables without values, only for understanding. To see how I define the variables, and add many more filters for validation of the content, see my playing code inside a GitHub repo.

    registerFormatType(type, {
        title,
        tagName,
        tagEnd,
        className,
        edit({isActive, value, onChange}) {
            // Get the selected string.
            const text = getTextContent(slice(value));
            const toInsert = tagName + text + tagEnd;
            //const onClick = () => onChange( insert( value, toInsert ) );
            const onClick = () => {
                element = create({
                    'html' : toInsert
                });
                if( element.formats.length === 0 ) {
                    return;
                }
                for ( let i = element.formats[0].length - 1; i >= 0; i-- ) {
                    value = toggleFormat(value, element.formats[0][i]);
                }
                onChange(value);
            };

            return (
                createElement(Fragment, null,
                    createElement(RichTextShortcut, {
                        type : 'primary',
                        character,
                        onUse: onClick
                    }),
                    createElement(RichTextToolbarButton, {
                        icon,
                        title,
                        onClick          : onClick,
                        isActive         : isActive,
                        shortcutType     : 'primary',
                        shortcutCharacter: character,
                        className        : `toolbar-button-with-text toolbar-button__advanced-${name}`
                    })
                )
            )
        }
    });

You should also see this code from the core of this topic.
https://github.com/WordPress/gutenberg/blob/4741104c2e035a6b80ab7e01031a9d4086b3f75d/packages/rich-text/src/register-format-type.js#L17