Add button to Block toolbar: toggleFormat is undefined

Now I can see what the problem is. You are not using the onChange properly. It comes from the props of the edit component:

Here you have a complete example that works:

registerFormatType(
    'nelio/keyboard', {
        title: __( 'Key', 'nelio' ),
        tagName: 'kbd',
        className: null,
        edit: ( { value, onChange } ) => (
            <RichTextToolbarButton
                icon="editor-removeformatting"
                title={ __( 'Key', 'nelio' ) }
                onClick={ () => onChange( toggleFormat( value, { type: 'nelio/keyboard' } ) ) }
            />
        ),
    }
);