Add Item to Custom TinyMCE Menu

addMenuItem() adds to tinymce’s toolbar, which isn’t used by WP by default, and uses context to add to the particular (sub)menu. You’re adding a MenuButton, and you can access the button through the editor’s buttons array, keyed by the button name:

add_action( 'admin_print_footer_scripts', function () {
    ?>
    <script type="text/javascript">
    jQuery(function ($) {
        tinymce.on('SetupEditor', function (editor) {
            if (editor.id === 'content') {
                editor.on('init', function () {
                    var button = this.buttons['nto_shortcode_button1'];
                    if (button) {
                        button.menu.push({
                            text: 'Tooltip',
                            icon: 'wp_help',
                            onclick: function() {
                                editor.insertContent('Hello World!');
                            }
                        });
                    }
                });
            }
        });
    });
    </script>
    <?php
} );