Customizing HTML Editor Quicktags button to open a dialog for choosing insert options

According to the Codex, you can’t do this directly through the API. However, you can do it by using your own quicktags.js file as shown below.

function sample_load_admin_scripts()
{
  if ( is_admin() ) {
    wp_deregister_script('quicktags');
    wp_register_script('quicktags', ("/path/to/your/quicktags.js"), false, '', true);
  }
}

if (is_admin()) {
  add_action('init', sample_load_admin_scripts);
}

Then just add Javascript to do your work. The formatting is relatively easy, but since you know how to do everything with TinyMCE I’m not going to go into that. Go get a quicktags.js file of your own here.