Add new “Insert Into Post” button with another function

There’s a get_image_send_to_editor() function in wp-admin/includes/media.php that runs this: apply_filters( 'image_send_to_editor', $html, $id, $caption, $title, $align, $url, $size, $alt );. Try hooking that filter.

Edit: Help with the filter call…
Your call to hook the callback would look like this:

add_filter('image_send_to_editor', array(&$MyClassReference, 'filter_iste'), 10, 8);

The 10 is the priority, you might need to adjust that…10 is the default.
Your filter function needs to look pretty much like this:

function filter_iste($html, $id, $caption, $title, $align, $url, $size, $alt) {
    ...
    return $html;
}

To derive what you need to do in the callback, i’d pretty much first dump all the parameters you’re getting passed through to the screen. If you need something more, you can always fetch the whole attachment data and metadata using the $id you got passed through.

Basically, you could now try two different approaches:

1. You check for the mime-type of the attachment and if it’s a video, replace the whole $html content with your shortcode. You could add an option to the media settings that allows the user to configure if they want you to do so or not. You only need to hook the whole thing if they do.
   Edit: to check for video mime-types:

   $attachment = get_post($id);
   $mime_type = $attachment->post_mime_type;
   if (substr($mime_type, 0, 5) == 'video' && get_option('use_video_shorty_on_insert')) {
       ...
   }
   

2. You run your $html content through a regexp replace to remove the bitsies the shortcode is wrapped in. What that regexp would look like depends on what the $html content typically looks like.
   I’d go for the first option, it’s more solid and better UI. I understand that you might want to give the user the ability to change options for your shortcode, but this is another topic and i do believe it’s possible to tweak the fields on the media popup to add some options there.

Edit 3: full example:

// this seems to be an additional filter running for images only
add_filter('image_send_to_editor', 'my_filter_iste', 20, 8);

function my_filter_iste($html, $id, $caption, $title, $align, $url, $size, $alt) {
    $attachment = get_post($id); //fetching attachment by $id passed through
    $mime_type = $attachment->post_mime_type; //getting the mime-type
    if (substr($mime_type, 0, 5) == 'video') { //checking mime-type
        //if a video one, replace $html by shortcode (assuming $url contains the attachment's file url)
        $html="';
    }
    return $html; // return new $html
}

Ok…wrong filter, apparently…try this:

// this seems to run when inserting a video via the 'From URL' tab
add_filter('video_send_to_editor_url', 'my_filter_vsteu', 20, 3);

function my_filter_vsteu($html, $href, $title) {
    $html="';
    return $html;
}

Ahh well, we’ve got one more:

// this seems to run generically for any media item from the 'Upload' tab
add_filter('media_send_to_editor', 'my_filter_mste', 20, 3);

function my_filter_mste($html, $send_id, $attachment) {
    if (substr($attachment->post_mime_type, 0, 5) == 'video') {
        $href = wp_get_attachment_url($attachment->ID);
        $html="';
    }
    return $html;
}