How do I create a custom add media button modal?

You can inspire from this tutorial here:
http://www.sitepoint.com/adding-a-media-button-to-the-content-editor/

For videos you can change the library type to ‘video’:

function open_media_window() {
    var window = wp.media({
        title: 'Insert a media',
        library: {type: 'video'},
        multiple: false,
        button: {text: 'Insert'}
    });
}

Hope this helps.