Responsive embed for the video shortcode

You could create a new callback for the embed_oembed_html filter and target the third input argument, the oembed $url.

Then you could e.g. create boolean helper functions like (untested):

function is_oembed_from_video_specific_hosts_wpse274552( $url )
{
    return in_array(
        parse_url( $url, PHP_URL_HOST ),
        [
             'youtube.com',
             'youtu.be',
             'vimeo.com',  // ... etc
        ],
        true
    );
}

or a more detailed (more expensive) checks like (untested):

function is_video_oembed_wpse274552( $url )
{
    $video_providers = [
        '#https?://((m|www)\.)?youtube\.com/watch.*#i',
        '#https?://((m|www)\.)?youtube\.com/playlist.*#i',         
        '#https?://youtu\.be/.*#i',        
        '#https?://(.+\.)?vimeo\.com/.*#i',                        
        '#https?://(www\.)?dailymotion\.com/.*#i',                 
        '#https?://dai\.ly/.*#i',
        '#https?://videopress\.com/v/.*#',
        '#https?://wordpress\.tv/.*#i',
        '#https?://(www\.)?funnyordie\.com/videos/.*#i',
        '#https?://(www\.)?(animoto|video214)\.com/play/.*#i',
        '#https?://www\.facebook\.com/.*/videos/.*#i',
        '#https?://www\.facebook\.com/video\.php.*#i', // ... etc
    ];

    $is_video_oembed = false;
    foreach( $video_providers as $video_provider )
    {
        if ( preg_match( $video_provider , $url ) ) {
        {
             $is_video_oembed = true;
             break;
        }
    }
    return $is_video_oembed;
}

Note that oEmbed providers can be modified, added and deleted in the WordPress core and through filters. That could affect the above checks.

Hope you can adjust it to your needs!