Change the default video shortcode

Here’s an untested way to override the output of the shortcode, with the output of your custom rtmp_video() function, when the mp4 video is hosted on amazonaws.com:

/**
 * Override the video shortcode output for mp4 hosted on 'amazonaws.com'          
 * with the rtmp_video output.
 *
 * @uses rtmp_video()
 */
add_filter( 'wp_video_shortcode_override', function( $output, $attr, $content, $instance )
{
    if( 
          isset( $attr['mp4'] ) 
       && 'amazonaws.com' === parse_url( $attr['mp4'], PHP_URL_HOST )
       && function_exists( 'rtmp_video' )
    ) 
        $output = rtmp_video( $attr, $content );

    return $output;

}, 10, 4 );

Note that the extract() isn’t recommended here in general and it’s not clear if your custom rtmp_video() callback supports the mp4 attribute, like you’ve shown in the shortcode example. A workaround for that could be $attr['src'] = $attr['mp4'];

You mentioned that you placed your shortcode in the function.php, file. I guess you mean the functions.php file in the current theme directory? You should really consider adding it in a custom plugin, because next time you switch themes, your content will not support that shortcode anymore.