There is no way to limit the gallery query in the shortcode_gallery function. However, you can get the child attachments of a post using the following.
$attachments = get_children( array('post_parent' => $post->id, 'post_status' => 'inherit', 'post_type' => 'attachment', 'post_mime_type' => 'image', 'order' => $order, 'orderby' => $orderby, 'numberposts' => 5) );
You will need to iterate through the attachments and build out your gallery HTML.