Sounds like you need a custom query in order to pull X amount of images. You could do the following (untested):
<?php
function custom_gallery_display( $number_of_images = 4 ) {
//make sure you have access to the WPDB object
global $wpdb;
//pull 4 random attachments
$images = $wpdb->get_results("SELECT ID from wp_posts WHERE post_type="attachment" ORDER BY RAND() LIMIT $number_of_images");
//may need to tweak this;
//not sure what format the results are returned
$image_ids = implode( ',', $images );
//now print with random images
echo do_shortcode('');
return;
}
?>
That should do the trick, although it may need a bit of tweaking.
You should also check out the documentation http://codex.wordpress.org/Gallery_Shortcode
According to it, you can easily create the randomized order by setting the orderby attribute to RAND
<?php echo do_shortcode(''); ?>
However, if you are forced to use a custom query in order to use X amount of images, you may as well just randomize it at that point, as I did above in the query.