The wpsc_start_category_query() function basically calls the native WP get_terms() functions that is used to make a wp query:
$category_list = get_terms(‘wpsc_product_category’,’hide_empty=0&parent=”.$category_id);
Since WPEC uses custom post types you can just build your own queries easily, here using http://codex.wordpress.org/Function_Reference/get_terms
But get_terms() doesn”t do random, so you have to get them all and then shuffle the array into a random order, then output the first 4 (any number you set to $max) fo them. So adjusting your layout code it should be:
<ul>
<?php
//display random sorted list of wpsc product categories
$counter = 0;
$max = 4; //number of categories to display
$terms = get_terms('wpsc_product_category');
shuffle ($terms); //makes list random
if ($terms) {
foreach($terms as $term) {
$counter++;
if ($counter <= $max) { ?>
<li>
<div>
<div class="image"><img src="https://wordpress.stackexchange.com/questions/95659/<?php echo wpsc_category_image($term->term_id); ?>" width="<?php echo get_option('category_image_width'); ?>" height="<?php echo get_option('category_image_height'); ?>" /></div>
<div class="caption-title transparent_class">
<?php echo $term->name; ?>
</div>
<div class="caption transparent_class">
<a href="<?php get_term_link( $term->slug, 'wpsc_product_category' ); ?>" class="wpsc_category_link"><?php echo $term->name; ?></a>
<?php if(get_option('wpsc_category_description')) :?>
<?php echo '<div class="wpsc_subcategory">'.$term->description.'</div>'; ?>
<?php endif;?>
</div>
</div>
</li>
<?php }
}
}
?>
</ul>
I’ve tested the code and works like described.