the_post_thumbnail() only retrieves the specified thumbnail size for the current post. You’ll need to write a conditional function or use a custom query to randomize the posts shown or thumbnail sizes from your theme’s loop.
If you are familiar with where your loop starts, then:
$tile_sizes = array('small','medium','category-thumb');
LOOP STARTS
$random_size = rand(0,2);
$chosen_size = $tile_sizes[$random_size];
the_post_thumbnail($chosen_size);
LOOP ENDS