use a conditional statement, for example:
$sub_cats = get_categories('parent=".get_query_var("cat'));
if( $sub_cats ) :
//show list of child categories, for instance with wp_list_categories()
echo '<ul>;
wp_list_categories('title_li=&child_of=".get_query_var("cat'));
echo '</ul>';
//or possibly using $sub_cats and a foreach loop//
echo '<ul>';
foreach( $sub_cats as $sub_cat ) {
echo '<li><a href="'.get_category_link($sub_cat->term_id).'">'.$sub_cat->name.'</a></li>';
}
echo '</ul>';
else:
//the 'normal' LOOP of category.php//
endif;
http://codex.wordpress.org/Function_Reference/get_categories
http://codex.wordpress.org/Template_Tags/wp_list_categories
edit: list code for sub categories added