Show subcategories and hide posts or show posts if not exists subcategories

I would use a combination of get_categories and get_category_children to achieve this. For example:

On the home page include something like this:

<?php
  $args=array(
    'orderby' => 'name',
    'order' => 'ASC'
  );
  $categories=get_categories($args);
  foreach($categories as $category) {
    $catChildren = get_term_by($category->ID, 'category');
    if(!empty($catChildren)) {
      echo '<h3><a href=www.yourwebsite.com/"' . $category->slug . '">' . $category->name . '</h3>';
    }
  } 
?>

And then on the sub category page:

<?php

  $currentCat = get_the_category();

  $args=array(
    'orderby' => 'name',
    'order' => 'ASC',
    'child_of' => $currentCat->cat_ID
  );

  $categories=get_categories($args);

  if($categories) {
    foreach($categories as $category) {
      echo '<h3><a href=www.yourwebsite.com/"' . $category->slug . '">' . $category->name . '</h3>';
    }
  }
  else {
    $args = array( 
      'numberposts' => 10, 
      'category' => $currentCat 
    );
    $catPosts = get_posts( $args );
    foreach ( $catPosts as $post ) {
      setup_postdata($post);
      the_title();
      the_content();
    }
  }

?>

[I haven’t tested this but its the way I have done it previously – obviously you’ll need to change a few bits and pieces. Make sure to read the function docs do you dont just end up frustrated!]

You will need to have your categories and ‘the loop’ set up properly for this to work [get_the_category(); for example must be called from within the loop unless a category ID is supplied].

Make sure to read the docs on all the functions used to understand what they’re doing:
get_category_children – http://hitchhackerguide.com/2011/02/12/get_category_children/
get_categories – http://codex.wordpress.org/Function_Reference/get_categories
get_posts – http://codex.wordpress.org/Template_Tags/get_posts

Comment if you have troubles and I’ll do my best to elaborate further.