Change link in get_terms if there is only one post in the category

This could be done in the template that displays your archive page when only one post is returned by wp_query for a given category. Something like this:

if ( $wp_query->post_count == 1 && $wp_query->max_num_pages == 1 ) {
    wp_redirect( get_permalink( $wp_query->posts['0']->ID ) );
    exit;
}