Sort Custom Post Type by Category ID then Post Name

To order taxonomy terms by ID, you have to set the orderby argument:

$tax_terms = get_terms( 'portfolio', array( 'orderby' => 'id' ) );

See the get_terms Codex page for a complete list of arguments for the function.

To order the posts by name (slug), set orderby to 'name' in your query arguments:

$args = array(
    'post_type' => $post_type,
    "$tax" => $tax_term->slug,
    'post_status' => 'publish',
    'posts_per_page' => -1,
    'ignore_sticky_posts'=> 1,
    'orderby'=> 'name',
    'order'=> 'ASC'
);

Note that 'name' is in quotes, as it is a string. I’ve also fixed 'ASC' to be a string as well, and changed the deprecated caller_get_posts to ignore_sticky_posts. You should always develop with debugging enabled to be alerted about these errors. Refer to WP_Query in Codex for a full list of valid query arguments and their use.