browse by category and tags?

Maybe change the permalinks for the tags in the sidebar to point to the category page they’re already viewing then pass the tag as a url parameter. If the parameter is set use pre_get_posts to merge the tag into the main query so it filters by both the category and tag.

Something along these lines

function filter_by_tag_too( $query ) {
  if ( isset( $_GET['tag'] ) ) {
    $query->set( 'tag', $_GET['tag'] );
  }
}
add_action( 'pre_get_posts', 'filter_by_tag_too' );