Multiple, nested tax_query relation

The cat-parameter is not overwritten by tax_query. Instead, it is added to tax_query. Thus, using 'cat' => -5, the following array is appended to tax_query:

(
    [taxonomy] => category
    [terms] => Array
        (
            [0] => 5
        )

    [include_children] => 1
    [field] => term_id
    [operator] => NOT IN
)

The resulting taxonomy query is built up of three different clauses with the OR-operator. Thus, in your case, adding 'cat' => -5 simply adds the array described above to tax_query, resulting in the query returning that posts that are either

• not in category 5,
• in any of the categories in $category, or
• in any of the tags in $tags

Nested tax_query

There is no way to use nested conditionals in tax_query. To achieve the conditional you are looking for (posts in a category from $category or having a tag from tags, but in all cases not being in category 5), you would have to hook into posts_where and possibly posts_join.

For anyone interested, the same goes for meta_query: this does not support nested queries either.

EDIT

Using posts_where

In response to your comment: to achieve this using posts_where, you will want to apply the filter to just the one query for which you want to exclude a category. To do so, add the filter before creating your query, and remove it afterwards:

add_filter( 'posts_where', 'myplugin_posts_where' );
$query = new WP_Query( $args );
remove_filter( 'posts_where', 'myplugin_posts_where' );

EDIT

Constructing the additional WHERE-clause

Using get_tax_sql to construct the additional WHERE-clause is useful as you do not have to construct your SQL manually, and the term IDs are automatically converted to term-taxonomy IDs.

$clauses = get_tax_sql( array(
    array(
        'taxonomy' => 'category',
        'field' => 'id',
        'terms' => 2,
        'operator' => 'NOT IN'
    )
), $wpdb->posts, 'ID' );

$where .= $clauses['where'];