How can I modify the loop from a plugin without wiping out other filters?

Use $tax_query = $query->get( 'tax_query' ); to get the existing tax_query, modify it by adding your changes, then use $query->set( 'tax_query', new_$tax_query ); to put the whole thing back.