The behavior you described is how WordPress works by design, if you provide a 'p' parameter, then 'tax_query' is ignored, see inside this else statment in wp-includes/query.php:L1643, this is where 'tax_query' is processed and added to the SQL statement, and this only happens if this if (line 1634) is skipped.
