Why does accessing url by category cause issues with post types?

Your condition is off and will always be true on category pages due to how logic operators work.

This might be more clear if I write what is happening like this:

is_category() || ( is_tag() &&  $query->is_main_query() )

Once is_category() is determined to be true the rest of condition doesn’t matter.

You should be more precise with your logic, along the lines of (brackets matter!):

$query->is_main_query() && ( $query->is_category() || $query->is_tag() )