Post & Category Archives URL structure

The issue with your desired structure (and that categories without base are common request but commonly unreliable) is ambiguity.

Each rewrite rule is like a tiny program, explaining WP how to turn URL into query variables. It is very important that WP can reliably distinguish which of the rules should handle particular URL.

In plain words the usual case would be following rules:

1. blog, followed by post name.
2. blog, followed by taxonomy name, followed by term name.

With your configuration rules become the following:

1. blog, followed by post name.
2. blog, followed by term name.

Notice how this is much less clear case. How is WP supposed to guess if that is post or term at the end? How is it supposed to guess which taxonomy that term belongs to?

In a nutshell such structure is inherently unreliable and is not supported by WP core. You might beat it into submission eventually, but I would recommend just to stick with robust and unambiguous rewrite rules.