It’s a little hard to untangle this, but I’ll try to address your points one by one.
create a separate 404.php page to control the output of that page. (and the silence about any attempt to do it otherwise)
404.php template controls presentation of 404 page. This is highly in line with rest of WP template hierarchy. It’s not mandatory either, without it the template logic proceeds to use index.php template for it.
It’s basically because they provided no good fallback (or I haven’t found it?) to control the second query: the fallback query run if nothing is returned by the initial query.
There is no fallback, because there is logical branch for it to proceed on. Main query receives input of query variables and either comes up with set of posts or doesn’t. Depending on specific context latter might or might not be 404 (single post will be, archives not, which I always find curious).
By directing us to create another document, WITHOUT A LOOP in it, you’d never notice the loop is still returning all the blog posts.
It doesn’t. I can’t easily think of situation of main query both producing 404 and succesfully querying posts (much less “all” of them). This sounds like a work of a bugged theme/plugin.
Yes, I can modify the main query but it makes no sense since all the posts will be retrieved anyway once nothing is found in the initial loop.
As above — they won’t be. Or at least they shouldn’t be.
Does anyone know how to prevent/change the fallback query?
The first step would be figuring out where it is coming from, because it isn’t from WordPress core. 🙂