I think most pagination issues stem from a misunderstanding of the query and template loading process that happens before you are executing code within a template file.
When someone requests a page:
-
WordPress parses the request and decides what it’s for based on the incoming URL. For example, is it the main root page, does it begin with
category, does it match the pattern of a single post, does it have a page number, etc.. The correct query vars are set from this process. -
WordPress runs the main query on the database, using the query vars set in the last step.
-
WordPress looks at the results of the main query, and determines whether it was successful or produced no results. If it was successful, WordPress loads the correct template for that type of request, or loads the 404 template if it wasn’t.
So now you’re finally in the template, if everything above went as expected. But now we can see the source of one problem – if that main query produced no results, you’ll never reach the template to run your custom query. That main query has no relation to the custom query in your template. Whether or not your custom query has results for that page number is irrelevant, if it doesn’t have a corresponding page in the main query. The answer in this case, is to modify the main query before it happens via pre_get_posts, to make that query give us the results we want instead of running another query in the template.
The other option, which you mention in your question, is to use a static page. This is the exception to the above behavior regarding the main query – a static page will return that single page’s content regardless of what page number you set in the URL. This is how you can get around the 404 issues with main queries. In that case, you must reference the correct query object when using the next/previous posts link functions, as Kaiser mentions in his answer.