Why do I have to use pre_get_posts with custom WP_Query to reset posts_per_page?

The number of pages you have in an archive is based on the results of the main query. You can’t paginate a custom query beyond the number of pages the main query contains, because it is determined to be a 404 before the template that runs your custom query is ever loaded. Altering posts_per_page and running your custom query works because you’ve made the number of pages match on both queries.

If the only thing you want to do differently with your custom query is alter posts_per_page, then you should remove the custom query entirely and use only pre_get_posts with some sort of is_ check to make sure you only alter the archive query you want to change.