Jetpack Infinite scroll conflicting with theme’s pre_get_posts custom posts_per_page

You’ve specified a priority of 1 for your pre_get_posts function here:

add_action( 'pre_get_posts', 'mytheme_portfolio_archive_pages', 1 );

Jetpack doesn’t specify a priority:

add_action( 'pre_get_posts', array( $this, 'posts_per_page_query' ) );

So that gets added with the default priority for add_action, which is 10:

$priority
(int) (optional) Used to specify the order in which the functions associated with a particular action are executed. Lower numbers correspond with earlier execution, and functions with the same priority are executed in the order in which they were added to the action.
Default: 10

So for each of these requests, your code is executed and sets posts per page to 20, then Jetpack’s code is executed and overwrites your value with its own.

To fix this, just specify a lower priority so yours runs later and overwrites Jetpack’s:

add_action( 'pre_get_posts', 'mytheme_portfolio_archive_pages', 20 );