I tried another solution and that did the trick. I found in the WooCommerce docs this little snippet:
// Display 24 products per page. Goes in functions.php
.
add_filter( 'loop_shop_per_page', create_function( '$cols', 'return 24;' ), 20 );
It is overriding the settings in the customizer. I only have to change the number ’24’ to my liking.