new WP_Query messes up pagination

If I understand you correctly you just want to display a different amount of posts at archives, for this you don’t need a custom secondary query at all. You can control it via pre_get_posts hook, either making use of posts_per_page or posts_per_archive_page parameter.

Example for pre_get_posts

function wpse188786_different_number_of_posts_for_archive_ppp( $query ) {
    if ( !is_admin() && $query->is_main_query() ) {
        if ( $query->is_archive ) {
            $query->set( 'posts_per_page', 4 );
        }
    }
}
add_action( 'pre_get_posts','wpse188786_different_number_of_posts_for_archive_ppp' );

Example for posts_per_archive_page

function wpse188786_different_number_of_posts_for_archive_ppap( $query ) {
    if ( !is_admin() && $query->is_main_query() ) {
        // no $query->is_archive check needed, because posts_per_archive_page
        // does override, when $query->is_archive or $query->is_search is true
        $query->set( 'posts_per_archive_page', 4 );
    }
}
add_action( 'pre_get_posts','wpse188786_different_number_of_posts_for_archive_ppap' );