Varying the number of posts per page from the first one

Thanks to s_ha_dum for the tip. I managed to solve it by setting the offset parameter in the special case of 2nd page onwards for the front page as follows:

function limit_posts_per_home_page() 
{
    $paged = (get_query_var('paged')) ? get_query_var('paged') : 1;

    $first_page_limit = 5;
    $limit = get_option('posts_per_page');

    if (is_front_page())
    {
        if ($paged == 1)
        {
            $limit = $first_page_limit;
        }
        else
        {
            $offset = $first_page_limit + (($paged - 2) * $limit);
            set_query_var('offset', $offset);   
        }
    }

    set_query_var('posts_per_archive_page', $limit);
    set_query_var('posts_per_page', $limit);
}
add_filter('pre_get_posts', 'limit_posts_per_home_page');