Varying the number of posts per page from the first one

I want to have the index page to have a different number of posts than the rest of the pages (page 2, page 3 of the index page, as well as the archive pages). In my case I want to limit the number of posts on the front page to 5, while the rest will follow the configuration parameter from wordpress admin (in my case 8).

I followed the example on this page here, and hooked in the pre_get_posts filter as follows:

function limit_posts_per_home_page() 
{
    $paged = (get_query_var('paged')) ? get_query_var('paged') : 1;

    if ( is_front_page() && ($paged == 1))
        $limit = 5;
    else
        $limit = get_option('posts_per_page');

    set_query_var('posts_per_archive_page', $limit);
    set_query_var('posts_per_page', $limit);
}
add_filter('pre_get_posts', 'limit_posts_per_home_page');

This ‘seems’ to work fine. However in actual fact with the posts_per_page parameter set to 8, Page 2 of the index page shows item 9 to 16, rather than 6 to 13, thus missing 3 items.

Is there to modify the offset in some way to make wordpress count from 6 rather than 8 from the 2nd page onwards?

Thanks to s_ha_dum for the tip. I managed to solve it by setting the offset parameter in the special case of 2nd page onwards for the front page as follows:

function limit_posts_per_home_page() 
{
    $paged = (get_query_var('paged')) ? get_query_var('paged') : 1;

    $first_page_limit = 5;
    $limit = get_option('posts_per_page');

    if (is_front_page())
    {
        if ($paged == 1)
        {
            $limit = $first_page_limit;
        }
        else
        {
            $offset = $first_page_limit + (($paged - 2) * $limit);
            set_query_var('offset', $offset);   
        }
    }

    set_query_var('posts_per_archive_page', $limit);
    set_query_var('posts_per_page', $limit);
}
add_filter('pre_get_posts', 'limit_posts_per_home_page');