WP_QUERY loop, offset in the arguments and the paginate_links – can these work together?

I have a page where the first 4 posts from category 7 are displayed with their own query and style. Then I have a small box were I want to display the rest of the posts from category 7 and to use a pagination. Basically i want to have pagination working only for the box while having the first four as pinned.
The problem is that i don’t want to have the first 4 duplicating in the box – therefore I’m using the offset in the query. Unfortunately when i do so each page from the pagination shows the same post from the previous one. What can i do to avoid this?

    <?php 
        $paged = ( get_query_var( 'paged' ) ) ? absint( get_query_var( 'paged' ) ) : 1;
        $args = array( 'cat'=>7, 'offset'=>4,'paged' => $paged);

        $the_query = new WP_Query($args);

        if ( $the_query->have_posts() ){ ?>

        <ul class="more-latest-news">

            <?php

                while ( $the_query->have_posts() ) { 
                    $the_query->the_post();

                    $title = get_the_title();
                    $permalink = get_the_permalink();

                    echo "<li><a href="".$permalink.""' title="".$title.""'>".$title."</a></li>";

                } 
            ?>

        </ul>
        <div class="navigation">
            <?php
            $big = 999999999; // need an unlikely integer

            echo paginate_links( array(
                'base' => str_replace( $big, '%#%', esc_url( get_pagenum_link( $big ) ) ),
                'format' => '?paged=%#%',
                'current' => max( 1, get_query_var('paged') ),
                'total' => $the_query->max_num_pages
            ) );
            ?>
        </div>

 $args = array( 'cat'=>7, 'offset'=>4,'paged' => $paged); 

in this array use posts_per_page insted of offset

$args = array( 'cat' => 7, 'posts_per_page' => 4,'paged' => $paged);