How to exclude latest x posts from a paginated query?

I have a WordPress loop, that is part of an AJAX call. Every time a button is clicked, this function is called, runs the below loop and outputs the next 6 posts. There are already five posts shown by default. How can I exclude the five latest (most recent published) posts from the WordPress loop? So that page 1 of the loop starts with the 6th, 7th, 8th, 9th, 10th & 11th post?

This is the loop that I use. It would be great if someone could tell me how to alter it so that it starts with the 6th post. Thanks in advance!

$args = array(
    'post_type' => 'post',
    'post_status' => 'publish',
    'posts_per_page' => '4',
    'paged' => $paged, //Starts with 1 (or a different number if that's better)
);
$query = new WP_Query( $args );
?>

<?php if ( $query->have_posts() ) : ?>
        <?php while ( $query->have_posts() ) : $query->the_post();?>

        <div class="amb-home-post-single override">
            <div class="amb-home-post-contents override">       
                <img class="amb-home-thumbnail-container override" src="https://wordpress.stackexchange.com/questions/364847/<?php echo the_post_thumbnail_url(); ?>">
                <h2><a href="<?php the_permalink() ?>"><?php the_title() ?></a></h2>
                <div class="amb-home-post content override"><?php the_excerpt() ?></div>
            </div>
        </div>


        <?php endwhile; ?>
<?php endif;

One way is to use the offset parameter in WP_Query, but it overrides the paged parameter according to the docs.

Example formula for an offset with pagination:

'posts_per_page' => $posts_per_page,
'offset'         => ( $paged - 1 ) * $posts_per_page + $offset_per_page,

Example:

Then for $posts_per_page as 4 and $offset_per_page as 5 we get the post numbers:

first page:   6,7,8,9    (skip first 5 posts)
second page: 10,11,12,13 (skip first 9 posts)
third page:  14,15,16,17 (skip first 13 posts)
...

Notes:

In this previous answer I’ve added a way to play with another type of formula.

There might be some performance issues for really large offset numbers.

Note that WordPress also has it’s own REST API that can in many cases be used instead of Ajax.

I also wonder if the offset is really needed here, if the original loop + Ajax can be replaced with a single REST API setup instead.

ps: above is untested.