Custom query in archive.php with pagination nightmare

I want to run a custom query showing only subcategories of a specific category inside my archive.php with pagination.

But, for some reason, it keeps counting (and paginating) by the number of post inside of all subcategories.

ie: If my custom code says that I want 5 child cats per page and I have 10, than I should get only 2 pages but it keeps showing me 3 pages like thi: 30 posts (inside all subcategories) / 10 (per page) = 3 pages

By doing my research, I found out that the pagination is calculated before my template run. And, because of that, I tried to use pre_get_posts hook to modify the main query but that seems not to work either…

How can I do this? (bellow some snippets I’m using)

My pre_get_posts function:

function showposts_only_5_on_cat( $query ) {
    if ( ! is_main_query() ) {
        return $query;
    } else {
        if ( is_category(3)) {
            $query->set( 'showposts',5 );
        }
      return $query;
    }
}
add_filter( 'pre_get_posts', 'showposts_only_5_on_cat' );

The structure to select the code to present:

<?php if (is_category(3)) {?>
<!-- bof special cat query -->

HERE IS THE PLACE WHERE I HAVE THE CUSTOM CODE

<!-- eof special cat query -->
<?php } else {?>
<!-- bof regular cat query -->

HERE I HAVE MY THEME'S CODE (BELLOW)

<!-- eof regular cat query -->
<?php } ?>

This is archive.php loop of my theme

<?php if (have_posts()) : ?>
            <?php while (have_posts()) : the_post(); ?>

// posts in here

            <?php endwhile; ?>

<?php       // Pagination
        if(function_exists('wp_pagenavi')) :
            wp_pagenavi();
        else : ?>
            <div class="navigation">
                <div class="alignleft"><?php previous_posts_link() ?></div>
                <div class="alignright"><?php next_posts_link() ?></div>
            </div>
<?php       endif; ?>

<?php       else :  

//no posts content

        endif;
        //Reset Query
        wp_reset_query(); ?>