Assign a random avatar to every user?

I have 10k users on my site and only around 1k of them are using custom avatars (through a shortcode avatar upload I have).

The rest are using a default (and custom) singular gravatar image.

I want to try assign all of the remaining users with a random avatar from my server.

I’ve tried two methods for this:

add_filter( 'avatar_defaults', 'wpb_new_gravatar' );

function wpb_new_gravatar ($avatar_defaults) {
    $myavatar="https://example.net/avatars/avatar%20%28".rand( 1 , 411 ).'%29.png';
    $avatar_defaults[$myavatar] = "Random Site Gravatar";
    return $avatar_defaults;
}

This obviously won’t work as it just randomly picks an avatar from my destination and sets that avatar for every user.

Then there is this method:

add_filter( 'pre_option_avatar_default', 'random_default_avatar' );

function random_default_avatar ( $value ) {
    return 'https://example.net/avatars/avatar%20%28'.rand( 1 , 411 ).'%29.png';
}

This will assign a random avatar to them but it will continually do it.

Would there be a way where I can randomly assign an avatar, set it permanently and then move on from that user?