How to prevent action in ajax request, when in admin side?

I have a plugin that hides certain posts in the frontend. For this to be consistent, I need posts to also be hidden in searches and AJAX searches (needs to work with any theme and search).

To make this work with AJAX search, I am using:

if ( wp_doing_ajax() ){
    add_action( 'pre_get_posts', array($this, 'plugin_function_set_visibility') );
}

My only problem is, this also hides the posts in the admin dashboard, posts can no longer be edited.

is_admin() does not work in this situation, it returns true for any admin-ajax request

After testing a lot of things, I found this function, which works:

function is_admin_request() {
                        /**
                         * Get current URL.
                         *
                         * @link https://wordpress.stackexchange.com/a/126534
                         */
                        $current_url = home_url( add_query_arg( null, null ) );

                        /**
                         * Get admin URL and referrer.
                         *
                         * @link https://core.trac.wordpress.org/browser/tags/4.8/src/wp-includes/pluggable.php#L1076
                         */
                        $admin_url = strtolower( admin_url() );
                        $referrer  = strtolower( wp_get_referer() );

                        /**
                         * Check if this is a admin request. If true, it
                         * could also be a AJAX request from the frontend.
                         */
                        if ( 0 === strpos( $current_url, $admin_url ) ) {
                            /**
                             * Check if the user comes from a admin page.
                             */
                            if ( 0 === strpos( $referrer, $admin_url ) ) {
                                return true;
                            } else {
                                /**
                                 * Check for AJAX requests.
                                 *
                                 * @link https://gist.github.com/zitrusblau/58124d4b2c56d06b070573a99f33b9ed#file-lazy-load-responsive-images-php-L193
                                 */
                                if ( function_exists( 'wp_doing_ajax' ) ) {
                                    return ! wp_doing_ajax();
                                } else {
                                    return ! ( defined( 'DOING_AJAX' ) && DOING_AJAX );
                                }
                            }
                        } else {
                            return false;
                        }
                    }