Search only blog posts (default WP search widget)

I was using this code:

add_filter( 'pre_get_posts','search_only_blog_posts' );

function search_only_blog_posts( $query ) {

    if ( $query->is_search ) {

        $query->set( 'post_type', 'post' );
    }
    return $query;
}

..until I realized that it applies to pretty much any default search in WordPress (including search in post list page in admin area etc).

How could I make the search widget to only search blog posts (not custom posts, taxonomies, images etc) so that it doesn’t apply to any default WP search (only widget search) ?

Or is it easier to just make my own search widget?

I would prefer to utilize everything WordPress has to offer and not to reinvent the wheel.

@PieterGoosen has a good description on why your pre_get_posts callback is giving you a problem.

Here’s an alternative workaround to restrict the native search widget to the post post type:

/**
 * Restrict native search widgets to the 'post' post type
 */
add_filter( 'widget_title', function( $title, $instance, $id_base )
{
    // Target the search base
    if( 'search' === $id_base )
        add_filter( 'get_search_form', 'wpse_post_type_restriction' );
    return $title;
}, 10, 3 );

function wpse_post_type_restriction( $html )
{
    // Only run once
    remove_filter( current_filter(), __FUNCTION__ );

    // Inject hidden post_type value
    return str_replace( 
        '</form>', 
        '<input type="hidden" name="post_type" value="post" /></form>',
        $html 
    );
}  

where we use adjust the output of the get_search_form() function but only for the search widgets.