Removing Image Sizes for Custom Post Type

I’m trying to go about this without the use of a plugin, but actions and filters

Is there a way to remove extra image sizes generated by WordPress based on the Custom Post Type? I’ve been trying to work with the intermediate_image_sizes_advanced filter but it doesn’t look like it has access to $post or post_type. So I’m using the function like so:

function filter_image_sizes($sizes) {
    global $post;
    global $post_type;

    if($post->post_type == 'cpt_slides' || $post_type == 'cpt_slides'){
        unset( $sizes['thumbnail']);
        unset($sizes['medium']);
        unset( $sizes['large']);
    }

    return $sizes;
}
add_filter('intermediate_image_sizes_advanced', 'filter_image_sizes');

The Reasoning / Situation

I’ve created a Slides Custom Post Type (like an Image Slider) that uses the Featured Image to allow the user to upload “Slides”. I also have a few custom image sizes, but one specifically for the slides alone. Since the slide images are going to be one static size, I don’t want WordPress to generate all these extra image sizes I’ll never use, hence the question.

May be this filter should work intermediate_image_sizes

Note: This solution will work if you are uploading an image from post edit screen. (tested on localhost with WP-3.8.1)

add_filter( 'intermediate_image_sizes', 'ravs_slider_image_sizes', 999 );
function ravs_slider_image_sizes( $image_sizes ){

    // size for slider
    $slider_image_sizes = array( 'your_image_size_1', 'your_image_size_2' ); 

    // for ex: $slider_image_sizes = array( 'thumbnail', 'medium' );

    // instead of unset sizes, return your custom size for slider image
    if( isset($_REQUEST['post_id']) && 'your_custom_post_type' === get_post_type( $_REQUEST['post_id'] ) )
        return $slider_image_sizes;

    return $image_sizes;
}