Restrict taxonomy dropdown to post type

I have 5 different custom post types and one custom taxonomy. I’ve been able to get a taxonomy dropdown using <?php wp_dropdown_categories( $args ); ?> but I noticed that I was not able to restrict the taxonomy dropdown to show the terms which have posts from a particular post type.

Some of the comments that I got asked me to create a different taxonomy for each post type but that is not possible in my case since my custom taxonomy is for defining a location and I’d like all the 5 custom post types to use this one taxonomy.

I know that I can load the posts from a post type and get the terms from those posts and use a foreach loop to loop through these and output them as options in a select field but I don’t want to use this since I would lose the hierarchy that I had set up for the taxonomy. If I can get the hierarchy using this method then I’m open to that too.

Is this possible or do I have to create separate taxonomies for each post type?

UPDATE
I wanted to be able to add a custom field value also as a parameter. TheDeadMedic’s answer does that.

You can use a snippet I found for querying terms that are attached to a particular post type:

function wpse_207655_terms_by_post_type_or_meta( $clauses, $taxonomy, $args ) {
    $meta_query = new WP_Meta_Query;
    $meta_query->parse_query_vars( $args );
    $post_types="";

    if ( ! empty( $args['post_type'] ) ) {
        if ( $_post_types = array_intersect( ( array ) $args['post_type'], get_post_types() ) )
            $post_types = implode( "','", $_post_types );
    }

    if ( $post_types || $meta_query->queries ) {
        global $wpdb;

        $clauses['join']   .= " INNER JOIN $wpdb->term_relationships AS r ON r.term_taxonomy_id = tt.term_taxonomy_id INNER JOIN $wpdb->posts AS p ON p.ID = r.object_id";
        $clauses['orderby'] = "GROUP BY t.term_id {$clauses['orderby']}";

        if ( $post_types )
            $clauses['where'] .= " AND p.post_type IN ('$post_types')";

        if ( $meta_query->queries ) {
            $sql = $meta_query->get_sql( 'post', 'p', 'ID' );

            $clauses['join']  .= $sql['join'];
            $clauses['where'] .= $sql['where'];
        }
    }

    return $clauses;
}

add_filter( 'terms_clauses', 'wpse_207655_terms_by_post_type_or_meta', 10, 3 );

Now you can use a post_type argument in wp_dropdown_categories (that actually just gets passed onto get_terms, which is where our filter comes into play):

wp_dropdown_categories( 'taxonomy=location&post_type=X' ); 

Update: Now you can also use WP_Query meta arguments:

wp_dropdown_categories( 'meta_key=foobar' );

…or a full blown meta query:

wp_dropdown_categories( array(
    'post_type'  => 'foobar',
    'meta_query' => array(
        array(
            'key'   => 'this',
            'value' => 'that',
        ),
        array(
            'key' => 'foobar',
        ),
    ),
));