Loop through all tags & output posts in alphabetical list

I have a bunch of posts that each have multiple tags, and I’m trying to find a way to output all of them on a single page, organized under an alphabetical listing of their respective tags. E.g. if Post1 has the tags A, B and D and Post2 has the tags A, C and D, the output would look like this:

Tag A
Post1
Post2

Tag B
Post 1

Tag C
Post2

Tag D
Post1
Post2

EDIT: I’ve gotten it working with categories, but I’d still love to have it work with tags instead. (All of the excluded IDs are because I’m technically using categories for other organization.) The functional code is:

<?php $cat_args = array(
    'orderby' => 'title',
    'order' => 'ASC',
    'exclude' => '26,27,32,52,36,31,42,38,41'
    );

$categories = get_categories($cat_args);
    foreach ($categories as $category)
    {
    $catID = $category->term_id;
    $catName = $category->name;
    echo '<strong>'.$catName.'</strong>';
        global $post; // required
        $pArgs = array('category' => $catID,'post_type' => 'shows','orderby' => 'title', 'order' => 'ASC');
        $custom_posts = get_posts($pArgs);
        foreach($custom_posts as $post) : setup_postdata($post);  ?>
            <div class="show">
            <a href="https://wordpress.stackexchange.com/questions/46830/<?php the_permalink(); ?>">
                            <?php the_post_thumbnail("show"); ?>
                <h3 class="center"><?php the_title(); ?></h3>
            </a>
            </div>
        <?php endforeach; ?>
        <?php } ?>

(Untested) but should works with any taxonomy including the ‘tag’ taxonomy (post_tag). The following example uses the taxonomy with name ‘my-taxonomy’.

<?php
//Get terms for this taxonomy - orders by name ASC by default
$terms = get_terms('my-taxonomy');

//Loop through each term
foreach($terms as $term):

   //Query posts by term. 
   $args = array(
    'orderby' => 'title', //As requested in comments
    'tax_query' => array(
        array(
            'taxonomy' => 'my-taxonomy',
            'field' => 'slug',
            'terms' => array($term->slug)
        )
     ));
    $tag_query = new WP_Query( $args );

    //Does tag have posts?
    if($tag_query->have_posts()):

        //Display tag title
        echo '<h2> Tag :'.esc_html($term->name).'</h2>';

        //Loop through posts and display
        while($tag_query->have_posts()):$tag_query->the_post();
            //Display post info here
        endwhile;

    endif; //End if $tag_query->have_posts
    wp_reset_postdata();
 endforeach;//Endforeach $term

?>