Custom Post Types on Category Pages

I’m working with a custom post type that uses the normal WordPress categories.
I am able to modify my category template to display my custom post types exactly how I want them.

The problem is, once I do this, normal blog posts are also affected.

Of course I can limit either normal blog posts or the post type to just one category and create a category-ID.php to work with just that category.

But I am wondering whether there is a more flexible solution?
Is it possible to make category templates that are specific for post types i.e. one to show posts, one to show custom post types, etc?

is this possible at all? If not, what is the best way around this?

For custom post types with custom taxonomy you can make a template for that taxonomy. Your custom taxonomy template would be named taxonomy-{taxonomy}.php. For example, if your taxonomy was called “cooking-categories” you would name your file taxonomy-cooking-categories.php.

More information on template files for special taxonomy archives can be found in the codex here: http://codex.wordpress.org/Template_Hierarchy#Custom_Taxonomies_display