Add custom field to category of custom post type

I want to add a custom field (for a icon) to categories of a custom post type.
I use the default category type and not a own create category taxonomy.

How can I add the custom field only to the category of these custom post types and not to the general categories (for blog posts).