I would make Edit, with the pencil, the primary action in this list-view and the you give 2 options:
Use swipe gestures to delete the category, with leave-behinds, as stated by @zach.
Add a contextual menu in the top navigation bar (3 vertical dots), with the menu-item: “Remove categories”, after which you can select the categories you want to delete as a user. This way users can also remove multiple categories at once & they are not prone to accidentally removing categories.
You may Also Like: