How to remove an item for a OR’d enum?

I have an enum like:

public enum Blah
{
    RED = 2,
    BLUE = 4,
    GREEN = 8,
    YELLOW = 16
}

Blah colors = Blah.RED | Blah.BLUE | Blah.YELLOW;

How could I remove the color blue from the variable colors?