Shortcode attribute escaping

I’m trying to set shortcode attribute value to ' " (single and double quote) by using escaped values ' ".

Example:
[shortcode attr="' ""]test[/shortcode]

As soon as I switch to visual editor, they get converted back to ' " and shortcode is now broken.
How does one properly escape shortcode attributes so that they can work even after switching between visual and text editor?

I suggest using keywords for them. so you would do something like [shortcode attr="single_quote double_quote"]test[/shortcode]. Then, while parsing the shortcode you can change them back.