I did a google search and this is what I came up with:

The pollen problem
Some consumers dislike lilies because pollen on the lilies anthers rubs off easily and can stain clothing. Florists can either remove the anthers as the blooms open or spray the anthers with a fresh flower sealer or fixative.
Following are tips for removing pollen from clothing.
1. Do not brush the pollen away with your hands; oils from your skin will set the stain.
2. Let the pollen "dry," then carefully brush it away with a dry, soft brush, chenille stem, or facial tissue. Adhesive tape also works well; gently dab pieces of tape on the fabric until the pollen is gone. Do not use water or a wet cloth; this will spread and set the stain.
3.If some pollen remains after completing these procedures, place the item in direct sunlight for a few hours; the stain should "magically" disappear.
4. Pollen stains can often be removed from washable fabrics by pretreating them with an enzymatic detergent such as Era.
The International Flower Bulb Centre

What about Pollen Stains?
Be mindful of pollen. If you remove all the pollen-bearing anthers just as the buds begin to open, your flowers will last longer, and pollen cannot stain tablecloths, or your guest's nose.
Should pollen brush upon clothing, it can stain if not treated correctly. Resist the temptation to mearly brush it off with your hands, for the natural oil on your hands will spread, and set the color. Simply use a medium stiff hairbrush, and if the pollen grains have not been smeared, you can safely knock them off. If noticed too late, use a washday pre-treatment before laundering, and hang the freshly washed clothing in bright sun for several hours, which usually bleaches any remaining color.
Snow Creek Gardens

Other sites suggested using scotch tape, and/or a vacuum cleaner nozzle over the pollen to "lift" the pollen off.
Whatever you choose, they all warned against our natural instinct to immediately attempt to rub or brush the pollen off!

Good luck!
~Meg

I work in a flower shop where we routinely remove this pollen as soon as the lily opens. When we get on clothing, we use Shout or something similar to get off clothing. I agree that you do not try to brush off. It only makes it worse. Also, if the pollen is pulled off as soon as the lily opens, it should be "firm" or not fuzzy and there will be little or no "fuzzy" stuff to get on hands or clothing.

If you do happen to get the stuff brushed in or wet and the stain won't just brush off, I've had good luck with Fantastik.

Help...I have red lily stamen stains on a light gray leather seat on a "borrowed" wheelchair. Of course I was sitting, with my bare leg, and the stain is smeared into the leather. I have used a Shout like product, and Oxyiclean with toothbrush. Still no luck. Please let me know what I can do.