How long did you wait before taking a scraper to it?

Saturday -- Finish drywall mud
Sunday -- Prime
Monday -- Paint first coat
Tuesday -- Paint second coat
Wednesday -- Install lighting fixtures
Thursday -- Touchup, that's when I happened to scrape off a couple nibs

Did it scrape off to raw drywall or just to the primer? Don't forget that paint takes a month to cure so it will be much more durable than it is now in a few weeks. Also, if you are using the shower and subjecting the walls to mass amounts of moisture, this can be like re-wetting the paint film and this can affect durability as well.

paintguy,

It separated between the primer and the drywall compound -- back side of the paint "souvenir" was white and top side was "almond."

I understand that primer dries in minutes but cures in weeks.

Shower is not yet operational.

Inside the shower area I used 1/2in Hardieboard and covered it with 1/4in Swanstone. The paint peeled outside of the shower area.

???

It could be drywall dust then. I think this is pretty rare to actually prime over enough dust to cause the paint not to stick though. If this is the case, hopefully it's just a spot here and there and you shouldn't have much to worry about. That's all I can think of.