100 grit may have been too rough---150 usually works better, IMHO.

Try one and see if the paint sticks---give it a couple of days. The only problem may be on spindles where kids grab, otherwise there is no wear and tear.

First, make sure they are washed with warm soapy water and rinsed well. As Mac says, 100 is too rough and will show sanding marks when the paint dries. Finish sand with 150 grit to smooth the surface. A sanding sponge is much easier to use on spindles. Get a fine grit one.

Primer should be used over poly. Poly is a hard sealant/barrier. A primer like Kilz or Bulls Eye will adhere to the primer, hence providing an adhesion base for the paint.

For painting spindles, a skiff brush works best.

Michael