do I need to use an epoxy to fill in those spots where the wood has been worn? It does not have dry rot, it is still solid wood.

what should I do to fix this? Need to do several window sills before the rain comes.

Then (I like Flood products) so this is one you could use to clean all your exterior wood before refinishing, thus removing the residue left in the wood, good for decks and all exterior wood that needs refinishing
http://www.floodco.com/wood-care-solutions/products/view-product.do?pr oductId=8&retailer=

I would use a wood sealer next since you do not want to stain this window sill.
http://www.preservation-solutions.com/exterior-wood-sealer.php

Then I would use a good primer, and exterior paint giving at least a couple of coats and let dry well before applying top coats.

If the sill looks like this, then the windows must also need re-glazing, if you live in a place with a winter climate on the horizon, you need to address this or you will lose precious $ out the windows! (also A/C use in summer)

I would address all the windows in the home, the sills and any air leaks they are causing. You could just paint with exterior paint, BUT you will have to do it again sooner than if you address the real issue head on, rain and environmental attacks on the wood on the exterior of your house.

Good luck with your project!

Just check the other sills and make sure they are all intact and weather proofed.

Good luck with your new house and your good maintenance practice!