The steps and materials sound just fine.

Poly is an interior finish---unless it specifies exterior---and I would not use poly outside anyway---it will crack.

Exterior matte or satin paint is designed to 'chalk' and will get dusty after a while. Using gloss exterior will make washing easier, no chalking, and dirt does not stick as easily.

If I dont use a clear top coat -- will the finish rub off onto clothing with an exterior paint (semigloss?)

The only top coat I know of that will not degrade quickly in an outside environment is marine spar varnish-----and even that will need recoating periodically.

Gloss exterior paint will last the longest without some kind of periodic maintenance and not cause a shedding problem.

Using the right exterior paint, one that will not "chalk", it will not rub off on your clothes. It is not necessary nor desirable to add a clear coat. First, you risk adhesion problems of incompatible finishes. Second, most clear finish have very poor UV resistance. Just ask yourself what type of clear finish would you put on your house after you've painted it.

Second the gloss or semi-gloss exterior enamel paint.