Did you reinstall the chain with the trolley in the correct position?
I'd operate the GDO without the chain attached, observe the direction that the chain-drive cog turns and reinstall the chain accordingly.
This is the same process as when installing a new GDO.

I tried that, too. Even with the chain uninstalled, the trolly only moves 4-5 inches and then reverses. I'm stumped.

Make sure you put the black cup back on the end of the motor shaft and that it is pressed all the way on. Make sure the RPM sensor is plugged in. If neither of these are the issue, then you probably did not have the motor shaft pushed all the way back when you tightened the coupler after replacing the worm gear. Loosen that coupler up and from the front of the motor (opposite end in which the coupler is on) push the shaft back as far as it will go. Hold it there while you tighten the coupler back up and then try it.

GARAGE OPENER FIXED!! Thanks for all the help. Without it I would not have found the problem. When I replaced the gears I had (stupidly) removed the up/down limiter and got it all out of whack. Once I realized what was going on I was able to adjust that and, along with readjusting the up/down travel and the up/down force, it is now working as well as a 12 year old opening can. Thanks for the input, I really appreciate it.