Yeah, you screwed up with a rotary. You always want to use a palm sander, and with the grain. Or a block sander with the grain. The grit you need to sand with again will depend on how deep the scratches are. If you started with 80, they're probably pretty deep, and you will probably need to start with at least 80 again. And hope there are no veneers involved.

It only takes one stray grit particle from a coarse paper to ruin the sanding when using a ROS. For pine it's usually not an issue, but on cherry (in particular) it can be a disaster.
To fix it, I'd start with a cabinet scraper which will get the swirls out faster than sanding. Then block sand, by hand, with the grain, with 220.
A cabinet scraper does require a learning curve for both sharpening and usage, but it's a miraculous little device.
Casey