You can try knocking off the lumps and bumps with a "brick chisel" and a heavy mallet.

Wear full-face protection - chunks fly everywhere.

Place the edge of the chisel at the base of the lump and tap with the mallet. Or whack it hard ... sometimes the lump pops right off along the stone's layers.

Sure, you can whack at it, but that's just going to leave you rougher spots than you already have. Flagstone isn't "smooth" by definition. It's a rough surface, suitable for exterior applications where slip resistance is important, like around a pool deck or for a pathway. If you want something smoother, you may have to go with slate, but beware of how slippery it can get when it gets a bit of moss on it or rain on it. Or, try a man made surface.