You can't rub out a finish with pumice until you have formed a significantly thick film. After only two coats of waterlox, I very much doubt that it is so. The worst thing that could happen is that the pumice gets in the pores of the wood and never comes out, short of stripping it again.
Waterlox cures slower than varnish, and is IMO a poor candidate for rubbing out any way; it is on the soft side when fully cured.
Maybe try rubbing out with a gray 3-m pad instead.
I prefer a felt block for rubbing anyway, except on curves and turnings. It helps you get even pressure.
Rottenstone + oil gives a finer cut, and less chance of burning through. It's safer while learning and getting a feel for how much you can rub.
Casey

Casey, thanks for the gray3-M, rottenstone alternative. I just have to ditch that glossy look. I can't bear to add any more waterlox. It seems each layer changes(darkens) the color ALOT.

I also researched another finish alternative: 0000 steel wool + baby oil (I assume mineral oil is the active ingredient). It seems non-messy. What do you think
about the effectivenss of that?

I will do a test on the side...

Crispy