"Now it doesn't want to come off the tiles themselves"

You wipe it away before it hardens with a LOT of sponges and buckets of clean water.

Now that it appears set it is going to be much more difficult to remove without damaging the tile and the grout you want in the joints.

In my opinion the only ways to get cured grout off installed tile, especially modified grout, is lots of rubbing with water and one of those lightly abrasive scratchy pads or with an acid bath (recommend only a professional do this).

ACIDS AND ACID CLEANING - FRIEND OR FOE

"lots of rubbing with water and one of those lightly abrasive scratchy pads or with an acid bath "

Scrubbing will dull the tile gloss as material comes loose and is dragged back and forth over the surface.

Acid can get it off, but you need to check VERY carefully if it will damage the face of the tile, and how much damage will occur to th egrout inthe joints.

A weal acid solution and time is needed, but it also attacks the grout in the joints.

Why wasn't the grout cleaned before is was allowed to set?

You keep washing till it is gone.

Then come back in a few minutes after the water evaporates and wash again.