Try a drop of Clorox on a hidden brick and see what happens. We used bleach on saltillo tile (basically an unfired brick that absorbs moisture like a sponge). We tried sealing them and it didn't work. We moved and promised ourselves to never use saltillo for anything ever again. There are too many ceramic and porcelain tiles that look as good or better than saltillo anyway.

Whay are they moldy? Are the bricks wet all the time? Mold will grow on anything that is wet all the time. You're not watering every day are???