if you let it dry completely, it should have all just pulled off. sounds like you tried to clean it up before it dried. a piece of sandpaper should do it.

Thanks davidandkasie, i'll try sand paper. I actually didn't try to clean it off before using the razor blade. It's like it sunk into the brick.

brick is porous, so it could have sunk in a good bit. a light sanding should take the stain off or at least make it less noticeable.

it will fade in time, but that could take years if the area is not exposed to direct sunlight/weather!

mellody-
It sounds like you're getting close to a situation I've been in a number of times: Working on a problem so much that it ended up looking worse than when I started. You might try a light scrubbing with a wire bristle brush if you've gotten off what you can with the razor blade. But then I'd leave it alone. Once you've abraded the brick surface very much it's probably going to look worse.

I'm wondering if you ever found a solution to your problem. I have a brick wall inside my house and I'm thinking about childproofing the corners by using some silicone adhesive to install some padding, but I'm worried that when we go to take it off, the wall will be permanently stained. There are lots of silicone solvents on the market...How did things work out in the end for you?