You probably should get rid of all that paint - it was obviously either not the right kind of paint for the brick, or else the brick wasn't washed prior to painting. You're only going to run into more trouble and sealing it in will cause worse trouble in future.

I love the look of partially white-washed brick and in the past have tried to figure out how to replicate it on my house. Lucky you, to actually have it!

I'll bet if you just leave the brick alone the whitewashed look will last a long time. Sealing it might make the white areas turn yellow, and it's hard to imagine a sealant that would work well on painted areas AND brick.

Scrape of the worst of the flakes, and then sit back and enjoy.