Put a primer over the pen marks and then paint.

Pen is an ink so consider it to be a stain that will bleed through latex paints and primers. You can either try to remove the pen marks or use a stain killing primer like Kilz or Bin and then touch up. Either way, you will likely need to touch up. I would probably use lacquer thinner to get the pen marks off and then touch up (easier than actually using the stain killing primer). I'm sure you don't have lacquer thinner on hand, so try some rubbing alcohol, Goo Gone, Soft Scrub with Bleach, hairspray, toothpaste, magic eraser, etc...so many things here that people have claimed to have worked. And then touch up.

Thanks, paintguy! I will work on cleaning it off. Once I get up the energy to plane the door, I hope to do all the touch up at once. I may end up sanding and spackeing a bit too. The more I live with the walls the more it annoys me, especially since I know I could have done it better the first time if I did it myself. Not looking forward to sandng through the two coats plus primer I insisted on, but I guess that's the price I pay for not being able to supervise better.

Hairspray works wonders in terms of removing ink - I've done this on clothing, but not on paint. Just spray and dab.

Trouble is paint does NOT go over the hairspay.

That's why you wash the hairspray off after you've removed the ink. Prime & paint. Hairspray isn't going to saturate dried paint if you wash it off right away.

I would first try a Mr Clean Magic Eraser just using the very edge of the sponge. It should come right off. If not prime and paint. You may also want to thin your paint with a little water just for touch ups.