Sure you can do a broker only open house. I suggest that you serve food (agents are hungry!). ;)
You could print up flyers and distribute them to every RE office in your area. You could probably also find many agent's email addresses by looking on their company websites.

"If you do a flat fee MLS listing, and thus are not represented by an agent, can you still have an open house for agents? Would they come?"

I would be very surprised if they didn't.

When I sold our last home (successfully by owner in two months), we were flat fee-listed and I held an open house for agents. I emailed probably 200 agents in my area (and let them know I'd have refreshments!). This was in 2007, right before the housing market completely crashed. I was offering a three percent buyer's agent commission on a home priced in the high $400s.

One agent showed up. And she didn't eat anything. :-)

I should have added to my post above that, despite the poor response to the last time I tried an agent open house, I'm going to try it again. We're flat fee-listed on our current home, planning to lower the price soon. I'll do it then. The only way to sell a home in this market is to work hard at it. Good luck.