I'd be tempted to say, "I'm sorry you feel that way. I won't ask you to subject yourself to it a minute longer."
Or, "I'm sorry you feel that way. Of course, I have no idea how it compares to your home, as we've never been invited there."
How about, "Now, why would you say that?"
You're right, it was inexcusably rude. Next time he calls around (and every time after that), in your shoes I'd say "I'm sorry, it's not convenient to have guests at the moment," and send him on his way.

The best retort to a remark like that is none....just a steady unblinking stare. If he doesn't at least lower his gaze....you might blink a few times and say...very sweetly of course..."I'm sorry...I don't think I know what you mean...."
Force him to see how out of line that remark was....without actually being rude yourself.
Linda C

I would have simply said "It may not be the Taj Majal but we like it and it suits our needs." And then I would have just given him my mom stare. Nancy Louise

My reaction is to stop dead in my tracks and say, "I beg your pardon? I know I didn't hear you correctly." With a smile, of course. I've never had the rude commented repeated.

My method of responding to rude comments is to lock eyes and firmly with warmth say something like " isn't it wonderful ... it's just right for us and we LOVE it" . I wouldn't take it seriously at all... people just don't think and some like to pull others down-I try not to let them. Congratulations on your new townhome!

I think in this case, the best would be to say nothing. You aren't going to teach him anything, no matter what you say. If you just can't restrain yourself, I like rhizo_1's suggestion. It isn't nasty or snarky, and it gives him another chance to get it right.

Definitely, don't say something rude or that will put him on the defensive. That just validates his making the rude comment in the first place, in a way, if you pretty much respond in kind. You be the bigger person.

I'm the type of person that would have said "Well, that was rude!" He probably then would have laughed and explained himself, then the matter would have been rectified. If he's rude enough to say such a thing, then he can face the consequences. I'm all for "if you can dish it out, then you can take it" attitude.

I guess Mr. Rude comes to your house because you are obviously too nice to turn him away or be rude like him. He probably knows this, but you may not. I'll bet you sleep a lot better at night than he does.

I mean, at the end of the day, my behavior is all that I am ultimately responsible for.

For what it's worth, that is my 2cents.

How well do you know this guy? How did your dad respond?

I ask because we have a friend like this who likes to give it and take it in the same manner. I would have responded: "well it's nicer than yours!" And we would have had a good laugh about it.

You haven't had the benefit of the golf course discussions they have shared and their own interaction. Unless your dad was offended I wouldn't give this another thought. And there are a million reasons why he might not have invited your dad to his house. Unless you know, don't assume.

My husband and I share an office suite even though we are in different professions so I sometimes hear comments he and his clients or friends make to each other in the reception area. They have inside jokes and rude comments and its their way of communicating. Guys talk to each other differently. My adult son and his buddies are another case--they call each other "loser" and other names, but the tone is teasing.
Women on the other hand communicate with each other about personal matters. we were at a recent neighborhood gathering and I heard more about medical treatments, who was in therapy/treatment &childbirth experiences in 20 minutes than I want to hear the rest of my life. Meanwhile the men are joking about some football game.
So my point is that as dgmarie says, if you Dad isn't offended I wouldn't give this another thought. Your Dad may like friends dropping in to visit with him.

I may have said something like "And thank goodness for that! We love it".

Rudeness always sort of catches me off guard. I just don't expect it and I'm quite taken aback by it. So my normal reaction is the "deer in the headlights" syndrome. I sort of stare blankly at the person for about a second or two. Then my wits return and my MO is just to ignore the comment and either continue the normal conversation or divert my attention elsewhere, like to another person if possible.

It's always easy to think of great comebacks later, but for me, no comment works best. I'm usually fairly reactionary and have said things I regret so this sort of natural reaction is maybe a survival thing for me. It prevents me from saying something that I'll wish I hadn't.

Now, in the case of a close friend I might let them know that the comment was hurtful later, just because we're friends and we have that comfort level. Depends on what was said, how it was said, and my perceived intent of the speaker. In the case of a relative stranger, or an acquaintance whom I wouldn't consider to be a friend, I'm happiest keeping my mouth shut.

I will admit that later I'll tell the story to DH and go over the rudeness and my feelings, etc. way more than he wants to hear! But that's my way of getting it off my chest while maintaining my composure when needed.