A seated person will take up about 24'' of depth. Add in an 18'' single one of a built in french door (narrowest possible door to clear) and 42'' might not hit a seated person. A 48'' is really minimum, but 54'' would be better for someone to squeeze past. 60'' if you expect traffic to pass through. And that's if you are doing a built in that's 24'' deep. If it's deeper than 24'', add that to the space needed.

I have something similar. It sucks. Can you turn the fridge so it opens towards the kitchen wall?

But if there are normally 5 of you, you can put that extra chair against the wall (or keep it tucked in) and just not seat someone there most of the time. The end of the table will be a handy landing space, so having the chair in the corner against the wall might work better.

The problem isn't the person sitting--it's the person trying to get into the fridge elbowing the diner in the noggin.

Who comprises "traffic" when the family is all seated at the table eating?

I saw a post on this site that cited a TOH blog that called for 40" between an island and cabinets, and 60" "walkway space needed between wall and dining counter."

Here is a link that might be useful: This Old House drawing