Install a back draft damper in the line.

http://www.batticdoor.com/bathfandraftblocker.html

"Maybe it should have a flap that closes when the fan is off or maybe we just need a better exhaust fan. Any advice?"

It should have a flap and better fans have better flaps.
You can sometimes see the flap after pulling the cover off the fan. If you can see it AND this problem only started after several years,you can try freeing the flap up. If this has always been a problem,you can (A.) help it some by installing about 4' of flexiable hose and mount it with curves like an inverted J , ( or maybe reconnect origional one) (B) Replace it with a better fan AND add a hose if the present fan is loud. Cheap,loud fans have blades like airplane props where quility fan blades look like a squirrl cage and are much quiter.

Thanks everyone for the advice. I have a better understanding of our situation now. Looks like when this winter storm clears up we will be doing some work. Gonna inspect our present fan and maybe replace it if needed and contact a general contractor about the backdraft if DH feels like he can't do it. And bpChill thanks for the link.

If your bathroom exhaust fan dumps INTO the attic that is a terrible installation... think mold.

If you are going to address the back flow issues, I'd suggest also have the contractor route the exhaust to the outside of the house.

When doing so, be aware of and prevent any low points where condensate can accumulate.

We have the same problem. Our exhaust fans are NuTone/Broan and were installed in 2019. They were not cheap. They are vented to the outside on the roof. I do feel the heat coming out on hot days here in Connecticut, and in the winter the cold air comes out. Where does the draft damper go? I'm assuming it can be added to the current system. Thanks for all the advice.