Sounds like the damper is non existant or stuck open. the damper is a flap that closes when the fan is not running. Take out the grease screen and take a look up the pipe with a flashlight to see if it's there.

Here is a link that might be useful: back draft damper

There will always be a draft coming out of this area. You can reduce it, but it won't be eliminated.
The first thing you can do is make sure the vent piping is well insulated after it leaves the conditioned space. In a standard exhaust setup there are two dampers. One as it leaves the vent fan and one attached to the roof cap. Make sure they are both installed. You can hear the one over the stove close when the fan is turned off. The roof one will need to be inspected.
Ron