This problem is common and has been discussed on this forum recently. You might want to do a search on the forum, but I'm not sure what word or words will locate those previous discussions. So I'll give a brief summary.

Many fireplaces, especially prefabs, are equipped with an air intake opening somewhere in the firebox. Its purpose is to provide fresh air and oxygen to feed the combustion and to reduce or eliminate the need to open a window to compensate for the negative air pressure that results from a chimney fire. Look around inside the firebox. Use a flash light, because it may be hidden and covered with soot. If the fireplace is on an outside wall, you should see an opening on the exterior of the wall. There is usually a sort of trap door that you can close to stop the draft when you do not have a fire going (or there should be one).

I covered the front of my fireplace with a sheet of wood painted the same color as the wall.

My fireplace/house is 33 years old and I was told fireplaces built in that era were often not insulated around the firebox and cold air can come in the chimney around the firebox straight down through not only the fireplace opening, but the walls as well. My fireplace is so cold the floor on the edge of the room right above is too cold to set our feet on without keeping a heating pad on the floor beside the chair for our feet. I am so fed up I am ready to have the whole thing closed up and removed. From reading other posts on this, it sounds like a lot of people have these problems. I am hiring someone who specializes in building fireplaces and insulating to see if he can do something, but I will never own a home with a fireplace again.