By insulate I assume you mean 'thermal" not "electrical" insulation. They actually make a firestop putty (looks for all the world like modeling clay) that you can use. I don't see any problem with judicious use of "Great Stuff" either.

Do you have photos?

Pics would be good. What do you mean by unfinished? Is there no insulation? Are studs visible on the inside or does drywall or other material cover the cavities?

Studs are visible.

Something - not as big obviously - as the 1st picture on left here:

Here is a link that might be useful: Sealing gaps around electrical wiring

Like I said, expanding foam would be fine, just try to keep it so it doesn't flow into the box itself and make a mess there.