Is there room to squeeze in some construction adhesive all around the sides of the box? You might also be able to drive a screw or two in through the back of the box if there's a stud behind it. Of course, whatever the fix, throw the breaker first.

Polyurethane caulk will anchor it solidly.

Caulk as primary anchorage for j-box?... that may "work", but... does... NOT... sound... code-compliant...?

Polyurethane caulk is as much cement as using actual construction adhesive.