They probably used a bit that was too large. Get new spikes and see if they bite better.

How old is the home?

"Does that mean there is something wrong with the hole that was drilled in our exterior? "

What is the exterior?

10-12 year old home, brick exterior. Shutters flew off in a windstorm last year and hurricane this year, neither had much over 80 mph gusts by the time it reached us, before that they were fine. Shutters are very thin vinyl (1/8th inch) (one actually ripped when it came off), so maybe the homebuilder probably took some shortcuts. Then again the shutters were fine for 10 years or so. Anyway, I now gather that redrilling the holes isnt that big a deal as long as you dont drill too big?

I'd get some mortar and fill the holes. Then re-drill them.

They are probably attached via the plastic spikes. The spikes, while the easiest to use, are the most sensitive to hole size. Drill them too small, you can't drive the spike without breaking it. Too large, they don't grab very well.

If you are feeling industrious, you can either off set the holes you drill and use new holes, go to a larger spike, you ditch the spike altogether and either mount them directly to the brick via a concrete fastening screw/anchor or bolt up some cellular PVC to the brick and screw the shutter into that.