If that is the case, it would be to your benefit, before re-installing the vinyl siding, to buy new sheets of the foam board and replace what is there (be sure to buy the same thickness of foam board). It is relatively inexpensive, and you'll rest easier knowing that, at least in THAT location, you have full insulation.

Another option would be to replace the vinyl siding with the type that has foam insulation molded into the siding itself. The problem with that is you might have a hard time matching the pattern, texture, and color.

You might have a legitimate complaint against the manufacturer, if the siding was originally installed by them. HUD requires full foam board insulation under the siding, and it appears the manufacturer may have taken some short cuts.