If you are on a fixed budget, and the number is low, then your best bet is to save more money while you learn how to tackle some DIY home improvement projects at your current home. The most expensive portion of any build is the labor to do it, and the only significant way to cut that expense is to bring some skills to the job. Or choose builder's grade everything just to get you into the home and slowly upgrade over time.

And right now, if you are on a limited budget, it makes no sense at all to build. It's much cheaper to buy. National averages are around 75% of the cost of your build will get you an equivlent home in most areas.

1. Our first excavator was a disaster, got us in trouble with some agencies, caused problems we had to pay to fix, etc. That was a major, major overage to the tune of around $40,000

2. Our second excavator is good, he has done great work. However, we have hit lots of rock. An extra $10k for blasting foundation and also trench lines for electric.

3. Mason, due to first excavator issues (overdug) we had to have more block in our garage foundation walls. Extra $$

4. Due to the overdig, garage floor had to be engineered, Extra $$ here

5. Plumber calls to say house is so large he would suggest a recirculating hot water loop so we would have hot water in all areas of the house without a long wait. Extra $1500

6. Drywall guy calls, says drywall just went up and he has to raise his bid by $9000 or he cant do the job. Im currently having other drywall bids come in to see where we are.

Luckily, our "budget" was flexible o me reigning everything in would have killed me in stress!

The second house was a different story. It is our long term home, in fact I don't see any reason we would ever want to move. When we started planning the budget we went with the cheaper options for finishes and things that would be easy to upgrade later. Even with that in mind we went to the bank with 10% built in for overages. Even with the cusion we went over that budget by about $5000. Here is why:
We had soil issues and had to over excavate and have and engineered basement slab installed. That was a majority of the over runs in cost. Needless to say it was a little nerve racking since it was the beginning of the project.
We upgraded a few things that we weren't planning on because it just didn't make sense to put in something we knew we would change in a year.
Our lighting budget was very under estimated, I forgot to count several fixtures(not sure how I missed them but I guess I am human!)
Our lumber salesman was horrible and the framer he suggested wasted an enormous amount of lumber so our bill was a lot higher than the take off he provided. The waste pile was at least 3 times the size of our last home.

Even though we went over budget we still were able to absorb the difference. The biggest thing I have learned is to expect that you will want to upgrade things that you initially think you won't care about. Put extra money in the buget to put in the upgrades and if you decide you don't want them then that is just more money in the bank. Better to plan to spend more than what you really think you will. Another big factor is the ability of the builder to negotiate and rearrange subs. It is amazing how many deals you can make with subs for different things (we offered the extra boulders we dug up to the landscaper) etc. That can save you money or get you an upgrade you weren't planning on!

I also built and had the appraisal exactly at build cost. The appraisal was based on comparables that were definitely inferior. So we built for less that what truly comparable existing would cost. In fact, next door a spec house is being built with a separate investor - ie 2 people need to make money to make it work. So - RE is local and in some places it makes sense to build.

If you do not have some financial reserves, then building may not be for you.

Not everything can be 'known' when starting.

The excavator may find soil at the desired depth that is not suitable for your foundation.
It may be a localized pocket that did not show on maps.
You could have paid many thousands of dollars for square foot buy square foot soil testing before you started digging, but that was probably not in your budget either (if you could even find someone to do it).
you might as well dig and then deal with anything that comes up.

When a plumbing fixture you ordered is not delivered on time and the plumber must make another trip to install it, who pays?

Building without additional reserves over the initial budget is simply a disaster in slow motion.

You will creep closer and closer to the your limit, till you hit the budget with additional work to be completed and no money to pay for the work.

They are referred to as 'unknowns' since they are imposable or expensive to quantify in dollar terms.

We built our first home (a modular) 15 years ago, and are just now beginning the stages of building our "forever" home (which is just another really nice modular).

My honest opinion? While I think a really good modular builder will help you make good choices up front (and that hopefully elminates a lot of the cost over-runs later on), I don't think there is any good way to go into a build without a healthy contingency budget.

Some things you just don't know until you start digging...and some things are going to go wrong along the way. It is just the nature of things.

Without knowing the cost of your proposed home, and how close that is to the upper limit of your budget, it is hard to give a more precise opinion - but I agree you have reason to be concerned.