My home is 5000 sqft above grade (three floors). If I had 2000 just for the tankless equipment, what would be the best unit to think about?
You do not give nearly enough information for any suggestions, and if you want installation included in that price, your budget is not sufficient.
Why are you interested in retrofitting tankless instead of replacing with another tank? Many people have misconceptions of how a tankless works and what benefits they may receive from switching.
Are you an electric home or is gas available? Electric is truly cost prohibitive to retrofit, whereas gas retrofitting is merely exhorbitant.
Where are you located? You need to pick a unit that can give you enough temperature rise for your expected volume of use at the coldest time of the year. That will mean different size units for Indiana vs. Florida.
Are you very handy, or will you have to pay labor for everything? Typical labor charges are $800-$1200K, depending on how much retrofitting needs to occur. A tankless won't just fit into your tank's spot without some alterations.
Are you willing to change some behavior on your part to be able to go with a tankless? Take inline showers instead of running two at once or running that load of laundry after you've run the DW at night. If you can reduce your demands for simultaneous hot water, a smaller unit can meet those and be less expensive.
How long will you be in your home? Payback costs have a LONG timeline. Payback also may depend on whether or not your local utility has any rebates available as incentives. Here, my local electric supplier will give a $100 rebate to anyone replacing a tanked electric with another tanked electric. And my gas company doesn't have any incentives at all. (It's still cheaper to have gas appliances, even so.)
You are gonna have to sit down and do some math to figure this out. It's not as simple as replacing an existing tank heater and deciding to get the 50 instead of the 40.
So to answer some of the questions. The 2k was for the unit itself. I don't care about operating costs, as a hundred bucks here or there doesn't matter (not to sound silly).
We are just above New York City. What I want is consistent hot water. Right now we have a 38 gallon indirect off of the oil steam boiler (which we are also replacing), and two showers later, no hot water for two hours.
It is my understanding that we will always have consistent hot water moving to a tankless, which is what I want. If I can do the same thing with a tank, ok by me too.
Old house with no ng? Install cost could be several thousand dollars. A normal electric tank would be fine and solar would be cheaper in the long run. The problem with electric tankless is that you have to upgrade your electric service.