Could you provide pictures?

Buy foam backed vinyl and glue it over the tile, you have to put down a leveler which you can easily do. You could even cut and install it yourself. Then just take off the doors of the vanity and sand and brace them, paint the entire unit a nice white, add new hardware. You can buy complete sets at Target very cheaply. A toilet is very easy to replace. You can watch it on Utube and just follow the steps. Same with redoing the vanity. We just finished doing an entire gut on our little bathroom and we're in our seventies. Every step was done by us. I used Utube for all the tutes like installing the Swanstone tub surround. I even made my own paints as I couldn't find what I wanted.
Learn as much as you can, so much available on line to walk you through every process. We did our entire remodel under three grand. That may not be what you plan to spend but retiling the shower floor and grouting is just grunt work and not much money. Same with the toilet.
We have always owned multiple properties and we did most of the work over the years. Trust me you can do it. Good luck.

gmp, I will!

yayagal, thank you for the encouragement! I think my DH would be OK with the floor and toilet. You Tube is a great suggestion! I would definitely like to replace the vanity . . .it is so old and was never nice to begin with. The question is what to do with the sink and attached countertop. I am not against getting a new sink, but the problem is how to get it loose from the tile w/o damaging the tile. (Granted, I HATE the tile, but it's got to stay for now, I guess.)

Re: gutting the bathroom-- it would involve plumbing, yes? Because we'd have to put in new copper pipes, I would think? (Can you tell I know next to nothing about this?) We had our upstairs bathroom done and it involved plumping/electrical. I don't think I want DH experimenting with that-- not yet anyway!

It will be easier to see all of this with pictures!

Numbers 1 and 3 might be related. I suspect there is stinky stuff soaked into the subfloor around the toilet.

If the tub itself is fine, and since you have another bathroom to use, it might be worth the time and pain to rip it all out (except the tub). You can reuse the plumbing pretty much. Remove the floor tile because you might have some serious rot in the subfloor and even if you don't, you'll want to seal out the stink around the toilet. Put in a new vanity and sink (big box stores have pretty nice ones that are reasonably priced), vinyl floor or do some tile, and think about what to do with the tub surround. If there's a chance water has been getting into the wall behind it, you'll want to get back there too. Eek!

fori, that is a really good point about the smell. I bet you are right. I wish I had known that the bathroom was not really functional when we bought this house. I can pretty much live with the ugly in there, but not the fact that things are rotting/there is a smell, etc. I didn't see the house when we bought it (DH did), and I was here only for the inspection . . .and was taking care of my then 8-month old so not so focused.

I feel like just ripping out everything! A little tricky right now since we have a newborn + 4 other kids. Ack.

you really need to pull up the flooring in there to check that the subfloor isn't ruined by any leakage. Could be the seal under the toilet has been broken. You'll need to put in a new wax seal.

It should also be easier to redo the flooring before resetting the toilet.

if you post a pic of the sink area it'll be easier to understand what you mean by taking out the sink w/o ruining the tile...

also, if you suspect leaks in the shower area don't use it until it is properly fixed/replaced or it'll cause more damage to the walls behind it - along with mold.

desertsteph, let's assume there IS already mold there. (We have not used the shower for a long, long time-- once we figured out that it was leaking).

What then? Do we need to get someone with haz mat gear on to take out the mold, or is this still something we could attempt to do ourselves?

Can you post some pictures? The kitchen forum's Picture Posting How To can tell you how if you haven't done it before. Pictures of problem areas, especially the sink could really go a long way to getting some suggestions on how to address the problems.

I am not an expert in any field of construction, but I can give you a little information from my own DIY experience. As yayagal already stated, research is key to DIY. Knowing how to do work the right way is key. I suggest reading books (library), searching online, searching this forum, YouTube videos, etc can be great resources for DIY information.

Personally, I think it's critical you identify the source(s) of the smell and leaking water, the original cause, and any damage. Once you've identified the problem and the damage, you can construct a plan to move forward. Water can be a MAJOR source of damage and deferring maintenance and/or cleanup may end up costing more in the long run, including serious health problems or structural damage. From your description, it sounds like you've already determined that there is at least one or multiple problems with water in the bathroom. First and foremost identify the source of the water. Leaking toilet? Leaking shower pan? Leaking fresh water or waste water pipes? If you prefer to avoid demolition as much a possible, I would recommend isolating each potential culprit and testing. Use scientific method. Check each variable individually by discontinuing use (and turning off water sources) and testing each component individually. You may need to allow sufficient time between tests in order to allow materials to dry out enough to identify sources. For example, a year and a half ago I discovered wet subfloor in my laundry room. After following the moisture and individually testing potential sources, I discovered my main waste pipe (eww) in the wall of my laundry room was leaking. Without thoroughly exploring the problem and working towards a complete fix, more damage and even a potentially dangerous situation could have occurred. Because I was proactive about finding the cause, identifying and fixing the cause and the damage, I was able to remove only non-structural materials, eradicate mold and mildew, and create a much more functional, beautiful, fresh smelling laundry room. I enlisted some professional help for cleanup of some of the water damage for peace of mind. I removed most of the water damaged material myself (particle board subfloor, sheetrock) and dried out the area using fans to circulate the air and heat (space heater, cranked my furnace). The professionals removed some additional material under the framing and sprayed the area with chemicals to kill any mold and mildew. There was very minor mold so the danger to myself was low. However, I used a very heavy duty mask with interchangeable cartidges (not simply a dust mask) and covered my skin and eyes thoroughly, closed off the space from the rest of the house, and opened windows for ventilation. You should also check laws and regulations for your area to find out if there are any legal requirements for cleanup.

I also want to address a few specific points in your original post.

"Grout is barely there on the floor, so any water on the floor leaks to the basement."
Most bathroom floors are not designed to be waterproof. Why would they? Water should be contained to wet areas: in the toilet, in the tub, in the shower, in the sink. Why is there water on your floor? If enough water has been on or under your floors, you may have damage underneath the tile (subfloor, joists, etc). If not, floors can be fairly easily re-grouted. You need to remove all of the remaining grout and then apply new grout. The removal of the old grout may be labor intensive, but application of new grout is fairly easy.

"Vanity is rotting."
Why is the vanity rotting? Is water getting around or under the vanity? If so, this problem needs to be addressed first. Or do you have termites or dry rot? Or simply a poorly constructed vanity that is falling apart from age?

"Toilet always smells, no matter what I clean it with."
This does not sound like a cleaning issue. As others speculated, perhaps your toilet is leaking? Or perhaps there is an issue with venting? I'm not an expert on plumbing (especially plumbing ventilation) but sewer gases can escape from improper plumbing and cause smells. Again, the source needs to be identified.

"Grout in shower is barely there, so water leaks."
Is this a shower only or tub and shower? Do you know anything about the construction? Where is the water coming through? Grout is not and never will be water proof. It is expected that water and/or moisture will get behind tile in bath/shower combos and showers. Shower pans should be constructed to provide sufficient water proofing prior to tile installation. There are numerous correct methods and there are some methods that have been used that are incorrect and will allow water to escape. Shower walls should either have water proofing (products such as Kerdi, RedGuard, etc) directly under the tile or vapor barriers (plastic, tar paper, etc) behind materials not susceptible to water (such as mortar or cement board). And no, green board does not count.

"Replace toilet. How difficult is this?"
Replacing a toilet isn't terribly difficult, assuming the current installation doesn't have serious problems. Essentially, you need to safely remove the toilet including shutting the water off completely, draining the water in the toilet, and unfastening it. Re-installation should include proper seating of the toilet (new wax ring) to waste pipe, bolting down (not too tight - don't crack the toilet), reconnecting and turning on the water. If any of the plumbing, subfloor, etc have stability problems or leaks, the problem can get more complex because you need to fix those issues prior to new toilet installation.

prickly, that is a lot to think about. I've been forwarding these messages to DH so he can get a better idea . . .it isn't any surprise that there is so much that truly needs to be done, so now the question is when and how. I do NOT want to do more damage-- the only reason we redid our other bathroom before this one is because water was leaking onto the ceiling of the room beneath it, so it was an emergency.

RE: the grout, that is very interesting. I always thought grout was a barrier that kept water from passing through. Granted, I am not saying that I would let an inch of water sit before drying it up, but if my kids get even a little bit of water on the floor from too much playing with the sink, it goes straight to the basement.

Also wanted to address your comment about green board not counting. UH OH. I am not sure, but I think that this is all our last contractor used for our other bathroom. Sigh.
I do not even know what a shower pan is!

There is so much I do not know. In talking with DH last night, we are thinking about just gutting it and getting it done right, but now I am worried, too, about finding someone who is going to do it right!

Your help has been invaluable!

I had a similar smell-around-toilet problem in my old house, so decided to investigate by ripping out the toilet and old vinyl floor. Previous owners had a thing for carpet in the bathrooms, which does have the virtue of hiding all the scarey flaws! It was so damp and rotted in the subfloor that it was utterly disgusting. Thank goodness I got it out before going to the trouble of replacing the toilet and floor. I removed and replaced the subfloor, put leveling compound on the floor, glued down real Marmoleum (an inexpensive remnant), and installed a new Toto toilet that I bought for $50 on craigslist (out of a condo show unit). I'm a girl, and did it by myself :) I barely own tools, and had no idea how to do this. But as noted above, it's possible to learn - and it's easy to shop for second-hand deals at salvage stores or online. You can do it, and it's worth it!

I just now saw this post, and not sure if the OP will even be coming back to see it.

When we moved into our house,we did need to replace a toilet. We needed a plumber anyway for other remodeling, and had him install the toilet. We were glad we did. When he tried to turn off the water to the toilet, the handle/pipe sort of sheered off in his hand. He was holding his hand on it to stop the water like the little Dutch boy story, and I was able to run down and turn off the water to the house.

My suggestions for you are:
1. Find out (if you don't know) where the faucet handle is to turn off the water to the entire house.
2. Have one of you by that faucet handle waiting when turning off the water or removing any faucets, toilets etc.
3. Pay your most responsible child to relay messages back and forth between you and your husband if the house-turnoff is too far for shouting.

With what you describe, I would be surprised if you don't fall through the floor and end up in the basement. That is a longstanding damage producing situation, Putting ANY money into bandaids over the top is a waste of time and money.

The exception would be the toilet ... stinky toilets are usually because they aren't set properly and need a new flange and wax ring ... so lift it and either replace the whole thing oir put a stainless steel flange and new ring in it.

Go into the basement and check the joists under the bathroom .. try stabbing them with a blunt piece of metal, such as a Philips screwdriver. If you can sink a screwdriver into the wood, it's gone from being ugly to dangerously weakened by long-term water leaks.

I would take the bathroom out of service for a while. Remove ALL the tile, check for subfloor soundness and them make a decision on what to do. Joists can be "sistered" (bolting a new, sturdy one on both sides of the unsound one like a splint , starting at least several feet away from the damaged spot actually as far as possible into sound wood.) Same with bad studs ... one by one a contractor can cut out bad sections and replace them without bringing the wall down.

WHY is the vanity "rotting" ... what is leaking?

MOLD NOTE: There are two kinds of mold ... only the slimy black stuff found where it's always wet has the possibility of being nasty ... it's a specific species. It needs close to running water over its surface to get going.

Other mold, green fuzzy, white streaky, even blackish discoloration on dry wood, is a nuisance, not a HAZMAT situation