I have been thinking of improving the insulation in the floor of the attic -- the house was built in 1985 and has only the original insulation -- but I'm concerned about 2 things. First, is there a risk that if the new insulation clogs up vents along the edge of the floor of that attic could create new problems. Second, the smaller attic space above the vaulted ceiling poses new challenges -- probably the only new insulation would have to be blown in, and I'm concerned because that might cover over recessed light fixtures, which might then catch fire if they get too hot.

How serious are these risks, and how can I make sure that insulation is installed properly?

If you are referring to the soffit vents, you can purchase inexpensive "soffit insulation baffles" at any Home Depot or Lowes and install those before you insulate. Google "soffit insulation baffles" or "attic rafter baffles" and you will find many articles and videos how to do this very easily and cheap task.

Insulating over recessed light fixtures can be a big fire hazard and other issues come into play as well. Are the light fixtures "IC" rated???? "IC" stands for Insulated Contact. If they are "IC" rated, the light fixture cans are rated for insulation to be touching the fixture. If the cans are not "IC" rated, you must keep the insulation at least 3" away from the light fixture cans to prevent overheating.

Again, Google "insulating recessed light fixtures" and you will find many articles and videos how to make dams out of wood or aluminum to achieve this 3" space around these fixtures. If space is limited due to your vaulted ceilings, and you don’t have "IC" rated cans, you may need to have the cans replaced if you are going to add the insulation.

Also, the wiring to these light fixtures could be UL rated for 60 degree Celsius or 90 degree since the UL rating was changed around the year your house was built. If the UL rating for the wiring is only 60 degrees and you are adding insulation could be problematic. If the light bulbs are 75 or 100 watt bulbs and the fixture is rated for 60 watt maximum and you add insulation, you could have a fire just waiting to happen.

If you don’t know what you are doing electrically, I would have an electrician make sure the wiring and the lights can handle what you want to do.

On the can lights.
The best solution would be to replace any can lights with new ones that are rated for direct insulation contact, and are sealed for air leaks. You could also get ones made for energy efficient lights at the same time.

This will cost a few bucks, but offer a good savings in the long run, and you can stack the insulation right over the lights.

If you don't want to do that, some people put fiberglass bats around the can lights, so the loose fill does not come in direct contact with the cans.

My existing can lights were rated for direct contact with insulation, but not sealed. I sealed up all the openings in the fixture with aluminum tape, and then sealed (from the attic side) the area around the fixtures where it meets the sheetrock with Great Stuff polyurethane foam. I use CFL lights in my can lights so they don't generate a lot of heat, and the temperatures in the fixtures do not run hot.
This is a non-standard kind of solution, and you will have to make your own judgement about whether is right for you.

I "can" tell you that the old can lights are a large source of heat loss -- they leak a lot of air, are not insulated, and have a big heat source built right into them -- just about as bad a thermal design as one could imagine. So, you should do something about them.
Fiber glass insulation also leaks a lot of air, so putting it around the can light without sealing the can well, is probably not a good solution.

I would suggest that to the extent possible you seal up any penetrations from the living area to the attic before insulating. This would include the lights, but also plumbing and wiring penetrations, bath vents, ... The EPA says that these living space to attic unsealed openings are THE major source of infiltration/exfiltration in most homes (that is, more on average than windows, ...).

If you have heating or cooling ducts going through the attic, these should also have all the joints sealed with duct mastic (not duct tape), and be insulated.

For a lot of reasons, I think cellulose loose fill insulation is better than FG in either bats or loose fill.

A very good book on all this stuff is "Insulate and Weatherize" by Harley -- Amazon has it.

Gary